Serving HTML as C array from Arduino code - String size limit problem

Question

I've been working on a HTML / websocket server on a Wiznet W5100S-EVB-Pico, programmed in the Arduino IDE. It all worked fine up until now but I'm running into, I think, a string size limit. I guess it is in the way the code handles the const char but I don't know how to do it properly.

I hope someone is willing to help :)

Let me explain:

• I convert the index.html to a index_html.h file containing a const char array:

const char c_index_html[] = {
0x3c,0x21,0x44,0x4f,0x43,..., ..., 0x6d,0x6c,0x3e};

• In my code I include the index_html.h file:

#include "index_html.h"

Now the code that actually serves the "HTML"

if (web_client){
    
    Serial.println("New client");
    // an http request ends with a blank line
    bool currentLineIsBlank = true;

    while (web_client.connected()){
      
      if (web_client.available()){
        
        char c = web_client.read();
        
        if (c == '\n' && currentLineIsBlank)                // if you've gotten to the end of the line (received a newline
        {                                                   // character) and the line is blank, the http request has ended,
          Serial.println(F("Sending response"));            // so you can send a reply

          String strData;
          strData = c_index_html;
          web_client.println(strData);
          break;
        }

        if (c == '\n')
        {
          // you're starting a new line
          currentLineIsBlank = true;
        }
        else if (c != '\r')
        {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }

This is not the prettiest code, it's smashed together from examples and now the main culprit seems to be:

String strData;
strData = c_index_html;
web_client.println(strData);

When I add extra code to the HTML and view the page source, the code is incomplete. I tested reducing the HTML to a minimum and that solves the problem.

So my main question is:

• How do I serve the 'const char c_index_html' without use of 'String'?

But also:

• How could I prettify the whole 'if (web_client)' serving function?

Thank you very much for making it all the way through this post and if you have a suggestion I would very much appreciate it ;)

Answer 1

Edit: There is a bug in the ethernet library shown in this post.

I don't know if it affects you; you should look at your library implementation.

---

I'm assuming that web_client is an instance of EthernetClient from the Arduino libraries.

EthernetClient::println is inherited from Print via Stream and is defined in terms of write, which is:

size_t EthernetClient::write(const uint8_t *buf, size_t size)
{
    if (_sockindex >= MAX_SOCK_NUM) return 0;
    // This library code is not correct:
    if (Ethernet.socketSend(_sockindex, buf, size)) return size;
    setWriteError();
    return 0;
}

So we see that it asks the socket to send the buffer up to some size. The socket can respond with a size or 0 (see edit); if it responds with 0 then there's an error condition to check.

Edit: This is how it's supposed to work. Since write is always returning the requested size and not telling you how much was written, you can't fix your problem using the print/write facilities and need to directly use socketSend.

You're not checking the result of this write (which is supposed to come through println) so you don't know whether the socket sent size bytes, 0 bytes, or some number in between.

In EthernetClient::connect we see that it's opening a TCP stream:

_sockindex = Ethernet.socketBegin(SnMR::TCP, 0);

When you call socketSend you're actually just copying your buffer into a buffer in the network stack. The TCP driver writes out that buffer when it can. If you're writing into that buffer faster than it's being flushed to the network then you'll fill it up and your socketSend calls will start returning < size bytes. See Does send() always send whole buffer?.

So you're probably right that your string is too long. What you need to do is spread your writes out. There are countless tutorials covering this on the web; it's roughly like this in your example:

...

size_t bytesRemaining = 0;

while (web_client.connected()){

  if (bytesRemaining > 0) {
    // Still responding to the last request
    char const* const cursor = c_index_html 
                             + sizeof(c_index_html) 
                             - bytesRemaining;
    size_t const bytesWritten = web_client.write(cursor, bytesRemaining);

    if (!bytesWritten) {
      // check for error
    }
    bytesRemaining -= bytesWritten;

    if (bytesRemaining == 0) {
      // End the message. This might not write! 
      // We should add the '\n' to the source array so 
      // it's included in our write-until-finished routine.
      web_client.println();
      // Stop listening
      break;
    }

  } else if (web_client.available()){
    // Time for a new request
    
    char c = web_client.read();
    
    if (c == '\n' && currentLineIsBlank)               
    {                                                   
      Serial.println(F("Sending response"));            
      // Start responding to this request
      bytesRemaining = sizeof(c_index_html);
      continue;
    }

    ...

Answer 2

This is what I think is going on. I'm not an expert so I might be wrong, but it seems to make sense.

This is not an answer as in "solution" but I found out there is a 2k buffer size limit using the W5100S-EVB-Pico. And indeed, if I keep the HTML below 2k it works. Turns out that I actually got Matt Murphy's suggestion to work but the 2k limit was the problem. It looks like a hardware/library limitation, not completely sure on that.

For now I'll shrink my HTML and Javascript to a minimum and compact it even more with for example textfixer.com. I think I might write some python code to do that

Maybe there is a path to a solution in the link below but at this moment I'll try to live with the limitations

Link: https://github.com/khoih-prog/EthernetWebServer/issues/7