How I do an ESP8266 HTTPupdate via private Github repository?

Question

I try to do an update firmeware via Git repo from an ESP8266. But I don't know how. The repo is private, that mean I need a password, I read that I can use HTTPclient library for authentication. How Github's authentication works?

Also, do I need some extra code for Update library? HTTPclient supports HTTPS?

EDIT: Here some example of my code, but is for a public repo:

update.cpp (I have it in a separate header file)

//#define repo "https://github.com/username/reponame/branch/path/to/file?raw=true"
#define repo "https://raw.githubusercontent.com/username/reponame/branch/path/to/file"

t_httpUpdate_return ret = ESPhttpUpdate.update(client, repo);
// Or:
//t_httpUpdate_return ret = ESPhttpUpdate.update(client, "server", 80, "file.bin");

I have configured httpUpdate error message, it show the next error:

CALLBACK:  HTTP update fatal error code -5
HTTP_UPDATE_FAILD Error (-5): HTTP error: connection lost

Answer 1

There is a different way to do an update from GitHub, first GitHub use a HTTPS connection, that mean you need configure before a TLS/SSL setting, Also, port 80 is for insecure connections, 443 is designed for secure connections.

Public repository (insecure)

This is the easier way, just add .setInsecure() to the wifi client from WiFiClientSecure.h library, this allow you to stablish a connection ignoring all alerts from http connection.

WiFiClientSecure client;
client.setInsecure();

Is insecure, only do this for testing, not for production.

You must use https://raw.githubusercontent.com, this is for download raw data from GitHub's public repos, just the file. Your full link to the file must be:

#define repo "https://raw.githubusercontent.com/<user>/<repo>/master/<path to the .bin>"
t_httpUpdate_return ret = ESPhttpUpdate.update(client, repo);

Replace <user> with your user name, and <repo> with your repository's name. <path to the .bin> is something like "folder/folder/firmware.bin"

Public repository (secure):

There is an example in the official GitHub repository of ESP8266/Arduino. https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266httpUpdate/examples/httpUpdateSecure/httpUpdateSecure.ino

You can follow this example for a secure connection with httpUpdate. Also you will need to download the certs, this can do by executing the next script in the same folder of your project: https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/BearSSL_CertStore/certs-from-mozilla.py

If you are using windows, to run this script, you will need add "OpenSSL.exe" to the path, Git comes with it, you add the Git's bin folder to the path. Also, you will need one more file "ar.exe", this come with the ESP8266 core. You can also put this two .exe files in the same folder of the script.

For Arduino IDE is something like:

%userprofile%\AppData\Local\Arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\2.5.0-4-b40a506\xtensa-lx106-elf\bin

For PlaformIO is:

%userprofile%\.platformio\packages\toolchain-xtensa\xtensa-lx106-elf\bin

When the script finish, this will create a folder named data with certs.ar inside. Upload this file system image to ESP8266 with LittleFS.

Private repository:

This is the same as the previous one, just couple of things to change, and we will make change to the ESP8266httpUpdate library. We use the same example to httpupdatesecure and you will need to configure a token in your GitHub account.

Follow the instructions of the help page of GitHub for create a token: https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line

You only need this option selected

Copy the token and save it, you only able see one time.

You can't use raw.githubusercontent.com, will give you error 404, this only work for public repos. You need: api.github.com. You full link looks like:

https://api.github.com/repos/<user>/<repo>/contents/<path to the .bin>

And you need to add the headers to the http request, in the ESP8266httpUpdate.cpp, you must put it in the function HTTPUpdateResult ESP8266HTTPUpdate::handleUpdate in the part where start add headers:

http.addHeader(F("Accept"), "application/vnd.github.v3.raw");
http.addHeader(F("authorization"), "Bearer <your token>");

Replace <your token> with the one that you created and saved before.

Remember, edit this library will affect all your future projects, so, when you are done revert or comment the two headers that you added to the library.

NOTE: Using a token will allow your application access to all your repos, even if your token is read-only. Currently, is not possible generate a token for a specific repo on GitHub. Be careful if you share your code. Better use a dummy account with only one repo for this.

Answer 2

[Edit] - It's wokring now, it was an arduino core bug, installed 2.7.4 and now it works(come form 3.0.2-dev)

Doesn't work here "Verify Bin Header Failed", tryied hosting bin on github, 000webhosting, surge, not seens anything related to web server problem :(

    #include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>
#include <ESP8266httpUpdate.h>
//#include <WiFiClientSecure.h>
#include <CertStoreBearSSL.h>
BearSSL::CertStore certStore;
#include <time.h>
#include <FS.h>
#include <LittleFS.h>
 
const String FirmwareVer={"1"}; 
//#define URL_fw_Version "/teste/key/version.txt"
//#define URL_fw_Bin "https://pacauiot.surge.sh/teste/key/firmware.bin"
//const char* host = "pacauiot.surge.sh";

#define URL_fw_Bin "https://fourieristic-prison.000webhostapp.com/meucu.php"
const char* host = "fourieristic-prison.000webhostapp.com";


const int httpsPort = 443;
const char* ssid = "wifi";
const char* password = "wifipass";
#define RTC_UTC_TEST 1510592825



void setClock()
// *******************************************************************************************
{
  // see https://github.com/esp8266/Arduino/issues/4637
  time_t now; 
  now = time(nullptr); // if there's no time, this will have a value of 28800; Thu Jan  1 08:00:00 1970 
  Serial.print("Initial time:"); Serial.println(now);
  Serial.println(ctime(&now));

  int myTimezone = -7;
  int dst = 0;
  int SecondsPerHour = 3600;
  int MAX_TIME_RETRY = 60;
  int i = 0;

  // it is unlikely that the time is already set since we have no battery; 
  // if no time is avalable, then try to set time from the network
  if (now <= 1500000000) {
    // try to set network time via ntp packets
    configTime(0, 0, "pool.ntp.org", "time.nist.gov"); // see https://github.com/esp8266/Arduino/issues/4749#issuecomment-390822737

                // Starting in 2007, most of the United States and Canada observe DST from
    // the second Sunday in March to the first Sunday in November.
    // example setting Pacific Time:
    setenv("TZ", "EST4EDT", 1); // see https://users.pja.edu.pl/~jms/qnx/help/watcom/clibref/global_data.html
    //                     | month 3, second sunday at 2:00AM
    //                                    | Month 11 - firsst Sunday, at 2:00am  
    // Mm.n.d
    //   The dth day(0 <= d <= 6) of week n of month m of the year(1 <= n <= 5, 1 <= m <= 12, where 
    //     week 5 means "the last d day in month m", which may occur in the fourth or fifth week).
    //     Week 1 is the first week in which the dth day occurs.Day zero is Sunday.

    tzset();
    Serial.print("Waiting for time(nullptr).");
    i = 0;
    while (!time(nullptr)) {
      Serial.print(".");
      delay(1000);
      i++;
      if (i > MAX_TIME_RETRY) {
        Serial.println("Gave up waiting for time(nullptr) to have a valid value.");
        break;
      }
    }
  }
  Serial.println("");

  // wait and determine if we have a valid time from the network. 
  now = time(nullptr);
  i = 0;
  Serial.print("Waiting for network time.");
  while (now <= 1500000000) {
    Serial.print(".");
    delay(1000); // allow a few seconds to connect to network time.
    i++;
    now = time(nullptr);
    if (i > MAX_TIME_RETRY) {
      Serial.println("Gave up waiting for network time(nullptr) to have a valid value.");
      break;
    }
  }
  Serial.println("ok");

  // get the time from the system
  char *tzvalue;
  tzvalue = getenv("TZ");
  Serial.print("Network time:");  Serial.println(now);
  Serial.println(ctime(&now));
  Serial.print("tzvalue for timezone = "); Serial.println(tzvalue);

  // TODO - implement a web service that returns current epoch time to use when NTP unavailable (insecure SSL due to cert date validation)

  // some networks may not allow ntp protocol (e.g. guest networks) so we may need to fudge the time
  if (now <= 1500000000) {
    Serial.println("Unable to get network time. Setting to fixed value. \n");
    // set to RTC text value
    // see https://www.systutorials.com/docs/linux/man/2-settimeofday/
    //
    //struct timeval {
    //  time_t      tv_sec;     /* seconds */
    //  suseconds_t tv_usec;    /* microseconds */
    //};
    timeval tv = { RTC_UTC_TEST, 0 };
    //
    //struct timezone {
    //  int tz_minuteswest;     /* minutes west of Greenwich */
    //  int tz_dsttime;         /* type of DST correction */
    //};
    timezone tz = { myTimezone * 60 , 0 };  

    // int settimeofday(const struct timeval *tv, const struct timezone *tz);
    settimeofday(&tv, &tz);
  }

  now = time(nullptr);
  Serial.println("Final time:");  Serial.println(now);
  Serial.println(ctime(&now));
}

  
void FirmwareUpdate()
{  
  //WiFiClientSecure client;
  BearSSL::WiFiClientSecure client;
  bool mfln = client.probeMaxFragmentLength(host, 443, 1024);  // server must be the same as in ESPhttpUpdate.update()
    Serial.printf("MFLN supported: %s\n", mfln ? "yes" : "no");
    if (mfln) {
      client.setBufferSizes(1024, 1024);
  }
  client.setCertStore(&certStore);
  //client.setTrustAnchors(&cert);
//  if (!client.connect(host, httpsPort)) {
//    Serial.println("Connection failed");
//    return;
//  }
//  client.print(String("GET ") + URL_fw_Version + " HTTP/1.1\r\n" +
//               "Host: " + host + "\r\n" +
//               "User-Agent: BuildFailureDetectorESP8266\r\n" +
//               "Connection: close\r\n\r\n");
//  while (client.connected()) {
//    String line = client.readStringUntil('\n');
//    if (line == "\r") {
//      //Serial.println("Headers received");
//      break;
//    }
//  }
//  String payload = client.readStringUntil('\n');
//
//  payload.trim();
//  if(payload.equals(FirmwareVer) )
//  {   
//     Serial.println("Device already on latest firmware version"); 
//  }
  if(1==2){
    
  }
  else
  {
    Serial.println("New firmware detected");
    ESPhttpUpdate.setLedPin(LED_BUILTIN, LOW); 
    t_httpUpdate_return ret = ESPhttpUpdate.update(client, URL_fw_Bin);
        
    switch (ret) {
      case HTTP_UPDATE_FAILED:
        Serial.printf("HTTP_UPDATE_FAILD Error (%d): %s\n", ESPhttpUpdate.getLastError(), ESPhttpUpdate.getLastErrorString().c_str());
        break;

      case HTTP_UPDATE_NO_UPDATES:
        Serial.println("HTTP_UPDATE_NO_UPDATES");
        break;

      case HTTP_UPDATE_OK:
        Serial.println("HTTP_UPDATE_OK");
        break;
    } 
  }
}  
void connect_wifi();
unsigned long previousMillis_2 = 0;
unsigned long previousMillis = 0;        // will store last time LED was updated
const long interval = 10000;
const long mini_interval = 5000;
 void repeatedCall(){
    unsigned long currentMillis = millis();
    if ((currentMillis - previousMillis) >= interval) 
    {
      // save the last time you blinked the LED
      previousMillis = currentMillis;
      setClock();
      FirmwareUpdate();
    }

    if ((currentMillis - previousMillis_2) >= mini_interval) {
      static int idle_counter=0;
      previousMillis_2 = currentMillis;    
      Serial.print(" Active fw version:");
      Serial.println(FirmwareVer);
      Serial.print("Idle Loop(5s)...");
      //Serial.println(idle_counter++);
     if(idle_counter%2==0)
      digitalWrite(LED_BUILTIN, HIGH);
     else 
      digitalWrite(LED_BUILTIN, LOW);
     if(WiFi.status() == !WL_CONNECTED) 
          connect_wifi();
   }
 }

  
void setup()
{
  Serial.begin(115200);
  Serial.println();
  Serial.println();
  Serial.println();
  for (uint8_t t = 4; t > 0; t--) {
    Serial.printf("[SETUP] WAIT %d...\n", t);
    Serial.flush();
    delay(1000);
  }
  Serial.println("Start Xuxu");
  WiFi.mode(WIFI_STA);
  connect_wifi();  
  setClock();
  pinMode(LED_BUILTIN, OUTPUT);
  LittleFS.begin();
  int numCerts = certStore.initCertStore(LittleFS, PSTR("/certs.idx"), PSTR("/certs.ar"));
  Serial.print(F("Number of CA certs read: "));
  Serial.println(numCerts);
  if (numCerts == 0) {
    Serial.println(F("No certs found. Did you run certs-from-mozill.py and upload the LittleFS directory before running?"));
    return; // Can't connect to anything w/o certs!
  }
   //repeatedCall();  
 FirmwareUpdate();
}
void loop()
{
   
}

void connect_wifi()
{
  
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED)
  {
    delay(500);
    Serial.print("O");
  }                                   
  Serial.println("Connected to WiFi");
}