【IoT】How to realize OTA online upgrade of ESP32 firmware

1. Background

In the actual product development process, online upgrade can remotely solve the problems introduced by product software development and better meet user needs.

2. Introduction to OTA

An OTA (over-the-air) update is the process of loading firmware to an ESP module using a Wi-Fi connection instead of a serial port.

2.1. There are three ways to upgrade ESP32 via OTA:

  • Arduino IDE: Mainly used in the software development stage to realize firmware programming without wiring
  • Web Browser: Manually deliver application update modules through a web browser
  • HTTP Server: Automatically use http server - for production applications 

In the three upgrade cases, the first firmware upload must be done through the serial port. 

The OTA process has no imposed security and needs to ensure that developers can only get updates from legitimate/trusted sources. After the update is complete, the module will restart and execute the new code. Developers should ensure that applications running on modules are closed and restarted in a safe manner.

2.2, Confidentiality Security

Modules must be displayed wirelessly to be updated with new sketches. This allows the module to be forcibly hacked and loaded with additional code. To reduce the chances of being hacked, consider password protecting your uploads, choosing certain OTA ports, etc.

ArduinoOTA library interface for improved security:

  1. void setPort(uint16_t port);
  2. void setHostname(const char* hostname);
  3. void setPassword(const char* password);
  1. void onStart(OTA_CALLBACK(fn));
  2. void onEnd(OTA_CALLBACK(fn));
  3. void onProgress (OTA_CALLBACK_PROGRESS (fn));
  4. void onError(OTA_CALLBACK_ERROR (fn));

Certain protections are already built in and do not require any additional coding by the developer. ArduinoOTA and espota.py use Digest-MD5 to verify uploads. The integrity of the transmitted data is verified on the ESP side using the MD5 checksum.

2.2, OTA upgrade strategy - for http

ESP32 connects to the HTTP server and sends a request Get to upgrade the firmware; each time it reads 1KB of firmware data and writes it to Flash.

There are (at least) four partitions related to the upgrade in the ESP32 SPI Flash: OTA data, Factory App, OTA_0, OTA_1. The FactoryApp memory contains the factory default firmware.

When the OTA upgrade is performed for the first time, the OTA Demo burns the target firmware to the OTA_0 partition, and after the burn is completed, the data of the OTA data partition is updated and restarted.

When the system restarts, the data of the OTA data partition is obtained for calculation, and it is decided to load the firmware of the OTA_0 partition for execution (instead of the firmware in the default Factory App partition) to realize the upgrade.

Similarly, if the ESP32 is already executing the firmware in OTA_0 after an upgrade, the OTA Demo will write the target firmware to the OTA_1 partition when upgrading again. After rebooting, perform the OTA_1 partition to upgrade. By analogy, the upgraded target firmware is always burned interactively between the two partitions OTA_0 and OTA_1, which will not affect the Factory App firmware when it leaves the factory.

 

3. OTA instance analysis

3,1, Arduino IDE solution firmware update

 

Uploading modules wirelessly from the Arduino IDE is suitable for the following typical scenarios: 

Faster alternative to loading via serial during firmware development - for updating a small number of modules, only the modules are available on the same network as the Arduino IDE's computer.

Reference example:

  1. #include <WiFi.h>
  2. #include <ESPmDNS.h>
  3. #include <WiFiUdp.h>
  4. #include <ArduinoOTA.h>
  5. const char* ssid = "..........";
  6. const char* password = "..........";
  7. void setup() {
  8. Serial.begin(115200);
  9. Serial.println("Booting");
  10. WiFi.mode(WIFI_STA);
  11. WiFi.begin(ssid, password);
  12. while (WiFi.waitForConnectResult() != WL_CONNECTED) {
  13. Serial.println("Connection Failed! Rebooting...");
  14. delay(5000);
  15. ESP.restart();
  16. }
  17. // Port defaults to 3232
  18. // ArduinoOTA.setPort(3232);
  19. // Hostname defaults to esp3232-[MAC]
  20. // ArduinoOTA.setHostname("myesp32");
  21. // No authentication by default
  22. // ArduinoOTA.setPassword("admin");
  23. // Password can be set with it's md5 value as well
  24. // MD5(admin) = 21232f297a57a5a743894a0e4a801fc3
  25. // ArduinoOTA.setPasswordHash("21232f297a57a5a743894a0e4a801fc3");
  26. ArduinoOTA
  27. .onStart([]() {
  28. String type;
  29. if (ArduinoOTA.getCommand() == U_FLASH)
  30. type = "sketch";
  31. else // U_SPIFFS
  32. type = "filesystem";
  33. // NOTE: if updating SPIFFS this would be the place to unmount SPIFFS using SPIFFS.end()
  34. Serial.println("Start updating " + type);
  35. })
  36. .onEnd([]() {
  37. Serial.println("\nEnd");
  38. })
  39. .onProgress([](unsigned int progress, unsigned int total) {
  40. Serial.printf("Progress: %u%%\r", (progress / (total / 100)));
  41. })
  42. .onError([](ota_error_t error) {
  43. Serial.printf("Error[%u]: ", error);
  44. if (error == OTA_AUTH_ERROR) Serial.println("Auth Failed");
  45. else if (error == OTA_BEGIN_ERROR) Serial.println("Begin Failed");
  46. else if (error == OTA_CONNECT_ERROR) Serial.println("Connect Failed");
  47. else if (error == OTA_RECEIVE_ERROR) Serial.println("Receive Failed");
  48. else if (error == OTA_END_ERROR) Serial.println("End Failed");
  49. });
  50. ArduinoOTA.begin();
  51. Serial.println("Ready");
  52. Serial.print("IP address: ");
  53. Serial.println(WiFi.localIP());
  54. }
  55. void loop() {
  56. ArduinoOTA.handle();
  57. }

3,2, Web Browser  solution firmware update

 

Scenarios for using this program:

Loading directly from the Arduino IDE is inconvenient or impossible

Users cannot expose OTA's modules from external update servers

Provide post-deployment updates to a small number of modules when setting up an update server is not feasible

Reference example:

  1. #include <WiFi.h>
  2. #include <WiFiClient.h>
  3. #include <WebServer.h>
  4. #include <ESPmDNS.h>
  5. #include <Update.h>
  6. const char* host = "esp32";
  7. const char* ssid = "xxx";
  8. const char* password = "xxxx";
  9. WebServer server(80);
  10. /*
  11. * Login page
  12. */
  13. const char * loginIndex =
  14. "<form name='loginForm'>"
  15. "<table width='20%' bgcolor='A09F9F' align='center'>"
  16. "<tr>"
  17. "<td colspan=2>"
  18. "<center><font size=4><b>ESP32 Login Page</b></font></center>"
  19. "<br>"
  20. "</td>"
  21. "<br>"
  22. "<br>"
  23. "</tr>"
  24. "<td>Username:</td>"
  25. "<td><input type='text' size=25 name='userid'><br></td>"
  26. "</tr>"
  27. "<br>"
  28. "<br>"
  29. "<tr>"
  30. "<td>Password:</td>"
  31. "<td><input type='Password' size=25 name='pwd'><br></td>"
  32. "<br>"
  33. "<br>"
  34. "</tr>"
  35. "<tr>"
  36. "<td><input type='submit' onclick='check(this.form)' value='Login'></td>"
  37. "</tr>"
  38. "</table>"
  39. "</form>"
  40. "<script>"
  41. "function check(form)"
  42. "{"
  43. "if(form.userid.value=='admin' && form.pwd.value=='admin')"
  44. "{"
  45. "window.open('/serverIndex')"
  46. "}"
  47. "else"
  48. "{"
  49. " alert('Error Password or Username')/*displays error message*/"
  50. "}"
  51. "}"
  52. "</script>";
  53. /*
  54. * Server Index Page
  55. */
  56. const char* serverIndex =
  57. "<script src='https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js'></script>"
  58. "<form method='POST' action='#' enctype='multipart/form-data' id='upload_form'>"
  59. "<input type='file' name='update'>"
  60. "<input type='submit' value='Update'>"
  61. "</form>"
  62. "<div id='prg'>progress: 0%</div>"
  63. "<script>"
  64. "$('form').submit(function(e){"
  65. "e.preventDefault();"
  66. "var form = $('#upload_form')[0];"
  67. "var data = new FormData(form);"
  68. " $.ajax({"
  69. "url: '/update',"
  70. "type: 'POST',"
  71. "data: data,"
  72. "contentType: false,"
  73. "processData:false,"
  74. "xhr: function() {"
  75. "var xhr = new window.XMLHttpRequest();"
  76. "xhr.upload.addEventListener('progress', function(evt) {"
  77. "if (evt.lengthComputable) {"
  78. "var per = evt.loaded / evt.total;"
  79. "$('#prg').html('progress: ' + Math.round(per*100) + '%');"
  80. "}"
  81. "}, false);"
  82. "return xhr;"
  83. "},"
  84. "success:function(d, s) {"
  85. "console.log('success!')"
  86. "},"
  87. "error: function (a, b, c) {"
  88. "}"
  89. "});"
  90. "});"
  91. "</script>";
  92. /*
  93. * setup function
  94. */
  95. void setup(void) {
  96. Serial.begin(115200);
  97. // Connect to WiFi network
  98. WiFi.begin(ssid, password);
  99. Serial.println("");
  100. // Wait for connection
  101. while (WiFi.status() != WL_CONNECTED) {
  102. delay(500);
  103. Serial.print(".");
  104. }
  105. Serial.println("");
  106. Serial.print("Connected to ");
  107. Serial.println(ssid);
  108. Serial.print("IP address: ");
  109. Serial.println(WiFi.localIP());
  110. /*use mdns for host name resolution*/
  111. if (!MDNS.begin(host)) { //http://esp32.local
  112. Serial.println("Error setting up MDNS responder!");
  113. while (1) {
  114. delay(1000);
  115. }
  116. }
  117. Serial.println("mDNS responder started");
  118. /*return index page which is stored in serverIndex */
  119. server.on("/", HTTP_GET, []() {
  120. server.sendHeader("Connection", "close");
  121. server.send(200, "text/html", loginIndex);
  122. });
  123. server.on("/serverIndex", HTTP_GET, []() {
  124. server.sendHeader("Connection", "close");
  125. server.send(200, "text/html", serverIndex);
  126. });
  127. /*handling uploading firmware file */
  128. server.on("/update", HTTP_POST, []() {
  129. server.sendHeader("Connection", "close");
  130. server.send(200, "text/plain", (Update.hasError()) ? "FAIL" : "OK");
  131. ESP.restart();
  132. }, []() {
  133. HTTPUpload& upload = server.upload();
  134. if (upload.status == UPLOAD_FILE_START) {
  135. Serial.printf("Update: %s\n", upload.filename.c_str());
  136. if (!Update.begin(UPDATE_SIZE_UNKNOWN)) { //start with max available size
  137. Update.printError(Serial);
  138. }
  139. } else if (upload.status == UPLOAD_FILE_WRITE) {
  140. /* flashing firmware to ESP*/
  141. if (Update.write(upload.buf, upload.currentSize) != upload.currentSize) {
  142. Update.printError(Serial);
  143. }
  144. } else if (upload.status == UPLOAD_FILE_END) {
  145. if (Update.end(true)) { //true to set the size to the current progress
  146. Serial.printf("Update Success: %u\nRebooting...\n", upload.totalSize);
  147. } else {
  148. Update.printError(Serial);
  149. }
  150. }
  151. });
  152. server.begin();
  153. }
  154. void loop(void) {
  155. server.handleClient();
  156. delay(1);
  157. }

3.3, HTTP server implementation update

 

The ESPhttpUpdate class can check for updates and download binaries from HTTP web servers. Updates can be downloaded from every IP or domain name address on the network or the Internet, mainly for remote server updates and upgrades.

Reference example:

  1. /**
  2. AWS S3 OTA Update
  3. Date: 14th June 2017
  4. Author: Arvind Ravulavaru <https://github.com/arvindr21>
  5. Purpose: Perform an OTA update from a bin located in Amazon S3 (HTTP Only)
  6. Upload:
  7. Step 1 : Download the sample bin file from the examples folder
  8. Step 2 : Upload it to your Amazon S3 account, in a bucket of your choice
  9. Step 3 : Once uploaded, inside S3, select the bin file >> More (button on top of the file list) >> Make Public
  10. Step 4 : You S3 URL => http://bucket-name.s3.ap-south-1.amazonaws.com/sketch-name.ino.bin
  11. Step 5 : Build the above URL and fire it either in your browser or curl it `curl -I -v http://bucket-name.ap-south-1.amazonaws.com/sketch-name.ino.bin` to validate the same
  12. Step 6: Plug in your SSID, Password, S3 Host and Bin file below
  13. Build & upload
  14. Step 1 : Menu > Sketch > Export Compiled Library. The bin file will be saved in the sketch folder (Menu > Sketch > Show Sketch folder)
  15. Step 2 : Upload bin to S3 and continue the above process
  16. // Check the bottom of this sketch for sample serial monitor log, during and after successful OTA Update
  17. */
  18. #include <WiFi.h>
  19. #include <Update.h>
  20. WiFiClient client;
  21. // Variables to validate
  22. // response from S3
  23. int contentLength = 0;
  24. bool isValidContentType = false;
  25. // Your SSID and PSWD that the chip needs
  26. // to connect to
  27. const char* SSID = "YOUR-SSID";
  28. const char* PSWD = "YOUR-SSID-PSWD";
  29. // S3 Bucket Config
  30. String host = "bucket-name.s3.ap-south-1.amazonaws.com"; // Host => bucket-name.s3.region.amazonaws.com
  31. int port = 80; // Non https. For HTTPS 443. As of today, HTTPS doesn't work.
  32. String bin = "/sketch-name.ino.bin"; // bin file name with a slash in front.
  33. // Utility to extract header value from headers
  34. String getHeaderValue(String header, String headerName) {
  35. return header.substring(strlen(headerName.c_str()));
  36. }
  37. // OTA Logic
  38. void execOTA() {
  39. Serial.println("Connecting to: " + String(host));
  40. // Connect to S3
  41. if (client.connect(host.c_str(), port)) {
  42. // Connection Succeed.
  43. // Fecthing the bin
  44. Serial.println("Fetching Bin: " + String(bin));
  45. // Get the contents of the bin file
  46. client.print(String("GET ") + bin + " HTTP/1.1\r\n" +
  47. "Host: " + host + "\r\n" +
  48. "Cache-Control: no-cache\r\n" +
  49. "Connection: close\r\n\r\n");
  50. // Check what is being sent
  51. // Serial.print(String("GET ") + bin + " HTTP/1.1\r\n" +
  52. // "Host: " + host + "\r\n" +
  53. // "Cache-Control: no-cache\r\n" +
  54. // "Connection: close\r\n\r\n");
  55. unsigned long timeout = millis();
  56. while (client.available() == 0) {
  57. if (millis() - timeout > 5000) {
  58. Serial.println("Client Timeout !");
  59. client.stop();
  60. return;
  61. }
  62. }
  63. // Once the response is available,
  64. // check stuff
  65. /*
  66. Response Structure
  67. HTTP/1.1 200 OK
  68. x-amz-id-2: NVKxnU1aIQMmpGKhSwpCBh8y2JPbak18QLIfE+OiUDOos+7UftZKjtCFqrwsGOZRN5Zee0jpTd0=
  69. x-amz-request-id: 2D56B47560B764EC
  70. Date: Wed, 14 Jun 2017 03:33:59 GMT
  71. Last-Modified: Fri, 02 Jun 2017 14:50:11 GMT
  72. ETag: "d2afebbaaebc38cd669ce36727152af9"
  73. Accept-Ranges: bytes
  74. Content-Type: application/octet-stream
  75. Content-Length: 357280
  76. Server: AmazonS3
  77. {{BIN FILE CONTENTS}}
  78. */
  79. while (client.available()) {
  80. // read line to / n
  81. String line = client.readStringUntil('\n');
  82. // remove space, to check if the line is end of headers
  83. line.trim();
  84. // if the the line is empty,
  85. // this is end of headers
  86. // break the while and feed the
  87. // remaining `client` to the
  88. // Update.writeStream();
  89. if (!line.length()) {
  90. //headers ended
  91. break; // and get the OTA started
  92. }
  93. // Check if the HTTP Response is 200
  94. // else break and Exit Update
  95. if (line.startsWith("HTTP/1.1")) {
  96. if (line.indexOf("200") < 0) {
  97. Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
  98. break;
  99. }
  100. }
  101. // extract headers here
  102. // Start with content length
  103. if (line.startsWith("Content-Length: ")) {
  104. contentLength = atoi((getHeaderValue(line, "Content-Length: ")).c_str());
  105. Serial.println("Got " + String(contentLength) + " bytes from server");
  106. }
  107. // Next, the content type
  108. if (line.startsWith("Content-Type: ")) {
  109. String contentType = getHeaderValue(line, "Content-Type: ");
  110. Serial.println("Got " + contentType + " payload.");
  111. if (contentType == "application/octet-stream") {
  112. isValidContentType = true;
  113. }
  114. }
  115. }
  116. } else {
  117. // Connect to S3 failed
  118. // May be try?
  119. // Probably a choppy network?
  120. Serial.println("Connection to " + String(host) + " failed. Please check your setup");
  121. // retry??
  122. // execOTA ();
  123. }
  124. // Check what is the contentLength and if content type is `application/octet-stream`
  125. Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));
  126. // check contentLength and content type
  127. if (contentLength && isValidContentType) {
  128. // Check if there is enough to OTA Update
  129. bool canBegin = Update.begin(contentLength);
  130. // If yes, begin
  131. if (canBegin) {
  132. Serial.println("Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!");
  133. // No activity would appear on the Serial monitor
  134. // So be patient. This may take 2 - 5mins to complete
  135. size_t written = Update.writeStream(client);
  136. if (written == contentLength) {
  137. Serial.println("Written : " + String(written) + " successfully");
  138. } else {
  139. Serial.println("Written only : " + String(written) + "/" + String(contentLength) + ". Retry?" );
  140. // retry??
  141. // execOTA ();
  142. }
  143. if (Update.end()) {
  144. Serial.println("OTA done!");
  145. if (Update.isFinished()) {
  146. Serial.println("Update successfully completed. Rebooting.");
  147. ESP.restart();
  148. } else {
  149. Serial.println("Update not finished? Something went wrong!");
  150. }
  151. } else {
  152. Serial.println("Error Occurred. Error #: " + String(Update.getError()));
  153. }
  154. } else {
  155. // not enough space to begin OTA
  156. // Understand the partitions and
  157. // space availability
  158. Serial.println("Not enough space to begin OTA");
  159. client.flush();
  160. }
  161. } else {
  162. Serial.println("There was no content in the response");
  163. client.flush();
  164. }
  165. }
  166. void setup() {
  167. //Begin Serial
  168. Serial.begin(115200);
  169. delay(10);
  170. Serial.println("Connecting to " + String(SSID));
  171. // Connect to provided SSID and PSWD
  172. WiFi.begin(SSID, PSWD);
  173. // Wait for connection to establish
  174. while (WiFi.status() != WL_CONNECTED) {
  175. Serial.print("."); // Keep the serial monitor lit!
  176. delay(500);
  177. }
  178. // Connection Succeed
  179. Serial.println("");
  180. Serial.println("Connected to " + String(SSID));
  181. // Execute OTA Update
  182. execOTA ();
  183. }
  184. void loop() {
  185. // chill
  186. }
  187. /*
  188. * Serial Monitor log for this sketch
  189. *
  190. * If the OTA succeeded, it would load the preference sketch, with a small modification. i.e.
  191. * Print `OTA Update succeeded!! This is an example sketch : Preferences > StartCounter`
  192. * And then keeps on restarting every 10 seconds, updating the preferences
  193. *
  194. *
  195. rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
  196. configsip: 0, SPIWP:0x00
  197. clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
  198. mode:DIO, clock div:1
  199. load: 0x3fff0008, len: 8
  200. load: 0x3fff0010, len: 160
  201. load: 0x40078000, len: 10632
  202. load: 0x40080000, len: 252
  203. entry 0x40080034
  204. Connecting to SSID
  205. ......
  206. Connected to SSID
  207. Connecting to: bucket-name.s3.ap-south-1.amazonaws.com
  208. Fetching Bin: /StartCounter.ino.bin
  209. Got application/octet-stream payload.
  210. Got 357280 bytes from server
  211. contentLength : 357280, isValidContentType : 1
  212. Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!
  213. Written : 357280 successfully
  214. OTA done!
  215. Update successfully completed. Rebooting.
  216. ets Jun 8 2016 00:22:57
  217. rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
  218. configsip: 0, SPIWP:0x00
  219. clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
  220. mode:DIO, clock div:1
  221. load: 0x3fff0008, len: 8
  222. load: 0x3fff0010, len: 160
  223. load: 0x40078000, len: 10632
  224. load: 0x40080000, len: 252
  225. entry 0x40080034
  226. OTA Update succeeded!! This is an example sketch : Preferences > StartCounter
  227. Current counter value: 1
  228. Restarting in 10 seconds...
  229. E (102534) wifi: esp_wifi_stop 802 wifi is not init
  230. ets Jun 8 2016 00:22:57
  231. rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
  232. configsip: 0, SPIWP:0x00
  233. clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
  234. mode:DIO, clock div:1
  235. load: 0x3fff0008, len: 8
  236. load: 0x3fff0010, len: 160
  237. load: 0x40078000, len: 10632
  238. load: 0x40080000, len: 252
  239. entry 0x40080034
  240. OTA Update succeeded!! This is an example sketch : Preferences > StartCounter
  241. Current counter value: 2
  242. Restarting in 10 seconds...
  243. ....
  244. *
  245. */

refer:

http://www.yfrobot.com/wiki/index.php?title=OTA_Updates

http://www.yfrobot.com/thread-11979-1-1.html

https://www.arduino.cn/thread-41132-1-1.html

http://www.ifindbug.com/doc/id-44691/name-Brief Analysis of esp32 Flash Partition and OTA Function.html

Related: 【IoT】How to realize OTA online upgrade of ESP32 firmware