TinyML < IoT_Course_English

Tags: view all tags

---+ TinyML
---++ Introduction

Machine learning (ML) algorithms usually work on huge datasets and require an enormous amount of resources. On the other hand many mobile phone use face recognition to improve photographs or speech recognition for natural language translation. This can be achieved by training machine learning models, in particular neural networks, on the big computer, which may use a GPU (Graphical Procession Unit) available on many of them, while the trained model is scaled down and used on the mobile phone.

Not long ago Google has released [[https://github.com/tensorflow/tensorflow][ TensorFlow,]] the Artificial Intelligence library it uses for its own algorithms, as !OpenSource and therefore made it available to everybody. The full !Tensorflow library is used to create and train models and the TFLite library allow the down-scaling and use on smaller machines. Only very recently TFLite-micro (tflm) has been released which strips down the system even further to make it available of embedded systems.
---++ [[TinyML]] with MicroPython on the ESP32

The TinyML library is described in a [[https://tinymlbook.com/wp-content/uploads/2020/11/TinyML_preview.pdf][book published by O'Reilly. ]]This book demonstrates how to use tflm models on three different micro-controllers:
   * Arduino Nano 33 BLE Sense
   * Sparkfun Edge
   * ST Microelectronics !STM32F746G Discovery
and uses the C++ programming language.

It describes 4 different projects:
   * A <b>hello world example </b>which uses a regression algorithm to predict values of a sine function
   * <b>Wake word detection, </b>a model which can detect the spoken words "yes" and "no". The on-board microphone is used to sample the audio data.
   * <b>People detection, </b>a system that gets images from a camera and determines if a person is in sight of the camera
   * *Magic Wand*, a system that recognizes gestures using an accelerometer
In the meantime the examples have been ported to the ESP32 and the !ESP32S3 as well. These examples are written in C++ using either the Espressif development environment esp-idf or the Arduino SDK. Unfortunately the ESP32 !TensorFlow library for the Arduino SDK is already a little outdated but I was able to create a !TensorFlow Lite Micro source tree from the latest TFLM repository, which can be used with the ESP32 Arduino SDK.

The magic wand demo is only available for the Arduino Nano 33 sense module.

Here we go through the [[ESP32 C++ examples]]

Michael O'Cleirigh has started a project on github trying to port these examples to !MicroPython (https://github.com/mocleiri/tensorflow-micropython-examples). Since there are several ESP32 modules that include a simple camera, !I2S microphones are supported in !MicroPython and several accelerometers with !I2C interface are available, it should be possible to run these examples on the ESP32.
---++ The hardware

For the person detection example several esp32-cam boards integrating a camera with the ESP32 CPU are available. Some provide an SD card interface, such that photos or videos from the camera can be recorded onto an SD card. Some of these boards have only 16 output pins and are therefore very limited in the number of GPIO lines available to interface external devices and others use 40 output pins. Only the [[ESP32-S3-FN8]] has a camera and a microphone and none have an accelerometer on board.

For all demos except the magic wand example I therefore use the !ESP32-S3-FN8

<img alt="esp32-s3-fn8.png" height="408" src="%ATTACHURL%/esp32-s3-fn8.png" title="esp32-s3-fn8.png" width="460" />

Of course it is possible to interface also the accelerometer to the above board, but in this case a bread board and quite a bit of external cabling is needed. I therefore prefer to use the !WeMos D1 mini eco-system with an !ESP32S3 CPU, a triple base and prototype boards onto which the microphone and the accelerometer are mounted.

The photos below show the CPU and the backplane board.
| <img alt="esp32s3_cpu.png" height="442" src="%ATTACHURL%/esp32s3_cpu.png" title="esp32s3_cpu.png" width="420" /> | <img alt="triple_board.png" height="151" src="%ATTACHURL%/triple_board.png" title="triple_board.png" width="400" /> | <img alt="proto.png" height="282" src="%ATTACHURL%/proto.png" title="proto.png" width="448" /> |
| !ESP32-S3FH4R2 CPU | back plane (triple base) | prototype board |
The pinout of the !ESP32S3 CPU looks as follows:

<img alt="esp32s3_pinout.png" height="410" src="%ATTACHURL%/esp32s3_pinout.png" title="esp32s3_pinout.png" width="736" />

and the circuit diagram can be found here:

%ATTACHURL%/sch_s3_mini_v1.0.0.pdf

in addition we need the microphone and the accelerometer. The !I2S driver in !MicroPython support the MH-ET Live and the SPH0645 microphones. There are many accelerometers on the market and I selected the !LSM6DS3 because this is also used in the !TinyML book.
| <img alt="sph0645.png" height="289" src="%ATTACHURL%/sph0645.png" title="sph0645.png" width="363" /> | <img alt="mh-et-live.png" height="255" src="%ATTACHURL%/mh-et-live.png" title="mh-et-live.png" width="467" /> | <img alt="lsm6ds3.png" height="249" src="%ATTACHURL%/lsm6ds3.png" title="lsm6ds3.png" width="174" /> |
| !SPH0645 microphone | MH-ET Live microphone | !LSM6DS3 accelerometer |

The photo below shows the MH-ET Live microphone mounted on a prototype board. The connections are made with small cables on the back. The prototype board can then simply be plugged onto the base board.
| <img alt="microphone.png" height="400" src="%ATTACHURL%/microphone.png" title="microphone.png" width="400" /> | <img alt="micro_back.png" height="402" src="%ATTACHURL%/micro_back.png" title="micro_back.png" width="400" /> |
| top view of the microphone mounted on the prototype board | cabling on the back |
Setting up the PC and !MicroPython for !TensorFlow

The following pages show how to set up the PC to run !TensorFlow, how to create models and how to down-scale them for use on the ESP32. Then a custom version of !MicroPython must be created including the esp32-camera driver and its !MicroPython interface as well as *ulab*, a stripped down version of numpy and scipy as well as the tflm library and its !MicroPython access routines.

Finally the installation of the !TinyML examples on the ESP32 is explained.

   * [[Preparing the PC]] for !TensorFlow
   * [[Custom !MicroPython]] with ulab, tflm and the esp32-camera driver
   * [[The Hello World]] example
   * [[Wake Word Detection]]
   * [[People Detection]]
   * [[Magic Wand old]]
   * [[Magic Wand]]
   * [[C++ examples]]

-- %USERSIG{UliRaich - 2022-01-31}%

---++ Comments

<br />%COMMENT%

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who
png	ESP32S3_kit.png	r1	manage	290.1 K	2023-09-02 - 09:20	UliRaich
png	ESP32S3_pinout.png	r1	manage	351.4 K	2023-09-02 - 09:28	UliRaich
png	esp32-s3-fn8.png	r1	manage	344.9 K	2024-02-21 - 16:28	UliRaich
png	esp32s3_cpu.png	r1	manage	283.9 K	2023-09-11 - 07:59	UliRaich
png	esp32s3_pinout.png	r1	manage	222.2 K	2023-09-12 - 12:57	UliRaich
png	lsm6ds3.png	r1	manage	69.6 K	2023-09-11 - 08:11	UliRaich
png	mh-et-live.png	r1	manage	155.6 K	2023-09-11 - 08:10	UliRaich
png	micro_back.png	r2 r1	manage	358.7 K	2023-09-11 - 08:41	UliRaich
png	microphone.png	r2 r1	manage	340.9 K	2023-09-11 - 08:18	UliRaich
png	proto.png	r1	manage	171.8 K	2023-09-11 - 08:14	UliRaich
pdf	sch_s3_mini_v1.0.0.pdf	r1	manage	161.6 K	2023-09-12 - 12:57	UliRaich
png	sph0645.png	r1	manage	118.9 K	2023-09-11 - 07:59	UliRaich
png	triple_board.png	r1	manage	105.6 K	2023-09-11 - 07:59	UliRaich

Topic revision: r13 - 2024-02-21 - UliRaich