The Magic Wand Example

In the Magic Wand example a "magic wand" is used to paint digits into the air. An accelerometer and gyroscope is used to record the movement and from the acceleration and gyroscope data the gesture is reconstructed.

The hardware

In order to get a stable hardware configuration we use the ESP32-S3FH4R2 CPU in conjunction with an LSM6DS3 accelerometer and gyroscope module. The LSM6DS3 is mounted on a WeMos D1 mini prototype board, which in turn is connected to the CPU via the triple base backplane. The backplane is attached to the magic wand with screws.

The magic wand On top: the LSM6DS3 accelerometer and gyroscope at the bottom: the ESP32-S2FH4R2 CPU

Interfacing the LSM6DS3 to the WeMos D1 mini bus

The magic wand example is the only one for which no ESP32 version can be found on the internet. The version available uses an Arduino Nano 33 BLE sense board with a LSM9DS1 accelerometer, gyroscope and magnetometer. This chip is substantially more expensive that the LSM6DS3, that misses the magnetometer, which however is not needed to run the example.

The LSM6DS3 has an I2C as well as an SPI interface. The connections are shown in the table below. The driver only uses the I2C interface but could be easily extended to support SPI as well.

LSM6DS3	WeMos D1 D pin	GPIO on ESP32S3FH4R2
3V3	3V3
GND	GND
SCL	D5	GPIO 12
SDA	D6	GPIO 13
CS	D7	GPIO 11
SA0	D8	GPIO 10
DCS	D1	GPIO 36
INT2	D2	GPIO 35
INT1	D3	GPIO 18

The software

The software is based on the demo written by Pete Warden: https://petewarden.github.io/magic_wand/. He gave a tutorial at the tinyML Summit 2021 which is published on youtube: https://www.youtube.com/watch?v=vKRdQHO7tIY. The code can be found on github: https://github.com/petewarden/magic_wand.

As can be seen from the documentation, he used an Arduino Nano 33 BLE Sense which incorporates a LSM9SD1 accelerometer, gyroscope and magnetometer. Since the Arduino Nano 33 Sense is rather expensive (some 50 US$) and the magnetometer on the LSM9DS1 is not used in the application, I tried to port the demo to an LoLin S3 mini board coupled with a LSM6DS3 accelerometer and gyroscope.

The original software consists of several parts:

• an Arduino Sketch reading the acceleration and gyroscope values from the LSM9DS1
  The sketch incorporates a pre-trained Tensorflow Lite model and gestures can be analysed to find a digit that has been drawn in the air.
  The accelerometer and gyroscope data are converted into position and stored in "stroke arrays". The strokes are then rasterized and save in pseudo image arrays which are fed into the model.
• In addition to the above strokes can be transferred to a WEB site via BlueTooth and plotted there. If the strokes make sense, they can be stored in json files ready to be used with the model. The browser must however be able to handle BlueTooth. I use Google Chrome on which BlueTooth must be enabled
  
  
• The model definition, training and conversion to Tensorflow Lite micro with quantization implemented as a jupyter notebook. Pre-registered strokes are available for model training and the trained model is tested with a number of test strokes.

Porting the code immediately to MicroPython seemed exceedingly difficult. Therefore I decided to first try a port to ESP32 Arduino.

Before trying to port the application the new hardware (the Lolin ESP32 S3 mini board and all its interfaces and the LSM6DS3 accelerometer) must be fully understood. I therefore wrote small hardware test sketches for

• the ws2812 LED (this can be used to show the BlueTooth state (advertizing or connected)
• BlueTooth Low Energy
• the LSM6DS3. There is an Arduino driver for this chip, but this driver is too simplistic and it does not make use on the on-chip FIFO. Pete Warden's code uses the LSM9DS1 FIFO. I therefore extended the LSM6DS3 driver adding the use of the FIFO and wrote test sketches to make sure things are working correctly.

The hardware tests can be found on github: https://github.com/uraich/TinyML/tree/main/hardwareTests.

-- Uli Raich - 2023-12-18

Comments