❬2021-04-28❭

Going Wireless

I've been battling with reducing the power line noise for too long, so I thought screw it, let's go off the power line entirely. I put the circuit on two 3V CR2032 coin cells and wrote some code to transmit the signals via BLE (Bluetooth Low Energy) using the ArduinoBLE library.

Since I can not plot the signals via the Arduino IDE plotter anymore, I switched to GNURadio and wrote a plugin that establishes the BLE connection and acts as a signal source in the GNURadio companion software

My new "electrodes" also arrived: Simple prong snap buttons. They don't have sharp edges like the pyramidal studs I used before, and allow me to easily remove the wires from the electrodes and plug them in somewhere else as needed.

I also employed INA128 instrumentation amplifiers, drastically reducing the complexity of the circuit. It's a tiny SMD chip, which I plan to embed in hot glue, along with the 3-4 capacitors and 3-5 resistors required for processing/de-noising, and place 8 of these processing units across the glove/wristband, connected to two electrodes each.

Now I'm battling the problem that I can only get about 1kB/s across the ether. How am I supposed to put 12kB/s worth of signal in there? (8 channels, 1k samples/s, 12 bit per sample) Let's see if I can find some nice compression method, but I fear that it's going to be lossy. :-/

❬2021-04-24❭

First Amplifier Circuit

I had my head stuck in electronics lectures, datasheets, and a breadboard to figure out a decent analog circuit for amplifying the signal. It sounds so straight forward, just plug the wires into the + and - pin of an operational amplifier, add a few resistors to specify the gain of the OpAmp, and feed the output to the analog input pin of the Arduino... But reality is messy, and it didn't quite work out like that.

Here's a list of problems:

• The voltage I measured from the electrodes seemed incredibly fragile. As soon as I wanted to do something with it, it seemed to change. This could be due to the high impedance of the skin, causing a drop in voltage as soon as one draws any current.
  • Solution: Used buffer amplifiers, in an instrumentation amplifier arrangement. Patrick Mercier from UCSD has a great lecture on that
• One electrode may have a DC voltage offset compared of the other electrode. When this gets large (~50mV+), the amplified voltage difference gets off the scale.
  • Solution: Added capacitors that filter out the DC component. Through experimentation I found that 100pF worked best. Curiously, this is the same capacitance that's sometimes used to model the skin
• The OpAmp amplifies not just the signal but also the noise, like:
  • Power line hum from USB connection
  • Electromagnetic Interference
  • Fluctuations in resistance/capacitance between electrode and muscle
  • Solutions:
    • Disconnect Laptop from main grid
    • Keep wires short
    • Add decoupling capacitors
    • Perform occult protection magic ceremony
    • In the future, perhaps some shielding or bandpass filtering
• I ordered a part that requires min. +/-2.25V. The Arduino supplies 3.3V, so all is well, right? Nope. It means that I need negative 2.25V as well as positive 2.25V.
  • Solution: Increased the voltage of the entire circuit to 5V, which the Arduino conveniently supports by changing a solder jumper. The middle-ground reference voltage rose from 1.65V to 2.5V, leaving enough room for the required +/-2.25V. I don't actually use the part yet, but I wanted to prepare for it.
• The 2.5V reference voltage from the voltage divider strongly fluctuated, messing up the output from the OpAmp.
  • Solutions:
    • Buffer amplifier after the voltage divider
    • Bypass capacitors close to the OpAmps
• The LM324N OpAmp that I used has an output voltage limit of 3.6V (at a supply voltage of 5V.) That cuts off a good chunk of the signal.
  • Solution: I added a second reference voltage at 1.66V so the output centers around that. (Conveniently, the output limit of 3.6V is close to the Arduino's ADC reference voltage of 3.3V.)
• Should I even do any of this? I'm limiting the neural network by introducing my bias about what a clean signal looks like. Any circuit will invariably filter out certain information, enhance other information, and add irrelevant noise. How do I know that the information that I filter out (e.g. the DC offset voltage between electrodes, or even what I consider irrelevant noise) isn't useful to the neural network?
  • Solution: Keep the signal processing reasonably minimal

I also connected the electrode signal to ground with a 1MΩ resistor which greatly improved the signal, and I have no idea why.

One peculiar thing I noticed was that the signal seemed stronger when my laptop was connected to the power supply. It superimposed noise, but also seemed to increase differences in electrode voltages. I don't quite understand this yet, but 2 things follow from that:

• For replication purposes, I'm using a Lenovo Thinkpad T460p switched to the Intel GPU, which creates it's own particular noise patterns, even when unplugged from the grid.
• I should try out modulating the ground electrode voltage with a controlled low frequency pattern to see if this improves signal to noise ratio. Ideally <30Hz or >500Hz so I can easily filer it out later.

Some of the references I used:

• Olimex "SHIELD-EKG-EMG"
• EEVblog OpAmps Tutorial
• EEVblog Bypass Capacitor Tutorial
• Patrick Mercier's lectures on Instrumentational Amplifiers
• BioPhysical Modeling, Characterization and Optimization of Electro-Quasistatic Human Body Communication, arXiv:1805.05200v1

The resulting circuit:

And the signals look like:

Yellow and green are two electrodes, right after their respective OpAmp, and purple is (yellow-green)*20.

This should be good enough to move forward, but I bought some INA128 instrumentation amplifiers and perhaps I will tinker some more to get an even better signal. Can't wait for the next prototype though :).

In other news, I watched Dr. Gregory House explain forearm muscles, so next time my electrode placement will be better than random!

And since I learned KiCad for creating the above schematic, I thought I'd add schematics for the previous models as well, see circuits.

❬2021-04-19❭

Amplifiers

I have the feeling that before building the next prototype, I should figure out some way of enhancing the signal in hardware before passing it to the microcontroller. It's fun to hook the 'trodes straight to the ADC and still get results, but I don't think the results are optimal. So these days I'm mostly researching and tinkering with OpAmps.

❬2021-04-15❭

Multiplexers

The analog multipexers (5x DG409DJZ) and other stuff arrived! I almost bought a digital multiplexer, because I didn't know there were various types... But I think that these will work for my use case. The raw signal that I get out of it looks a little different, but when I filter out the low & high frequencies with TestMultiplexer2.ino, the direct signal and the one that goes through the multiplexer looks almost identical =D.

❬2021-04-14❭

Data Cleaning

The arduino code now produces samples at a consistent 1kHz. I also moved the serial read operations of the calibrator software into a separate thread so that it doesn't slow down on heavy load, causing the buffer to fill up, and the labeling to desynchronize. I am once again confused and surprised that I got ANY useful results before.

I disconnected analog input pin 7 from any electrode, and used it as a baseline for the other analog reads. By subtracting pin 7 from every other pin, the noise that all reads had in common was cancelled out. Hope this doesn't do more harm than good.

I also connected the ground line to one of the wrist electrodes rather than to the palm, since the palm electrode tended to move around a bit, rendering all the other signals unstable.

And did you know that the signals looks much cleaner when you unplug the laptop from the power grid? :p

I'll finish with a video of me trying to play the frustrating one-button jumping game Sienna by flipping my wrist. This doesn't go so well, but maybe this game isn't the best benchmark :D My short-term goal is to finish level 1 of this game with my device.

❬2021-04-13❭

Cyber Gauntlet +1

So if you ever worked with electromyography, this will come to no surprise to you, but OMG, my signal got so much better once I added a ground electrode and connected it to the ground pin of the Arduino. I tried using a ground electrode before, but connected it to AREF instead of GND, which had no effect, so I prioritized other branches of pareto improvement.

I am once again confused and surprised that I got ANY useful results before.

For prototype #3, I moved the electodes further down towards the wrist in hope that I'll be able to track individual finger movements. It had 17 electrodes, 2x8 going around the wrist, as well as a ground electrode at the lower palm. Only 9 of the 17 electrodes are connected, 8 directly to the ADC pins, and one to 1.65V, which I created through a voltage divider using two 560kΩ resistors between the 3.3V and GND pins of the Arduino, so that the electrode signals will nicely oscillate around the middle of the input voltage range.

It all started out like a piece of goth armwear:

Photo from the testing period:

Soldering wires to the electrodes:

The "opened" state shows the components of the device:

But it can be covered by wrapping around a layer of cloth, turning it into an inconspicuous fingerless glove:

If you look hard at this picture, you can see the LED of the Arduino glowing through the fabric, the voltage divider to the right of it, appearing like a line pressing through the fabric, the ground electrode on the lower right edge of my palm, and the food crumbs on my laptop :)

The signal seems to be much better, and as I move my arm and hand around, I can see distinct patterns using the Arduino IDE signal plotter, but for some reason the neural network doesn't seem to process it as well. Will need some tinkering. I hope it was not a mistake to leave out the electrodes at the upper forearm.

I already ordered parts for the next prototype. If all goes well, it's going to have 33 'trodes using analog multiplexers. The electrodes will be more professional & comfortable as well. Can't wait!

❬2021-04-11❭

Adding some AI

Most neural interfaces I've seen so far require the human to train how to use the machine. Learn unintuitive rules like "Contract muscle X to perform action Y", and so on. But why can't we just stick a bunch of artificial neurons on top the human's biological neural network, and make the computer train them for us?

While we're at it, why not replace the entire signal processing code by a bunch of more artificial neurons? Surely a NN can figure out to do a bandpass filter and moving averages, and hopefully come up with something more advanced than that. The more I pretend that I know anything about signal processing, the worse this thing is going to get, so let's just leave it to the AI overlords.

The Arduino Part

The Arduino Nano 33 BLE Sense supports TensorFlow Lite, so I was eager to move the neural network prediction code onto the microcontroller, but that would slow down the development, so for now I just did it all on my laptop.

The Arduino code now just passes through the value of the analog pins to the serial port.

Calibrating with a neural network

For this, I built a simple user interface, mostly an empty window with a menu to select actions, and a key grabber. (source code)

The idea is to correlate hand/arm movements with keys that should be pressed when you perform those hand/arm movements. To train the AI to understand you, perform the following calibration steps:

1. Put on the device and jack it into your laptop
2. Start the Calibrator
3. Select the action "Start/Resume Recording" to start gathering training data for the neural network
4. Now for as long as you're comfortable (30 seconds worked for me), move your hand around a bit. Hold it in various neutral positions, as well as positions which should produce a certain action. Press the key on your laptop whenever you intend your hand movement to produce that key press. (e.g. wave to the left, and hold the left arrow key on the laptop at the same time) The better you do this, the better the neural network will understand wtf you want from it.
   • Holding two keys at the same time is theoretically supported, but I used TKinter which has an unreliable key grabbing mechanism. Better stick to single keys for now.
   • Tip: The electric signals change when you hold a position for a couple seconds. If you want the neural network to take this into account, hold the positions for a while during recording.
5. Press Esc to stop recording
6. Save the recordings, if desired
7. Select the action "Train AI", and watch the console output. It will train it for 100 epochs by default. If you're not happy with the result yet, you can repeat this step until you are.
8. Save the AI model, if desired
9. Select the action "Activate AI". If everything worked out, the AI overlord will now try to recognize the input patterns with which you associated certain key presses, and press the keys for you. =D

Results

I used this to walk left and right in 2020game.io and it worked pretty well. With zero manual signal processing and zero manual calibration! The mathemagical incantations just do it for me. This is awesome!

Some quick facts:

• 8 electrodes at semi-random points on my forearm
• Recorded signals for 40s, resulting in 10000 samples
• I specified 3 classifier labels: "left", "right", and "no key"
• Trained for 100 epochs, took 1-2 minutes.
• The resulting loss was 0.0758, and the accuracy was 0.9575.
• Neural network has 2 conv layers, 3 dense, and 1 output layer.

Video demo:

Still a lot of work to do, but I'm happy with the software for now. Will tweak the hardware next.

Now I'm wondering whether I'm just picking low hanging fruits here, or if non-invasive neural interfaces are really just that easy. How could CTRL-Labs sell their wristband to Facebook for $500,000,000-$1,000,000,000? Was it one of those scams where decision-makers were hypnotized by buzzwords and screamed "Shut up and take my money"? Or do they really have some secret sauce that sets them apart? Well, I'll keep tinkering. Just imagine what this is going to look like a few posts down the line!

❬2021-04-09❭

F-Zero

The look of the first device was way too unprofessional, so I pulled out my sewing machine and made a custom tailored sleeve from comfortable modal fabric.

On the inside, I attached some recycled studs that served as electrodes. Who needs that expensive stuff they sell as electrodes when a piece of iron suffices?

This time it had 4 electrodes. I targeted the middle and the distal end of two muscles, the Brachioradialis and the Extensor carpi radialis longus. I picked those muscles at random, because I honestly don't know what the fuck I am doing.

Software-wise, I played around with moving average and got reasonable signals, but it was clear that there was too much noise.

How to filter, though? I'm not going to solder some bandpass filter, that's too slow and inflexible. There are simple algorithms for doing it in software (link 1, link 2), but something seemed off about this method. In the end, I decided to learn how to do a Fourier transform on the Arduino.

With this code (inspired by this post), I took 64 samples at a sampling rate of 1kHz, performed the Fourier transform, cut out anything under 30% and over 50% of my frequency range, and then summed up the amplitudes of the remaining frequencies to generate the output.

Still very crude, but it allowed me to get distinctive signal patterns for various positions of my arm:

I was genuinely surprised that I got information of this fidelity and usefulness from just hooking up 4 ADC's to semi random places of my forearm and a software bandpass filter. This was good enough to use it as a basic input device!

I wondered, can I control a racing car game with this?

To test that, I built this program to read out the signals and convert certain ranges of values to keyboard presses of the keys Left and Right. The value ranges need to be calibrated before each use: I held my left arm like I'm grabbing an invisible steering wheel, moved it left and right, and looked hard at the signal values to find correlations like "signal A is always below X if and only if I steer left". Once the calibration is done, the invisible steering wheel turned into a magical keyboard with 2 keys =D

Right away I tried it out to steer in my favorite racing game, F-Zero:

Note that in addition to the steering wheel, I used my other hand to accelerate.

I loved it, but there is still a lot of work to be done. The calibration is a pain, especially since it needs to be repeated if the electrodes move too much, which happens a lot with this kind of sleeve. Also I want more electrodes, better signal processing, and data transfer via Bluetooth so I can run it off a battery.

❬2021-04-08❭

Baby Steps

The Arduino arrived. I have no electrodes though. But what are electrodes, just some pieces of metal taped to your skin, right? Let's improvise that:

There are two pieces of aluminum foil taped to my skin, held together with blue medical wrap.

The educational material about electromyographs that I've seen described a chain of hardware elements to process and clean up the signal:

• amplification
• filtering
• rectification
• antialiasing
• smoothing
• averaging
• etc.

But I thought, let's focus on the MVP. Why not simply hook the electrodes straight to the analog input pins of the Arduino with some alligator clamps? Worked fine. I did minimal signal processing in software though, you can find the source code here.

This video shows the myoelectric signal on Arduino IDE's built-in signal plotter:

❬2021-04-03❭

The Idea

On this day, I got the idea and started researching EMG design and signal processing, motor neurology basics, as well as existing projects.

Soon I realized that I will need a microcontroller to record and process the signals. I considered the Raspberry Pi Pico and Arduino Nano 33 BLE Sense, and chose the Arduino because:

• Bluetooth
• More analog-to-digital converter inputs
• TensorFlow Lite support, which would allow me to leverage neural networks for signal processing. This is a bit of a stretch, can't wait to get disappointed by this :)

I wish there was a decent battery/UPS shield, couldn't find one so far.