The PsyLink project now has it's own website: psylink.me, and this is the place where I will continue the development log, as soon as I finish the basic structure of the website.❬2021-05-17❭
Gyroscope + Accelerometer
I fixed up the PsyLink UI. It was so broken after the rewrite to Bluetooth Low Energy, I'm once again stunned that I got ANY useful results before. But now it receives the transmissions from the Arduino properly.
I also added 6 more signal channels: The x/y/z-axes from the Gyroscope and from the Accelerometer that are built in to the Arduino Nano 33 BLE Sense.
All put together finally allowed me to singlehandedly drive through the finish line of my favorite racing game F-Zero! \o/
- For training the AI, I recorded 60k samples over 2 minutes (500Hz)
- Trained for 5 epochs, which took 1 minute without GPU acceleration
- 4 training labels ("left", "right", "accelerate", "nothing")
- Final training and validation loss: 0.04, accuracy: 98%
- 6 layer neural network with 2 convolutional layers
- Music by Mitch Murder
Hell yeah! The PCBs arrived:
Soldering & Sewing
I never soldered such tiny SMD parts before, and didn't have proper tools, way too thick soldering tin and solder iron tip. I was also too impatient to order some, so after hours of torture, I produced this batch:
The new prototype was to be a forearm sleeve of modal fabric once again, with snap buttons for electrodes which will also hold the signal processing PCBs in place.
But how to attach the Arduino and the power supply module to the sleeve? I thought, "why not Velcro?" (hook-and-loop fastener) and started sewing it to the circuit boards:
(Yes, doing it felt as weird as it looks)
So I sewed the sleeve, assembled one electrode pair along with its processing PCB, and wired everything together. Here's me being overly excited about the first wireless test run:
Then there was the question of where to put the electrodes. Using an improvised muscle map along with two flexible electrodes on individual straps, I could find spots whose electrical activity correlated with turning the arm, twisting the wrist, or pressing individual fingers onto the table:
Flexor Digitorum Superficialis was particularly interesting; I found 3 areas over that muscle which map to the index, middle and ring finger each. For turning the arm and wrist, the muscles with "Carpi" in their name (e.g. Extensor carpi ulnaris) worked pretty good.) A huge disappointment was Extensor Digitorum, which is supposed to be active when fingers move up, but I could not find such correlation. Then again, I use snap buttons for electrodes, so I'm not that surprised.
The final layout of the electrodes:
This piece is fully separable from the electronics and therefore machine washable.
Here are additional pictures of the inner side, the separated electronics, as well as everything combined. This nicely shows the tree topology of the green signal processing modules that pass through the power supply among each other to reduce the volume of wiring.
I could have had 8 electrode pairs, but only added electrodes for 7. On these pictures, the electrode pair for the middle finger is also missing the circuitry. That's mostly a testament to my laziness.
Actually I regret where I placed the Arduino, since there would be some great spots for electrodes, but I noticed that too late. Will try to remove the Velcro and maybe add an 8th electrode pair there.
The final cyb3rware:
While the signal was quite strong with the test straps, I found that the amplitude of the signal went way down once I had everything attached to the sleeve. Maybe there was some kind of interference from the Arduino or the power supply being closer to my skin, or maybe the modal fabric messes with the signal somehow. I hope I can compensate for this by increasing the signal amplification multiplier, but I leave that for later.
This issue occurred with a single electrode pair already, but was aggravated when attaching more of them. It might help if I add some flux capacitors to the power supplies to prevent cross-interference.
I drove F-Zero with Prototype #2 before, but back then I cheated a little bit. It only recognized 2 keys, left and right, and I accelerated with the keyboard using my other hand.
This time I hoped I could do better, and trained the AI to recognize 3 different keys (left, right, accelerate) from my muscle signals. It even kinda worked!
This was after recording ~2000 muscle signal samples over 1-2 minutes and training a convolutional neural network for 25-50 epochs (<1 minute) on the data using the PsyLink UI. I used 4 electrode pairs, all of which are on the dorsal side of the forearm.
In the racing game, I didn't make it to the finish line yet, and it does look pretty clumsy, but I blame it on the software still having some obvious flaws. It doesn't even account for packet loss or packet duplication when handling the Bluetooth packets yet. Hope it will go better once I fixed them. Also, the test drive was with only 4 electrode pairs.
The raw values as visualized with the GNURadio flowgraph while randomly moving my forearm/wrist/hand show that the correlations between the signals are low enough to be theoretically useful:
If you enlarge this image, you'll see that especially the black line is considerably different, which I suppose is because it's the only electrode pair that spans across several muscles. And that makes me wonder: Am I doing too much pre-processing in hardware before I feed the data into the AI? Sure, the differential amplification of this new prototype enhances small signals that the previous prototypes might have not detected, but a lot of information is lost too, like the voltage differences between electrodes from different electrode pairs.
Maybe I can compensate for this by simply adding some more electrode pairs that span muscles. I'm also thinking of switching to a design with 32-64 randomly placed electrodes -> buffer amplifiers -> multiplexers -> analog to digital converters of the Arduino. That way, the neural network can decide for itself which voltage differences it wants to look at.
New PCB layout
While soldering the PCB, I found some flaws and made these changes to the previous PCB:
- Added silkscreen labels to the connectors and components. I was sure I wouldn't mix up anything since there aren't many connectors, they're nicely symmetrical, and I'm the designer after all. But nope. I still mixed them up.
- Removed unnecessary vias. (I was actually not sure whether the pin holes will really conduct between the front and the back side of the board, so I added vias as a safety measure.)
- The label now shows the new name "psylink" instead of "myocular"
- A friend also gave me the tip to increase the thickness of power supply wires
Power Supply Module
I made an updated schematic (circuit 6) that shows more clearly how the modules are connected. Also corrected an error with the feedback of the voltage follower, and changed values of some resistors/capacitors:
I also constructed the power supply module:
but for some reason it didn't work. All the parts seemed to have been connected the right way, I couldn't find a short circuit, but the output voltage was ~0.5V instead of ~5V, and the reference voltage was just 0. I blame a possibly broken opamp.
Well, I didn't like the design and length of the circuit board anyway, so it didn't hurt trashing the thing and building this beauty instead:
I'll use female-to-female jumper wires to connect V+ and GND to the arduino, and 3 more wires to bootstrap the power supply of the mesh network of the signal processing modules.
- Yep, that's 2 coin cells in there
- Outputs: Black=Ground, Red=V+, Green=V+/2 (reference signal)
- There's an optional second green pin for the ground electrode
- If you're wondering why I'm using a big ass quad opamp when I just need a single output: I don't have a smaller one atm.
- I totally need to move this to a SMD PCB in the long run, this is still too bulky, but will do for now. It's about the dimensions of a 9V battery.
I wonder if some 深圳人 will read this, shake their head, and view me as a primate struggling to make fire with sticks. That's what it felt like to construct this thing anyway. Nevertheless, I'm one step closer to the next prototype :)❬2021-05-07❭
After some brainstorming, I changed the working title of this project from Myocular to ✨PsyLink✨. The close second favorite was FreeMayo (thanks to Vifon for the suggestion). Free as in free speech/software/hardware, and mayo as a play on myo (ancient greek for "muscle"). But somehow I liked PsyLink more. It's inspired by the Psionic Abilities from the 1999's game System Shock 2.
FYI, this is the list of words that I considered, although unfortunately many of the coolest combinations were taken:
Finished new UI
The new user interface now supports all previous features!
- Capturing muscle signals
- Capturing the keys that the user is pressing
- Training a neural network to predict key presses from given signals
- Auto-pressing keys based on incoming signals using said neural network to predict which keys the user wants to press
It's sooo much more pleasant to have a direct view on the state of the application and an instant visualization of the signals. The previous version was literally just a blank window, with a single menu called "File" that contained all the actions. :D I never even bothered to upload a screenshot, but here's one for documentation purposes:
Also, this time I used clean & efficient data structures to make the code easier to work with, a more reliable key capturing library (pynput), and threads to prevent one activity from blocking the others. The signals obviously go via Bluetooth instead of a wired serial connection.
I'm also thinking of changing the name for the project, since people are reading it as "my ocular" rather than recognizing the neologism made of "myo" (for "muscle") and "ocular" (from "eye"). But all the good names are taken, of course. -_-❬2021-05-04❭
Higher Bandwidth, new UI
Hah, I managed to raise the Bluetooth bandwidth from ~1kB/s to 6-7kB/s with this one magic line:
It raises the power consumption by 4% (3.5mW), but that's totally worth it. I can now get all 8 channels in 8-bit resolution at 500Hz across the aehter. Eventually I should aim for 10-bit at 1kHz, but I think that can wait.
This is the GNURadio flowgraph and the resulting output. (I only have hardware for 2 electrode pairs, so even-numbered and odd-numbered signals are wired to the same input. Still waiting for the PCBs.)
- No Bluetooth connection: 86.9mW (16.9mA x 5.14V)
- Transmitting at 1-2kB/s: 88.9mW (17.3mA x 5.14V)
- Transmitting at 6-7kB/s: 92.5mW (18.0mA x 5.14V)
Surprisingly to me, the LEDs were draining a good chunk of the power, and I saved 16mW by removing the external power LED (see previous photo) and by PWM-dimming the blue LED that indicated Bluetooth connections. It gives me approximately 15 hours run time with 2x CR2032 coin cells.
Also I'm in the process of rewriting the UI:
The colorful column graph is a live visualization of the signal. The columns correspond to electrode pairs, while the rows are time frames. The top row shows the amplitude of the signal at the current time, and the rows flow downward, allowing you to view changes back in time, as well as correlations between signals.
You'll also be able to change settings on the fly, view the status of e.g. key recordings or machine learning processes, and more. All of this is in a modular library that will also be usable from e.g. GNURadio.
I was thinking of changing the graphical user interface toolkit from Tkinter to a more modern one, because Tkinter looks a little shabby, and it has problems determining which keys are currently pressed, but I decided against it, because I made the experience of being unable to run my own software several years after writing it because the exact version of the GUI toolkit, along with all dependencies, was too annoying to set up. Tkinter has been around for decades and will probably stay, so I'll stick with it for now. Also, I can easily solve the key pressing issue with an external key tracking library like pynput.
Can't wait to try out the new UI with 8 individual electrode pairs, once the PCBs arrive! (assuming they work :'D)❬2021-04-30❭
Today I made a new version of the PCB that processes the signals from one electrode pair:
Actually, several versions. This is the 4th iteration, and let's not even look at the previous ones because they were just plain wrong. I stared at this design for a long time though and couldn't find another problem, so I went ahead and ordered 30 pieces of it. Can't wait to find out in what way I messed up :'D And hey, maybe it'll actally work.
- Dimensions: 20x17x1.6mm, rounded corners
- Tiny enough to fit between 2 electrodes!
- 2 connectors for electrodes, at the middle top & bottom
- 1 connector for the output at the bottom right corner (on the front side. The back side is mirrored)
- 3 power line connector ports on the other corners, with 3 pins each:
- The signal reference voltage
- +6V from the battery
- 3 capacitors, 3 resistors, 1 integrated circuit
To avoid having a kilogram of cables on the device, this board supports wiring in a mesh network topology, where the boards share the power lines amongst each other using the redundant power line connector ports. One board can power two other boards, which in turn can power 4, and so on.
The bypass capacitor between ground and V+ will hopefully keep the voltage stable, though I'm a bit worried about the reference signal. If necessary, I can "abuse" the reference signal pin of the power line connector ports to add extra ground electrodes. I considered adding an extra opamp on every board to generate a fresh reference voltage but that would make the circuit too big for my taste.❬2021-04-29❭
Soldering the Processing Units
The plan was to split the circuit into:
- 1 central part including the Arduino, power supply and the OpAmp that generates the signal ground, and
- 8 distributed signal processing units, embedded in hot glue for stability and electrical insulation, consisting of an instrumentation amplifier and related components, close to the electrodes to avoid signal degradation.
Here's my try to solder one of those units:
This took me over an hour, during which I began questioning various life choices, started doubting this whole project, poured myself a Manhattan cocktail, wondered how long it would take to complete all eight of these, whether it will even be robust enough to withstand regular usage of the device (NO, IT WON'T), and how I'm going to fix the inevitable broken solder joints when the entire thing is in fucking hot glue...
I gave up, and now my plan is to get PCBs for this instead. I have little experience with this, so I've been putting it off, but how hard can it be?
I removed the decouplying capacitor between ground and GNDS (signal ground) by the REF pin of the INA128 because mysteriously it made the signal worse, not better. Also removed the 1K resistors between electrodes 1+2 and the respective capacitors, because they served no apparent purpose.
Also, I was frustrated that GNURadio doesn't allow you to get a "rolling" view of a signal. The plot widget buffers as many samples as it can show, and only when the buffer is full, it updates the graph, clears the buffer and waits again. I wanted instant updates as soon as new samples are in, and as a quick&dirty workaround I wrote a GNURadio shift block which keeps filling up the buffer of the plotting widgets.
I'll finish with a nice picture of a finger snap, as recorded with one electrode pair on my dorsal wrist. Click to enlarge and view the frequency domain as well. (Just one electrode pair because that's all I can squeeze out of the poor bluetooth low energy bandwidth so far)
I've been battling with reducing the power line noise for too long, so I thought screw it, let's go off the power line entirely. I put the circuit on two 3V CR2032 coin cells and wrote some code to transmit the signals via BLE (Bluetooth Low Energy) using the ArduinoBLE library.
Since I can not plot the signals via the Arduino IDE plotter anymore, I switched to GNURadio and wrote a plugin that establishes the BLE connection and acts as a signal source in the GNURadio companion software
My new "electrodes" also arrived: Simple prong snap buttons. They don't have sharp edges like the pyramidal studs I used before, and allow me to easily remove the wires from the electrodes and plug them in somewhere else as needed.
I also employed INA128 instrumentation amplifiers, drastically reducing the complexity of the circuit. It's a tiny SMD chip, which I plan to embed in hot glue, along with the 3-4 capacitors and 3-5 resistors required for processing/de-noising, and place 8 of these processing units across the glove/wristband, connected to two electrodes each.
Now I'm battling the problem that I can only get about 1kB/s across the ether. How am I supposed to put 12kB/s worth of signal in there? (8 channels, 1k samples/s, 12 bit per sample) Let's see if I can find some nice compression method, but I fear that it's going to be lossy. :-/