Oído: Open-Source Speech Recognition on a $5 Chip

Speech recognition on a microcontroller has, in practice, meant one of two things: a wake word, or a short list of commands the device was programmed to expect. Say anything else and it hears nothing.

Today we’re releasing Oído: speech recognition for any English sentence, running entirely on an ESP32-S3. That’s a $5 microcontroller with a 240 MHz dual-core CPU, 8 MB of PSRAM, 16 MB of flash and no neural accelerator. No cloud, no command list, and it’s open source. It is not real time yet: on the board it runs at about twice the length of the audio (details below).

¡Oído! is what cooks call out in a Spanish kitchen to confirm an order: heard, got it.

How accurate it is

Word error rate on LibriSpeech, the standard English benchmark (lower is better):

System	Runs on	test-clean	test-other
Oído, int8	ESP32-S3	3.7%	8.2%
Oído, int4 (8.3 MB)	ESP32-S3	4.6%	10.0%
Espressif MultiNet7	ESP32-S3	8.5%	21.3%
Moonshine tiny	laptop	5.0%	12.1%
Whisper tiny.en	laptop	6.3%	15.9%

On the same chip, Oído makes 2.3 to 2.6 times fewer errors than Espressif’s own recognizer, which matches speech against a list of up to 200 predefined commands. It also makes fewer errors than Whisper tiny and Moonshine tiny running at full precision on a laptop. That holds in noise too: across 14 conditions built from real car, kitchen and cafeteria recordings, background chatter and room echo, Oído averages 8.4% against 12.1% for Whisper tiny.

Squeezing the model onto the chip costs almost nothing. The int8 version is within about a tenth of a point of the original full-precision model (3.70% vs 3.68% on test-clean).

The model is NVIDIA’s. The work is the engine.

Oído runs NVIDIA’s openly licensed Conformer-CTC Small, a 13-million-parameter speech model. We didn’t retrain or distill it. The hard part was making it run on a chip whose fast internal memory is 512 KB, when the model alone is 14 MB.

The obstacle isn’t arithmetic, it’s memory bandwidth. The weights live in flash, which the chip reads through a small cache at a few tens of megabytes per second. So we wrote a new inference engine in C for the ESP32-S3’s vector instructions, designed around that limit:

• Integer matrix kernels that do 16 multiply-adds per instruction.
• Integer attention, softmax included, so the whole model runs in int8 arithmetic.
• A bandwidth-aware schedule that processes 64 frames of audio at a time, so each weight is read from flash once per block instead of once per frame. This cut weight traffic from 18 to 7.5 MB per second of audio.
• Both CPU cores working in parallel.

There’s also an int4 version that takes 8.3 MB of flash instead of 14 MB, leaving 6 MB for your own application, and an optional on-chip language model that brings the error rate down to 3.3% and 7.2%.

Where it stands

We want to be precise about this. The accuracy numbers above come from the host build of the on-chip engine: the same C code with the same int8 arithmetic. We have now also run the firmware on a physical ESP32-S3 board (ESP32-S3-WROOM-1-N16R8), and the board’s transcripts are identical to those of Espressif’s instruction-exact emulator on all 27 clips we ran. The laptop’s math library rounds the last bit differently from the chip’s, which changes a word on some hard, noisy clips (22 of those 27 are identical to the host build, and the error rate on the 27 is the same).

The speed is the problem. With one core, the English model with the language model runs at 1.97 times the length of the audio (1.91 to 2.02 over 27 clips); the int4 model at 1.76, the Spanish model at 2.06. A 4-second command shows its text about 8.7 seconds after it ends, including the 0.8-second pause we wait to be sure you are done. Earlier we estimated 0.7 to 0.95 times real time from instruction counts, and that estimate was wrong: it assumed the second core would work and about 1.5 cycles per instruction, while the chip needs 2.0. The second core does not work yet: when both cores run the vector kernels, the transcripts come out wrong or the firmware crashes. So today Oído is a research prototype for short commands where a few seconds of delay are acceptable, not a live recognizer.

Two other limits: it handles English and Spanish only (see the update below), and in utterance mode it transcribes after each utterance rather than word by word.

Update: a streaming mode

Several of you asked about the wait. We added a streaming variant that processes audio as it arrives, in 1.28-second chunks, and shows partial text while you speak. The price is accuracy, because the model can’t see what comes after each chunk: 4.9% / 11.0% on LibriSpeech with the language model, against 3.3% / 7.2% in utterance mode. That is still ahead of Espressif’s recognizer (8.5% / 21.3%) and Whisper tiny (6.3% / 15.9%), and under noise it averages 9.0% across our 14 conditions against 12.1% for Whisper tiny, because we trained it with noise, music, babble and reverberation.

On the board it runs at 1.96 times real time, so it falls behind the speaker and the backlog grows with the length of what you said. Feeding clips at the rate of speech, the final text appeared 4.1 to 11.1 seconds after the clip ended for clips of 3.6 to 10.6 seconds (about 4 seconds for a short command), against 7.2 to 22.4 seconds in utterance mode. We had estimated 1.1 to 1.4 seconds from instruction counts before we had a board; that needs a speed below one, and we don’t have it yet. Try it with python live_demo.py --model stream; the model is on Hugging Face under CC-BY-SA-4.0.

Update: Spanish

There is now a Spanish model for the same chip and engine, and it is open too (CC-BY-4.0). It is NVIDIA’s Conformer with a new Spanish vocabulary, fine-tuned on about 2,500 hours of Spanish with noise and reverberation augmentation, plus a Spanish language model. On the same 400-utterance test subsets it scores 13.3% on Common Voice, 10.6% on Multilingual LibriSpeech, 15.5% on VoxPopuli and 11.3% on FLEURS with the language model, against 33.0, 21.5, 28.7 and 17.1 for Whisper tiny (multilingual) on a laptop.

Read that fairly: we trained on the training splits of those corpora and Whisper tiny is zero-shot, so the comparison favors us on these domains, and phone calls, strong regional accents or specialized vocabulary will be harder. On the board it runs at 2.06 times real time, and its transcripts match the host build on 9 of 12 clips (on those 12 the board scores 9.3% against 7.3% for the host build, a four-word difference). Try it with python live_demo.py --model es; the model is on Hugging Face.

Why we built it

Voice is moving off the cloud. A device that streams audio to a GPU pays for inference for as long as it exists, stops understanding people when the connection drops, and sends their voice somewhere else. On-device recognition removes all three problems, but until now it also meant giving up open-ended speech. Oído shows that a commodity $5 chip is enough for the real thing.

It’s the listening half of the on-device voice stack we’re building at Lokutor for this class of hardware.

Try it

Everything is on GitHub at github.com/lokutor-ai/oido, and the models are on Hugging Face: int8 and int4.

You don’t need a board to try it. The host build runs the same engine as the chip on your laptop, including a live microphone demo (Python with numpy, soundfile, sentencepiece and sounddevice):

git clone https://github.com/lokutor-ai/oido
cd oido/esp32/host && make
python live_demo.py

To run it on hardware you need an ESP32-S3-DevKitC-1 N16R8 and an INMP441 microphone. The README has the wiring, a one-line flash script, and the raw board measurements.

Open source, with a commercial option

The code is licensed under GPLv3. The int8 model is CC-BY-4.0 and the int4 model is CC-BY-SA-4.0. For products that can’t meet GPLv3 terms, we offer commercial licenses, models for other languages and integration support. Write to us at contact@lokutor.com.

We’re also writing up the full technical details as a paper. If you build something with Oído, we’d love to hear about it.