Lip reading transcription on a 2020 iPhone

Here's a screen recording of an iPhone 12 Pro (six years old now!) correctly reading my lips and transcribing what I mouthed, all in a couple of seconds end to end.

This builds on lipflow, Amy Zhou's lip-reading dictation for desktop computers, which uses a research model called Auto-AVSR. We've written a fair bit about getting speech models to run well on phones, and lip reading seemed like a neat fit for our app, Crosstalk, so we ported the pipeline to iOS to see how good we could make it.

It works! Sadly we can't ship it, because Auto-AVSR's published weights are for non-commercial use only. To release something like this we'd have to train our own model on video we're allowed to use. Maybe one day, if there's enough demand!

Here's a comparison of our iOS port against lipflow running on a laptop:

The size saving all comes from squashing the model's weights. We got the encoder (the bit that looks at crops of your mouth) running on the iPhone's Neural Engine, with its weights palettized down to 4 bits. The decoder and the language model (which write out the words and check they make a sensible sentence) run on the CPU, quantized to 8 bits.

Most of the speedup is in the search, the step that turns the model's output into sentences. We rewrote lipflow's Python search in Swift and had the decoder and language model cache their earlier steps, which took it from 1.16 seconds to 0.30. For the app demo above, we also tracked the face on every other frame rather than every frame, and did the tracking while the video was still recording for maximum speed.

The biggest jump in accuracy came from fine-tuning to my face (as expected). With 20 minutes of footage of me mouthing sentences and 2 minutes on my RTX 3090, the word error rate went down by more than 40% for both the original lipflow and our iOS port.

If you'd like to see this in Crosstalk one day, let us know!