How Aurora decides a live word is safe to show

Words appear on screen while you are still speaking. The newest ones are dimmed, because they may still change as Aurora hears more context; once a word is shown in full strength it is settled and will not be rewritten during the session. This article is the rule that moves a word from one state to the other, and the constants it is built from.

What this means for you

Three things are worth knowing before the mechanics.

• The dimmed tail is a draft. Those words are Aurora's current best guess at the last second or so of speech. They can change wording, and occasionally a word disappears entirely because the next pass heard the audio differently.
• Full-strength text is settled for the session. Once a word is promoted out of the tail, nothing in the live path rewrites it, and its speaker column is fixed.
• After you stop, the recording is transcribed once more. That pass runs on the saved audio with more context than the live path ever had, and it can replace what you read live. So "settled for the session" is not "final forever" — the final text is what you see after refinement has finished.

Why a live transcript rewrites itself at all

A live transcript rewrites itself because a streaming recognizer is repeatedly asked to decode audio it has already decoded. Aurora runs an intermediate decode every 500 milliseconds over the tail of the current utterance, capped at a 24-second window, and each of those decodes is an independent opinion about the whole window. Later audio legitimately changes an earlier hypothesis: a word heard as "for" becomes "four" once the number after it arrives.

Which leaves a product decision, not a modeling one. Publish every hypothesis and the line flickers and contradicts itself. Publish nothing until the utterance ends and the screen stays empty while somebody is talking. Aurora publishes both, labeled differently.

How Aurora decides a word is stable

Aurora confirms a word when two consecutive decodes agree on it — the LocalAgreement policy from the Whisper-Streaming work — and when enough audio has been captured after it. The agreement prefix is the longest run of words on which the previous hypothesis and the current one match, and the text that gets committed is taken verbatim from the newer hypothesis: normalization decides matching only, never what the reader sees.

Matching on exact strings was the first thing that had to go. Two decodes of the same audio routinely differ by a capital letter, a diacritic, an attached comma or one character, and under exact equality the agreement prefix collapses at that word and the confirmation frontier stalls behind speech the user finished saying seconds ago. The visible symptom is a line that stops growing and then lurches. So comparison folds case, diacritics, width and punctuation, and additionally tolerates a single-character edit on tokens of five characters or more — below that length one edit is a different word ("in" against "it"), not a re-decode wobble.

Why 0.6 seconds of audio instead of two more words

First, what that number is not. The 0.6 seconds is a hold-back rule: a word becomes eligible to confirm once the capture frontier has moved 0.6 seconds past that word's own end. It is not an end-to-end latency figure, and Aurora does not publish one. What you see on screen is this hold-back plus the decode cadence plus however long the agreement rule waits for a second matching hypothesis, and only a measured run on a real device could put a number on that sum.

The rule is audio-based because a word-count hold-back punishes exactly the moment people notice. The earlier version kept the last two agreed words back as the alignment anchor, which meant the final words of every sentence could not confirm until two more words were spoken or the utterance ended — and at the end of a sentence, neither happens. The result was the "nothing, then a chunk" behavior: a pause, then a burst.

Holding audio instead is the evidence that actually matters: the last word in a decode window is the one the recognizer times worst and may have heard clipped mid-syllable. The agreed suffix is still the anchor for clipping the next decode, so alignment is unchanged — only the question "is this word safe yet" moved from counting words to measuring audio.

The duplicated-phrase bug

Word timestamps inside a single decode were trusted to scope the next window, and they are not good enough for it. A timing misalignment can place the anchor's timestamp behind words that were already confirmed; the next decode then re-emits that audio, the re-timed copies pass the end-of-confirmed cutoff, and the phrase confirms a second time. On screen: "Can you Can you".

The fix aligns each hypothesis by text before agreement runs. With an anchor, the hypothesis is expected to start at the anchor; when the anchor appears later and everything before it restates confirmed text, that head is dropped. Without an anchor, the longest hypothesis prefix restating a confirmed suffix is dropped. A genuine repetition survives, because its second copy is the anchor, so the hypothesis already starts there and nothing is cut.

Why the unconfirmed tail is dimmed instead of hidden

Hiding it cost about a second of speech with nothing on screen, and that second is the feedback people record for. Making the committed zone append-only meant publishing only what agreement had confirmed; onset indicators could say "something is being heard", where the words themselves can say what was heard.

So the producer marks each word stable or not, and the reducer keeps both groups. The unconfirmed group is published beside the confirmed one rather than inside it — rendered at 0.35 opacity, never animated, and excluded from the traces that check display correctness, because a reader cannot be expected to rely on it. Plenty of products show a provisional tail; the difference worth having is saying which part of the line is provisional.

Why the speaker label lags behind the words

Because a label that changes as fast as the evidence does is unreadable. Raw diarization windows and stability-gated turns both used to write the live speaker immediately, so the live row flipped voice mid-utterance and dragged the line between speaker columns. Both now propose into a dwell: a candidate must survive 32,000 samples — two seconds at Aurora's 16 kHz capture rate — of audio progress before it may color the live row, and it changes only at turn granularity.

The dwell is measured on the sample clock of the newest confirmed word, not on wall-clock time, which makes it deterministic under replay: the same recording fed through the pipeline twice produces the same labeling decisions at the same positions. Committed rows are never revisited during the session — once a row is in the confirmed zone its speaker is permanent for the live pass, and a late diarization opinion changes the future rather than the past.

What changes after the conversation ends

Live confirmation and final text are two different guarantees, and conflating them is the easiest way to be surprised by Aurora. Everything above is about the live pass: what may be drawn on screen while the microphone is open, and what may never be redrawn once it is.

When a session stops, Aurora queues an offline refinement pass over the recorded audio and works through it in the background. That pass is not bound by the live rules: it sees whole utterances rather than a rolling window, and the transcript blocks it produces supersede the blocks committed live. Words you watched confirm can therefore read differently in the finished transcript, and speaker runs can be relabeled by the same pass.

That is a feature rather than a contradiction: the live text is optimized for keeping up with the room, the refined text for being right. It does mean the honest way to describe the two live zones is "settled for this session", not "final". If you are checking a quote, check it after refinement has finished.

What this costs, and what Aurora still gets wrong

The confirmed line runs at least 0.6 seconds behind the room, plus the decode cadence, plus whatever the agreement rule is waiting for. A word can also appear in the dimmed tail and then not appear at all, because the next decode heard the audio differently — which is the tail's stated contract, and still surprising the first time it happens. And Aurora deliberately optimizes for the transcript rather than for captioning: a live caption system would publish faster and accept rewriting text a reader has already read.

The remaining weakness is not the confirmation rule, it is the speaker. A two-second dwell keeps the live row calm but cannot rescue an attribution the ownership stage got wrong; what Aurora's speaker attribution actually scores publishes those numbers, including which stage of the pipeline loses them.

References

1. Macháček, Dabre, Bojar. Turning Whisper into Real-Time Transcription System. arXiv:2307.14743 — the local-agreement streaming policy Aurora's confirmation rule builds on.
2. Radford et al. Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2212.04356 — Whisper, whose window-based decoding produces the shifting hypotheses described here.
3. Medennikov et al. Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering. arXiv:2507.18446 — the source of the live speaker proposals the dwell smooths.

Related

1. What Aurora's speaker attribution actually scores — measured WDER, and the stage that damages it.
2. What on-device transcription costs on an iPhone — the throughput this cadence is spending.
3. What happens to your audio after you press record — where the samples these decodes read are written.