# What Aurora's speaker attribution actually scores > Measured word-level speaker error on four public-corpus suites, the stage of Aurora's own pipeline that damages it, and the downloadable per-case results behind every number. ENGINEERING · 18 SEPTEMBER 2026 · 12 MIN READ How often Aurora puts a word in the wrong speaker's column, measured on four public-corpus suites, plus the stage of Aurora's own pipeline that damages the result and what both mean if you are about to record a four-person meeting. Aurora engineering **Aurora** separates speakers on the device, from one mixed microphone signal, while the conversation is still happening. This article publishes what that scored on four public-corpus suites, which layer of Aurora's own pipeline damages the result, and how many of the thresholds making those decisions have any recorded evidence behind them. The short version: separating the audio into voices is competitive, putting the right word in the right column is not yet, and Aurora's own source says so. ## What this means for you Two or three people at a table, an interview, a lecture with questions: this is where Aurora's separation behaves the way the product implies, and where a wrong speaker label is an occasional edit rather than a running annoyance. Four voices in a hard room is the hard case in the research literature and the hard case here. On the worst suite in this set — four people around one distant microphone, talking over each other — roughly two words in three carried the wrong speaker label. On real four-speaker earnings calls, with clean remote audio and dense turn-taking, it was roughly one word in three. If your meeting looks like that, plan on correcting speaker labels afterwards. One thing Aurora does not do is guess *who* anybody is. Voices are separated automatically and named by hand, so an attribution error costs you an edit — it does not put a stranger's name on a sentence in a searchable archive. The rest of this article is the measurement, its limits, and the files you can check it against. Every number below was measured on 17 June 2026 and has not been re-run since. ## How Aurora scores speaker quality Aurora's load-bearing metric is **word diarization error rate** (WDER): the share of recognized words assigned to the wrong speaker. A WDER of 0.34 means about 34 of every 100 words are attributed to the wrong person. Lower is better, and 0 would mean every word landed in the right column. The older metric, **diarization error rate** (DER), scores time rather than words: how much of the audio was labeled with the wrong voice, counting speech the system missed, silence it attributed to somebody, and speech it gave to the wrong person. The two are not interchangeable. A system can score a respectable DER while still putting the first three words of every answer in the wrong column, because those words are short and the surrounding silence is long. For a transcript product the word is the unit the user reads, so the word is the unit that has to be scored. Aurora therefore reports both, separately: DER for comparability with published diarization work, WDER for the thing people actually notice. Where a corpus ships only time-domain reference labels, the scorer reports WDER as unavailable rather than estimating it. ## What Aurora scored, corpus by corpus Four suites, four acoustic regimes. Three of them ship word-level speaker references, so they can be scored with WDER; the fourth, MSDWild, ships time-level segments only and is therefore a DER-only suite. The spread across them is the honest picture of what on-device separation does today. Word-level speaker error (WDER) — the three suites with word-level speaker references. Lower is better. Aurora end-to-end, recognition model openai\_whisper-base, unified pipeline configuration, one run per case, measured 17 June 2026 at app commit c029825e. [Per-case results (CSV)](https://auroraisyours.com/evidence/speaker-attribution-2026-06-17.csv). | Corpus | Cases scored | WDER (min–median–max) | Transcript word error (median) | | --- | --- | --- | --- | | LibriSpeech, synthetic turns | 10 of 10 | 0.05 – 0.09 – 0.19 | 0.04 | | Earnings-21, real calls | 6 of 8 | 0.20 – 0.34 – 0.49 | 0.16 | | AMI, single distant mic | 3 of 3 | 0.63 – 0.66 – 0.71 | 0.50 | The second table is the time-domain metric, which is the one published diarization work usually quotes. The anchor column is pyannote 3.1 as published by Argmax OpenBench: a different pipeline, running different models, on the same corpora. It is context for what these corpora cost a diarization system in general — not a like-for-like comparison with Aurora, and nothing here should be read as Aurora beating or losing to pyannote. Time-level speaker error (DER) — all four suites. Lower is better. Same run as above. Anchor: pyannote 3.1 as published by Argmax OpenBench, a different pipeline and different models, quoted as context only. | Corpus | Cases scored | DER (median) | Anchor, different pipeline | | --- | --- | --- | --- | | LibriSpeech, synthetic turns | 10 of 10 | 0.08 | — | | Earnings-21, real calls | 6 of 8 | 0.28 | 0.10 | | AMI, single distant mic | 3 of 3 | 0.30 | 0.23 | | MSDWild, in the wild | 4 of 4 | 0.10 | 0.32 | Two rows deserve to be read against each other. On MSDWild — the highest-overlap public corpus in the set — Aurora's median DER of 0.10 is low in absolute terms. On AMI, DER of 0.30 is also in the region the anchor occupies, while WDER on the same audio is 0.66. Cutting the audio into voices is not the problem. Putting words on the right side of a boundary is. These figures are a snapshot, not a status. They were produced on 17 June 2026 at app commit c029825e. Since that date 118 commits have touched Aurora's transcription and diarization sources — among them an opt-in, on-device speaker-attribution correction pass (commit 3905fc7d, 24 August 2026) aimed at exactly the failure described below — and the suite has not been re-run. Read this as the last published measurement, not as a description of today's build. ## Why the AMI number is so much worse AMI single-distant-microphone audio compounds two failures at once. Recognition on those slices scores a word error rate of 0.47 to 0.52 — four people around a table, one microphone in the middle, crosstalk throughout — and WDER is computed over an alignment between reference words and recognized words. When half the words are wrong, the alignment that speaker labels are scored against is itself poor, and the errors multiply rather than add. For a user, that regime is the one to be wary of: one phone on a table in a reverberant room with four people interrupting each other degrades the transcript first and the speaker columns second, and the two make each other worse. A microphone closer to the talkers, or one device per side of the table, is worth more than any setting in the app. For the engineering, AMI is a parity probe and not a quality target. The useful signal in it is comparative: if Aurora's DER were drastically worse than published anchors on audio the diarization model has seen in training, the gap would be pipeline-inflicted. It is not drastically worse, which points the investigation at attribution instead of at the acoustic model. ## Why Aurora can't quote its best-looking numbers Because the best-looking numbers here are measured on audio the diarization model was trained on. Aurora's live speaker model is a streaming Sortformer variant, and its published training data includes AMI, ICSI, VoxConverse, DIHARD-III dev, AliMeeting, DiPCo and simulated mixtures built from LibriSpeech. A suite built from any of those can be a regression gate or a parity probe; it cannot be a quality claim, and it must never become a tuning target — tuning against training data optimizes for a number nobody will experience. Contamination verdicts, recorded per suite in Aurora's suite README and in each suite manifest, with the model card checked before import (verified 11 June 2026) | Suite | In the diarization model's training data | What it may be used for | | --- | --- | --- | | LibriSpeech, synthetic turns | Yes, as simulated mixtures | Determinism and regression floor | | AMI, single distant mic | Yes, by design | Parity probe against published anchors | | Earnings-21 | Verified absent | Quality numbers, and the only tuning target | | MSDWild | Verified absent | Wild-acoustics stress, DER only | So of the four suites above, exactly two can carry a quality statement at all, and the 0.34 on the cover of this article is the one that comes from them. The rule in Aurora's repository is that the model card of every inference model is checked before a dataset is imported, and the verdict recorded in the suite manifest — which is why the table above exists at all. We have not audited anyone else's benchmarks and make no claim about them. ## Which layer loses the speakers Aurora's own attribution stage does. A layer-by-layer audit on one four-speaker round-table fixture measured DER at every hand-off between the diarization model's raw output and the words displayed to the user, and the damage is visible at a single stage: the word-ownership policy takes speaker evidence at DER 0.15 and produces word decisions at DER 0.41. Time-level speaker error (DER) at each stage, one four-speaker round-table fixture, run boundary\_v6\_013\_boundary\_finalizer\_rerun\_20260609, 9 June 2026 — a single fixture, not a corpus average. [Stage results (CSV)](https://auroraisyours.com/evidence/speaker-stage-der-2026-06-09.csv). | Stage | DER | | --- | --- | | Raw live diarizer output, collapsed | 0.194 | | After the stability gate | 0.150 | | Word-ownership policy decisions | 0.407 | | Live committed words | 0.214 | | Refined transcript words | 0.262 | | Displayed transcript words | 0.228 | The stability gate improves the diarization model's own output, from 0.194 to 0.150. Then the stage that converts speaker regions into per-word owners throws most of that away. The recovery further down the table is not a fix; it is later stages partially undoing an earlier decision, which is the shape of a bug rather than of a tuning opportunity. The caveat matters as much as the numbers: this is one synthetic fixture from the regression-floor suite, measured once. It localizes a defect; it does not size it. ## How many of those thresholds are justified Nine of thirty-seven, and none of the nine by a measurement. Aurora's turn-assembly decisions are driven by a single tuning table in which every threshold carries its value, its unit, its rationale and its provenance, and provenance is one of three things: validated by a named eval fixture, derived from another documented invariant, or inherited without recorded evidence. Provenance of every attribution threshold — TurnAssemblerTuning.swift, app commit 8c827033, unchanged as of 18 September 2026. [All 37 thresholds (CSV)](https://auroraisyours.com/evidence/attribution-thresholds-2026-09-18.csv). | Provenance | Count | Examples | | --- | --- | --- | | Validated by a named eval fixture | 0 | — | | Derived from a documented invariant | 9 | 16 samples per millisecond at 16 kHz; 40 s in-memory retention of committed blocks | | Inherited, no recorded evidence | 28 | 500 ms boundary zone; 0.50 overlap share; 0.15 decision margin; 3-word boundary phrase | Those twenty-eight numbers are not arbitrary — each has a rationale written beside it, and each was arrived at while fixing something real. What none of them has is a run proving the value is better than a neighboring one. Keeping that distinction in the type system, where a unit test asserts the table cannot drift from the constants it documents, is what stops a repair cascade from being retold later as a calibration study. A threshold is only labeled eval-validated when a named fixture demonstrably justified its value. Twenty-eight of the thirty-seven in Aurora's attribution table are not, and the code says so rather than inventing a story for them. Aurora's turn-assembly tuning table, honesty rule The next measurable step is therefore not a larger diarization model. It is the ownership stage in the table above, and the twenty-eight thresholds inside it that nothing has yet justified. ## Methods, and how to check this yourself Everything needed to reproduce or contradict the figures above is listed here, and the aggregate results are published as files you can download. Those files carry scores and corpus provenance only: no audio, no transcript text, no personal data. Measurement conditions for the corpus scores in this article | Field | Value | | --- | --- | | Run date | 17 June 2026 | | Build | Aurora app repository, commit c029825e (*test(evals): TDT/unified suite reports, eval runner, and coverage map*) | | Recognition model | openai\_whisper-base, 145 MB, d\_model 512. Aurora's suite reports record it under the internal tier id *medium*; the tier-to-model mapping is documented in the app repository. | | Speaker model | FluidInference/diar-streaming-sortformer-coreml — NVIDIA Streaming Sortformer, four speaker slots, Core ML | | Pipeline configuration | uni\_full, the unified transcription-and-diarization path, all repair stages enabled | | Device | The maintainer's Apple silicon Mac, through Aurora's Swift eval harness with silent file-based audio injection. These are **not** iPhone measurements. | | Audio | 16 kHz mono, 16-bit, corpus slices between 28 s and 67 min | | Datasets | LibriSpeech clean validation (synthetic turns, 300 ms inter-turn silence), 10 cases; Earnings-21 from rev.ai speech-datasets, 8 cases of which 6 completed; AMI manual annotations v1.6.2, SDM condition, Array1-01, 3 cases; MSDWild few.val published evaluation list, 4 cases | | Sample size | 25 cases, 23 scored. Every case was run once. | | Metrics | WDER, DER and word error rate as defined above, computed by Aurora's own scorer against each corpus's published reference | Limitations, in the order they would change a reader's mind: - **One run per case.** The harness's own repeatability check reports *not-enough-current-runs* for every case, so none of these figures has a confidence interval. A median of six cases is a median of six cases. - **Every case fails its gate.** The suite's boundary-quality gate is stricter than these results, so each case carries the verdict *fail*. That is a gate outcome, not a sign the measurement is invalid — and it is the reason this work is being done. - **Two Earnings-21 cases did not complete.** The 18-minute and 25-minute slices are reported as incomplete rather than quietly dropped, so the Earnings-21 median rests on six cases, not eight. - **Provisional.** No independent verification pass has run, and Aurora's suite README still marks the Earnings-21 and AMI imports as pending completion. - **Measured on a Mac.** The harness runs the pipeline on a desktop, not on a phone. Accuracy is not expected to differ much; the article makes no speed claim from this run. - **Two suites cannot carry a quality claim** because the diarization model was trained on their audio, as the contamination table above records. ## Published results 1. [speaker-attribution-2026-06-17.csv](https://auroraisyours.com/evidence/speaker-attribution-2026-06-17.csv) — one row per case: suite, corpus, contamination verdict, audio length, speakers, reference turns and words, model, WER, DER, WDER and the three DER components. 2. [speaker-attribution-2026-06-17.json](https://auroraisyours.com/evidence/speaker-attribution-2026-06-17.json) — the same rows plus the run metadata, metric definitions, per-suite aggregates and the limitations listed above, in machine-readable form. 3. [speaker-stage-der-2026-06-09.csv](https://auroraisyours.com/evidence/speaker-stage-der-2026-06-09.csv) and [its JSON twin](https://auroraisyours.com/evidence/speaker-stage-der-2026-06-09.json) — the per-stage DER walk, with the fixture and its caveats. 4. [attribution-thresholds-2026-09-18.csv](https://auroraisyours.com/evidence/attribution-thresholds-2026-09-18.csv) and [its JSON twin](https://auroraisyours.com/evidence/attribution-thresholds-2026-09-18.json) — all 37 attribution thresholds with value, unit, rationale and provenance. ## References 1. El Shafey, Soltau, Shafran. *Joint Speech Recognition and Speaker Diarization via Sequence Transduction.* [arXiv:1907.05337](https://arxiv.org/abs/1907.05337) — defines word diarization error rate. 2. Medennikov et al. *Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering.* [arXiv:2507.18446](https://arxiv.org/abs/2507.18446) — the streaming diarization architecture Aurora runs on device. 3. Del Rio et al. *Earnings-21: A Practical Benchmark for ASR in the Wild.* [arXiv:2104.11348](https://arxiv.org/abs/2104.11348) — the corpus Aurora uses for load-bearing numbers. 4. Argmax. *OpenBench diarization benchmarks.* [github.com/argmaxinc/OpenBench](https://github.com/argmaxinc/OpenBench/blob/main/BENCHMARKS.md) — the pyannote 3.1 DER anchors quoted above. 5. University of Edinburgh. *The AMI Meeting Corpus.* [groups.inf.ed.ac.uk/ami/corpus](https://groups.inf.ed.ac.uk/ami/corpus/) — the single-distant-microphone condition. 6. X-LANCE. *MSDWild: multi-modal speaker diarization in the wild.* [github.com/X-LANCE/MSDWILD](https://github.com/X-LANCE/MSDWILD) — the highest-overlap corpus in the set. ## Related 1. [How Aurora decides a live word is safe to show](https://auroraisyours.com/blog/live-transcript-confirmation) — the confirmation rules that publish these words at all. 2. [Why Aurora does not send a bot to your meeting](https://auroraisyours.com/blog/no-meeting-bot) — why the input is one mixed signal instead of a stream per speaker. 3. [What on-device transcription costs on an iPhone](https://auroraisyours.com/blog/on-device-transcription-cost) — the compute these models are given. ## More from the blog 18 Sep 2026 · Engineering · 8 min ## How Aurora decides a live word is safe to show https://auroraisyours.com/blog/live-transcript-confirmation 14 Jul 2026 · Engineering · 9 min ## What on-device transcription costs on an iPhone https://auroraisyours.com/blog/on-device-transcription-cost