An early cue to identify English or French helped a multilingual speech model improve on several sound- and word-learning tests, Apple researchers report in an arXiv preprint posted Sept. 27. Their controlled HuBERT experiment narrowed some gaps with single-language models trained for the same total number of audio hours, but did not measure speech-to-text performance.
The team trained HuBERT-Base, a self-supervised speech model that learns patterns from audio without word transcripts, on 500 hours of English read speech and 500 hours of French read speech. It compared this 1,000-hour bilingual setup with models trained on 1,000 hours in one language. Separate 500-hour, single-language controls used the same language-specific audio subsets as the bilingual model, testing whether less exposure to each language alone explained the gap. Each condition was trained three times with different random seeds.
At HuBERT’s first training stage, an auxiliary classifier learned to tell whether each speech segment was English or French. The team removed that classifier before the model’s second stage and its final evaluations, so the resulting encoder did not need a language label at test time.
In the continuous phone-ABX test, which measures whether similar speech sounds remain distinguishable, the early-classifier model’s error was 10.4%, down from 11.6% for the bilingual baseline and slightly below the 10.8% monolingual reference. The paper reports means across three training seeds; lower error is better.
On sWUGGY, a model chooses which of two clips contains a real spoken word rather than a similar-sounding made-up one. The early-classifier model made the correct choice on 55.2% of English- and French-test pairs on average, up from 52.1% for the bilingual baseline but below the monolingual reference’s 58.5%.
ProsAudit’s English-only lexical task compares a natural pause between words with one inserted inside a word. The model favored the natural pause in 72.9% of test pairs, up from 68.9% and just above the monolingual reference’s 72.6%. These benchmark scores are not speech-recognition accuracy.
A separate intervention changed HuBERT’s second-stage prediction targets: instead of one shared set of 500 sound-pattern labels, it used distinct English and French sets of 250 labels each. It scored 56.7% on sWUGGY, above the classifier’s 55.2% and the bilingual baseline’s 52.1%. Timing also mattered: applying the classifier in the second stage or both stages made the model’s representations cluster more by language and produced weaker lexical scores than using it only in the first stage. The authors interpret the pattern as a possible balance: shared representations may let languages help each other, while distinguishing them may keep similar sounds from being treated as equivalent. The experiments did not measure those effects separately.
The results come from one English-French pair, read-speech data and one HuBERT architecture, so they do not establish better performance in a deployed speech recognizer or across other languages.
Sources
- Original paper, arXiv:2609.33345v1 (posted Sept. 27, 2026; CC BY 4.0)
- Zero Resource Speech Benchmark 2021 paper (sWUGGY task definition)
- ProsAudit benchmark paper (English lexical task definition)
- Illustrative voice-recording photo by Kaboompics via Pexels, under the Pexels License.