Interspeech 2026, Sydney

PiDAPhonetically-informed Data Augmentation for Robust Vietnamese Speech Translation

Giang Son Nguyen1, Tung X. Nguyen1, Hieu Minh Truong1, Nhu Vo1,2,
Wray Buntine1,3, Dung D. Le1
1VinUniversity 2University of Technology Sydney 3Monash University
Paper arXiv Medical MT Project

Vietnamese ASR errors are not random noise — they are systematic phonetic confusions. Injecting the same kind of confusion into clean training text buys +2.04 BLEU on real ASR output while clean-text translation also improves, with no audio and no LLM in the loop.

+2.04
BLEU on PhoWhisper ASR output
+0.78
BLEU on wav2vec2 ASR output
+0.68
BLEU on clean text — no tradeoff
0
hours of audio and LLM calls needed
Fine-tuning data for the NMT component Clean text
BLEUMT		 PhoWhisper output
BLEUST		 wav2vec2 output
BLEUST
clean & PiDA (ours) 33.72 28.29 23.18
clean pairs only (standard fine-tuning) 33.04 26.25 22.40
real ASR output only (needs audio) 32.00 28.06 23.65
clean & MEDSAGE (LLM-generated) 32.59 26.68 22.88

Training on real ASR output does buy robustness — but it costs 1.04 BLEU on clean text and needs audio you may not have. Clean + PiDA is the only condition that improves both ASR settings and clean-text translation at once.

Summary

Cascaded speech translation suffers from error propagation: when the ASR front-end mis-transcribes, the NMT model — trained only on clean text — has to translate a sentence unlike anything it saw in training. On FLEURS Vi–En this costs 6.79 to 10.64 BLEU. Augmenting with synthetic noise is the standard fix, but its benefit depends on matching the real error distribution — so the question is what kind of error to inject.

We present the first systematic categorization of ASR errors for Vietnamese ST, classifying substitution errors by phonetic cause and quantifying their impact with a linear mixed-effects model. Most substitutions turn out to be systematic phonetic confusions rather than random noise, and those are the ones that degrade translation. So the synthetic corruption should be phonetic too: PiDA substitutes syllables with phonetically similar alternatives retrieved from XPhoneBERT embeddings — to our knowledge the first use of explicit phonetic embeddings to generate ASR-style training data for MT in any language. Because PiDA is text-only, it extends to domains with parallel text but little speech data.

Vietnamese ASR errors are phonetic

ASR error analysis pipeline and the resulting error taxonomy FLEURS Vietnamese audio is transcribed by two ASR systems, aligned to the reference by minimum edit distance, sorted into six phonetic categories by an LLM classifier validated against human labels, and regressed on the change in translation edit rate. The three within-vocabulary phonetic categories carry the largest coefficients; phonetically unrelated substitutions carry none. FLEURS Vi–En train 3k audio + reference transcript Two ASR systems PhoWhisper 8.62% WER · wav2vec2 15.26% Align & categorize min. edit distance; LLM labels, κ = 0.95 Mixed-effects model ΔTER on per-category error rate 12,096 SUBSTITUTION ERRORS, BY PHONETIC CAUSE — COUNT · ΔTER COEFFICIENT Vowel confusion 2,193 · 97.66 sau “after” → sao “star” Tonal confusion 563 · 79.12 ma “ghost” → mà “but” Consonant confusion 1,870 · 71.84 sử “history” → xử “execute” OOV (foreign terms) 5,665 · 58.82 feet → phít NA (phonetically unrelated) 1,336 · n.s. the closest thing to random noise — no measurable effect No error (orthographic) 469 · excluded hoà “draw” → hòa Phonetic confusion — what PiDA simulates Significant but out of scope Not a phonetic error

Counts are summed over both ASR systems; coefficients are on one length-normalised scale, so they compare directly (deletions 133.95 and insertions 127.51 are larger still). The ordering is the point: all three phonetic categories land above OOV, while phonetically unrelated substitutions are statistically indistinguishable from zero. So injecting random noise simulates the one error type that does not matter.

The PiDA pipeline

The PiDA augmentation pipeline A one-off precomputation phase converts a Vietnamese syllable inventory to IPA, embeds it with XPhoneBERT and builds a FAISS index of each syllable's nearest phonetic neighbours. The per-sentence augmentation phase marks words for corruption using real ASR error statistics, then replaces each marked syllable with a neighbour drawn from a temperature-scaled softmax over cosine similarity, producing a noisy training pair with the unchanged English translation. PRECOMPUTATION — ONCE PER LANGUAGE 1 Syllable inventory Top 50k Vietnamese words from wordfreq → ~9,400 syllables 2 Phoneme conversion CharsiuG2P maps each syllable to an IPA phoneme sequence 3 Embedding extraction XPhoneBERT states, mean-pooled → one 768-d vector per syllable 4 Similarity index FAISS; cache the top-50 phonetic neighbours AUGMENTATION — PER SENTENCE Clean Vietnamese FLEURS train, paired with its English translation 5 Error annotation Mark each word with prob. p = WER; delete or substitute, per real ASR mix 6 Phonetic corruption Draw the replacement from the top-k (k=5) neighbours by softmax over cosine similarity, temperature τ=0.5 Noisy training pair corrupted Vietnamese + unchanged English target phonetic neighbours Clean text Corrupted text Processing step Reusable artefact Where the baselines differ: random augmentation draws the replacement from a frequency-weighted syllable list, ignoring pronunciation; MEDSAGE asks an LLM to write it.

Steps 1–4 depend only on the language, so the index is built once and reused; only step 5 — the error rate and the deletion/substitution mix — is fitted to a particular ASR front-end. Insertions are not simulated, since the pipeline has no language model to generate plausible inserted words. The corruptions it produces have the shape of the real confusions above: sử → xử, sau → sao, ma → mà.

Results

We corrupt the FLEURS Vi–En training split with each method, fine-tune VinAI-Translate on it, and evaluate on the 0.9k test split against clean transcripts (MT) and against both ASR outputs (ST). Mixed conditions use a 1:1 clean-to-noisy ratio.

Fine-tuning data Text
only		 LLM
indep.		 MT (clean text) ST (PhoWhisper) ST (wav2vec2)
BLEU ↑COMET ↑ BLEU ↑COMET ↑ BLEU ↑COMET ↑
No fine-tuning——28.050.8423.730.8022.150.73
+ clean pairs (baseline)✓✓33.040.8626.250.8122.400.75
+ clean & freq-based subs✓✓33.130.8627.450.8122.770.77
+ freq-based subs only✓✓32.900.8627.460.8222.520.78
+ clean & real noisy—✓33.090.8627.430.8222.960.76
+ real noisy only—✓32.000.8528.060.8323.650.78
+ clean & MEDSAGE✓—32.590.8626.680.8222.880.77
+ MEDSAGE only✓—32.450.8626.430.8222.480.78
+ clean & PiDA (ours)✓✓33.720.8628.290.8223.180.77
+ PiDA only (ours)✓✓33.580.8628.190.8223.430.77

Green marks a significant gain over the clean-pairs baseline, red a significant loss (p < 0.05). Text-only = no audio needed; LLM indep. = no external language model needed.

Domain adaptation alone does not buy robustness. Clean fine-tuning helps, but the MT–ST gap survives it — 6.79 BLEU for PhoWhisper, 10.64 for wav2vec2. The weaker the ASR front-end, the wider the gap.

Real ASR output works, at a price. It gives the best wav2vec2 robustness in the table and costs 1.04 BLEU on clean text — and it needs audio, which is exactly what is scarce for Vietnamese ST.

LLM-generated corruption does not transfer. MEDSAGE gives no significant gain over the baseline for either ASR system; its corruptions rest on the LLM's implicit sense of phonetic plausibility, which appears inadequate for a phonologically distant language.

PiDA removes the robustness–accuracy tradeoff — +2.04 BLEU on PhoWhisper output (+0.84 over random substitutions), +0.78 on wav2vec2, while clean-text BLEU also improves and COMET holds. Results are stable across the sampling grid k ∈ {3, 5, 10} and τ ∈ {0.3, 0.5, 1.0} — under one BLEU point of spread — so the method does not depend on careful tuning.

Limitations

We evaluate on a single dataset, because high-quality Vietnamese ST resources are scarce — the only other candidate, MultiMed-ST, has substantial audio–transcript misalignments that make error analysis on it unreliable. OOV errors, the largest single category, involve cross-lingual phonetic mapping our within-vocabulary method does not address, and PiDA does not simulate insertions. Future work will extend to OOV handling and validate on further Vietnamese ST benchmarks.

BibTeX

@misc{nguyen2026pidaphoneticallyinformeddataaugmentation,
      title={PiDA: Phonetically-Informed Data Augmentation for Robust Vietnamese Speech Translation},
      author={Giang Son Nguyen and Tung X. Nguyen and Hieu Minh Truong and Nhu Vo and Wray Buntine and Dung D. Le},
      year={2026},
      eprint={2606.12911},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.12911},
}

Acknowledgments

This work is part of our wider medical machine translation project at VinUniversity, under the Cross-College project Robust Vietnamese–English Clinical and Educational Medical Translation (Project ID: VUNI.2324.CC06), between the College of Engineering & Computer Science and the College of Health Sciences. Giang Son Nguyen, Tung X. Nguyen, Hieu Minh Truong and Dung D. Le are partly supported by the Center for AI Research, VinUniversity; Nhu Vo is supported by the Vingroup Scholarship. Giang Son Nguyen thanks the AUMOVIO-NTU Corporate Lab for lending GPU resources.