M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis

Abstract

NAR zero-shot TTS often relies on rigid duration modeling or pseudo-alignment, limiting prosodic expressiveness and increasing computational overhead. We propose M3-TTS, an efficient dual-stage MMDiT framework that establishes robust cross-modal correspondence. Self-Adaptive Joint Representation enables implicit monotonic alignment, enhancing both temporal modeling and pronunciation fidelity. Adaptive Distribution Modulator ensures explicit content-timbre disentanglement, while a Mel-VAE-based Distilled Latent Prior provides spatiotemporal compression—accelerating training threefold and enabling native 44.1 kHz synthesis. Extensive evaluations on Seed-TTS-eval demonstrate that M3-TTS attains SOTA performance, achieving the lowest WER of 1.36% for English and 1.33% for Chinese