MaskGCT翻译站点

Abstract The recent large-scale text-to-speech (TTS) systems are usually grouped as autoregressive and non-autoregressive systems. The autoregressive systems implicitly model duration but exhibit certain deficiencies in robustness and lack of duration controllability. Non-autoregressive systems require explicit alignment information between text and speech during training and predict durations for linguistic units (e.g. phone), which may compromise their naturalness. In this paper, we introduce Masked Generative Codec Transformer (MaskGCT), a fully non-autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision, as well as phone-level duration prediction. MaskGCT is a two-stage model: in the first stage, the model uses text to predict semantic tokens extracted from a speech self-supervised learning (SSL) model, and in the second stage, the model predicts acoustic tokens conditioned on these semantic tokens. MaskGCT follows the mask-and-predict learning paradigm. During training, MaskGCT learns to predict masked semantic or acoustic tokens based on given conditions and prompts. During inference, the model generates tokens of a specified length in a parallel manner. Experiments with 100K hours of in-the-wild speech demonstrate that MaskGCT outperforms the current state-of-the-art zero-shot TTS systems in terms of quality, similarity, and intelligibility.
抽象最近的大规模文本转语音 （TTS） 系统通常分为自回归和非自回归系统。自回归系统隐式地模拟了持续时间，但在稳健性和缺乏持续时间可控性方面表现出一定的缺陷。非自回归系统在训练期间需要文本和语音之间的显式对齐信息，并预测语言单位（例如电话）的持续时间，这可能会损害它们的自然性。在本文中，我们介绍了 Masked Generative Codec Transformer （MaskGCT），这是一种完全非自回归的 TTS 模型，无需在文本和语音监督之间提供明确的对齐信息，也无需电话级别的持续时间预测。MaskGCT 是一个两阶段模型：在第一阶段，模型使用文本来预测从语音自我监督学习 （SSL） 模型中提取的语义标记，在第二阶段，模型预测以这些语义标记为条件的声学标记。MaskGCT 遵循掩码和预测学习范式。在训练过程中，MaskGCT 学习根据给定的条件和提示预测掩蔽的语义或声学标记。在推理过程中，模型以并行方式生成指定长度的 Token。对 100K 小时野外语音的实验表明，MaskGCT 在质量、相似性和可理解性方面优于当前最先进的零样本 TTS 系统。

System Overview  系统概述

Figure 1. An overview of our MaskGCT system. MaskGCT consists of four main parts: (1) a speech semantic representation codec converts speech to semantic tokens; (2) a text-to-semantic model predicts semantic tokens with text and prompt semantic tokens; (3) a semantic-to-acoustic model predicts acoustic tokens conditioned on semantic tokens; (4) a speech acoustic codec reconstructs speech waveform from acoustic tokens.
图 1.我们的 MaskGCT 系统概述。MaskGCT 由四个主要部分组成：

（1） 语音语义表示编解码器将语音转换为语义标记;

（2） 文本到语义模型使用文本和提示语义标记预测语义标记;

（3） 语义到声学模型预测以语义标记为条件的声学标记;

（4） 语音声学编解码器从声学标记重建语音波形。

Zero-shot In-context Learning
零样本上下文学习

The first four prompt audios are from the demo page of Seed-TTS.