mirror of
https://github.com/storytold/storyteller-ml.git
synced 2026-10-16 00:13:07 +00:00
Michael Chung
33 lines
1.4 KiB
Markdown
33 lines
1.4 KiB
Markdown
# Model Card: VALL-E X
|
|
|
|
**Author**: [Songting](https://github.com/Plachtaa).<br>
|
|
<br>
|
|
This is the official codebase for running open-sourced VALL-E X.
|
|
|
|
The following is additional information about the models released here.
|
|
|
|
## Model Details
|
|
|
|
VALL-E X is a series of two transformer models that turn text into audio.
|
|
|
|
### Phoneme to acoustic tokens
|
|
- Input: IPAs converted from input text by a rule-based G2P tool.
|
|
- Output: tokens from the first codebook of the [EnCodec Codec](https://github.com/facebookresearch/encodec) from facebook
|
|
|
|
### Coarse to fine tokens
|
|
- Input: IPAs converted from input text by a rule-based G2P tool & the first codebook from EnCodec
|
|
- Output: 8 codebooks from EnCodec
|
|
|
|
### Architecture
|
|
| Model | Parameters | Attention | Output Vocab size |
|
|
|:------------------------:|:----------:|------------|:-----------------:|
|
|
| G2P tool | - | - | 69 |
|
|
| Phoneme to coarse tokens | 150 M | Causal | 1x 1,024 |
|
|
| Coarse to fine tokens | 150 M | Non-causal | 7x 1,024 |
|
|
|
|
### Release date
|
|
August 2023
|
|
|
|
## Broader Implications
|
|
We anticipate that this model's text to audio capabilities can be used to improve accessbility tools in a variety of languages.
|
|
Straightforward improvements will allow models to run faster than realtime, rendering them useful for applications such as virtual assistants. |