storytold/storyteller-ml
2
0
Fork 0
mirror of https://github.com/storytold/storyteller-ml.git synced 2026-10-16 00:13:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update README. Root-level readme is going to go out of date

Browse Source 
Since the root readme can't give us much information on each model,
we'll just add some ASCII art.
This commit is contained in:
Brandon Thomas
2023-03-13 04:26:27 -04:00
parent 9fe76017d4
commit c4ec6ffc08
4 changed files with 118 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+53 -74
View File
@@ -1,79 +1,58 @@
storyteller-ml
==============
The ML models that power FakeYou and other Storyteller functions.
This repository is in a bit of a rough shape, but should be improving.
TTS models (Tacotron, HifiGan, WaveGlow, etc.) live under [`tts/`, and you can find documentation there](tts/).
We expect to add video, posture estimation, phoneme prediction, and other models soon.
Run Dockerized in Development
-----------------------------
### TTS
```bash
docker build .
docker run --rm --gpus all -it \
-p 8000:8000 \
--mount type=bind,source=/tmp,target=/tmp \
--entrypoint ./start_tts_server.sh [image name]
```
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡟⠆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣼⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡰⠋⠔⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡠⠊⡠⢊⣤⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠈⡟⢆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡰⠒⠤⣀⣀⠀⠀⠀⡜⢁⠊⠀⢸⣿⣿⣿⣷⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠘⢎⠳⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡔⠀⠀⠀⠀⠀⠉⠑⠲⠅⠀⠀⠀⠘⣿⣿⣿⣿⣿⣿⣷⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠈⠣⡈⠢⡀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣠⣤⣴⡋⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠐⠀⠀⢸⣿⣿⣿⣿⣿⣿⣿⣿⣷⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠲⢌⣑⠒⠒⣒⣊⡉⠥⠤⠔⠢⣤⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⣦⠀⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⣶⣦⡀⠀⠉⠉⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢺⣄⣱⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⢿⣿⣄⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⢠⣾⣿⣿⣿⣿⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠔⢁⡰⠄⠀⠀⠀⠀⠀⠀⠈⠻⣿⣿⣿⣿⣿⣿⡿⠟⠛⠉⠀⠀⠀⠀⠀⠉⠓⠢⢄⣀⠀⠀
⠀⠀⠀⠀⠀⢀⣴⣿⣿⣿⣿⣿⣿⣿⣧⡀⠀⠀⠀⠀⠀⠀⠀⠇⠀⠀⢃⠀⣦⣾⡀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠹⣿⠋⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢹⡀⠀
⠀⠀⠀⠀⢀⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣦⠀⠀⠀⠀⠀⠀⢠⠀⠀⠀⠁⠻⠿⠃⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⣿⠶
⠀⠀⠀⢀⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣷⣤⡀⠀⠀⠀⢀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⣾⡄⠀
⠀⠀⢀⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣦⣀⠀⠸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡀⠀⢰⡜⢀⣠⣴⣶⣾⣿⡿⠛⠙⠻⣿⣿⣿⣿⠇⠈⠁⠀
⠀⢀⣾⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠿⠂⠀⠀⠀⠀⢈⠦⠴⢄⡀⠀⠀⠀⠀⠈⢉⣠⣾⣿⠿⠛⠛⠛⠻⠛⠁⠀⠀⠀⠈⠛⢿⣿⠀⠀⠀⠀
⠀⣸⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠿⠟⠋⠉⠁⠀⠀⠀⠀⠀⠀⠀⢸⣀⡀⠀⠈⠉⠉⠉⠉⣁⠤⢿⡿⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠋⠀⠀⠀⠀
⢠⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⠿⠋⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡌⠀⠈⠉⠉⠉⠉⠉⠉⠁⠀⠘⡅⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⢸⣿⣿⣿⣿⣿⣿⣿⣿⡿⠋⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡗⠤⣀⠀⠀⠀⠀⠀⣀⣀⠤⠚⠣⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⣾⣿⣿⣿⣿⣿⣿⣿⡟⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠠⣾⠃⠀⠀⠀⠀⠀⡇⠀⠀⠈⠉⠉⠁⠀⠀⠀⠀⠀⠀⣡⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⣿⣿⣿⣿⣿⣿⠟⠻⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠄⠀⠀⡏⠀⠀⠀⠀⠀⠀⠦⢄⡀⠀⠀⠀⠀⠀⠀⠀⣀⡠⠔⠁⠑⠄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⣿⣿⣿⣿⠟⠁⠀⡅⠀⠀⠀⠀⠀⠀⠀⠀⠀⠂⠀⠀⢀⠇⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠒⠊⠉⠉⠉⠁⠀⠀⠀⠀⠀⡔⠂⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ Storyteller ML
⢸⣿⣿⠋⠀⠀⠀⣿⡀⠀⠀⠀⠀⠀⠀⡐⠁⠀⠀⠀⡜⠀⠀⠀⠀⠀⠀⠀⣆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⡠⠔⠋⠀⠀⢀⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀Our Machine Learning Monorepo⠀
⠘⣿⠃⠀⠀⠀⠀⠛⠛⣦⣀⠀⢀⣠⡞⠀⠀⠀⠀⠀⠃⠀⠀⠀⠀⠀⠀⠀⡇⠉⠉⠉⠀⠀⠀⠀⠀⠉⠉⠁⠀⠀⠀⠀⢠⠎⠐⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀(Begone, TensorFlow and CUDA errors!)
⠀⠹⡆⠀⠀⠀⠀⠀⠀⠿⠟⠉⢿⡿⠀⠀⠀⠀⠀⡘⠀⠀⠀⠀⠀⠀⠀⠀⠰⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⡠⠔⠚⠁⠀⠀⢸⠓⠠⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠀⠀⠀⠀⠀⠀⠃⠀⠀⠀⠀⠀⠀⠀⠀⠀⢫⠉⠉⠒⠒⠒⠒⠒⠊⠉⠉⠀⠀⠀⠀⠀⢀⡠⠊⡆⠀⠈⠢⡀⠀⠀⠀⠀⠀ Specific docs under each project
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡜⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠑⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣀⠤⠒⠉⠀⠀⠁⠀⠀⠀⠈⠆⠀⠀⠀⠀Global docs under toplevel `docs` folder.
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠃⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠳⡒⠒⠒⠒⠒⠒⠒⠉⠁⠀⠀⠀⠀⠀⢀⡆⠀⠀⠀⠀⠈⡄⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡘⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⡀⠀⠂⠀⠃⠀⠀⠀⠀⠀⢠⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⡦⠄⠀⠀⠀⠐⠂⠀⠁⠀⠀⠀⡠⠀⠀⠀⠀⠀⠀⡘⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠸⡀⠀⠀⠀⠀⠀⠀⢀⡠⠤⢺⠃⠀⠀⠀⠀⠀⢠⠃⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢧⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡗⠒⠒⠒⠀⠈⠈⠀⠀⢀⠆⠀⠀⠀⠀⠀⣠⠃⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⡀⣴⠃⠀⠀⠀⢀⡠⢾⡁⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⠇⠀⠒⠒⠂⠁⣀⡜⠁⠀⠀⠄⠒⠉⠀⠀⠈⠒⢄⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣠⣧⣤⣠⣤⣤⡶⠋⠁⠑⠢⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠁⢄⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⣿⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡼⠿⠛⠛⠉⠉⡹⠀⠀⠀⠀⠀⠈⠐⢄⡀⠀⠀⢀⣀⣤⣶⣶⣾⣧⡀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣿⣿⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⣇⠀⠀⢀⠠⢴⡇⠀⠀⠀⠀⠀⠀⠀⠀⠈⠐⠰⢿⣿⣿⣿⣿⣿⠛⠁⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⣿⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⡟⠁⠈⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠛⠀⠉⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⣿⣷⡀⠀⠀⠀⠀⠀⠀⢀⣠⠃⠀⠀⠀⢀⣸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢻⣿⣷⣤⣴⣶⣤⣤⣾⣿⣿⠤⠐⠒⠉⠀⢀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢿⣿⣿⣿⣿⣿⠛⣿⣿⠇⠀⠀⠀⠀⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⣿⣿⣿⣿⡇⠀⠈⠋⠀⠀⠀⠀⣀⠤⠚⡆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⣿⣿⣿⣿⡖⠒⠒⠒⠉⠉⠁⠀⠀⠀⢡⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⣿⣿⣿⣷⡀⠀⠀⠀⠀⠀⠀⠀⠀⣸⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢿⣿⣿⣷⡄⠀⠀⠀⠤⠀⠀⠁⠀⠱⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠻⣿⣿⣷⡄⠀⠀⠀⠀⠀⠀⠀⠀⢂⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠻⣿⣿⣦⡀⠀⠀⠀⠀⠤⠂⠀⢆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠻⣿⣿⣄⠀⠀⠀⠀⠀⠀⢈⣦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠻⣿⣷⣄⠀⠀⠠⠔⠉⠀⠡⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⠻⢷⣦⣀⠀⠀⠀⠠⠚⠆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠙⠻⢷⣤⣀⠀⠀⠀⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠛⠶⠦⣤⡷⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀
```
How This Works
--------------
We have a Rust monolith that controls the user interface, account system, and all the database CRUD operations.
It doesn't do any ML work itself or have any attached GPUs.
There are a series of worker pods ("jobs") that pull from work queues and run inference, then upload the results.
![](docs/diagrams/microservices.png)
The jobs are written in Rust and either shell out to Python code or call an in-container Python server that attempts
to LRU cache models in memory.
![](docs/diagrams/jobs1.png)
![](docs/diagrams/jobs2.png)
TODO
----
Core cleanup
- [ ] Cleanup Tacotron code
- [ ] Integrate WaveGlow and HifiGan more cleanly
- [ ] Define the HTTP server interface
- [ ] Define the model checking interface
- [ ] Clean up in-memory LRU caching
- [ ] There are two copies of Nemo. Why? Clean this up.
- [ ] Deployability
- [ ] Clean up requirements.txt and upgrade packages
- [ ] Clean up Dockerfile
Docker image cleanup
- [ ] Remove docker-base-images-nvidia-cuda-experimental
- [ ] Remove inessential build pieces
Make the models sound better
- [ ] Arpabet
- [ ] G2P
- [ ] Arpabet custom dictionary
- [ ] User-supplied Arpabet strings in braces
- [ ] Word break down 888 -> eight hundred eighty eight
- [ ] Support I18N
- [ ] Accent character support
Introduce new models
- [ ] TalkNet
- [ ] GlowTTS + MelGan (vocoder)
- [ ] Make these work under the same venv
Lots more I haven't covered yet...
docs/README.md
+26
View File
@@ -0,0 +1,26 @@
storyteller-ml
==============
The ML models that power FakeYou and other Storyteller functions.
This repository is in a bit of a rough shape, but should be improving.
TTS models (Tacotron, HifiGan, WaveGlow, etc.) live under [`tts/`, and you can find documentation there](tts/).
We expect to add video, posture estimation, phoneme prediction, and other models soon.
FakeYou Backend Architecture
----------------------------
We have a Rust monolith that controls the user interface, account system, and all the database CRUD operations.
It doesn't do any ML work itself or have any attached GPUs.
There are a series of worker pods ("jobs") that pull from work queues and run inference, then upload the results.
![](./diagrams/microservices.png)
The jobs are written in Rust and either shell out to Python code or call an in-container Python server that attempts
to LRU cache models in memory.
![](./diagrams/jobs1.png)
![](./diagrams/jobs2.png)
tts/tacotron2_v1_early_fakeyou/.dockerignore
+25
View File
@@ -0,0 +1,25 @@
# Secrets (which shouldn't be in git anyway)
*secret*sh
.env-secrets
secrets.toml
secrets.yaml
# Python Virtual Environments
python/
tacotron2/python/
# Python build
*.pyc
*.pyo
__pycache__/
# Rust build
target/
# VIM
*~
# Everything else
.env
.git
.idea/
tts/tacotron2_v1_early_fakeyou/README.md
+14
View File
@@ -13,6 +13,20 @@ The following files are entrypoints and an essential part of FakeYou.com functio
files).
Run Dockerized in Development
-----------------------------
### TTS
```bash
docker build .
docker run --rm --gpus all -it \
-p 8000:8000 \
--mount type=bind,source=/tmp,target=/tmp \
--entrypoint ./start_tts_server.sh [image name]
```
To Run Tests
------------