diff --git a/animation/SadTalker/docs/FAQ.md b/animation/SadTalker/docs/FAQ.md new file mode 100644 index 0000000..6451a22 --- /dev/null +++ b/animation/SadTalker/docs/FAQ.md @@ -0,0 +1,46 @@ + +## Frequency Asked Question + +**Q: `ffmpeg` is not recognized as an internal or external command** + +In Linux, you can install the ffmpeg via `conda install ffmpeg`. Or on Mac OS X, try to install ffmpeg via `brew install ffmpeg`. On windows, make sure you have `ffmpeg` in the `%PATH%` as suggested in [#54](https://github.com/Winfredy/SadTalker/issues/54), then, following [this](https://www.geeksforgeeks.org/how-to-install-ffmpeg-on-windows/) installation to install `ffmpeg`. + +**Q: Running Requirments.** + +Please refer to the discussion here: https://github.com/Winfredy/SadTalker/issues/124#issuecomment-1508113989 + + +**Q: ModuleNotFoundError: No module named 'ai'** + +please check the checkpoint's size of the `epoch_20.pth`. (https://github.com/Winfredy/SadTalker/issues/167, https://github.com/Winfredy/SadTalker/issues/113) + +**Q: Illegal Hardware Error: Mac M1** + +please reinstall the `dlib` by `pip install dlib` individually. (https://github.com/Winfredy/SadTalker/issues/129, https://github.com/Winfredy/SadTalker/issues/109) + + +**Q: FileNotFoundError: [Errno 2] No such file or directory: checkpoints\BFM_Fitting\similarity_Lm3D_all.mat** + +Make sure you have downloaded the checkpoints and gfpgan as [here](https://github.com/Winfredy/SadTalker#-2-download-trained-models) and placed them in the right place. + +**Q: RuntimeError: unexpected EOF, expected 237192 more bytes. The file might be corrupted.** + +The files are not automatically downloaded. Please update the code and download the gfpgan folders as [here](https://github.com/Winfredy/SadTalker#-2-download-trained-models). + +**Q: CUDA out of memory error** + +please refer to https://stackoverflow.com/questions/73747731/runtimeerror-cuda-out-of-memory-how-setting-max-split-size-mb + +``` +# windows +set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128 +python inference.py ... + +# linux +export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128 +python inference.py ... +``` + +**Q: Error while decoding stream #0:0: Invalid data found when processing input [mp3float @ 0000015037628c00] Header missing** + +Our method only support wav or mp3 files as input, please make sure the feeded audios are in these formats. diff --git a/animation/SadTalker/docs/best_practice.md b/animation/SadTalker/docs/best_practice.md new file mode 100644 index 0000000..7670050 --- /dev/null +++ b/animation/SadTalker/docs/best_practice.md @@ -0,0 +1,94 @@ +# Best Practices and Tips for configuration + +> Our model only works on REAL people or the portrait image similar to REAL person. The anime talking head genreation method will be released in future. + +Advanced confiuration options for `inference.py`: + +| Name | Configuration | default | Explaination | +|:------------- |:------------- |:----- | :------------- | +| Enhance Mode | `--enhancer` | None | Using `gfpgan` or `RestoreFormer` to enhance the generated face via face restoration network +| Background Enhancer | `--background_enhancer` | None | Using `realesrgan` to enhance the full video. +| Still Mode | ` --still` | False | Using the same pose parameters as the original image, fewer head motion. +| Expressive Mode | `--expression_scale` | 1.0 | a larger value will make the expression motion stronger. +| save path | `--result_dir` |`./results` | The file will be save in the newer location. +| preprocess | `--preprocess` | `crop` | Run and produce the results in the croped input image. Other choices: `resize`, where the images will be resized to the specific resolution. `full` Run the full image animation, use with `--still` to get better results. +| ref Mode (eye) | `--ref_eyeblink` | None | A video path, where we borrow the eyeblink from this reference video to provide more natural eyebrow movement. +| ref Mode (pose) | `--ref_pose` | None | A video path, where we borrow the pose from the head reference video. +| 3D Mode | `--face3dvis` | False | Need additional installation. More details to generate the 3d face can be founded [here](docs/face3d.md). +| free-view Mode | `--input_yaw`,
`--input_pitch`,
`--input_roll` | None | Genearting novel view or free-view 4D talking head from a single image. More details can be founded [here](https://github.com/Winfredy/SadTalker#generating-4d-free-view-talking-examples-from-audio-and-a-single-image). + + +### About `--preprocess` + +Our system automatically handles the input images via `crop`, `resize` and `full`. + +In `crop` mode, we only generate the croped image via the facial keypoints and generated the facial anime avator. The animation of both expression and head pose are realistic. + +> Still mode will stop the eyeblink and head pose movement. + +| [input image @bagbag1815](https://twitter.com/bagbag1815/status/1642754319094108161) | crop | crop w/still | +|:--------------------: |:--------------------: | :----: | +| | ![full_body_2](example_crop.gif) | ![full_body_2](example_crop_still.gif) | + + +In `resize` mode, we resize the whole images to generate the fully talking head video. Thus, an image similar to the ID photo can be produced. ⚠️ It will produce bad results for full person images. + + + + +| | | +|:--------------------: |:--------------------: | +| ❌ not suitable for resize mode | ✅ good for resize mode | +| | | + +In `full` mode, our model will automatically process the croped region and paste back to the original image. Remember to use `--still` to keep the original head pose. + +| input | `--still` | `--still` & `enhancer` | +|:--------------------: |:--------------------: | :--:| +| | | + + +### About `--enhancer` + +For higher resolution, we intergate [gfpgan](https://github.com/TencentARC/GFPGAN) and [real-esrgan](https://github.com/xinntao/Real-ESRGAN) for different purpose. Just adding `--enhancer ` or `--background_enhancer ` for the enhancement of the face and the full image. + +```bash +# make sure above packages are available: +pip install gfpgan +pip install realesrgan +``` + +### About `--face3dvis` + +This flag indicate that we can generated the 3d-rendered face and it's 3d facial landmarks. More details can be founded [here](face3d.md). + +| Input | Animated 3d face | +|:-------------: | :-------------: | +| | | + +> Kindly ensure to activate the audio as the default audio playing is incompatible with GitHub. + + + +#### Reference eye-link mode. + +| Input, w/ reference video , reference video | +|:-------------: | +| ![free_view](using_ref_video.gif)| +| If the reference video is shorter than the input audio, we will loop the reference video . + + + +#### Generating 4D free-view talking examples from audio and a single image + +We use `input_yaw`, `input_pitch`, `input_roll` to control head pose. For example, `--input_yaw -20 30 10` means the input head yaw degree changes from -20 to 30 and then changes from 30 to 10. +```bash +python inference.py --driven_audio \ + --source_image \ + --result_dir \ + --input_yaw -20 30 10 +``` + +| Results, Free-view results, Novel view results | +|:-------------: | +| ![free_view](free_view_result.gif)| diff --git a/animation/SadTalker/docs/changlelog.md b/animation/SadTalker/docs/changlelog.md new file mode 100644 index 0000000..03d1385 --- /dev/null +++ b/animation/SadTalker/docs/changlelog.md @@ -0,0 +1,29 @@ +## changelogs + + +- __[2023.04.06]__: stable-diffiusion webui extension is release. + +- __[2023.04.03]__: Enable TTS in huggingface and gradio local demo. + +- __[2023.03.30]__: Launch beta version of the full body mode. + +- __[2023.03.30]__: Launch new feature: through using reference videos, our algorithm can generate videos with more natural eye blinking and some eyebrow movement. + +- __[2023.03.29]__: `resize mode` is online by `python infererence.py --preprocess resize`! Where we can produce a larger crop of the image as discussed in https://github.com/Winfredy/SadTalker/issues/35. + +- __[2023.03.29]__: local gradio demo is online! `python app.py` to start the demo. New `requirments.txt` is used to avoid the bugs in `librosa`. + +- __[2023.03.28]__: Online demo is launched in [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/vinthony/SadTalker), thanks AK! + +- __[2023.03.22]__: Launch new feature: generating the 3d face animation from a single image. New applications about it will be updated. + +- __[2023.03.22]__: Launch new feature: `still mode`, where only a small head pose will be produced via `python inference.py --still`. + +- __[2023.03.18]__: Support `expression intensity`, now you can change the intensity of the generated motion: `python inference.py --expression_scale 1.3 (some value > 1)`. + +- __[2023.03.18]__: Reconfig the data folders, now you can download the checkpoint automatically using `bash scripts/download_models.sh`. +- __[2023.03.18]__: We have offically integrate the [GFPGAN](https://github.com/TencentARC/GFPGAN) for face enhancement, using `python inference.py --enhancer gfpgan` for better visualization performance. +- __[2023.03.14]__: Specify the version of package `joblib` to remove the errors in using `librosa`, [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Winfredy/SadTalker/blob/main/quick_demo.ipynb) is online! +- __[2023.03.06]__: Solve some bugs in code and errors in installation +- __[2023.03.03]__: Release the test code for audio-driven single image animation! +- __[2023.02.28]__: SadTalker has been accepted by CVPR 2023! diff --git a/animation/SadTalker/docs/example_crop.gif b/animation/SadTalker/docs/example_crop.gif new file mode 100644 index 0000000..550d348 Binary files /dev/null and b/animation/SadTalker/docs/example_crop.gif differ diff --git a/animation/SadTalker/docs/example_crop_still.gif b/animation/SadTalker/docs/example_crop_still.gif new file mode 100644 index 0000000..b023e47 Binary files /dev/null and b/animation/SadTalker/docs/example_crop_still.gif differ diff --git a/animation/SadTalker/docs/example_full.gif b/animation/SadTalker/docs/example_full.gif new file mode 100644 index 0000000..31bbf16 Binary files /dev/null and b/animation/SadTalker/docs/example_full.gif differ diff --git a/animation/SadTalker/docs/example_full_crop.gif b/animation/SadTalker/docs/example_full_crop.gif new file mode 100644 index 0000000..e59c7d1 Binary files /dev/null and b/animation/SadTalker/docs/example_full_crop.gif differ diff --git a/animation/SadTalker/docs/example_full_enhanced.gif b/animation/SadTalker/docs/example_full_enhanced.gif new file mode 100644 index 0000000..38b528a Binary files /dev/null and b/animation/SadTalker/docs/example_full_enhanced.gif differ diff --git a/animation/SadTalker/docs/face3d.md b/animation/SadTalker/docs/face3d.md new file mode 100644 index 0000000..394f4fe --- /dev/null +++ b/animation/SadTalker/docs/face3d.md @@ -0,0 +1,47 @@ +## 3D Face Visualization + +We use `pytorch3d` to visualize the 3D faces from a single image. + +The requirements for 3D visualization are difficult to install, so here's a tutorial: + +```bash +git clone https://github.com/OpenTalker/SadTalker.git +cd SadTalker +conda create -n sadtalker3d python=3.8 +source activate sadtalker3d + +conda install ffmpeg +conda install -c fvcore -c iopath -c conda-forge fvcore iopath +conda install libgcc gmp + +pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113 + +# insintall pytorch3d +pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu113_pyt1110/download.html + +pip install -r requirements3d.txt + +### install gpfgan for enhancer +pip install git+https://github.com/TencentARC/GFPGAN + + +### when occurs gcc version problem `from pytorch import _C` from pytorch3d, add the anaconda path to LD_LIBRARY_PATH +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/$YOUR_ANACONDA_PATH/lib/ + +``` + +Then, generate the result via: + +```bash + + +python inference.py --driven_audio \ + --source_image \ + --result_dir \ + --face3dvis + +``` + +The results will appear, named `face3d.mp4`. + +More applications about 3D face rendering will be released soon. diff --git a/animation/SadTalker/docs/free_view_result.gif b/animation/SadTalker/docs/free_view_result.gif new file mode 100644 index 0000000..ef364b8 Binary files /dev/null and b/animation/SadTalker/docs/free_view_result.gif differ diff --git a/animation/SadTalker/docs/install.md b/animation/SadTalker/docs/install.md new file mode 100644 index 0000000..807c8eb --- /dev/null +++ b/animation/SadTalker/docs/install.md @@ -0,0 +1,39 @@ +### macOS + +This method has been tested on a M1 Mac (13.3) + +```bash +git clone https://github.com/OpenTalker/SadTalker.git +cd SadTalker +conda create -n sadtalker python=3.8 +conda activate sadtalker +# install pytorch 2.0 +pip install torch torchvision torchaudio +conda install ffmpeg +pip install -r requirements.txt +pip install dlib # macOS needs to install the original dlib. +``` + +### Windows Native + +- Make sure you have `ffmpeg` in the `%PATH%` as suggested in [#54](https://github.com/Winfredy/SadTalker/issues/54), following [this](https://www.geeksforgeeks.org/how-to-install-ffmpeg-on-windows/) tutorial to install `ffmpeg` or using scoop. + + +### Windows WSL + + +- Make sure the environment: `export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH` + + +### Docker Installation + +A community Docker image by [@thegenerativegeneration](https://github.com/thegenerativegeneration) is available on the [Docker hub](https://hub.docker.com/repository/docker/wawa9000/sadtalker), which can be used directly: +```bash +docker run --gpus "all" --rm -v $(pwd):/host_dir wawa9000/sadtalker \ + --driven_audio /host_dir/deyu.wav \ + --source_image /host_dir/image.jpg \ + --expression_scale 1.0 \ + --still \ + --result_dir /host_dir +``` + diff --git a/animation/SadTalker/docs/resize_good.gif b/animation/SadTalker/docs/resize_good.gif new file mode 100644 index 0000000..de0be0c Binary files /dev/null and b/animation/SadTalker/docs/resize_good.gif differ diff --git a/animation/SadTalker/docs/resize_no.gif b/animation/SadTalker/docs/resize_no.gif new file mode 100644 index 0000000..4c4e54f Binary files /dev/null and b/animation/SadTalker/docs/resize_no.gif differ diff --git a/animation/SadTalker/docs/sadtalker_logo.png b/animation/SadTalker/docs/sadtalker_logo.png new file mode 100644 index 0000000..107aebf Binary files /dev/null and b/animation/SadTalker/docs/sadtalker_logo.png differ diff --git a/animation/SadTalker/docs/using_ref_video.gif b/animation/SadTalker/docs/using_ref_video.gif new file mode 100644 index 0000000..3ff4948 Binary files /dev/null and b/animation/SadTalker/docs/using_ref_video.gif differ diff --git a/animation/SadTalker/docs/webui_extension.md b/animation/SadTalker/docs/webui_extension.md new file mode 100644 index 0000000..5e64469 --- /dev/null +++ b/animation/SadTalker/docs/webui_extension.md @@ -0,0 +1,49 @@ +## Run SadTalker as a Stable Diffusion WebUI Extension. + +1. Install the lastest version of [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui) and install SadTalker via `extension`. +image + +2. Download the checkpoints manually, for Linux and Mac: + + ```bash + + cd SOMEWHERE_YOU_LIKE + + bash <(wget -qO- https://raw.githubusercontent.com/Winfredy/OpenTalker/main/scripts/download_models.sh) + ``` + + For Windows, you can download all the checkpoints [here](https://github.com/OpenTalker/SadTalker/tree/main#2-download-models). + +3.1. Option 1: put the checkpoint in `stable-diffusion-webui/models/SadTalker` or `stable-diffusion-webui/extensions/SadTalker/checkpoints/`, the checkpoints will be detected automatically. + +3.2. Option 2: Set the path of `SADTALKTER_CHECKPOINTS` in `webui_user.sh`(linux) or `webui_user.bat`(windows) by: + + > only works if you are directly starting webui from `webui_user.sh` or `webui_user.bat`. + + ```bash + # Windows (webui_user.bat) + set SADTALKER_CHECKPOINTS=D:\SadTalker\checkpoints + + # Linux/macOS (webui_user.sh) + export SADTALKER_CHECKPOINTS=/path/to/SadTalker/checkpoints + ``` + +4. Start the WebUI via `webui.sh or webui_user.sh(linux)` or `webui_user.bat(windows)` or any other method. SadTalker can also be used in stable-diffusion-webui directly. + + image + +## Questions + +1. if you are running on CPU, you need to specific `--disable-safe-unpickle` in `webui_user.sh` or `webui_user.bat`. + + ```bash + # windows (webui_user.bat) + set COMMANDLINE_ARGS="--disable-safe-unpickle" + + # linux (webui_user.sh) + export COMMANDLINE_ARGS="--disable-safe-unpickle" + ``` + + + +(If you're unable to use the `full` mode, please read this [discussion](https://github.com/Winfredy/SadTalker/issues/78).) diff --git a/animation/SadTalker/examples/driven_audio/RD_Radio31_000.wav b/animation/SadTalker/examples/driven_audio/RD_Radio31_000.wav new file mode 100644 index 0000000..3b04940 Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/RD_Radio31_000.wav differ diff --git a/animation/SadTalker/examples/driven_audio/RD_Radio34_002.wav b/animation/SadTalker/examples/driven_audio/RD_Radio34_002.wav new file mode 100644 index 0000000..6813e81 Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/RD_Radio34_002.wav differ diff --git a/animation/SadTalker/examples/driven_audio/RD_Radio36_000.wav b/animation/SadTalker/examples/driven_audio/RD_Radio36_000.wav new file mode 100644 index 0000000..c73adfe Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/RD_Radio36_000.wav differ diff --git a/animation/SadTalker/examples/driven_audio/RD_Radio40_000.wav b/animation/SadTalker/examples/driven_audio/RD_Radio40_000.wav new file mode 100644 index 0000000..88ce964 Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/RD_Radio40_000.wav differ diff --git a/animation/SadTalker/examples/driven_audio/bus_chinese.wav b/animation/SadTalker/examples/driven_audio/bus_chinese.wav new file mode 100644 index 0000000..8886477 Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/bus_chinese.wav differ diff --git a/animation/SadTalker/examples/driven_audio/chinese_news.wav b/animation/SadTalker/examples/driven_audio/chinese_news.wav new file mode 100644 index 0000000..168e92b Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/chinese_news.wav differ diff --git a/animation/SadTalker/examples/driven_audio/chinese_poem1.wav b/animation/SadTalker/examples/driven_audio/chinese_poem1.wav new file mode 100644 index 0000000..17c0871 Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/chinese_poem1.wav differ diff --git a/animation/SadTalker/examples/driven_audio/chinese_poem2.wav b/animation/SadTalker/examples/driven_audio/chinese_poem2.wav new file mode 100644 index 0000000..e3b294e Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/chinese_poem2.wav differ diff --git a/animation/SadTalker/examples/driven_audio/deyu.wav b/animation/SadTalker/examples/driven_audio/deyu.wav new file mode 100644 index 0000000..1a6bf3e Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/deyu.wav differ diff --git a/animation/SadTalker/examples/driven_audio/eluosi.wav b/animation/SadTalker/examples/driven_audio/eluosi.wav new file mode 100644 index 0000000..1ffe3c9 Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/eluosi.wav differ diff --git a/animation/SadTalker/examples/driven_audio/fayu.wav b/animation/SadTalker/examples/driven_audio/fayu.wav new file mode 100644 index 0000000..e3c67d8 Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/fayu.wav differ diff --git a/animation/SadTalker/examples/driven_audio/imagine.wav b/animation/SadTalker/examples/driven_audio/imagine.wav new file mode 100644 index 0000000..08a5dc7 Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/imagine.wav differ diff --git a/animation/SadTalker/examples/driven_audio/itosinger1.wav b/animation/SadTalker/examples/driven_audio/itosinger1.wav new file mode 100644 index 0000000..4937dbb Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/itosinger1.wav differ diff --git a/animation/SadTalker/examples/driven_audio/japanese.wav b/animation/SadTalker/examples/driven_audio/japanese.wav new file mode 100644 index 0000000..0b6146f Binary files /dev/null and b/animation/SadTalker/examples/driven_audio/japanese.wav differ diff --git a/animation/SadTalker/examples/ref_video/WDA_AlexandriaOcasioCortez_000.mp4 b/animation/SadTalker/examples/ref_video/WDA_AlexandriaOcasioCortez_000.mp4 new file mode 100644 index 0000000..5c06ff5 Binary files /dev/null and b/animation/SadTalker/examples/ref_video/WDA_AlexandriaOcasioCortez_000.mp4 differ diff --git a/animation/SadTalker/examples/ref_video/WDA_KatieHill_000.mp4 b/animation/SadTalker/examples/ref_video/WDA_KatieHill_000.mp4 new file mode 100644 index 0000000..1e295d5 Binary files /dev/null and b/animation/SadTalker/examples/ref_video/WDA_KatieHill_000.mp4 differ diff --git a/animation/SadTalker/examples/source_image/art_0.png b/animation/SadTalker/examples/source_image/art_0.png new file mode 100644 index 0000000..d8d9764 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_0.png differ diff --git a/animation/SadTalker/examples/source_image/art_1.png b/animation/SadTalker/examples/source_image/art_1.png new file mode 100644 index 0000000..4388abe Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_1.png differ diff --git a/animation/SadTalker/examples/source_image/art_10.png b/animation/SadTalker/examples/source_image/art_10.png new file mode 100644 index 0000000..5f6568b Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_10.png differ diff --git a/animation/SadTalker/examples/source_image/art_11.png b/animation/SadTalker/examples/source_image/art_11.png new file mode 100644 index 0000000..4caf17c Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_11.png differ diff --git a/animation/SadTalker/examples/source_image/art_12.png b/animation/SadTalker/examples/source_image/art_12.png new file mode 100644 index 0000000..e15306c Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_12.png differ diff --git a/animation/SadTalker/examples/source_image/art_13.png b/animation/SadTalker/examples/source_image/art_13.png new file mode 100644 index 0000000..1293741 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_13.png differ diff --git a/animation/SadTalker/examples/source_image/art_14.png b/animation/SadTalker/examples/source_image/art_14.png new file mode 100644 index 0000000..0f0489b Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_14.png differ diff --git a/animation/SadTalker/examples/source_image/art_15.png b/animation/SadTalker/examples/source_image/art_15.png new file mode 100644 index 0000000..a0af242 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_15.png differ diff --git a/animation/SadTalker/examples/source_image/art_16.png b/animation/SadTalker/examples/source_image/art_16.png new file mode 100644 index 0000000..7489eec Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_16.png differ diff --git a/animation/SadTalker/examples/source_image/art_17.png b/animation/SadTalker/examples/source_image/art_17.png new file mode 100644 index 0000000..acec2b9 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_17.png differ diff --git a/animation/SadTalker/examples/source_image/art_18.png b/animation/SadTalker/examples/source_image/art_18.png new file mode 100644 index 0000000..96358e0 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_18.png differ diff --git a/animation/SadTalker/examples/source_image/art_19.png b/animation/SadTalker/examples/source_image/art_19.png new file mode 100644 index 0000000..4f477a1 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_19.png differ diff --git a/animation/SadTalker/examples/source_image/art_2.png b/animation/SadTalker/examples/source_image/art_2.png new file mode 100644 index 0000000..9560673 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_2.png differ diff --git a/animation/SadTalker/examples/source_image/art_20.png b/animation/SadTalker/examples/source_image/art_20.png new file mode 100644 index 0000000..de1ea5c Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_20.png differ diff --git a/animation/SadTalker/examples/source_image/art_3.png b/animation/SadTalker/examples/source_image/art_3.png new file mode 100644 index 0000000..a67667d Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_3.png differ diff --git a/animation/SadTalker/examples/source_image/art_4.png b/animation/SadTalker/examples/source_image/art_4.png new file mode 100644 index 0000000..99cb57f Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_4.png differ diff --git a/animation/SadTalker/examples/source_image/art_5.png b/animation/SadTalker/examples/source_image/art_5.png new file mode 100644 index 0000000..81f73c4 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_5.png differ diff --git a/animation/SadTalker/examples/source_image/art_6.png b/animation/SadTalker/examples/source_image/art_6.png new file mode 100644 index 0000000..e9f6d8f Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_6.png differ diff --git a/animation/SadTalker/examples/source_image/art_7.png b/animation/SadTalker/examples/source_image/art_7.png new file mode 100644 index 0000000..d8cc380 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_7.png differ diff --git a/animation/SadTalker/examples/source_image/art_8.png b/animation/SadTalker/examples/source_image/art_8.png new file mode 100644 index 0000000..b407fc9 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_8.png differ diff --git a/animation/SadTalker/examples/source_image/art_9.png b/animation/SadTalker/examples/source_image/art_9.png new file mode 100644 index 0000000..3ac52d5 Binary files /dev/null and b/animation/SadTalker/examples/source_image/art_9.png differ diff --git a/animation/SadTalker/examples/source_image/full3.png b/animation/SadTalker/examples/source_image/full3.png new file mode 100644 index 0000000..40cd6d6 Binary files /dev/null and b/animation/SadTalker/examples/source_image/full3.png differ diff --git a/animation/SadTalker/examples/source_image/full4.jpeg b/animation/SadTalker/examples/source_image/full4.jpeg new file mode 100644 index 0000000..79f17f5 Binary files /dev/null and b/animation/SadTalker/examples/source_image/full4.jpeg differ diff --git a/animation/SadTalker/examples/source_image/full_body_1.png b/animation/SadTalker/examples/source_image/full_body_1.png new file mode 100644 index 0000000..4fca65c Binary files /dev/null and b/animation/SadTalker/examples/source_image/full_body_1.png differ diff --git a/animation/SadTalker/examples/source_image/full_body_2.png b/animation/SadTalker/examples/source_image/full_body_2.png new file mode 100644 index 0000000..b7bc622 Binary files /dev/null and b/animation/SadTalker/examples/source_image/full_body_2.png differ diff --git a/animation/SadTalker/examples/source_image/happy.png b/animation/SadTalker/examples/source_image/happy.png new file mode 100644 index 0000000..9d194ba Binary files /dev/null and b/animation/SadTalker/examples/source_image/happy.png differ diff --git a/animation/SadTalker/examples/source_image/happy1.png b/animation/SadTalker/examples/source_image/happy1.png new file mode 100644 index 0000000..b702974 Binary files /dev/null and b/animation/SadTalker/examples/source_image/happy1.png differ diff --git a/animation/SadTalker/examples/source_image/people_0.png b/animation/SadTalker/examples/source_image/people_0.png new file mode 100644 index 0000000..8895eeb Binary files /dev/null and b/animation/SadTalker/examples/source_image/people_0.png differ diff --git a/animation/SadTalker/examples/source_image/sad.png b/animation/SadTalker/examples/source_image/sad.png new file mode 100644 index 0000000..6584467 Binary files /dev/null and b/animation/SadTalker/examples/source_image/sad.png differ diff --git a/animation/SadTalker/examples/source_image/sad1.png b/animation/SadTalker/examples/source_image/sad1.png new file mode 100644 index 0000000..341e0cb Binary files /dev/null and b/animation/SadTalker/examples/source_image/sad1.png differ diff --git a/animation/SadTalker/scripts/download_models.sh b/animation/SadTalker/scripts/download_models.sh new file mode 100644 index 0000000..6898648 --- /dev/null +++ b/animation/SadTalker/scripts/download_models.sh @@ -0,0 +1,32 @@ +mkdir ./checkpoints + +# lagency download link +# wget -nc https://github.com/Winfredy/SadTalker/releases/download/v0.0.2/auido2exp_00300-model.pth -O ./checkpoints/auido2exp_00300-model.pth +# wget -nc https://github.com/Winfredy/SadTalker/releases/download/v0.0.2/auido2pose_00140-model.pth -O ./checkpoints/auido2pose_00140-model.pth +# wget -nc https://github.com/Winfredy/SadTalker/releases/download/v0.0.2/epoch_20.pth -O ./checkpoints/epoch_20.pth +# wget -nc https://github.com/Winfredy/SadTalker/releases/download/v0.0.2/facevid2vid_00189-model.pth.tar -O ./checkpoints/facevid2vid_00189-model.pth.tar +# wget -nc https://github.com/Winfredy/SadTalker/releases/download/v0.0.2/shape_predictor_68_face_landmarks.dat -O ./checkpoints/shape_predictor_68_face_landmarks.dat +# wget -nc https://github.com/Winfredy/SadTalker/releases/download/v0.0.2/wav2lip.pth -O ./checkpoints/wav2lip.pth +# wget -nc https://github.com/Winfredy/SadTalker/releases/download/v0.0.2/mapping_00229-model.pth.tar -O ./checkpoints/mapping_00229-model.pth.tar +# wget -nc https://github.com/Winfredy/SadTalker/releases/download/v0.0.2/mapping_00109-model.pth.tar -O ./checkpoints/mapping_00109-model.pth.tar +# wget -nc https://github.com/Winfredy/SadTalker/releases/download/v0.0.2/hub.zip -O ./checkpoints/hub.zip +# unzip -n ./checkpoints/hub.zip -d ./checkpoints/ + + +#### download the new links. +wget -nc https://github.com/OpenTalker/SadTalker/releases/download/v0.0.2-rc/mapping_00109-model.pth.tar -O ./checkpoints/mapping_00109-model.pth.tar +wget -nc https://github.com/OpenTalker/SadTalker/releases/download/v0.0.2-rc/mapping_00229-model.pth.tar -O ./checkpoints/mapping_00229-model.pth.tar +wget -nc https://github.com/OpenTalker/SadTalker/releases/download/v0.0.2-rc/SadTalker_V0.0.2_256.safetensors -O ./checkpoints/SadTalker_V0.0.2_256.safetensors +wget -nc https://github.com/OpenTalker/SadTalker/releases/download/v0.0.2-rc/SadTalker_V0.0.2_512.safetensors -O ./checkpoints/SadTalker_V0.0.2_512.safetensors + + +# wget -nc https://github.com/Winfredy/SadTalker/releases/download/v0.0.2/BFM_Fitting.zip -O ./checkpoints/BFM_Fitting.zip +# unzip -n ./checkpoints/BFM_Fitting.zip -d ./checkpoints/ + +### enhancer +mkdir -p ./gfpgan/weights +wget -nc https://github.com/xinntao/facexlib/releases/download/v0.1.0/alignment_WFLW_4HG.pth -O ./gfpgan/weights/alignment_WFLW_4HG.pth +wget -nc https://github.com/xinntao/facexlib/releases/download/v0.1.0/detection_Resnet50_Final.pth -O ./gfpgan/weights/detection_Resnet50_Final.pth +wget -nc https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.4.pth -O ./gfpgan/weights/GFPGANv1.4.pth +wget -nc https://github.com/xinntao/facexlib/releases/download/v0.2.2/parsing_parsenet.pth -O ./gfpgan/weights/parsing_parsenet.pth + diff --git a/animation/SadTalker/scripts/extension.py b/animation/SadTalker/scripts/extension.py new file mode 100644 index 0000000..e99fefe --- /dev/null +++ b/animation/SadTalker/scripts/extension.py @@ -0,0 +1,189 @@ +import os, sys +from pathlib import Path +import tempfile +import gradio as gr +from modules.call_queue import wrap_gradio_gpu_call, wrap_queued_call +from modules.shared import opts, OptionInfo +from modules import shared, paths, script_callbacks +import launch +import glob +from huggingface_hub import snapshot_download + + + +def check_all_files_safetensor(current_dir): + kv = { + "SadTalker_V0.0.2_256.safetensors": "sadtalker-256", + "SadTalker_V0.0.2_512.safetensors": "sadtalker-512", + "mapping_00109-model.pth.tar" : "mapping-109" , + "mapping_00229-model.pth.tar" : "mapping-229" , + } + + if not os.path.isdir(current_dir): + return False + + dirs = os.listdir(current_dir) + + for f in dirs: + if f in kv.keys(): + del kv[f] + + return len(kv.keys()) == 0 + +def check_all_files(current_dir): + kv = { + "auido2exp_00300-model.pth": "audio2exp", + "auido2pose_00140-model.pth": "audio2pose", + "epoch_20.pth": "face_recon", + "facevid2vid_00189-model.pth.tar": "face-render", + "mapping_00109-model.pth.tar" : "mapping-109" , + "mapping_00229-model.pth.tar" : "mapping-229" , + "wav2lip.pth": "wav2lip", + "shape_predictor_68_face_landmarks.dat": "dlib", + } + + if not os.path.isdir(current_dir): + return False + + dirs = os.listdir(current_dir) + + for f in dirs: + if f in kv.keys(): + del kv[f] + + return len(kv.keys()) == 0 + + + +def download_model(local_dir='./checkpoints'): + REPO_ID = 'vinthony/SadTalker' + snapshot_download(repo_id=REPO_ID, local_dir=local_dir, local_dir_use_symlinks=False) + +def get_source_image(image): + return image + +def get_img_from_txt2img(x): + talker_path = Path(paths.script_path) / "outputs" + imgs_from_txt_dir = str(talker_path / "txt2img-images/") + imgs = glob.glob(imgs_from_txt_dir+'/*/*.png') + imgs.sort(key=lambda x:os.path.getmtime(os.path.join(imgs_from_txt_dir, x))) + img_from_txt_path = os.path.join(imgs_from_txt_dir, imgs[-1]) + return img_from_txt_path, img_from_txt_path + +def get_img_from_img2img(x): + talker_path = Path(paths.script_path) / "outputs" + imgs_from_img_dir = str(talker_path / "img2img-images/") + imgs = glob.glob(imgs_from_img_dir+'/*/*.png') + imgs.sort(key=lambda x:os.path.getmtime(os.path.join(imgs_from_img_dir, x))) + img_from_img_path = os.path.join(imgs_from_img_dir, imgs[-1]) + return img_from_img_path, img_from_img_path + +def get_default_checkpoint_path(): + # check the path of models/checkpoints and extensions/ + checkpoint_path = Path(paths.script_path) / "models"/ "SadTalker" + extension_checkpoint_path = Path(paths.script_path) / "extensions"/ "SadTalker" / "checkpoints" + + if check_all_files_safetensor(checkpoint_path): + # print('founding sadtalker checkpoint in ' + str(checkpoint_path)) + return checkpoint_path + + if check_all_files_safetensor(extension_checkpoint_path): + # print('founding sadtalker checkpoint in ' + str(extension_checkpoint_path)) + return extension_checkpoint_path + + if check_all_files(checkpoint_path): + # print('founding sadtalker checkpoint in ' + str(checkpoint_path)) + return checkpoint_path + + if check_all_files(extension_checkpoint_path): + # print('founding sadtalker checkpoint in ' + str(extension_checkpoint_path)) + return extension_checkpoint_path + + return None + + + +def install(): + + kv = { + "face_alignment": "face-alignment==1.3.5", + "imageio": "imageio==2.19.3", + "imageio_ffmpeg": "imageio-ffmpeg==0.4.7", + "librosa":"librosa==0.8.0", + "pydub":"pydub==0.25.1", + "scipy":"scipy==1.8.1", + "tqdm": "tqdm", + "yacs":"yacs==0.1.8", + "yaml": "pyyaml", + "av":"av", + "gfpgan": "gfpgan", + } + + # # dlib is not necessary currently + # if 'darwin' in sys.platform: + # kv['dlib'] = "dlib" + # else: + # kv['dlib'] = 'dlib-bin' + + # #### we need to have a newer version of imageio for our method. + # launch.run_pip("install imageio==2.19.3", "requirements for SadTalker") + + for k,v in kv.items(): + if not launch.is_installed(k): + print(k, launch.is_installed(k)) + launch.run_pip("install "+ v, "requirements for SadTalker") + + if os.getenv('SADTALKER_CHECKPOINTS'): + print('load Sadtalker Checkpoints from '+ os.getenv('SADTALKER_CHECKPOINTS')) + + elif get_default_checkpoint_path() is not None: + os.environ['SADTALKER_CHECKPOINTS'] = str(get_default_checkpoint_path()) + else: + + print( + """" + SadTalker will not support download all the files from hugging face, which will take a long time. + + please manually set the SADTALKER_CHECKPOINTS in `webui_user.bat`(windows) or `webui_user.sh`(linux) + """ + ) + + # python = sys.executable + + # launch.run(f'"{python}" -m pip uninstall -y huggingface_hub', live=True) + # launch.run(f'"{python}" -m pip install --upgrade git+https://github.com/huggingface/huggingface_hub@main', live=True) + # ### run the scripts to downlod models to correct localtion. + # # print('download models for SadTalker') + # # launch.run("cd " + paths.script_path+"/extensions/SadTalker && bash ./scripts/download_models.sh", live=True) + # # print('SadTalker is successfully installed!') + # download_model(paths.script_path+'/extensions/SadTalker/checkpoints') + + +def on_ui_tabs(): + install() + + sys.path.extend([paths.script_path+'/extensions/SadTalker']) + + repo_dir = paths.script_path+'/extensions/SadTalker/' + + result_dir = opts.sadtalker_result_dir + os.makedirs(result_dir, exist_ok=True) + + from app_sadtalker import sadtalker_demo + + if os.getenv('SADTALKER_CHECKPOINTS'): + checkpoint_path = os.getenv('SADTALKER_CHECKPOINTS') + else: + checkpoint_path = repo_dir+'checkpoints/' + + audio_to_video = sadtalker_demo(checkpoint_path=checkpoint_path, config_path=repo_dir+'src/config', warpfn = wrap_queued_call) + + return [(audio_to_video, "SadTalker", "extension")] + +def on_ui_settings(): + talker_path = Path(paths.script_path) / "outputs" + section = ('extension', "SadTalker") + opts.add_option("sadtalker_result_dir", OptionInfo(str(talker_path / "SadTalker/"), "Path to save results of sadtalker", section=section)) + +script_callbacks.on_ui_settings(on_ui_settings) +script_callbacks.on_ui_tabs(on_ui_tabs) diff --git a/animation/SadTalker/scripts/test.sh b/animation/SadTalker/scripts/test.sh new file mode 100644 index 0000000..bcfecfd --- /dev/null +++ b/animation/SadTalker/scripts/test.sh @@ -0,0 +1,21 @@ +# ### some test command before commit. +# python inference.py --preprocess crop --size 256 +# python inference.py --preprocess crop --size 512 + +# python inference.py --preprocess extcrop --size 256 +# python inference.py --preprocess extcrop --size 512 + +# python inference.py --preprocess resize --size 256 +# python inference.py --preprocess resize --size 512 + +# python inference.py --preprocess full --size 256 +# python inference.py --preprocess full --size 512 + +# python inference.py --preprocess extfull --size 256 +# python inference.py --preprocess extfull --size 512 + +python inference.py --preprocess full --size 256 --enhancer gfpgan +python inference.py --preprocess full --size 512 --enhancer gfpgan + +python inference.py --preprocess full --size 256 --enhancer gfpgan --still +python inference.py --preprocess full --size 512 --enhancer gfpgan --still