storytold/FineTrainers-Conditioning
2
0
Fork 0
mirror of https://github.com/storytold/FineTrainers-Conditioning.git synced 2026-10-18 07:33:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

improve dataset preparation (#43)

Browse Source 
* update

* update

* update

* update
This commit is contained in:
Aryan
2024-10-18 02:43:22 +05:30
committed by GitHub
parent 8c12f34d4e
commit f2a1626de2
8 changed files with 129 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
assets/dataset.md
+2 -2
View File
@@ -18,7 +18,7 @@ The framework supports resolutions and frame counts that meet the following cond
- Any resolution as long as it is divisible by 32. For example, `720 * 480`, `1920 * 1020`, etc.
- **Supported Frame Counts (Frames)**:
- Must satisfy (4K + 1), i.e., multiples of 4 such as 16, 24, 32, 48, 64, 80.
- Must be `4 * k` or `4 * k + 1` (example: 16, 32, 49, 81)
It is recommended to place all videos in a single folder.
@@ -58,4 +58,4 @@ huggingface-cli download --repo-type dataset Wild-Heart/Disney-VideoGeneration-D
This dataset has been prepared in the expected format and can be used directly. However, directly using the video dataset may cause Out of Memory (OOM) issues on GPUs with smaller VRAM because it requires loading the [VAE](https://huggingface.co/THUDM/CogVideoX-5b/tree/main/vae) (which encodes videos into latent space) and the large [T5-XXL](https://huggingface.co/google/t5-v1_1-xxl/) text encoder. To reduce memory usage, you can use the `training/prepare_dataset.py` script to precompute latents and embeddings.
Fill or modify the parameters in `prepare_dataset.sh` and execute it to get precomputed latents and embeddings (make sure to specify `--save_tensors` to save the precomputed artifacts). When using these artifacts during training, ensure that you specify the `--load_tensors` flag, or else the videos will be used directly, requiring the text encoder and VAE to be loaded. The script also supports PyTorch DDP so that large datasets can be encoded in parallel across multiple GPUs (modify the `NUM_GPUS` parameter).
Fill or modify the parameters in `prepare_dataset.sh` and execute it to get precomputed latents and embeddings (make sure to specify `--save_latents_and_embeddings` to save the precomputed artifacts). If preparing for image-to-video training, make sure to pass `--save_image_latents`, which encodes and saves image latents along with videos. When using these artifacts during training, ensure that you specify the `--load_tensors` flag, or else the videos will be used directly, requiring the text encoder and VAE to be loaded. The script also supports PyTorch DDP so that large datasets can be encoded in parallel across multiple GPUs (modify the `NUM_GPUS` parameter).
assets/dataset_zh.md
+4 -3
View File
@@ -18,7 +18,7 @@ A black and white animated sequence on a ship’s deck features a bulldog charac
- 任意分辨率且必须能被32整除。例如，`720 * 480`, `1920 * 1020` 等分辨率。
- **支持的帧数（Frames）**：
- 满足 (4K +1)，即4的倍数，例如，16, 24, 32, 48, 64, 80。
- 必须是 `4 * k` 或 `4 * k + 1`（例如：16, 32, 49, 81）
所有的视频建议放在一个文件夹中。
@@ -66,6 +66,7 @@ OOM（内存不足），因为它需要加载 [VAE](https://huggingface.co/THUDM
文本编码器。为了降低内存需求，您可以使用 `training/prepare_dataset.py` 脚本预先计算潜在变量和嵌入。
填写或修改 `prepare_dataset.sh` 中的参数并执行它以获得预先计算的潜在变量和嵌入（请确保指定 `--save_tensors`
以保存预计算的工件）。在训练期间使用这些工件时，确保指定 `--load_tensors` 标志，否则将直接使用视频并需要加载文本编码器和
填写或修改 `prepare_dataset.sh` 中的参数并执行它以获得预先计算的潜在变量和嵌入（请确保指定 `--save_latents_and_embeddings`
以保存预计算的工件）。如果准备图像到视频的训练，请确保传递 `--save_image_latents`，它对沙子进行编码，将图像潜在值与视频一起保存。
在训练期间使用这些工件时，确保指定 `--load_tensors` 标志，否则将直接使用视频并需要加载文本编码器和
VAE。该脚本还支持 PyTorch DDP，以便可以使用多个 GPU 并行编码大型数据集（修改 `NUM_GPUS` 参数）。