storytold/FineTrainers-Conditioning
2
0
Fork 0
mirror of https://github.com/storytold/FineTrainers-Conditioning.git synced 2026-10-23 08:15:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update README and Contribution guide (#20)

Browse Source 
* multi resolutions support

* full chinese readme

* Update README.md

* Update README.md

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* reformat from pycharm

* dataset.md

* Update README.md

---------

Co-authored-by: Aryan <aryan@huggingface.co>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
This commit is contained in:
Yuxuan.Zhang
2024-10-12 05:07:57 +08:00
committed by GitHub
parent 9621d72628
commit 1323aeb5b1
6 changed files with 559 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
assets/contribute.md
+15 -2
View File
@@ -1,3 +1,16 @@
# 欢迎你们的贡献
# Contributions Welcome
本项目属于非常初级的阶段
This project is in a very early stage, and we welcome contributions from everyone. We hope to receive contributions and support in the following areas:
1. Support for more models. In addition to CogVideoX models, we also highly encourage contributions supporting other models.
2. Support for richer datasets. In our example, we used a Disney video generation dataset, but we hope to support more datasets as the current one is too limited for deeper fine-tuning exploration.
3. Anything in `TODO` we mention in our README.md
## How to Submit
We welcome you to create a new PR and describe the corresponding contribution. We will review it as soon as possible.
## Naming Conventions
- Please use English for naming, avoid using pinyin or other languages. All comments should be in English.
- Strictly follow PEP8 conventions, and use underscores to separate words. Please avoid using names like a, b, c.
assets/contribute_zh.md
+16
View File
@@ -0,0 +1,16 @@
# 欢迎你们的贡献
本项目属于非常初级的阶段，欢迎大家进行贡献。我们希望在以下方面得到贡献和支持：
1. 支持更多的模型，除了 CogVideoX 模型之外的模型，我们也非常支持。
2. 更丰富的数据集支持。在我们的例子中，我们使用了一个 Disney 视频生成数据集，但是我们希望能够支持更多的数据集，这个数据集太少了，并不足以进行更深的微调探索。
3. 任何我们在README中`TODO`提到的内容。
## 提交方式
我们欢迎您直接创建一个新的PR，并说明对应的贡献，我们将第一时间查看。
## 命名规范
- 请使用英文命名，不要使用拼音或者其他语言命名。所有的注释均使用英文。
- 请严格遵循 PEP8 规范，使用下划线分割单词。请勿使用 a,b,c 这样的命名。
assets/dataset.md
+61
View File
@@ -0,0 +1,61 @@
## Dataset Format
### Prompt Dataset Requirements
Create a `prompt.txt` file, which should contain prompts separated by lines. Please note that the prompts must be in English, and it is recommended to use the [prompt refinement script](https://github.com/THUDM/CogVideo/blob/main/inference/convert_demo.py) for better prompts. Alternatively, you can use [CogVideo-caption](https://huggingface.co/THUDM/cogvlm2-llama3-caption) for data annotation:
```
A black and white animated sequence featuring a rabbit, named Rabbity Ribfried, and an anthropomorphic goat in a musical, playful environment, showcasing their evolving interaction.
A black and white animated sequence on a ship’s deck features a bulldog character, named Bully Bulldoger, showcasing exaggerated facial expressions and body language...
...
```
### Video Dataset Requirements
The framework supports resolutions and frame counts that meet the following conditions:
- **Supported Resolutions (Width * Height)**:
- Any resolution as long as it is divisible by 32. For example, `720 * 480`, `1920 * 1020`, etc.
- **Supported Frame Counts (Frames)**:
- Must satisfy (4K + 1), i.e., multiples of 4 such as 16, 24, 32, 48, 64, 80.
It is recommended to place all videos in a single folder.
Next, create a `videos.txt` file. The `videos.txt` file should contain the video file paths, separated by lines. Please note that the paths must be relative to the `--data_root` directory. The format is as follows:
```
videos/00000.mp4
videos/00001.mp4
...
```
For developers interested in more details, you can refer to the relevant `BucketSampler` code.
### Dataset Structure
Your dataset structure should look like this. Running the `tree` command, you should see:
```
dataset
├── prompt.txt
├── videos.txt
├── videos
├── videos/00000.mp4
├── videos/00001.mp4
├── ...
```
### Using the Dataset
When using this format, the `--caption_column` should be set to `prompt.txt`, and the `--video_column` should be set to `videos.txt`. If your data is stored in a CSV file, you can also specify `--dataset_file` as the path to the CSV file, with `--caption_column` and `--video_column` set to the actual column names in the CSV. Please refer to the [test_dataset](../tests/test_dataset.py) file for some simple examples.
For instance, you can fine-tune using [this](https://huggingface.co/datasets/Wild-Heart/Disney-VideoGeneration-Dataset) Disney dataset. The download can be done via the 🤗 Hugging Face CLI:
```
huggingface-cli download --repo-type dataset Wild-Heart/Disney-VideoGeneration-Dataset --local-dir video-dataset-disney
```
This dataset has been prepared in the expected format and can be used directly. However, directly using the video dataset may cause Out of Memory (OOM) issues on GPUs with smaller VRAM because it requires loading the [VAE](https://huggingface.co/THUDM/CogVideoX-5b/tree/main/vae) (which encodes videos into latent space) and the large [T5-XXL](https://huggingface.co/google/t5-v1_1-xxl/) text encoder. To reduce memory usage, you can use the `training/prepare_dataset.py` script to precompute latents and embeddings.
Fill or modify the parameters in `prepare_dataset.sh` and execute it to get precomputed latents and embeddings (make sure to specify `--save_tensors` to save the precomputed artifacts). When using these artifacts during training, ensure that you specify the `--load_tensors` flag, or else the videos will be used directly, requiring the text encoder and VAE to be loaded. The script also supports PyTorch DDP so that large datasets can be encoded in parallel across multiple GPUs (modify the `NUM_GPUS` parameter).
assets/dataset_zh.md
+71
View File
@@ -0,0 +1,71 @@
## 数据集格式
### 提示词数据集要求
创建 `prompt.txt` 文件，文件应包含逐行分隔的提示。请注意，提示必须是英文，并且建议使用 [提示润色脚本](https://github.com/THUDM/CogVideo/blob/main/inference/convert_demo.py) 进行润色。或者可以使用 [CogVideo-caption](https://huggingface.co/THUDM/cogvlm2-llama3-caption) 进行数据标注：
```
A black and white animated sequence featuring a rabbit, named Rabbity Ribfried, and an anthropomorphic goat in a musical, playful environment, showcasing their evolving interaction.
A black and white animated sequence on a ship’s deck features a bulldog character, named Bully Bulldoger, showcasing exaggerated facial expressions and body language...
...
```
### 视频数据集要求
该框架支持的分辨率和帧数需要满足以下条件：
- **支持的分辨率（宽 * 高）**：
- 任意分辨率且必须能被32整除。例如，`720 * 480`, `1920 * 1020` 等分辨率。
- **支持的帧数（Frames）**：
- 满足 (4K +1)，即4的倍数，例如，16, 24, 32, 48, 64, 80。
所有的视频建议放在一个文件夹中。
接着，创建 `videos.txt` 文件。 `videos.txt` 文件应包含逐行分隔的视频文件路径。请注意，路径必须相对于 `--data_root` 目录。格式如下：
```
videos/00000.mp4
videos/00001.mp4
...
```
对于有兴趣了解更多细节的开发者，您可以查看相关的 `BucketSampler` 代码。
### 数据集结构
您的数据集结构应如下所示，通过运行`tree`命令，你能看到:
```
dataset
├── prompt.txt
├── videos.txt
├── videos
├── videos/00000.mp4
├── videos/00001.mp4
├── ...
```
### 使用数据集
当使用此格式时，`--caption_column` 应为 `prompt.txt`，`--video_column` 应为 `videos.txt`。如果您的数据存储在 CSV
文件中，也可以指定 `--dataset_file` 为 CSV 文件的路径，`--caption_column` 和 `--video_column` 为 CSV
文件中的实际列名。请参考 [test_dataset](../tests/test_dataset.py) 文件中的一些简单示例。
例如，使用 [这个](https://huggingface.co/datasets/Wild-Heart/Disney-VideoGeneration-Dataset) Disney 数据集进行微调。下载可通过🤗
Hugging Face CLI 完成：
```
huggingface-cli download --repo-type dataset Wild-Heart/Disney-VideoGeneration-Dataset --local-dir video-dataset-disney
```
该数据集已按照预期格式准备好，可直接使用。但是，直接使用视频数据集可能会导致较小 VRAM 的 GPU 出现
OOM（内存不足），因为它需要加载 [VAE](https://huggingface.co/THUDM/CogVideoX-5b/tree/main/vae)
（将视频编码为潜在空间）和大型 [T5-XXL](https://huggingface.co/google/t5-v1_1-xxl/)
文本编码器。为了降低内存需求，您可以使用 `training/prepare_dataset.py` 脚本预先计算潜在变量和嵌入。
填写或修改 `prepare_dataset.sh` 中的参数并执行它以获得预先计算的潜在变量和嵌入（请确保指定 `--save_tensors`
以保存预计算的工件）。在训练期间使用这些工件时，确保指定 `--load_tensors` 标志，否则将直接使用视频并需要加载文本编码器和
VAE。该脚本还支持 PyTorch DDP，以便可以使用多个 GPU 并行编码大型数据集（修改 `NUM_GPUS` 参数）。