FineTrainers-Conditioning/docs/training/optimization.md at fa422c4ca5780e4bf5ca6615e0c7451e508e5c56

mirror of https://github.com/storytold/FineTrainers-Conditioning.git synced 2026-10-09 00:09:45 +00:00

Files

T

Sayak Paul eeb4dd7afa [docs] refactor docs for easier info parsing (#175 )

* refactor docs for easier info parsing

* refactor readme

* anonym paths

* updates

* remove notes.

* add a toc.

* minor typos.

* move cog.md -> cogvideox.md

* add a note about the model-specific docs in training readme.

* add memory usage for CogVideoX.

Co-authored-by: a-r-r-o-w <contact.aryanvs@gmail.com>

* change to 5b from 2b for CogVideoX.

Co-authored-by: a-r-r-o-w <contact.aryanvs@gmail.com>

* more appropriate names.

* add headers to the model docs.

* fix adapter name

* minor

* updates

* fix cog training command example

---------

Co-authored-by: a-r-r-o-w <contact.aryanvs@gmail.com>

2025-01-05 20:50:30 +05:30

658 B

Raw Blame History

To lower memory requirements during training:

Use a DeepSpeed config to launch training (refer to accelerate_configs/deepspeed.yaml as an example).
Pass --precompute_conditions when launching training.
Pass --gradient_checkpointing when launching training.
Pass --use_8bit_bnb when launching training. Note that this is only applicable to Adam and AdamW optimizers.
Do not perform validation/testing. This saves a significant amount of memory, which can be used to focus solely on training if you're on smaller VRAM GPUs.

We will continue to add more features that help to reduce memory consumption.

658 B Raw Blame History

658 B

Raw Blame History