FineTrainers-Conditioning/docs/training/optimization.md at feature/full-finetuning

mirror of https://github.com/storytold/FineTrainers-Conditioning.git synced 2026-10-09 00:09:45 +00:00

Files

T

Sayak Paul eeb4dd7afa [docs] refactor docs for easier info parsing (#175 )

* refactor docs for easier info parsing

* refactor readme

* anonym paths

* updates

* remove notes.

* add a toc.

* minor typos.

* move cog.md -> cogvideox.md

* add a note about the model-specific docs in training readme.

* add memory usage for CogVideoX.

Co-authored-by: a-r-r-o-w <contact.aryanvs@gmail.com>

* change to 5b from 2b for CogVideoX.

Co-authored-by: a-r-r-o-w <contact.aryanvs@gmail.com>

* more appropriate names.

* add headers to the model docs.

* fix adapter name

* minor

* updates

* fix cog training command example

---------

Co-authored-by: a-r-r-o-w <contact.aryanvs@gmail.com>

2025-01-05 20:50:30 +05:30

658 B

Raw Permalink Blame History

To lower memory requirements during training:

Use a DeepSpeed config to launch training (refer to accelerate_configs/deepspeed.yaml as an example).
Pass --precompute_conditions when launching training.
Pass --gradient_checkpointing when launching training.
Pass --use_8bit_bnb when launching training. Note that this is only applicable to Adam and AdamW optimizers.
Do not perform validation/testing. This saves a significant amount of memory, which can be used to focus solely on training if you're on smaller VRAM GPUs.

We will continue to add more features that help to reduce memory consumption.

658 B Raw Permalink Blame History

658 B

Raw Permalink Blame History