storytold/FineTrainers-Conditioning
2
0
Fork 0
mirror of https://github.com/storytold/FineTrainers-Conditioning.git synced 2026-10-17 00:23:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

[optimization] support 8bit optims from bistandbytes (#163)

Browse Source 
* support 8bit optims from bnb.

* fix exist_ok

* use_8bit_bnb.

* fix.

* note in readme.
This commit is contained in:
Sayak Paul
2024-12-30 14:03:56 +05:30
committed by GitHub
parent e9fba4ab7f
commit 191ea8b0cc
4 changed files with 12 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+1
View File
@@ -394,6 +394,7 @@ If you would like to use a custom dataset, refer to the dataset preparation guid
> - Use a DeepSpeed config to launch training (refer to [`accelerate_configs/deepspeed.yaml`](./accelerate_configs/deepspeed.yaml) as an example).
> - Pass `--precompute_conditions` when launching training.
> - Pass `--gradient_checkpointing` when launching training.
> - Pass `--use_8bit_bnb` when launching training. Note that this is only applicable to Adam and AdamW optimizers.
> - Do not perform validation/testing. This saves a significant amount of memory, which can be used to focus solely on training if you're on smaller VRAM GPUs.
## Memory requirements