diff --git a/README.md b/README.md index aacdf50..de2cdd8 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,7 @@ Current test rig: 1x B200 (180 GB) pod — connection details in `pod-info.txt`. (SGLang / vLLM-Omni / diffusers), concurrency model, RunPod serverless - `docs/BENCHMARKS.md` — measured results on the B200 - `docs/CONCURRENCY.md` — concurrency model + cost-per-video math across GPU tiers +- `docs/MITIGATIONS.md` — per-machine-class OOM map and required ComfyUI flags/patches ## Quick start diff --git a/docs/MITIGATIONS.md b/docs/MITIGATIONS.md new file mode 100644 index 0000000..44747cc --- /dev/null +++ b/docs/MITIGATIONS.md @@ -0,0 +1,66 @@ +# MiniMax H3 — per-machine-class mitigations & OOM map + +Compiled 2026-08-06 from measured failures and fixes across five pods. +Companion to `docs/BENCHMARKS.md` (runtimes) and `bench/failures.py` (stats). + +## The two resources that fail, and their fixes + +**1. Container RAM (cgroup cap — check `/sys/fs/cgroup/memory/memory.limit_in_bytes`, +cgroup v1 on RunPod consumer pods, or `/sys/fs/cgroup/memory.max` for v2).** +ComfyUI stages every loaded model's weights in host RAM ("prepared for dynamic +VRAM loading"). Task-switching between fl2va and ref2va stacks WITHOUT eviction +stages TE (15 GB) + two DiTs (20 GB each) + overhead ≈ 60 GB+. Exceeding the +cap = silent SIGKILL, no traceback, "server died" mid-run. +Fix: `--cache-none` (one model staged at a time) + `--disable-pinned-memory` +(staging stays pageable/reclaimable). Cost: weights reload per run — measured +~2.3× slowdown on the RTX 4000 (551 s vs an est. ~240 s), ~1.4× on the 4090. + +**2. VRAM, in two distinct phases:** +- *TE vision encode* (multi-ref): budget bug, fixed by ComfyUI **PR #15316** + (apply: `git fetch origin pull/15316/head:pr15316 && git checkout pr15316 -- + comfy/text_encoders/minimax.py`) + `--reserve-vram 4`–`5`. Ref pre-downscale + (`--ref-downscale 0.2` in our harness) also clears this phase alone. +- *DiT sampling with ref tokens / long clips / large canvas*: needs the + activations to fit beside the (partially loaded) 20 GB DiT. On 24 GB this is + the hard wall: 15 s clips, 1344×768, and multi-ref all exceeded it pre-patch. + `--lowvram` made it WORSE (crashes) — do not use for H3. + +Hygiene for all capped machines: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` +(set automatically by `start-comfyui.sh`), torch ≥ 2.5 (`enable_gqa`), and for +Blackwell (B200/5090/RTX PRO 4000) torch ≥ 2.7 with cu128 wheels. + +## Machine-class matrix (all measured, not theoretical) + +| Pod class | VRAM | RAM cap | Required config | What works | What fails | +|----------------------|--------|---------|--------------------------------------------------------------|---------------------------------------------------|-------------------------------| +| B200 ($6.94/hr) | 180 GB | ~3 TB | none (defaults) | everything incl. 9-ref, 15 s, full-res, bf16 | nothing observed (96/96) | +| RTX 5090 ($0.99/hr) | 32 GB | 124 GB | #15316 patch + `--reserve-vram 4` | multi-ref up to 8 full-size refs confirmed; 9-ref + max sizing under test | TBD (matrix running) | +| RTX 4090 ($0.74/hr) | 24 GB | 62 GB | #15316 patch + `--reserve-vram 5` + `--cache-none` + `--disable-pinned-memory` | t2v/i2v ≤10 s @ ≤864×480, 1-ref ref2v; multi-ref under re-test with patch | 15 s clips, 1344×768, multi-ref (pre-patch) | +| RTX PRO 4000 ($0.57/hr) | 24 GB | 29 GB | same as 4090 (mandatory even for t2v) | t2v/i2v, 1-ref (551–588 s — uneconomical) | multi-ref untested post-fix; 15 s died | + +Key procurement insight: **the container RAM cap, not the GPU, decides the +mitigation burden.** A 24 GB card in a ≥96 GB-RAM container would skip +`--cache-none` and its ~1.4–2.3× tax. When renting, prefer pod configs with +RAM ≥ 4× VRAM. + +## Reference-image envelope (node schema + measured) + +- The node accepts up to **9 reference images** + 3 reference videos (each + with optional soundtrack) + 3 standalone audio clips — "12 refs" = 9 images + + 3 videos. +- `match` sizing (refs scaled down to canvas area) is the default and the + cheap path; `max` (2048 px short edge) multiplies sampling cost ~9× at + 8 refs and is the most VRAM-hungry config in the whole matrix. +- Ref count scaling on B200: +4 s per extra ref (match). Counts 1→9 and the + ref-resolution ladder (0.1 MP → native) are being measured on all three + GPU classes right now (`--suite refmatrix`). + +## Open items + +- 5090: 9-ref and 8-ref-`max` verdicts (running). +- 4090 with full stack (#15316 + cache-none + reserve 5): multi-ref verdict + (running). +- RTX PRO 4000: dropped from fleet consideration on economics; would need + re-test only if its price class matters. +- Upstream: file the multi-ref OOM issue referencing #15316 (user decision — + outward-facing).