cloud-worker/docs/BENCHMARKS.md

MiniMax H3 — measured benchmarks

Measured 2026-08-06 with bench/minimax_bench.py through the ComfyUI API. Raw data: bench/results.jsonl (B200), bench/4090/, bench/4000/. Regenerate the tables with python3 bench/summarize.py.

Method. ComfyUI 0.30 headless; template-reference sampling everywhere: res_multistep sampler, simple scheduler, 20 steps, no CFG, 24 fps, nvfp4_awq Qwen3-VL-32B text encoder, fp16 video VAE + fp32 audio VAE. Times are ComfyUI execution time (execution_start→execution_success), which excludes client/queue overhead (wall time tracked separately in the raw data; typically +1–2 s). "Warm" = model already resident; first-load runs are recorded but excluded from the means. Repeated configs use distinct seeds so ComfyUI's node cache can't short-circuit. Reference inputs are real images (film stills, anime characters, forests/volcanos, up to 23040×3840 — scripts/pod/fetch-bench-images-v2.py).

Hardware: B200 180 GB (US, $6.94/hr) · RTX 4090 24 GB (EU, $0.74/hr) · RTX PRO 4000 Blackwell 24 GB (EU, $0.57/hr). The EU cards run pruned int8 only (bf16/int8 don't fit 24 GB usefully) with --reserve-vram 2.5; the RTX 4000 additionally needs --cache-none --disable-pinned-memory because its container has a 29 GB RAM cap that otherwise OOM-kills weight staging.

B200: weight-family comparison (warm, 20 steps)

The headline: bf16 is the fastest family on the B200 — quantization costs speed here (dequant overhead) and only pays when VRAM is scarce.

Config	bf16	int8	pruned int8	bf16 VRAM	int8 VRAM	pruned VRAM
t2v 864×480 5 s	51 s	63 s	63 s	159 GB	63 GB	69 GB
t2v 864×480 15 s	240 s	270 s	270 s	182 GB	101 GB	75 GB
t2v 1344×768 5 s	169 s	197 s	194 s	177 GB	99 GB	73 GB
ref2v 1 ref 864×480 5 s	54 s	68 s	~68 s¹	186 GB	95 GB	68 GB
ref2v 4 refs match	63 s	86 s	81 s	154 GB	95 GB	70 GB
ref2v 8 refs match	79 s	99 s	95 s	185 GB	96 GB	68 GB
ref2v 8 refs match 1344×768	282 s	335 s	339 s	158 GB	103 GB	77 GB
ref2v 4 refs max	91 s	117 s	112 s	156 GB	96 GB	70 GB
ref2v 8 refs max	785 s	891 s	841 s	177 GB	112 GB	92 GB

¹ pruned 1-ref mean is 90±41 s over 3 runs because the first run absorbed a model swap; the steady-state runs are ~68 s, matching int8.

bf16 peaks at 155–187 GB — it only runs on ≥180 GB cards. int8 fits under 112 GB (H200-class); pruned int8 under ~95 GB worst-case, ~70 GB typical (H100-class without offload, 24 GB cards with offload).

Scaling behavior (B200, pruned int8)

Duration (864×480 t2v): 5 s → 63 s, 10 s → 154 s, 15 s → 270 s. Cost per output-second rises from 12.6→15.4→18.0 s — mildly superlinear (attention).

Resolution (5 s t2v): 608×352 → 32 s, 864×480 → 63 s, 1344×768 → 194 s. Roughly ∝ pixels^1.2. The corner case 1344×768×15 s = 1108 s (18.5 min).

Reference count (864×480 5 s, match sizing): 1 ref ≈ 68 s, 4 refs ≈ 81 s, 8 refs ≈ 95 s — ~+4 s per extra reference image. Cheap.

Reference sizing max is the expensive lever: with 8 large refs it goes 95 s → 841 s (~9×), because 2048px reference tokens ride through every sampling step. Use match unless identity fidelity demands otherwise.

Cross-GPU (pruned int8, warm, 20 steps — final measured)

t2v 864×480 5 s:

GPU	$/hr	time/video	videos/hr	$/video	notes
B200 (bf16)	$6.94	51 s (n=2)	70	$0.098	fastest family on this card
B200 (pruned)	$6.94	63 s (n=3)	57	$0.121
RTX 4090	$0.74	222 s (n=3, ±0)	16	$0.046	--reserve-vram 2.5
RTX PRO 4000	$0.57	553 s (n=3, ±3)	6.5	$0.087	--cache-none tax (29 GB RAM cap)

ref2v single ref, 864×480 5 s:

GPU	time/video	$/video
B200 (bf16)	54 s (n=3)	$0.104
RTX 4090	287–392 s	$0.059–0.081
RTX PRO 4000	581 s (n=3, ±6)	$0.092

RTX 5090 (32 GB, $0.99/hr) — added later, the fleet workhorse:

Config (pruned int8, warm)	time	$/video	notes
t2v 864×480 5 s	143 s (n=3, ±1)	$0.039	cheapest t2v measured
ref2v 1 ref	153 s (n=3, ±0)	$0.042
ref2v 8 refs full-size	220 s (n=3, ±1)	$0.060	vs $0.152 B200-bf16: 2.5× cheaper
ref2v 9 refs full-size	237 s	$0.065	max ref count
t2v 15 s	582 s	$0.160	works (killed 24 GB pre-patch)
t2v 1344×768	455 s	$0.125	works
9:16 480×864 / 1:1 640×640	187 s / 141 s	—	aspects fine
ref2v 9 refs max sizing	2031 s (34 min)	$0.559	works; avoid in production

bf16 on the 5090 (66 GB DiT layer-streamed through 32 GB): warm t2v = 110 s — faster than its own pruned int8 (143 s); ref2v 1/4/8 refs = 310/336/369 s (needs --cache-none when switching between bf16 models — two 66 GB stacks break the 124 GB container RAM cap). B200-quality output at $0.99/hr for everything except 15 s + full-res extremes.

Verdict: the 5090 dominates cost AND capability among consumer cards — cheapest per video on every task shape, full reference envelope, and a credible bf16 quality tier. The patched 4090 (see MITIGATIONS.md) is a viable budget multi-ref tier (~8-9 refs at 429–444 s). The RTX PRO 4000's low price is consumed by its slow, reload-heavy runs — dropped. Fleet recipe: 5090 pools for volume, B200 pool only for 15 s/full-res premium jobs and fastest-latency needs.

Concurrency (2 ComfyUI instances, one B200)

See docs/CONCURRENCY.md for the model. Experiment: two resident instances, sequential pair vs concurrent pair of identical warm t2v jobs.

Measured (pruned int8, t2v 864×480 5 s, both instances warm):

Mode	Total for 2 videos	Per-video effective
Sequential (one instance)	128 s	64 s
Concurrent (two instances)	145 s	72.5 s

Concurrent was 13% slower for the same work — each co-scheduled job ran 142 s vs 63 s alone (2.27×). There is no pipeline-overlap win; cache and scheduler contention make co-scheduling strictly worse. One worker per GPU, queue in front, scale horizontally. (This also removes any argument for MPS/MIG partitioning for throughput.)

Reliability & failure statistics

Every run's end state and error is recorded in the JSONL files; regenerate the failure tables with python3 bench/failures.py, and validate output files with python3 bench/validate_outputs.py (downloads each MP4 and ffprobes streams + duration). Snapshot as of 2026-08-06 ~09:00 UTC:

GPU	success rate	dominant failures
B200 180 GB	96/96 (100%)	none
RTX 4090 24 GB	9/53 (17%)	VRAM OOM: TE vision (multi-ref), DiT sampling (refs/15s/full-res); cascading server death
RTX PRO 4000 24 GB	7/12 (58%)	container-RAM OOM (pre-flags); corrupt ref image (fixed); server death on 15 s t2v

Output validation: 100% of successful runs produced valid videos (h264 + stereo AAC at exactly the requested duration) — every success on all three GPUs checks out. (Three early B200 outputs 404 on re-download because they were written to the container disk that the pod reset wiped — outputs now go to the persistent volume via --output-directory.)

The 24 GB failure pattern is reference images, not duration/resolution. t2v succeeds reliably on both consumer cards at every size tried; ref2v OOMs in torch._int_mm during Qwen3-VL image encoding: the 15 GB text encoder plus multi-image vision activations exceeds 24 GB (observed 22.1 GB peak at OOM). With aggressive flags (--cache-none --disable-pinned-memory --reserve-vram 2.5) a single reference works (4090: 392 s); 4-ref and 8-ref configurations fail consistently, and repeated OOMs eventually killed the server process, cascading failures across the rest of that suite (harness now survives this and records server_died).

A second environment trap on cheap tiers: the RTX 4000 pod's container has a 29 GB RAM cap that OOM-killed weight staging until --cache-none --disable-pinned-memory was applied; both EU pods also shipped torch 2.4.1, too old for H3 (needs ≥2.5; Blackwell needs ≥2.7+cu128).

Fleet implication: consumer 24 GB cards are viable for t2v/i2v and single-ref ref2v only. Multi-reference work — the priority modality — needs ≥32 GB (5090-class, unverified) and realistically ≥48 GB for headroom, or it stays on big-GPU pools.

Multi-ref rescue status (in progress)

Community research (see agent report summarized in commit history) identified the root cause as ComfyUI PR #15316: VRAM is budgeted for TE weights but not image-encode activations. Two OOM phases were isolated on the 4090:

1. TE vision encode — fixed by ref pre-downscaling (--ref-downscale, now in the harness) and/or the #15316 patch + higher --reserve-vram. Confirmed: with 0.1–0.2 MP refs the TE phase completes.
2. DiT sampling with ref tokens — the 20 GB DiT + ref-token-enlarged attention doesn't fit 24 GB. --lowvram made things worse (crashes). Remaining candidates: #15316 patch active + --reserve-vram 5 (patched file is applied on both EU pods, pending verification), TE eviction via ComfyUI-H3-Multishot, or a smaller DiT quant (community INT4/W4A8, ~10–12 GB, quality TBD).

Verification was interrupted when all three pods went offline simultaneously (apparent RunPod account event ~09:30 UTC). Resume plan lives in the repo: restart pods → scripts/local/push-to-pod.sh → rescue matrix via bench/minimax_bench.py --suite refheavy --ref-downscale 0.2.

Takeaways

1. On the B200, run bf16. Fastest and highest quality; quantized families exist for smaller cards, not for speed.
2. Many references are cheap; big references are not. 8 refs at match costs ~1.4× a single ref. max sizing costs up to ~9×. Default match.
3. Duration and resolution both scale superlinearly; the trained envelope corner (15 s @ 1344×768) costs ~18 min even on a B200 — treat full-res long clips as premium jobs.
4. 24 GB consumer cards work but pay a heavy offload tax (~3.5× B200 latency rather than the ~3× raw-compute ratio), and cheap pod tiers bring operational traps: container RAM caps (29 GB on the RTX 4000 pod) and stock torch builds too old for H3 (needs ≥2.5 for enable_gqa; Blackwell needs ≥2.7+cu128).