reliability: failure-stats + output-validation tooling; persistent output dir

- bench/failures.py: per-GPU success rates and error-class breakdown by config
- bench/validate_outputs.py: ffprobe every recorded output via /view
- finding: 24GB cards fail on multi-ref (TE vision OOM), not duration/size;
  100% of successful runs produce valid videos
- outputs now written to /workspace (survives pod resets)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Hanashi
2026-08-06 14:41:00 -04:00
parent fbc68bb749
commit a293d0a2c4
13 changed files with 308 additions and 2 deletions
+3 -2
View File
@@ -7,6 +7,7 @@
set -euo pipefail
LOG=/workspace/minimax-h3/comfyui-$(hostname).log
mkdir -p /workspace/minimax-h3/output
ARGS=${COMFY_ARGS:-}
# wait out any process that is mid-shutdown before deciding it's "running"
@@ -21,9 +22,9 @@ fi
if command -v tmux >/dev/null; then
tmux new -s comfyui -d \
"cd /ComfyUI && python3 main.py --listen 127.0.0.1 --port 8188 $ARGS 2>&1 | tee -a $LOG"
"cd /ComfyUI && python3 main.py --listen 127.0.0.1 --port 8188 --output-directory /workspace/minimax-h3/output $ARGS 2>&1 | tee -a $LOG"
else
cd /ComfyUI
nohup python3 main.py --listen 127.0.0.1 --port 8188 $ARGS >>"$LOG" 2>&1 &
nohup python3 main.py --listen 127.0.0.1 --port 8188 --output-directory /workspace/minimax-h3/output $ARGS >>"$LOG" 2>&1 &
fi
echo "started; tail $LOG"