storytold/cloud-worker
2
0
Fork 0
mirror of https://github.com/storytold/cloud-worker.git synced 2026-10-17 00:23:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

reliability: failure-stats + output-validation tooling; persistent output dir

Browse Source 
- bench/failures.py: per-GPU success rates and error-class breakdown by config
- bench/validate_outputs.py: ffprobe every recorded output via /view
- finding: 24GB cards fail on multi-ref (TE vision OOM), not duration/size;
  100% of successful runs produce valid videos
- outputs now written to /workspace (survives pod resets)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Hanashi
2026-08-06 14:41:00 -04:00
parent fbc68bb749
commit a293d0a2c4
13 changed files with 308 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bench/4000/results.csv
+5
View File
@@ -1,3 +1,8 @@
ts,label,task,model_file,width,height,seconds,length,steps,sampler,scheduler,ref_image_size,ref_count,warm,ok,wall_s,exec_s,peak_vram_gb,outputs,error
2026-08-06T07:51:24Z,g4000_smoke_t2v,t2v,minimax_h3_fl2va_pruned_int8_convrot.safetensors,608,352,5.0,124,20,res_multistep,simple,match,1,False,False,135.0,,15.0,,server unreachable while polling: [Errno 104] Connection reset by peer
2026-08-06T07:58:16Z,g4000_smoke_t2v_b,t2v,minimax_h3_fl2va_pruned_int8_convrot.safetensors,608,352,5.0,124,20,res_multistep,simple,match,1,False,True,343.4,341.0,22.7,g4000_smoke_t2v_b_00001_.mp4,
2026-08-06T17:58:17Z,g4000_pi8_key_t2v_5s_r0,t2v,minimax_h3_fl2va_pruned_int8_convrot.safetensors,864,480,5,124,20,res_multistep,simple,match,1,False,True,559.7,557.0,20.4,g4000_pi8_key_t2v_5s_r0_00001_.mp4,
2026-08-06T18:07:31Z,g4000_pi8_key_t2v_5s_r1,t2v,minimax_h3_fl2va_pruned_int8_convrot.safetensors,864,480,5,124,20,res_multistep,simple,match,1,True,True,553.4,551.4,20.4,g4000_pi8_key_t2v_5s_r1_00001_.mp4,
2026-08-06T18:16:44Z,g4000_pi8_key_t2v_5s_r2,t2v,minimax_h3_fl2va_pruned_int8_convrot.safetensors,864,480,5,124,20,res_multistep,simple,match,1,True,True,553.4,550.6,20.4,g4000_pi8_key_t2v_5s_r2_00001_.mp4,
2026-08-06T18:26:34Z,g4000_pi8_key_ref2v_1ref_r0,ref2v,minimax_h3_ref2va_pruned_int8_convrot.safetensors,864,480,5,124,20,res_multistep,simple,match,1,False,True,589.8,587.8,19.0,g4000_pi8_key_ref2v_1ref_r0_00001_.mp4,
2026-08-06T18:36:16Z,g4000_pi8_key_ref2v_1ref_r1,ref2v,minimax_h3_ref2va_pruned_int8_convrot.safetensors,864,480,5,124,20,res_multistep,simple,match,1,True,True,582.4,580.1,19.0,g4000_pi8_key_ref2v_1ref_r1_00001_.mp4,
1 ts label task model_file width height seconds length steps sampler scheduler ref_image_size ref_count warm ok wall_s exec_s peak_vram_gb outputs error
2 2026-08-06T07:51:24Z g4000_smoke_t2v t2v minimax_h3_fl2va_pruned_int8_convrot.safetensors 608 352 5.0 124 20 res_multistep simple match 1 False False 135.0 15.0 server unreachable while polling: [Errno 104] Connection reset by peer
3 2026-08-06T07:58:16Z g4000_smoke_t2v_b t2v minimax_h3_fl2va_pruned_int8_convrot.safetensors 608 352 5.0 124 20 res_multistep simple match 1 False True 343.4 341.0 22.7 g4000_smoke_t2v_b_00001_.mp4
4 2026-08-06T17:58:17Z g4000_pi8_key_t2v_5s_r0 t2v minimax_h3_fl2va_pruned_int8_convrot.safetensors 864 480 5 124 20 res_multistep simple match 1 False True 559.7 557.0 20.4 g4000_pi8_key_t2v_5s_r0_00001_.mp4
5 2026-08-06T18:07:31Z g4000_pi8_key_t2v_5s_r1 t2v minimax_h3_fl2va_pruned_int8_convrot.safetensors 864 480 5 124 20 res_multistep simple match 1 True True 553.4 551.4 20.4 g4000_pi8_key_t2v_5s_r1_00001_.mp4
6 2026-08-06T18:16:44Z g4000_pi8_key_t2v_5s_r2 t2v minimax_h3_fl2va_pruned_int8_convrot.safetensors 864 480 5 124 20 res_multistep simple match 1 True True 553.4 550.6 20.4 g4000_pi8_key_t2v_5s_r2_00001_.mp4
7 2026-08-06T18:26:34Z g4000_pi8_key_ref2v_1ref_r0 ref2v minimax_h3_ref2va_pruned_int8_convrot.safetensors 864 480 5 124 20 res_multistep simple match 1 False True 589.8 587.8 19.0 g4000_pi8_key_ref2v_1ref_r0_00001_.mp4
8 2026-08-06T18:36:16Z g4000_pi8_key_ref2v_1ref_r1 ref2v minimax_h3_ref2va_pruned_int8_convrot.safetensors 864 480 5 124 20 res_multistep simple match 1 True True 582.4 580.1 19.0 g4000_pi8_key_ref2v_1ref_r1_00001_.mp4