PipelineScore v4

71 tasks. Real tools. Real repos.

v4 tests what a model does with work you would actually hand it: fixing a bug in a real repository, finishing a multi-step job by calling tools in the right order, and answering from long documents. Code runs in a sandbox and every answer is checked by a program, not another model. The score is quality only; run time sits beside it.

v4 is harder than v3, so scores are lower and the two boards are not comparable. See the v3 board.

5 runs · best run per model and hardware · click a column to re-rank

#	Model · hardware	PipelineScore ▾	Code	Repo fix	Agent	Tool calls	Reason	Long docs	Instructions	Run time
1	qwen3.8:27bm5-max-48gb	94.8	99.3	100.0	100.0	100.0	91.7	100.0	60.0	76 min
2	gemma4:31b-it-q4_K_Mm3-ultra-96gb	90.3	89.5	100.0	100.0	100.0	83.3	95.0	48.9	5.0 h
3	muse-glimmer:30bm5-max-48gb	89.4	80.0	100.0	76.7	100.0	91.7	100.0	98.8	3.2 h
4	MiniMax-M2.7minimax	83.2	93.8	50.0	83.3	100.0	66.7	100.0	87.6	2.0 h
5	openai/gpt-oss-20bm5-max-48gb	70.6	94.7	75.0	53.3	70.8	66.7	75.0	65.4	9 min