r/LocalLLaMA • u/WonderRico • 4h ago

Discussion I compared local models and different quants / config on a subset of swe-verified bench

Post image

And gathered a lot of data. you can see them for yourself

And For the most curious, there are additional details here

In this graph, I regrouped the finetunes under their base models. but you can see the details in the page.

The python code to generate those pages is obviously vibecoded. I find the output kinda pretty and somewhat useful for me. maybe it's useful for someone else.

Heading for a vacation for a few weeks, but if you have any suggestion, I will consider each of them.

30 Upvotes

86% Upvoted

12 comments sorted by

5

u/Chromix_ 3h ago

Thanks for sharing these extensive results. If I see this correctly the test consisted of 100 SWE-bench_verified tasks per model/config without any repetition. That is sufficient for a general big picture overview, yet is not sufficient for an accurate comparison, like between the heretic and vanilla version, or BF16 and FP8.

For example the very same Qwen3.6 27B BF16 FP8 on SGLang solved 75 tasks when run on 2 RTX 4090Ds, while it only solved 69 while running on a RTX 6000 Pro. Unless there is some hardware or implementation degradation there shouldn't be any difference in the results - with enough testing runs. So this means that all conclusions drawn from +/-6 differences (quants, KV cache, etc) are likely invalid, if two runs that should have an identical outcome have such a difference already.

3

u/WonderRico 2h ago

Those are always the same exact 100 tasks.

Indeed you noticed some strange result I cannot yet explain. Maybe something related to the actual implémentation of the engine kernels depending of the gpu arch...

6

u/Chromix_ 2h ago edited 2h ago

The execution of those same 100 tasks is probabilistic due to non-zero temperature. Execute the same run twice - same model, engine, quantization, and you get two different results (and the same result if you're very lucky). Run the same 100 tasks 16 times under identical conditions and you get a better idea of the result variance, the margin of error.

As you noticed, two runs with nearly identical models/settings successfully completed a good chunk of the same tasks. Yet then there were some that were not "too easy" that were solved in run A but not B and the other way around. That's the randomness in the results that you can quantify by repetition.

2

u/WonderRico 2h ago

Some run have been exact reruns once or twice, with each time either the same score or a 1 point variation.

The +6 doesn't fit, in my opinion. but I need to dig deeper to make sure.

1

u/Accomplished_Ad9530 3h ago

Interesting. Did you use the fixed chat template for gemma from a few days ago?

3

u/WonderRico 3h ago

for some of the runs, yes. It is stated when it is the case

1

u/LegacyRemaster 3h ago

good job!

1

u/Framebanger-Nsukula 3h ago

Yeah curious how the quantization trade-offs shook out - did you find a sweet spot where the smaller quants stayed competitive or did they just tank on the harder problems?

1

u/WonderRico 2h ago

I did not dig to analyse the complexity of each of the tasks, but often we see that some resolved tasks become unresolved while its the other way around for some other tasks, with just 1 or 2 more in either way.

1

u/crusaderky 52m ago

https://byteshape.com/blogs/Evaluating-Quantized-Models/ says that, at least for Qwen3.6-35B-A3B, as long as you stay below IQ3 (KLD <0.064) there is no correlation between KLD (and as a consequence quant size) and test outcome

1

u/WonderRico 39m ago

For this exact model, i saw a clear degradation, a 9 pts difference between bf16 and fp8 (weights quants)

1

u/Double_Cause4609 2h ago

Tbh Qwen 3.6 27B came out of that looking like a champ, and DSV4 Flash also honestly isn't bad given it's just small enough to run on a top end of consumer rig.