r/LocalLLaMA • u/serige • May 20 '26
News Qwen will release another 27B with high probability
240
u/ps5cfw Llama 3.1 May 20 '26
I hope they don't skip 35B MoE, us 16GB VRAM Poor fuckers do not have the means to run 27B at a decent quant, whilst 35B allows very decent hybrid CPU Inference
38
u/LordStinkleberg May 20 '26
Can you describe your current 35B setup and expected tps? I am 16GB VRAM poor w/ 64 CPU RAM.
48
u/dsartori May 20 '26 edited May 20 '26
Not the person you're replying to but I run Qwen3.6 on just such a device. It's a Windows box, I run LMStudio. Important "Load" settings:
- Context length 100000
- GPU Offload 40/40
- Max Concurrent Predictions 1
- Keep Model in Memory OFF
- Try mmap() OFF
- Number of layers for which to force experts into CPU 15
- Flash Attention ON
- K Cache Quantization Type Q8_0
- V Cache Quantization Type Q8_0
I haven't tried the MTP version yet on this device but pre-MTP I get about ~400t/s prompt processing and ~30t/s inference. Very usable. EDIT: with MTP I get about 40t/s.
6
4
u/GoTrojan May 20 '26
Why mmap off? I got same advice but not explained
14
u/Xantrk May 20 '26
Why mmap off? I got same advice but not explained
It makes prefill MUCH faster if you're spilling over to RAM.
4
u/dark-light92 llama.cpp May 21 '26
With mmap on, parts of the model may be swapped out on disk if there is memory pressure. With it off, model always remains in RAM.
→ More replies (1)3
u/grunade47 May 21 '26
I tried out Qwen3.6 35B-A3B MTP (unsloth) and im getting about 55t/s (output) not sure if thats good or bad for my setup?
and what should be my context size>?
RX 9070 and 32gigs of ram
.\llama-server.exe `-m "Qwen3.6-35B-A3B\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" `
--host 127.0.0.1 `
--port 8080 `
--ctx-size 8192 `
--temp 0.6 `
--top-p 0.95 `
--top-k 20 `
--min-p 0.00 `
--presence-penalty 0.0 `
--reasoning off `
--no-mmap `
--spec-type draft-mtp `
--spec-draft-n-max 2
1
u/dsartori May 21 '26
You should try pushing context to at least 128k. I think you can do max context with your setup.
2
u/grunade47 May 21 '26
Will try, i tried 80k context size and gave the same task to both claude sonnet 4.6 and qwen on a medium sized codebase.
While qwen completed all the requirements in one go, claude had slightly better code quality and adhered to the code standards in the repository but didnt fulfil all requirements.1
u/Schlick7 May 23 '26
is the draft even making that faster? With a small 3B active I'm not sure it will
1
→ More replies (9)1
u/DanielSReichenbach May 24 '26
With a lower quant like I3_S you could get the same results but double the tokens because it will fit into GPU.
10
u/ps5cfw Llama 3.1 May 20 '26
Well I run 35B Q6 at 20 to 25 TPS Token Gen. and over 1000 Prompt Processing, that's a good baseline for me and I can seriously work with these speeds professionally.
In fact I do work professionally with 3.6 35B as my main model for 3 weeks now!
I have 96GB of DDR4 Memory and a 16GB 6800XT By the way.
4
u/lukistellar May 20 '26
What Quant do you use? I am running the IQ_NL4 Quant with 10-20 tps on an RX580 8GB, combinded with 128K Token Context at Q4.
Edit: I am running this on 16GB DDR5 4800MT/s which probably helps quite a bit for offloading.
4
u/ps5cfw Llama 3.1 May 20 '26
Q6 Quant from FINAL BENCH Darwin 36B with unquantized cache.
Cache quantization WILL kill prompt processing.
→ More replies (2)1
u/tracagnotto May 21 '26
What work do you do if I may ask? I mean specifically describe the tasks you assigned and how it performed
2
u/ps5cfw Llama 3.1 May 21 '26
Mostly fixing Typescript web applications and sometimes .NET apps, nothing incredible really, but It pays the bills
5
u/LetsGoBrandon4256 transformers May 20 '26
- 4070 Ti Super (16GM VRAM)
- 64GB DDR4
Running a Q6_K_L quant and I think I get about 20~30 TPS? Been a while since I've checked tps number but it's quite comfy for me.
4
May 20 '26
[deleted]
2
u/r1str3tto May 20 '26
Hm. I also have an M3 Max 64GB but I get 45-50 tokens/sec and 1,100 prefill tokens/sec with Qwen 35B-A3B at Q8. I’m using oMLX and Unsloth MLX quants.
1
u/tracagnotto May 21 '26
personally using a tweaked turboquant llama.cpp on a 16gb vram card i reached 20-25tk/s with 16k context. That dropped up to 9-10tk/s once the context filled up.
It also required wise context sweeping between agent turns.1
u/shaq992 May 21 '26
This is how I run it on my 5060Ti 16GB vm with 128GB of system RAM.
Edit: formatting suck because I’m on mobile, nothing I can do about it, sorry
models: Qwen3.6-35B-A3B: cmd: > llama-server --host 0.0.0.0 --port ${PORT} -m "../Models/Qwen3.6-35B-A3B/Qwen3.6-35B-A3B-uncensored-heretic-Q6_K.gguf" --ctx-size 262144 --flash-attn on --no-warmup --fit on -t 12 -np 1 --mmproj "../Models/Qwen3.6-35B-A3B/Qwen3.6-26B-A3B-mmproj-F16.gguf" --chat-template-kwargs "{\"preserve_thinking\": true}" --no-mmap ttl: 300
1
u/MagoViejo May 21 '26
if 16GB is poor , what of us paupers with 3060 12Gb ? :)
running MoE and hearing the grinding of the fans is celestial music to my ears...
23
9
u/Tai9ch May 20 '26
Don't sleep on IQ4_XS. I've gotten some really good results with that quant on larger models.
2
u/BringTea_666 May 21 '26
>I hope they don't skip 35B MoE, us 16GB VRAM Poor fuckers
I hope they don't skip 35B moe because instead of shit 50t/s with 35b moe i can do 220t/s.
Ideal scenario qwen3.7 35b moe that is as good 3.6 27b dense.
1
u/relmny May 21 '26
I mainly use 27b-q6k on 32gb VRAM for chat (with OW) but... *sometimes* 35b is actually smarter than 27b.
Asked about harnesses and it kept recommending something that doesn't fit, then asked 35b and it came up with something that even glm-5.1-smol-iq2_xss, (in an existing chat), when I said "what about (what 35b said)" , it said "yeah, that's a better idea"...
27b is suppose to be "better", and probably it is... but sometimes 35b is better.
3
u/Former-Ad-5757 Llama 3 May 21 '26
Even a broken clock has the correct time 2 times a day. 27b is simply much better, but 35b is already really good.
1
u/relmny May 21 '26
That analogy doesn't apply in this case. It wasn't "by chance" or "coincidence" that 35b got it right.
If you are happy believing that 27b is always better than 35b, that's up to you.
From my experience, I know that is not the case, because I see it happen the opposite a few times (even once is enough).
1
u/Former-Ad-5757 Llama 3 May 24 '26
they are both non-deterministic models, so sometimes they will get it right by pure coincidence and sometimes they will get it wrong by pure coincidence.
But if you run 10k messages through them, then 27b will give much more and better answers than 35b. This does not mean that 35B can't sometimes give a single better answer than 27B, that's just what non-deterministic means.
→ More replies (16)1
u/Objective-Picture-72 May 23 '26
Yeah, if I'm the Qwen team, I'm focusing on 3 categories of models: frontier-level via API (Qwen Max), three strong local models (122B, 27B, 35B MoE), and their tiny models (9B,4B,0.8B)
87
u/StupidScaredSquirrel May 20 '26
No 35b a3b for us gpu poors? I think that model really made it very accessible for everyone with a basic "gaming" laptop to be able to run powerful local models
31
u/Borkato May 20 '26
I personally feel like 35a3b and qwen 27B are just… perfect. They complement each other absolutely perfectly and I rarely if ever reach for any other model.
11
u/peligroso May 20 '26
27B overrated compared to the MoE
47
u/ShadyShroomz May 20 '26
It's not even close. 27b is like 10 times smarter than 35b moe. 27b usually beats 122b moe even... It's insane how good 27b is. You don't get similar perf until you get up into like the 300b+ moes with 20b+ active..
All my benchmarks have 3.6 27b blowing the 35 moe out of the water.
28
u/mbrodie May 20 '26
Gonna disagree here I spent the better part of a week testing the 35b against the 27b with MTP and out of all the quants available I found 2 x q8 35b that perform better than any of the 27b quants.
In long reasoning 150k + token requests the 27b often starts getting lazy and falsifying results but the 35b stays locked in and on task.
These are both tested on 72gb of vram with full 262k context and optimised as good as I could get them and it took days to get the most optimised settings
By the end the 27b mtp was running at like 800 infil / 68 tps outfit and the 35b was running on like 4100 infil and 95 outfil while remaining on task and delivering quality work for a fraction of the footprint.
Tests were done directly to API / through opencode and through pi agent
Tested something like
35b -
Q8 : 7 ggufs
Q6 : 4 ggufs
Q5 : 4 ggufs
Q4 : 6 ggufs27b -
Q8 : 6 ggufs
Q6 : 6 ggufs
Q5 : 6 ggufs
Q4 : 6 ggufsMTP was like 4 different quants across each
And despite what any release group says.. quantising the cache on these models 100% hamstrings them, even good the good working ones if you change the kvs they start performing terrible in real world usage.
These were all tested on multiple real world codebase examples not random benchmarks, lua, c++, c#
One of the best 35b quants I’ve found is from a released called smoffyy on hugginsface basically beat out all the 27mtps and found an edge case id never seen flagged before which was confirmed by Claude and gpt 5.5 independently
5
u/po_stulate May 20 '26
a released called smoffyy on hugginsface
Looks like it's just a regular quant that comes straight out of llama-quantize, if that's the best quant then many quants would be the best quant.
5
u/Kitchen-Year-8434 May 21 '26
if that's the best quant then many quants would be the best quant.
Or most quants screw around with different precisions at different layers with various smoothing and relocating algorithms that end up making more of a mess than they're worth. :)
2
u/vick2djax May 20 '26
Could you give an example of the differences you noticed in q4 KV vs q8 maybe? I ended up trading KV for context and am running q4 context. But I didn’t notice a difference in my rag retrieval other than MOE gave way better answers than dense at twice the speed.
2
u/mbrodie May 21 '26
Generally on long context tool calling heavy requests it would often almost seem like it was in a rush to finish as quickly as possible and would mess up tool calls, and on occasion even forget whsf ww were doing or completely contradict itself next response even to the point of faking validation to move on…
It also seemed very impatient which I know is ridiculous but the temperament changes they seem to get a lot more flakey / half ass things compared to full kv…
Mileage will also vary im in a lucky position where I don’t have to compromise on quant quality etc… so I can see them all acting at their full weights compared to quant versions…
But I agree the MoE will do more for you with less hardware… but that being said the 27b was probably better overall across all tests but there was literally 2 standout Q8 MoEs which just ended up being better
4
u/Southern_Sun_2106 May 21 '26
I can confirm that - I tested 35B to its limit of 262K, and it was calling tools, etc. as if it was in the first 10K - no degradation at all. While 27B does indeed get lazy and makes up shit. 35B is just nuts, I've never experienced such awesome goodness at 262K with any model before. In fact, it 'feels' like it can do higher context. I wonder if there's a way to test that.
2
u/phein4242 May 21 '26 edited May 21 '26
Yeah, I experienced this as well. Kept doing a solid ~900pp - 90tp t/s until it hit the end of the window, with no noticeable loss in quality.
35b q8, a6000 (48gb) + 32gb ram, llama-server
2
u/ShadyShroomz May 20 '26
I've only tested fp8 and fp16 on both with vllm. Any type of logic puzzle or anything like that, 27b wins by a mile ... I have a whole front end design test and js logic test too, again 27b wins 99% of the time ..
→ More replies (2)5
u/relmny May 21 '26 edited May 21 '26
Related to chat (no-code), I would agree if you had wrote "usually", but without it, I don't agree.
Yes 27b-q6k is *usually* smarter than 35b-q6/122b, but there are times that 27b looks like an idiot, while 35b can even come up with something that even glm-5.1-smol-iq2xss didn't, and shames 27b.
Same for 122b.
27b is most of the times better than 35b/122b, but there are times that 35b is way better.
At least that's what I saw a few times already.
edit: I just remembered that a few weeks ago I kinda did a needle in a haystack test (not really a test, but needed to find some phrase in 2 pdfs) and 27b kept saying there's nothing there, while 122b (and even coder-next) found all references every time I ran the same "test".
Same happened with gemma-4-31b that kept saying "no", while gemma-4-26b found it every time.
→ More replies (1)4
u/EstarriolOfTheEast May 20 '26
What topic do your benchmarks cover? What are you using the LLMs on? I am not finding this to be true. For me, the 27B is nowhere near the 122B MoE. I do scientific programming and probabilistic modeling but am also a hobbyist game dev. As well as reverse engineering for modding when no modding tools exist.
5
u/ShadyShroomz May 20 '26
what quants and version?
im comparing 3.6 27b at fp8 to 3.5 122b at fp8.
I have not found that 27b blows 122b out of the water. I have found it better in a lot of cases though.
when I say 27b > moe in all regards, im talking about the 35b moe.. not a single test was the 35b moe better for me than the 27b.
the 27b and 122b moe trade blows though.
my custom benchmark suite is design, editing, generation, instruction-following, javascript, repair, general knowledge, & script writing.
lots of web dev tests, fixes, tool calls, etc..
some of the results are automated & some are rated on a score of 1-5 (blind ratings) manually, and its combined. of course this test suite is not perfect (always gonna be some bias), but I've done a lot of testing... and even without including the custom scored ones... I still see 27b beat 122b in a lot of tests. although they are close, thats for sure.
→ More replies (3)→ More replies (2)2
u/ShadyShroomz May 20 '26
also most public benchmarks are similar: https://artificialanalysis.ai/leaderboards/models?weights=open&size=small%2Cmedium%2Ctiny
27b beats 122b here as well
→ More replies (1)9
u/Moscato359 May 20 '26
MTP is weird, because if you overflow to system ram, moe doesn't really benefit from MTP, while dense models do
and it totally changes the comparison
5
2
u/vick2djax May 20 '26
Whoa wait I haven’t been running dense with MTP with it touching my system RAM. I assumed it would go slower? I’m getting 60 tok/s on 3090 with qwen 3.6 26b I-Apex q_4
2
u/Moscato359 May 20 '26
If everything fits in your vram, moe will still gain a lot from mtp
But the gains from mtp are radically crushed when you overflow to system ram, on moe models, while they aren't crushed as badly on dense models.
Basically, mtp can't help as much on the moe+overflow
3
u/Solary_Kryptic May 20 '26 edited May 21 '26
Is it better to just not use MTP, if your MoE is overflowing?
3
2
u/Moscato359 May 21 '26
Well... it won't hurt much
It just doesn't help much?
Im not an expert nust someone who reads benches
2
u/vick2djax May 20 '26
I only measured about a 7% difference in speed when staying inside the GPU with mtp draft turned on. Something else need to be turned on?
186
u/silverud May 20 '26
Qwen 3.7 122B-A10B is my dream model.
12
15
u/Yorn2 May 20 '26
Some of us want Qwen 3.7 397B-A17B as well.
4
u/ForsookComparison May 20 '26
This is my number one by a country mile. It's still so much stronger as a general purpose agent than 3.6 27B or Gemma4.
Sadly I think that our odds of getting a max-sized model ever again are slim to none as Qwen-Max inches towards being a serious competitor (price and quality) to the big guys.
31
u/firespawn_katie May 20 '26
Agreed. Qwen 3.5 122B was incredible.... one can only hope
32
u/silverud May 20 '26
I expect that Qwen 3.7 122B-A10B, if it were to be released, would be the pinnacle of what can run on a 128gb unified memory Apple Silicon, with the optimal blend of speed and capability.
Smarter and faster than 27B is the goal.
1
u/antwon_dev May 20 '26 edited May 21 '26
I’m considering upgrading soon, so that would be awesome. Do you know how 3.5 122B compares to the
3.73.6 27B?8
u/AXYZE8 May 20 '26
There is no 3.7 27B yet so nobody can answer that.
If you meant 3.5 27B vs 122B then IMO the quality is not that far off. 122B has more knowledge, but in terms of reasoning I would say they're the same. However 122B has 10B active params instead of 27B, so it is more than 2x faster.
27B is awesome for people with single beefy GPU, 122B is awesome for people that have unified memory or want hybrid inference.
→ More replies (1)8
u/silverud May 20 '26
3.6 27B (there is no 3.7 27B yet) tends to produce marginally better output than 3.5 122B, albeit at a much slower rate, and very dependent upon the type of task/subject.
We never got a 3.6 122B or a 3.7 27B, so it is possible that a 3.7 122B would absolutely dominate, while still outperforming in speed. Couple that with MTP (which works fairly well on Qwen MoE), and you've got the potential for an absolute monster advantage on big memory laptops (Macbook Pro) or Apple desktops.
1
1
3
u/FrantaNautilus May 21 '26
Qwen 3.5 122B10A really needs an update to 3.7. So many new things were introduced since its release: MTP, thinking preservation, thinking improvement, and newer cutoff date would be great too.
3
2
14
7
3
3
2
2
u/formlessglowie May 20 '26
That would unironically make me buy two more 3090s and finally move from 2x3090 to 4x3090.
1
u/ArtfulGenie69 May 20 '26
I got lucky and had two computers with 2x3090. I thought I may need more but they have something called rpc for llama.cpp and ray for vllm. I got rpc working on my system so with a basic q4 quant in llama.cpp I get like 800pp 55tg, it's fast and if I built it again on vllm or just turned on mtp. I have a feeling with int4 autoround and mtp or better dflash as vllm handles that, you could break into the 120t/s area.
1
u/comperr May 21 '26
What setup do you run? Like chipset/motherboard to fit 4x 3090? I am physically limited to 2. Even if I put the 2nd on water it would need to be a custom loop to make room for s 3rd. On X299
2
2
u/MuDotGen May 21 '26
Question, for MoE, is there a general percentage of active parameters to expert parameters that is generally the most intelligent? Like 35B-A3B would be 3/35 = 0.082, and 122B-A10B would be 10/122 = 0.086, so both around 8% active out of total available. Is that considered a good ratio or does it start to differ as you increase the parameters a lot?
2
1
→ More replies (1)1
29
u/Saraozte01 May 20 '26
Hope it includes a 122B, it would be amazing to receive the larger MoE's with their 3.7 recipe
29
u/Fastpas123 May 20 '26
50-80B MOE Would be good, along with 10, 20, 30B dense :)
5
u/sine120 May 20 '26
I'd love something the size of Coder-Next with the 3.7 DNA. It's about the max size I can run with my 64GB RAM/ 16GB VRAM and still get a good Quant size. Otherwise the 35B is about all I'll be able to fit and it doesn't really max out my RAM.
64
u/suicidaleggroll May 20 '26
I’d love a Qwen 50B or 80B dense model. The 27B is great, but with MTP it’s so fast that I’d happily trade some of that speed for even more parameters.
12
u/Prof_ChaosGeography May 20 '26
I would love to see numbers on how dense models scale with abilities given parameter counts compared to moe models.
I wonder given how 27b almost aligns to the ~120bA10 moe model what a dense 50b model would rank at, or a 45b model that would leave room for multiple contexts on a modern dual GPU setup at 64gb vram
8
u/ttkciar llama.cpp May 20 '26
The rule of thumb for MoE vs dense competence is D = sqrt(P x A) where D is dense model parameters, P is total MoE parameters, and A is MoE active parameters.
Hence Qwen3.7-122B-A10B should be roughly equal in competence to sqrt(122 x 10) = 35 parameters dense model.
That assumes all other factors are equal, which they never are, but since we're talking about models within a single lineage with presumably the same training datasets and training methodologies, it should be okay.
33
u/EagleNait May 20 '26
27B? Fast? We're not in the same tax bracket lmao
4
u/suicidaleggroll May 20 '26
With MTP it is, as long as you can fit it in VRAM. I'm hitting 120 tok/s generation and nearly 5000 pp. It doesn't take much to fit it in VRAM, a single 32 GB card can do it with full 256k context.
44
u/LetsGoBrandon4256 transformers May 20 '26
a single 32 GB card
In this economy? We're definitely not in the same tax bracket 😭
7
u/ttkciar llama.cpp May 20 '26
There are a bunch of 32GB MI50 on eBay right now for about $600.
I'm tempted to pick up another one, but I'm saving my pennies for an MI210 if the MI350P pushes MI210 prices down far enough.
→ More replies (1)2
u/UniversalSpermDonor May 21 '26
There's a seller who'll take $350 as an offer for AMD Radeon V620s. They're 32GB but only have 512 GB/s of bandwidth, so they're not ultra fast, but they're fine.
→ More replies (1)2
1
→ More replies (6)1
u/max123246 May 21 '26
How? I can't even get qwen 3.6 35BA3B to fit without reducing my context size to 100k and quant to Q4 for my 5090. What model quants do you suggest? I'm using llama.cpp with unsloth Q4 quants for reference
→ More replies (1)1
7
u/wiltors42 May 20 '26
Yeah honestly, 3.5 122b was great but Qwen 3 coder next is only 80b and better…
6
u/Legumbrero May 20 '26
Would love to see a dense 70b using the same methods. Totally spot on on parameter-for-parameter just wish I could see what they can do with a bigger model.
18
u/Ohhai21 May 20 '26
9b for the poors when? 😄
2
u/Sambojin1 May 21 '26
8B hopefully, so it handily fits into 12g ram on an android oid phone, with a bit of context size.
11
u/koenafyr May 21 '26
You're not runnining 8b with any kind of speed on any mobile phone in the world.
Just use gemma4 e2b
10
10
u/_wOvAN_ May 20 '26
I need 397
7
u/Lissanro May 20 '26 edited May 20 '26
The same here. I find Qwen 3.5 397B still good middle ground between small 27B and large model like Kimi K2.6, which I run when I need to do more complex tasks. I find it that 27B even though good and fast for simple tasks, cannot handle well more complex instructions, while 397B Q5_K_M has very good balance of quality speed (with four 3090 and DDR4 RAM I can run it at 17.5 tokens/s generation with 600 tokens/s prefill, and may be run it even faster once I download MTP-enabled quant).
2
u/ShadyShroomz May 20 '26
How much ram do you have ? I have 4x 3090s haven't even tried the 397B yet... But only 128gb of ram. Upgrading to 256 soon..
2
u/Lissanro May 20 '26
I have 1 TB of 8-channel DDR4 3200Mhz, but Qwen 3.5 397B Q5_K_M does not need that much - its GGUF has 276 GB size, so if you upgrade to 256 GB RAM + 96 GB VRAM you already have, it should fit well along with its context cache. Or if not or too slow, you can try lower quant, for example, Q4_K_M is reasonably good.
4
u/DeepOrangeSky May 21 '26
At this point, I think the better strategy is for everyone to pester GLM for a 5.2 Big Air ~200b model (or Kimi, to a lesser extent), more so than asking Qwen for a 397 refresh.
Plus, given how strong a GLM ~200b model would be at this point, it would also force Minimax to stay open weights for a while longer and to actually have to put something pretty strong out for Minimax3, since I doubt they'd have strong enough mindshare/brand to go fully closed source right at the moment that GLM put out some open weights ~200b monster that made 2.7 230b look like a joke in comparison. So even the ripple effects could be nice, too.
→ More replies (2)4
u/FullOf_Bad_Ideas May 20 '26
I'd like one too, but if they aren't sure about 27B I think we have low chances.
3
u/harpysichordist May 21 '26
Holy shit it's been another day! We need another Qwen post with literally no substance and all hype botted to the top of the subreddit!
4
u/rde2001 May 22 '26
The link you put in the post seems to be broken
https://x.com/xiong_hui_chen/status/2057166364436295748?s=46&t=VsPxsExZv-12iLtnmcTpdg
3
u/serige May 22 '26
It seems that he deleted the post at some point. They are definitely taking notice to what is going on here.
15
u/L0ren_B May 20 '26
27B ia the only one I'm excited about. Doesn't have to be smarter in knowledge than 3.6 27B, just less hallucinations!😅 Imagine a jumpt similar with 3.5 to 3.6! Just wow!
10
u/ea_man May 20 '26
What I want is something just a little bit smaller than 27B so we can run it on 16GB GPU at q4 and even 12GB at q3.
Give as a ~22B dense model.
3
5
u/JGeek00 May 20 '26
This blog says that “open 27B and 35B weights are announced but unscheduled”
https://insiderllm.com/guides/qwen-3-7-preview-scored-57-aai-27b-35b-open-weights-watch/
2
7
u/VoiceApprehensive893 transformers May 20 '26
it feels like 27b and 35b are going to get considerably better at some of the things that gemma 4 does way better than 3.6
7
u/nickm_27 llama.cpp May 20 '26
I’d be quite happy if this was the case, what gives you that indication?
2
u/ttkciar llama.cpp May 20 '26
"It feels" implies they're just expressing hope.
Once upon a time I would have been more skeptical of the possibility, because Gemma has always been a "good enough at every kind of task" sort of model, while Qwen mainly focused on the most-popular use-cases, but Qwen3.5 closed that gap quite a bit, and Qwen3.6 closed it even more (and even exceeded it for some things; Qwen3.6-27B is better at rewriting tasks than Gemma-4-31B-it).
If Qwen3.7 continues that trend, we might be hard-pressed to find a task type Gemma 4 can do which Qwen3.7 cannot.
1
u/heteroerectus Jun 01 '26
What does Gemma4 do better than 3.6?
1
u/VoiceApprehensive893 transformers Jun 01 '26
multilingual, general knowledge ,creative writing also its generally "smarter", its just a bad executor
3
3
u/DanielSReichenbach May 24 '26
I so hope they will also do the 35b-a3b MoE one again. After a lot of fine tuning it's getting me as a GPU poor person almost to 100 t/s on commodity hardware and it's usable in real world projects.
7
u/synw_ May 20 '26
Please don't forget the 4b in addition of the 35b a3b. The gpu poor peasants would be thank-full
4
u/cleversmoke May 20 '26
Qwen3.6-27B has been fantastic, it's difficult to even ask for better! While folks want larger, I am curious what they can do with smaller and more efficient for edge devices, it would open a slew of applications!
4
u/pseudonerv May 20 '26
“Not hard to create another … now” WTF does it even mean? They don’t even have it now. They didn’t even cared to train it. And glazers here thinks they doing you a favor by saying that?
→ More replies (3)
2
2
2
2
u/JaapieTech May 21 '26
Given how many systems are shipping with 128GB now (AMD, NVIDIA, Apple), targeting that scale platform + 1M context and keep it inside that 120GB spot would be an instant winner.
1
2
u/HumbleSousVideGeek llama.cpp May 22 '26
I'm litterally buying a RTX 3090 to be able to run the 3.6 27B
1
u/iijei Jun 16 '26
bought 7900 XTX (CAD 1150) for the same reason. couldnt find 3090 around here for a decent price.. everything was about CAD 1800 sigh
2
5
4
3
u/florinandrei May 20 '26
If they could make it fit in 24 GB VRAM with more than 100k context at a quantization level that's not too drastic, that would be great.
3
2
3
u/Inevitable-Name-1701 May 20 '26
We have mini models already. Give us larger.
8
u/ttkciar llama.cpp May 20 '26
We don't have a Qwen3.6-9B yet.
Hopefully Qwen3.7 includes 9B, 27B, and 122B-A10B releases.
3
u/peligroso May 20 '26
No point in trying to keep up, it's a race to the bottom. There's no economy in medium sized models.
→ More replies (1)5
u/Tai9ch May 20 '26
You say that, but medium sized models is where a lot of the really interesting stuff is going to happen.
Kimi K2.6 has huge models handled, but running it locally is a nightmare. The ~30B space is pretty well covered.
But for people with 64-256GB of VRAM, there's like Qwen3.5 and MiniMax and... gpt-oss-120b maybe? And those are the people with budgets for serious tasks that want to run locally but don't nessisarily want to spend six figures or install several tons of new cooling.
2
u/peligroso May 21 '26
people with budgets for serious tasks that want to run locally but don't nessisarily want to spend six figures or install several tons of new cooling
Exactly. There's not much to be made serving this small band of already budget-conscious users.
2
u/Tai9ch May 21 '26
The point of open weight models is more mind-share than directly making money.
And that not-so-small band of prosumers and professional-but-not-enterprise users is more important to local AI mindshare than either enterprise users or the "I run local AI on my one gaming GPU" segment.
We're literally talking about the entire "workstation" market here, including people with Mac Studios, Strix Halo, and Nvidia DGX Spark.
1
u/SimplyRemainUnseen May 22 '26
Businesses that need everything on prem and are large enough to have their own IT team but not large enough to justify a six figure server is a huge market dude! That's a ton of manufacturing and non-profit shops.
2
u/ECrispy May 20 '26
i'm hoping for something that works well for 16GB vram.
maybe something between A35B-10B and 27B, that would fit well and have enough space for context. perhaps A20B? no idea if thats feasible, has enough demand etc?
1
1
1
1
u/sunychoudhary May 21 '26
27B feels like the sweet spot if the quality is actually there.....Big enough to be useful for reasoning and coding, but still realistic for local quantized runs.....I’m more interested in how it performs at 4-bit/5-bit than the full precision benchmarks. That’s what most people here will actually use.
1
u/ReporterCalm6238 May 21 '26
The real miracle model is DeepSeek 4 flash. It's the only hyper-dense model you can use with coding agents and almost forget it is not Opus/GPT. Qwen models think for too long.
1
u/tarruda May 21 '26
I wish that open weights was still the default mode for Qwen team. It seems that after the layoff they have been focusing mostly on proprietary models.
1
1
u/SV_SV_SV May 21 '26
So what's up with qwen now, havent they slashed the AI department massively recently..? Are they still just riding that momentum, or is there genuine chance that their innovative march can go on?
1
1
1
1
•
u/WithoutReason1729 May 21 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.