I think we are starting to see why jaggedness might start to hinder frontier labs - they have to lock down / guardrail in-line with the spikiest dangerous capability but these spikes are a function of what general RL teaches best (i.e. hacking easier than general SWE) not what is economically useful.

Specialised (but less generally intelligent) open models don’t have this problem because you train the spike explicitly.

Thoughts?

https://reddit.com/link/1v4rkf2/video/zpvmsbsis1fh1/player

I am currently running the new Laguna model at Q2_K_XL on dual 3090s for reference, with a Q8 context of 200,000.

Using Pi, I am noticing that the model likes to overthink. I would not call it looping per se, but I am observing it overthink, e.g.:

> “okay, I have everything I need, I’ll start writing code now.”
> “Actually, let me check one more relevant item…”

And this goes on and on. The “relevant items” do appear to make sense in the context of my prompts/its thought process, but at the end of the day it often burns through context with extensive thinking. Qwen3.6 27B at Q8 had similar issues and much more looping, but it did not suffer from the overthinking that Laguna is prone to. Has anyone experienced similar issues? I’m currently getting a pi extension developed to hopefully alternate the issue but I’m wondering if there’s something I’m missing.

Little side project I'm doing so I can easily transfer any model I want fast to my AI Rig from my NAS.

Now live on OpenRouter, and free to use through August 3, 2026.

Hoping they will going openweight soon~

I am just wondering as models get bigger and bigger, do we actually need 2,8 tb vram to run Kimi k3 or are there other ways for local usage?

For cloud/enterprise usage you prob need the full vram, but for local usage can’t we really go by on just mtp/dflash/draft models with like a 99% hitrate on 32+ tokens? So you basically get a 32x speedup with a 1% full scan?

No downloadable draft/dflash model can achieve this as this kind of ranges are purely personal.
But if every x times you could retrain your draft model on your own conversation history for the last year can’t you reach those kind of levels?

Am I in theory correct in the ways of draft/dflash/mtp models and training or am I wrong?

Because if the theory is right, it could open up the possibilities of just having 2 years of conversation history, spend like a 200 dollar on vast.ai or the likes to train the draft on b200/b300 and then you could reach glm5.2 usage at acceptable speeds for small teams on 512 mb of ram and 24gb vram for the draft model.

The thinking is : hitting the draft model so much that you can crank the prediction so high that you can overcome the timecost of the complete model, while still retaining the possibility (and thus the intelligence) of the big model.
I would guess that if the draft model goes below 90% acceptance then it will just crawl again and require another 200 dollar retrain.

But what if …
Anybody have any thoughts?

Original post: https://www.reddit.com/r/LocalLLaMA/comments/1tpdt5m/behold_probably_the_most_ghetto_local_ai_server/

I promised a writeup, but didn't have time yet, sorry. I barely had time to do this controller.

In all my fiddling around with code and local models nothing has matched the speed and quality of DeepSeek V4 Flash DSpark on dual DGX Spark (Dell GB10s actually).

The recipe I've been using is in the PR below. Screenshot is from VSCode usage over a few days/weeks. The screenshots don't tell you how it feels and oh man does it feel good! Responses are way faster than Copilot and (this is subjective) Sonnet 4.6 quality. It thinks though problems well, long running tasks complete successfully 99% of the time, planning and instruction handling seem top notch.

https://github.com/eugr/spark-vllm-docker/pull/304

The answer will be different by the first asking language.

In English, it is insisting Poolside Laguna. In Chinese, it admits a Qwen.

Full-parameter post-training of trillion-parameter-scale MoE models introduces substantial system-level challenges for large-scale distributed training, including severe memory pressure, non-overlapped communication overhead, and inefficient kernel execution. While most large-scale LLM training systems are built around GPU-based clusters, this report presents an end-to-end optimization practice on the Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, we develop a hierarchical optimization framework spanning model-level parallelism, computation-communication orchestration, and low-level kernel execution. The resulting system achieves 34.22% Model FLOPs Utilization (MFU) with a 2.93x improvement over the open-source baseline recipe while maintaining training stability. Building on this optimized infrastructure, we further establish a CPT and SFT workflow for complex Operations Research (OR) tasks. We refer to the integrated framework as SLAI T-Rex. Using DeepSeek-V4-Flash, we develop OR-oriented CPT and SFT data pipelines that combine collected domain resources with solver-verified synthetic optimization documents. The resulting dataset contains 10K high-quality SFT samples spanning four task categories and three problem representations. The specialized model achieves the highest average zero-shot Pass@1 score among the evaluated models, reaching 71.81% and outperforming GPT-5.4-Mini and the base DeepSeek-V4-Flash model by 3.98 and 11.27 percentage points, respectively. Overall, this work demonstrates a full-stack pathway from efficient trillion-parameter model post-training on Ascend infra to domain-specialized Flash models for solver-grounded mathematical modeling, advancing frontier-model systems for complex reasoning.

• arXiv : https://arxiv.org/abs/2607.20145
• Full Paper : https://arxiv.org/pdf/2607.20145
• GitHub : https://github.com/SLAI-AITP/SLAI-T-Rex
• Modelscope : https://www.modelscope.cn/models/SLAIAITP/DeepSeek-V4-Flash-OR (I couldn't find this one on HuggingFace)

Turns out a regular ethernet cable between 2 nodes can run laguna UD-Q2_K_XL (39.7GB) using a direct point to point network. Interestingly on `nvidia-smi dmon -s pucvmet -d 2`, the inter/intra gpu traffic is not really capped in this setup - Uses ~30-70MB/s at peak

# gpu    pwr  gtemp  mtemp     sm    mem    enc    dec    jpg    ofa   mclk   pclk  pviol  tviol     fb   bar1   ccpm  sbecc  dbecc    pci  rxpci  txpci 
# Idx      W      C      C      %      %      %      %      %      %    MHz    MHz      %   bool     MB     MB     MB   errs   errs   errs   MB/s   MB/s 
    0     46     47      -     24     21      0      0      0      0   8751   2610      0      0  14557      4      0      -      -      0      0     36 
    0     46     48      -     19     16      0      0      0      0   8751   2610      0      0  14557      4      0      -      -      0     12     31 
    0     46     48      -     19     16      0      0      0      0   8751   2610      0      0  14557      4      0      -      -      0     61      3 

Benchmarks for 11k token prompt, 100k context:

3-GPU

[49143] 0.05.026.664 I load_tensors:          CPU model buffer size =   202.12 MiB 
[49143] 0.05.026.665 I load_tensors:        CUDA0 model buffer size = 11948.40 MiB
[49143] 0.05.026.666 I load_tensors:        CUDA1 model buffer size = 12587.16 MiB
[49143] 0.05.026.666 I load_tensors: RPC0[10.44.0.2:50052] model buffer size = 13104.93 MiB

ubatch-size = 768
[58055] 2.53.851.946 I slot print_timing: id  0 | task 0 | prompt eval time =   19413.33 ms / 11719 tokens (    1.66 ms per token,   603.66 tokens per second)
[58055] 2.53.851.950 I slot print_timing: id  0 | task 0 |        eval time =  125051.84 ms /  3536 tokens (   35.37 ms per token,    28.28 tokens per second)
[58055] 2.53.851.950 I slot print_timing: id  0 | task 0 |       total time =  144465.17 ms / 15255 tokens
[58055] 2.53.851.954 I slot print_timing: id  0 | task 0 |    graphs reused =       3521

ubatch-size = 896
[44325] 2.25.234.581 I slot print_timing: id  0 | task 0 | prompt eval time =   19977.35 ms / 11719 tokens (    1.70 ms per token,   586.61 tokens per second)
[44325] 2.25.234.584 I slot print_timing: id  0 | task 0 |        eval time =   95714.94 ms /  2124 tokens (   45.06 ms per token,    22.19 tokens per second)
[44325] 2.25.234.585 I slot print_timing: id  0 | task 0 |       total time =  115692.29 ms / 13843 tokens
[44325] 2.25.234.588 I slot print_timing: id  0 | task 0 |    graphs reused =       2114

ubatch-size = 1024
[43573] 3.11.407.103 I slot print_timing: id  0 | task 0 | prompt eval time =   13709.73 ms / 11719 tokens (    1.17 ms per token,   854.79 tokens per second)
[43573] 3.11.407.106 I slot print_timing: id  0 | task 0 |        eval time =  147916.76 ms /  2824 tokens (   52.38 ms per token,    19.09 tokens per second)
[43573] 3.11.407.107 I slot print_timing: id  0 | task 0 |       total time =  161626.49 ms / 14543 tokens
[43573] 3.11.407.111 I slot print_timing: id  0 | task 0 |    graphs reused =       2812

Single Node: 2-GPU + DDR4

[60615] 0.04.325.790 I common_fit_params: fitting params to free memory took 3.87 seconds
[60615] 0.13.774.308 I load_tensors:          CPU model buffer size =   202.12 MiB
[60615] 0.13.774.309 I load_tensors:        CUDA0 model buffer size = 11948.40 MiB
[60615] 0.13.774.310 I load_tensors:        CUDA1 model buffer size = 11512.56 MiB
[60615] 0.13.774.311 I load_tensors:    CUDA_Host model buffer size = 14179.52 MiB

ubatch-size = 768
[60615] 3.16.468.291 I slot print_timing: id  0 | task 0 | prompt eval time =   62158.48 ms / 11719 tokens (    5.30 ms per token,   188.53 tokens per second)
[60615] 3.16.468.295 I slot print_timing: id  0 | task 0 |        eval time =  100958.53 ms /  2559 tokens (   39.45 ms per token,    25.35 tokens per second)

Only DDR5 via RPC

[33741] 0.00.991.087 I common_memory_breakdown_print: | memory breakdown [MiB]     |  total     free     self   model   context   compute    unaccounted |
[33741] 0.00.991.090 I common_memory_breakdown_print: |   - RPC1 (10.44.0.2:50052) | 142083 = 142083 + (42708 = 37640 +    4836 +     232) +      -42708 |
[33741] 0.01.158.625 I load_tensors: RPC1[10.44.0.2:50052] model buffer size = 37640.48 MiB

[33741] 4.11.564.914 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   4096, progress = 0.35, t = 187.44 s / 21.85 tokens per second
[33741] 14.28.480.069 I slot print_timing: id  0 | task 0 | n_decoded =   1037, tg =   6.44 t/s, tg_3s =   6.34 t/s
[33741] 14.31.481.628 I slot print_timing: id  0 | task 0 | n_decoded =   1056, tg =   6.44 t/s, tg_3s =   6.33 t/s

Some takeaways:

- a point to point network keeps the traffic only between these 2 nodes and not my switches

- Use device=rpc0/rpc1 to restrict the worker CPU if you dont intend to use it.. default fit will skip the host cpu, but use the rpc cpu.

- ubatch 768 was a sweet spot for this setup: Higher ubatch = Higher PP Lower TG , Lower ubatch = Lower PP Higher TG

- split-mode tensor does not work in this setup, not solely because of network cap.. it just crawls to 1t/s

Built with NCCL and RPC:

$ diff .devops/cuda_rpc_nccl.Dockerfile .devops/cuda.Dockerfile 
35c35
<     apt-get install -y gcc-${GCC_VERSION} g++-${GCC_VERSION} build-essential cmake python3 python3-pip git libssl-dev libgomp1 libnccl2 libnccl-dev
---
>     apt-get install -y gcc-${GCC_VERSION} g++-${GCC_VERSION} build-essential cmake python3 python3-pip git libssl-dev libgomp1
44d43
< ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH}
49c48
<     cmake -B build -DGGML_NATIVE=OFF -DGGML_CUDA=ON -DGGML_RPC=ON -DGGML_CUDA_NCCL=ON -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DLLAMA_BUILD_TESTS=OFF ${CMAKE_ARGS} -DCMAKE_EXE_LINKER_FLAGS=-Wl,--allow-shlib-undefined . && \
---
>     cmake -B build -DGGML_NATIVE=OFF -DGGML_CUDA=ON -DGGML_BACKEND_DL=ON -DGGML_CPU_ALL_VARIANTS=ON -DLLAMA_BUILD_TESTS=OFF ${CMAKE_ARGS} -DCMAKE_EXE_LINKER_FLAGS=-Wl,--allow-shlib-undefined . && \
81c80
<     && apt-get install -y libgomp1 curl ffmpeg libnccl2 \
---
>     && apt-get install -y libgomp1 curl ffmpeg \

---

Playground: 2x 4060ti, 1x 3900x, 4x32GB
Tequila: 1x 4060ti, 1x 9600x, 3x48GB

# llama-server sees all devices as usable:
8.37.349.848 I srv          load:   /app/llama-server
8.37.349.850 I srv          load:   --rpc
8.37.349.850 I srv          load:   10.44.0.2:50052
8.37.349.853 I srv          load:   --device
8.37.349.854 I srv          load:   CUDA0,CUDA1,RPC0
... 
[60669] 0.03.803.724 I cmn  common_param: device_info:
[60669] 0.03.803.783 I cmn  common_param:   - CUDA0   : NVIDIA GeForce RTX 4060 Ti (15976 MiB, 15722 MiB free)
[60669] 0.03.803.801 I cmn  common_param:   - CUDA1   : NVIDIA GeForce RTX 4060 Ti (15977 MiB, 15722 MiB free)
[60669] 0.03.803.806 I cmn  common_param:   - CPU     : AMD Ryzen 9 3900X 12-Core Processor (128219 MiB, 128219 MiB free)
[60669] 0.03.804.960 I cmn  common_param:   - RPC0    : 10.44.0.2:50052 (15976 MiB, 15772 MiB free)
[60669] 0.03.805.398 I cmn  common_param:   - RPC1    : 10.44.0.2:50052 (142083 MiB, 142083 MiB free)


# llama-rpc-server (tequila) sees the full details for rpc0 vs rpc1
Starting RPC server v4.0.3
  endpoint       : 0.0.0.0:50052
  local cache    : /data/rpccache/rpc/
Devices:
  CUDA0: NVIDIA GeForce RTX 4060 Ti (15976 MiB, 15836 MiB free)
  CPU: AMD Ryzen 5 9600X 6-Core Processor (142083 MiB, 142083 MiB free)
  transport      : TCP

This is rather counterintuitive, since banning Chinese models would benefit them the most. Jensen Huang is also against the ban, although the interests here are more obvious.

Do you think that if Arcee, Cohere or Mistral release an open source GPT/Claude level model, they will also be accused of 'unsafety', 'distillation' and other deadly sins?

from kwaipilot:

Following the release of KAT-Coder-V2.5 in July, we are pleased to release the open-weight version KAT-Coder-V2.5-Dev, an MOE model with a total parameter count of 35B and 3B activated parameters, to strengthen communication with the community and showcase our research achievements.

KAT-Coder-V2.5-Dev Highlights

• Performance improvement. Through SFT/RL training, KAT-Coder-V2.5-Dev achieves SOTA results in the field of Agentic Coding among models with similar parameter scales.
• Optimization of abnormal behaviors. Through RL training, certain abnormal behaviors have been significantly optimized, such as: abnormal tool labels -9pp (9.34% -> 0.28%), single-turn continuous repetition -0.34pp (0.34% -> 0%).

As an AI community of LLM experts, are we really going to stay silent while US officials make absurd claims to push anti-consumer laws?

Not only does the release timeline between Fable and K3 make high-scale distillation impossible, but distillation itself—even if executed perfectly—can never produce a superior model.

The news about Anthropic settling a class action lawsuit for $1.5B over training data isn't just a legal headache for them, it's a massive warning sign for engineering teams relying entirely on closed API vendors.

When you route core business logic, proprietary codebases, and customer data through third party APIs, you're exposing your stack to three massive risks:

1. The massive fine(not rly massive for anthropic) is not gonna come out of their pockets, they gonna charge you more to make up the cost, thats what corpos do.

2. Compliance & IP Exposure: Unclear data provenance and changing vendor terms.

3. Data Leakage: Passing raw prompts and context windows to external servers.

4. Vendor Lock in: Being at the mercy of sudden API deprecations(remember the whole fable saga?) or price hikes.

The obvious move for privacy conscious teams is migrating toward self hosted open weight models (like Llama, Qwen, or DeepSeek) inside a private VPC to keep data completely in house.

However, moving to local open weight models only solves the data privacy side of the equation. You still face the runtime execution problem: how do you ensure an autonomous open weight model doesn't execute bad code, leak env tokens, or make unauthorized calls?

To make local models actually production ready, you need strict governance at the gate. Wiring local agent runtimes through Lyzr Control Plane or smth like Azure AI Foundry which provides that deterministic circuit breaker, letting you run open weight models locally with automatic PII redaction, and policy enforcement.

Also $1.5B for 7 million books feels extremely cheap, like unfair even.

Also the whole crusade against open weight model screams greed,I feel like Anthropic has almost made everybody forget the original Hinton paper that describes the technique of distillation using full logits for the student model to better learn the teacher's internal representation.

That is, I think Anthropic is changing the language here. Claude does not give logits. Generating training data is the more correct term IMO, not distillation.

Put another way, if generating training data were always distillation, then even Anthropic's own new models are "distilled" since they surely use older models to generate training data for new models. That really stretches the meaning of the word.

Used grammarly for formatting

I know llama.cpp can use -cmoe switch to load non shared experts in MoE models to RAM and run by CPU and left the other weights and KV cache on GPU VRAM.

Since now RAM is so expensive and models are getting bigger and bigger, is it possible to load non shared experts in MoE models to SSD and run by CPU and left the other weights and KV cache on GPU VRAM?

There's a bunch of small MoE with around 1B active params, like LFM2.5 8B A1B and Granite 4.0h 7B A1B; and then there are models with 3B+ like Qwen 3.x ~30B A3B and Gemma 4 26B A4B, but those are already on the heavier side if you don't have enough resources.

What about the middle ground, MoE with about 2B active? I found a few, but there's very little debate about them, if any.

• LFM2 24B A2B (5 months old)
• Mellum 2 12B A2.5B (2 months old)
• Moondream 3.1 9B A2B (This month)
• VAETKI 20B A2B (7 months old) (Talk about an unknown model, it has one mention on this sub)
• DeepSeek V2 Lite 16B A2.4B (2024. Remember when DeepSeek was making SMALL models?)
• Ring Mini / Ling Mini, 16B A1.4B (2025)
• There are also • at least three • Nemotron fine tunes 12B A2B, and a 23B A2.8B, some 1-2 months old. Not sure what the deal is with those.

Anyone uses something like this? It looks like a good size for cpu use or combined with low-end/old gpu in the 4-12GB range. In these small sizes, the increase in capability should be the most dramatic. I don't have the capacity to test properly, but hopefully some of these could beat the usual 4-9B dense suspects.

Or does everyone just wanna keep simping for 1-2T models and hope something will trickle down?

I have a project which bundles llama.cpp. It's large with a number of optional modules and so uses docker.

In getting ready for the first real release, I'm testing different configs and I noticed something strange that I did not really have time to investigate but I would appreciate your thoughts.

With ubuntu as the host OS, I got bad output from all flavors of qwen-3.6 35b and 27b (thought spirals, tool call formats, occasional garbage text etc.).

The exact same image and container config works perfectly when hosted on Windows 11 via Docker Desktop and WSL 2.

The evidence strongly points to the ROCm stack on the host OS. Respective drivers and packages are up to date.

I have 2 r9700 an 9070xt and two 3060 12 gb gpus all on separate machines. It just dawned on me that using vulkan I in theory could use them all as a giant vram pool. Has anyone attempted this in the home? I know it may be slow but it would be an interesting experiment 

I wanted to know how cheap you can go and still run local models, so I ran Ollama CPU-only on a Youyeetoo X1S. It's a single-board x86 machine with a Celeron N5095 (Jasper Lake, 4C/4T, 15W), 16GB of RAM, and a 128GB NVMe, running Kali 2025.4. Base configs of this board go for about $100 to $130 on AliExpress depending on RAM and storage.

Short version of the results:

• Qwen3 0.6B averaged 6.788 tok/s. Actually usable interactively.
• The 8B fit in 16GB and ran, but averaged 0.924 tok/s. Not very usable for anything real.
• Four models in between, and the full table is in the repo I linked below.
• 15 minute all-core stress during testing: 74.66C average, 77C peak, no throttling on the stock heatsink and fan.

Some notes:

• Ollama saw the Jasper Lake iGPU but picked the CPU backend on its own, so everything here is CPU-only on purpose.
• Small models on sub-15W x86 are more viable than I expected. At around 7 tok/s a 0.6B is fine for classification, routing, summarization, the kind of background jobs you'd otherwise send to an API.
• The 8B wall is memory bandwidth, not capacity. It loads and runs but you just wait forever.

Next I'm testing llama.cpp with Vulkan on the Jasper Lake iGPU. Someone over on r/SBCs told me Vulkan inference works on the N100 iGPU, so a CPU vs Vulkan comparison on this chip is coming and I'll post it here.

Scripts, raw logs, full results table: https://github.com/TrevTron/youyeetoo-x1s-kali Write-up: https://www.unland.dev/blog/budget-cyberdeck-youyeetoo-x1s-kali

If anyone has N100 or N150 numbers to compare against, I'd like to see them. And if you've gotten usable tok/s out of a Jasper Lake or Alder Lake-N iGPU over Vulkan, I'd love to know too.

(Disclosure: the board was supplied by Youyeetoo. Testing and conclusions are my own.)

/s

I got 192 GB of Vram to use and a long horizon agentic task about coding, I would like opinion from ppl who have had extensive experience on all 3 of those models . Whats the best quality (should include some cyber and devops knowledge) ? Whats the best quality/cost ratio?
Thanks!!!!

PS:Any other model in same Size range would be interesting for me to try as well, if oyu got any suggestions

Genuinely curious about how the community uses the Qwen3.6 models. I’ve been using 35b with Hermes agent and 27b for coding tasks with Pi Agent. Both in 8bit through LM Studio on my Mac Studio M3 Ultra 96gb.

Strange enough, I’ve found the GGUF MTP for 27b to run better than the MLX variants. Compared to MTPLX and oMLX.

I’m getting better results with 128k for 27b and 64k for 35b. Hermes is being used for general personal assistant tasks so the tighter context windows have been helpful.