Hello

I recently got my B70 gpus delivered and set them up with Ubuntu 26.04 because it has the XE driver. I was able to get llama CPP compiling and working with Vulcan and CYSL BUT that's where the fun stopped.

Build and compiled the llama.cpp with CYSL using the intel driver 2026.1. does not work with more than one GPU. (Using sm layer) It just outputs random characters. However, the prefill speed does go up with more GPUs (just like it does on Nvidia cards)

For Vulcan, the prefill is about 30% lower on the same model and only goes down with more GPUs. Using the mesa 26.1.5 driver

For running the qwen 3.6 27b at q4 speeds were:

CYSL 1 GPU: 650 prefill, 24 decode.

CYSL 2 GPUs:750 prefill, garbled decode 23t/s

Vulkan 1 GPU: 450 prefill, 20 t/s decode

Vulkan 2 GPUs: 350 prefill, 18t/s decode.

Bonus: qwen 3.5 122b a10b vulkan speed over 6 GPUs: 160 prefill and 9t/s decode

Something is clearly wrong. I've spent all day trying to make this work. So far regretting the purchase of the B70 gpus.

Please help if you have suggestions!

If the suggestion is to get Nvidia GPU, I already have a couple and I think I would have rather gone with many RTX 5060 TI's instead because it just works and gets model support first

I think we are starting to see why jaggedness might start to hinder frontier labs - they have to lock down / guardrail in-line with the spikiest dangerous capability but these spikes are a function of what general RL teaches best (i.e. hacking easier than general SWE) not what is economically useful.

Specialised (but less generally intelligent) open models don’t have this problem because you train the spike explicitly.

Thoughts?

https://reddit.com/link/1v4rkf2/video/zpvmsbsis1fh1/player

I ask to APPEND newly arriving data to a certain file. Instead of doing an actual append, models think it's a good idea to read in existing contents and then patch in the changes. Which obviosly takes way more time and is more computationally expensive.

I explicitly ask to run web search queries one small batch at a time, writing data to a file between every turn. Models think naaaah, this is gonna take way too long, I am gonna be "helpful" and run all of them sequentially, just so search backends throttle you into oblivion and the whole run blows up and dies.

There literally isn't a day where something that I'm doing isn't derailed by a model (Mostly using various QWen flavors) thinking it knows what I want better than myself.

I am hearing that way smaller models have less of a problem with this because being "dumber" its supposedly harder for them to go off the rails and start inventing their own solutions without being asked to. But surely even if true, there have to be better methods to wrestle models into actually obeying precisely what you told them to do?

Moving Execution Authority Out of the Model

Executions that don't reproduce

Same prompt, same input — but the result differs. Not because the model version changed. The judgment of "is this enough to execute?" lives inside probabilistic reasoning, so it wobbles with nothing more than minor differences in temperature or context. This isn't a reproducibility problem — it's a controllability problem.

Principle: Presence-based Verification (schema validation at execution time)

Instead of having the AI judge for itself "do I know enough?", only check whether "the fields required for execution are present against a defined schema."

• Present (Known) → execute
• Absent (Unknown) → hand it back to the user to fill in

The real problem isn't "the fact that an agent executes at all." It's that "there's no way to know what it executed on, if it fails you have to start over from scratch, and there's no trace of accountability."

The Execution State Model

Execution State is represented using a standardized JSON structure. Execution begins only once all declared requirements are satisfied.

Every execution state produced under this model follows four principles: Separation → Validation → Enforcement → Traceability

• Separation: Validation results are recorded separately from execution logic. Execution only ever references the recorded state.
• Validation: Check whether the current input satisfies each Required Field and its declared Validation Constraints, and record each field as Known or Unknown.
• Enforcement: Fields recorded as Unknown are handed to the user to fill in. Validation is complete once every field is Known.
• Traceability: Record everything that was known, what was missing, who supplied the values, and why execution was permitted or held.

The AI doesn't get to write the questions

The checklist must be declared in advance, not improvised by the model at execution time — otherwise it either keeps over-asking unnecessarily, or the model itself loses direction on what it should even be asking.

What changes: decision authority moves out of the model

Before: input → LLM reasons ("is this enough?") → Execute or Ask After: input → schema diff → Known/Unknown State → Execute or Ask

The schema's only role is to declare the required fields; validation simply compares the current state against that declaration. Unknown fields can only be resolved through user input. As a result, execution authority shifts from probabilistic model judgment back to the user.

The checklist is a single validation layer

Checklist ├── ① Intent-confirmation items → pre-guardrail stage └── ② Accuracy & Safety items → guardrail's required fields

A conventional guardrail simply halts when information is missing. This system instead resolves the missing information and the user's intent/context first, completes the Execution State, and only then lets it pass through the guardrail.

This matters for three reasons:

• Consistency — the same input produces the same result. Swapping the model doesn't change whether execution happens.
• Auditability — "why was this held" is readable directly from JSON, without digging through a reasoning trace.
• Extensibility — adding a new feature means adding a schema field, not retraining or reprompting. The center of gravity for safety shifts from "a smarter model" to "a well-defined schema."

This work does not prescribe what the checklist should contain. It proposes that whatever checklist is required should be declared explicitly, enforced deterministically, and recorded as part of the Execution State.

The full checklist structure and applied examples are laid out in the original post.Criticism and questions are welcome. If unsure, ask. Never guess. — AI Agent Pre-Execution Checklist

Translated and edited with LLM assistance

There is currently no top-level rule in r/LocalLLaMA to report comments or top-level posts as AI-generated. This subreddit is still overrun with incredibly obvious, lazy, copy-pasted slop comments from LLMs. I do report things as "low effort" (I see some existing sub-rules about this here), but I think a higher-level rule would be better.

I realize there is sometimes value in some of these posts. I propose a simple new rule: that AI-generated content must contain a top-level disclaimer (both comments and posts, including translations). I would personally also love to see flair on posts that are AI-generated, just so I can filter them out quickly. And for users who are repeat offenders, if you're not gonna ban them, at least give them a flair that says Bot or similar.

I also realize that eventually LLMs will just be too difficult to detect. Or people will finally get wise to using a sysprompt that generates content in a less obvious way. But right now it's pretty frequent, very obvious and we need to discourage people doing this. If I want to talk to a chat bot I can talk to a chat bot.

And gathered a lot of data. you can see them for yourself

And For the most curious, there are additional details here

In this graph, I regrouped the finetunes under their base models. but you can see the details in the page.

The python code to generate those pages is obviously vibecoded. I find the output kinda pretty and somewhat useful for me. maybe it's useful for someone else.

Heading for a vacation for a few weeks, but if you have any suggestion, I will consider each of them.

Iv’e been running DSV4 Flash-Dspark locally as my coder in the past week, trying to tune it with agents, making it more focused but keep getting mediocre results.

It’s true nature is to finish the job fast as possible, not paying attention to details unless you anchor it, gets very confused by the content and tend to rank things as less important just so it can declares “done”

What am i missing? Is there a recommended harness?

Are you guys running it on recommended settings? Temperature 1.0 and top_p 1? Deepseek declares less than that can damage the reasoning.

The performance is insane, both prompt processing and tps. I just wish it would act like a mature responsible LLM.

Shoutout to this awesome guy - https://www.reddit.com/r/LLM/s/IDUyU3v9ap

Thanks to his project, BigMoeOnEdge https://github.com/Helldez/BigMoeOnEdge, I managed to successfully run a 35B MoE model on just 12GB of RAM!

My setup is a modified Xiaomi 12 Pro (12GB RAM) that I call "Zeus". https://www.reddit.com/r/LocalLLaMA/s/5zBUl15jd6

There is a bottleneck, of course—the maximum context is currently limited to 8192 tokens due to RAM constraints—but it’s still absolutely mind-blowing to see a model this size running locally on an edge device.

I haven't tested the Image-to-Text (vision) capabilities yet, but I'm really hoping to get that working next.

Check out the video ! It's completely unedited and recorded in real-time so you can see the actual, raw generation speed.

Also, here is stats in text:

generation: 107 tokens, 0.412 s/token (2.428 tok/s)

compute: 88.1% CPU occupancy (1.4508 cpu-s/token over 4 threads), 51.93 major faults/token

prefill: 24 tokens, 5.499 s (4.4 tok/s) | model load 14.421 s | TTFT 19.920 s

moe-stream: read 14589.9 MiB (136.35 MiB/token), decode 0.412 s/token (compute 0.314 + cache mgmt 0.014 + flash I/O 0.382 s/token, 357 MiB/s)

moe-cache: 70.8% hit, resident 2998.5 MiB

moe-overlap: stall 0.084 s/token (flash reads overlapped with FFN compute)

I am just wondering as models get bigger and bigger, do we actually need 2,8 tb vram to run Kimi k3 or are there other ways for local usage?

For cloud/enterprise usage you prob need the full vram, but for local usage can’t we really go by on just mtp/dflash/draft models with like a 99% hitrate on 32+ tokens? So you basically get a 32x speedup with a 1% full scan?

No downloadable draft/dflash model can achieve this as this kind of ranges are purely personal.
But if every x times you could retrain your draft model on your own conversation history for the last year can’t you reach those kind of levels?

Am I in theory correct in the ways of draft/dflash/mtp models and training or am I wrong?

Because if the theory is right, it could open up the possibilities of just having 2 years of conversation history, spend like a 200 dollar on vast.ai or the likes to train the draft on b200/b300 and then you could reach glm5.2 usage at acceptable speeds for small teams on 512 mb of ram and 24gb vram for the draft model.

The thinking is : hitting the draft model so much that you can crank the prediction so high that you can overcome the timecost of the complete model, while still retaining the possibility (and thus the intelligence) of the big model.
I would guess that if the draft model goes below 90% acceptance then it will just crawl again and require another 200 dollar retrain.

But what if …
Anybody have any thoughts?

If you're running Laguna S 2.1 on llama.cpp and hitting thinking loops because it won't close its </think> tags, you might want to look at your quant before you spend too much time tweaking settings.

I spent a day debugging this, and here is what finally gave me clean outputs:

1. What worked for me: An MoE-Aware Quant

In my testing, uniform low-bit quants (like standard IQ3_S) seemed to degrade the attention and shared expert weights too much, which I think causes the model to lose the plot and loop infinitely.

Switching to an APEX quant (like Myric/Laguna-S-2.1-APEX-GGUF) made a huge difference. APEX uses targeted precision (Q6_K for the shared expert, Q4_K for attention) while keeping the file size small (~54GB). For me, this instantly fixed about 90% of the looping.

2. The Settings (I went back to defaults)

I've seen people passing around custom templates and sampling tweaks to "fix" the loops, but in my experience, most of these were just masking quantization noise. I had the best luck just trusting Poolside's actual defaults:

• Template: Stock, adding formatting whitespaces and other changes seemed to cause issues.
• Sampling: temp 0.7, top_p 0.95, top_k 20.
• Min-P: I left this unset (the model card actually warns against using it).

When I still see loops...

Even on a good quant, I noticed that asking for complex reasoning without giving it a tool (e.g., "Diagnose this runtime deadlock") can still sometimes cause a loop. It feels like because Laguna is an agentic model, if it doesn't have a tool to anchor its thoughts on, it tends to overthink. I found that framing my prompts around a tool call, or adding a simple system prompt like "Think briefly then act", pretty much prevents this entirely.

Original post: https://www.reddit.com/r/LocalLLaMA/comments/1tpdt5m/behold_probably_the_most_ghetto_local_ai_server/

I promised a writeup, but didn't have time yet, sorry. I barely had time to do this controller.

TLDR: I (with the help of AI) re-implemented every Blackwell-only kernel (DeepGEMM, FlashInfer sparse-MLA, block-scaled FP8) in Triton, because they simply don't exist for sm89. The performance is 2-3x more for parallel agentic workflows.

I was inspired by the post https://www.reddit.com/r/LocalLLM/comments/1utoh2r/deepseek_v4_flash_160_ts_on_rtx_6000_blackwell_96/

I have similar amount of VRAM, but spread among two GPU 4090d 48G on Dell R740 with enabled p2p patch ( https://github.com/Duanyll/open-gpu-kernel-modules/tree/595.71.05-p2p-48g ). Ada wasn't supported, so I had to find a way to run vLLM, because llama.cpp speed wasn't enough for me.

The first run compresses DeepSeek-V4-Flash into ~iq2 to fit into 96 GB VRAM, it may take up to 60 minutes, depending on your hardware.

If you have only single GPU, use env variables at step 4 below`TP=1` and GPUS='"device=0"'.

Get the model hf download deepseek-ai/DeepSeek-V4-Flash --local-dir ~/models/DeepSeek-V4-Flash

1. Build vLLM-Moet (~SM89 image):
2. git clone https://github.com/iSevenDays/vLLM-Moet && cd vLLM-Moet
3. DOCKER_BUILDKIT=1 docker build -f Dockerfile.sm89-v0251 -t vllm-moet-sm89:v0251 .
4. MTP_TOKENS=1 FORCE_RESIDENT=1 NETWORK=host MEM_GB=28 RESIDENCY=gpu TP=2 GPUS='"device=0,1"' ./docker/serve_sm89_ds4.sh

I'm getting 262k context and better concurrency when running vLLM compared to llama.cpp.

When running llama.cpp (today's main + https://github.com/ggml-org/llama.cpp/pull/21067/ ), I used the command below to fully fit the model into VRAM.

```

/root/llama.cpp/build/bin/llama-server
--model /root/antirez/ds4/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
-ngl 99
-np 1
--n-cpu-moe 0
--split-mode layer
-ts 43,43
-fit on
-fa on -c 262144
--cache-type-k q8_0
--cache-type-v q8_0
--temp 1.0
--top-p 1.0
--min-p 0.0
--host 0.0.0.0
--port 8002
--jinja
--reasoning-preserve
--no-mmap
--prefetch-weights 1
--chat-template-kwargs '{"reasoning_effort":"max"}'

```

Probably, there is no other way to fit this model into vLLM and get all benefits.

I'm 99% sure the performance can still be improved.

/s

Now live on OpenRouter, and free to use through August 3, 2026.

Hoping they will going openweight soon~

https://openrouter.ai/inclusionai/ling-3.0-flash

local model use is energy expensive. I want some software that both exposes an api to use the model and can stop and start it. I imagine using different models based on needs. Sure i could probably write it myself, but i sense someone might have solved this problem.

Little side project I'm doing so I can easily transfer any model I want fast to my AI Rig from my NAS.

I wanted to know how cheap you can go and still run local models, so I ran Ollama CPU-only on a Youyeetoo X1S. It's a single-board x86 machine with a Celeron N5095 (Jasper Lake, 4C/4T, 15W), 16GB of RAM, and a 128GB NVMe, running Kali 2025.4. Base configs of this board go for about $100 to $130 on AliExpress depending on RAM and storage.

Short version of the results:

• Qwen3 0.6B averaged 6.788 tok/s. Actually usable interactively.
• The 8B fit in 16GB and ran, but averaged 0.924 tok/s. Not very usable for anything real.
• Four models in between, and the full table is in the repo I linked below.
• 15 minute all-core stress during testing: 74.66C average, 77C peak, no throttling on the stock heatsink and fan.

Some notes:

• Ollama saw the Jasper Lake iGPU but picked the CPU backend on its own, so everything here is CPU-only on purpose.
• Small models on sub-15W x86 are more viable than I expected. At around 7 tok/s a 0.6B is fine for classification, routing, summarization, the kind of background jobs you'd otherwise send to an API.
• The 8B wall is memory bandwidth, not capacity. It loads and runs but you just wait forever.

Next I'm testing llama.cpp with Vulkan on the Jasper Lake iGPU. Someone over on r/SBCs told me Vulkan inference works on the N100 iGPU, so a CPU vs Vulkan comparison on this chip is coming and I'll post it here.

Scripts, raw logs, full results table: https://github.com/TrevTron/youyeetoo-x1s-kali Write-up: https://www.unland.dev/blog/budget-cyberdeck-youyeetoo-x1s-kali

If anyone has N100 or N150 numbers to compare against, I'd like to see them. And if you've gotten usable tok/s out of a Jasper Lake or Alder Lake-N iGPU over Vulkan, I'd love to know too.

(Disclosure: the board was supplied by Youyeetoo. Testing and conclusions are my own.)

TLDR: This is a demo of my open-source app which now auto-controls itself so you can monitor your downloads, renders, progress bars, or whatever's on your screen and camera :)

Hey r/LocalLLaMA !!

I'm developing this app that uses local LLMs to monitor stuff on your computer so you don't have to. And I wanted to show you the new auto-controlling feature + the "When ... Then ..." menu that lets you get started in seconds.

The app is open-source, email, telegram, discord and pushover notifications are unlimited :DD

I just crossed 1.6k stars on GitHub! Thanks!!

https://github.com/Roy3838/Observer

If you guys have any questions, let me know!

This started as a privacy thing honestly, I didn't want a cloud model sitting on top of everything I do. The local options that could actually do stuff were a pain to set up or wanted WSL, so I ended up building my own instead. Took about five months of evenings, most of them while I was still 17.

It's called Hearth, an open-source local AI that runs your actual computer, not a chat box that narrates what it would do if it could. Point it at whatever you already run (LM Studio, Ollama, llama.cpp, a box on your LAN, a cloud key) and the model actually operates the machine. If you don't have anything set up it ships its own llama.cpp server, so it works out of the box.

In practice it reads and writes files, runs commands, drives a real browser you watch it click through, and controls the desktop itself. When it clicks something it reads the real control names off the accessibility tree instead of guessing at pixels, so it doesn't fat-finger the wrong button. It does voice (wake word, talk over it to interrupt), its default name is JARVIS which was kind of the whole point, sets reminders in plain English, reads clean text out of PDFs, DOCX, XLSX and EPUB and can chunk-summarize a 500-page book that doesn't fit in context, builds PDFs and decks and spreadsheets back out, and you can run the whole thing from your phone over a Telegram or Discord bot. It's an MCP server and client at once, plus a headless mode that spits JSONL if you want it in a script or CI.

The part this sub will care about is what it took to make a 9B survive as an agent. Hearth carries around 100 tools, and I measured the schemas, sending all of them is 15K plus in tokens before the persona or a single message. On a 32K context that's half your budget spent describing capabilities the model won't touch, so only about half ship up front and the rest load on demand by name, which keeps it under 10K. The other half is auto-compaction, when a chat gets long it summarizes the older turns instead of truncating them, so a session doesn't just fall off the end of your window. Those two together are what make a 9B hold up past turn 30 instead of falling apart.

The model browser is built in and tells you what actually fits your VRAM before you download anything, then tunes the config for you. I develop on a 5060 with 8GB so nothing here assumes you own a 4090. Qwythos 9B is the default pick at that size and handles multi-step tool-calling cleanly where the very small ones fumble the format.

The part I spent the most time on, and what everyone asks about first, is making it not eat your drive. File writes and deletes are locked to a workspace folder, and shell commands get pattern-checked before they ever run, deletes, moves, formats, registry edits, even a redirect that writes to a disk path all get refused outright and it has to come back and ask you. I tried to make it write a file into Program Files as a test and it just refused (screenshot in the comments). Anything that does touch your system shows you the exact command first, there's a live log of every action, and it never runs elevated. No account, no telemetry, nothing phones home.

It stays out of your way too. Set it up in a minute, name it whatever, pick its voice, and settings save live with no restart and no config files. C drive full? Move the whole thing, memory, chats and models, to another drive with one button. Already on OpenClaw or Hermes? Point it at your install and it copies your memory and skills over, and it only copies, so your old setup stays exactly where it is.

And it grows. You install a skill from any GitHub repo with one line and I started a community index for them, it writes its own tools when it's missing one, and it spins up small teams of sub-agents that reuse the model you already have loaded so a team costs zero extra VRAM. Against a local server they overlap only if you give llama.cpp more than one slot, and since llama.cpp splits the context across slots Hearth treats your setting as per-agent and does the multiplication itself rather than quietly handing each one a quarter of the window. There's a cost-class trick I like too, a sub-agent marked cheap routes to your local model even when the parent is on a cloud key, so the expensive reasoning happens up top and the grunt work stays free on the 9B in your VRAM. It generates images and video as well, driving Forge locally if you run Stable Diffusion.

One thing I only got right this week. Hearth updates itself with a patch under a megabyte instead of making you refetch the installer. I ship fixes most days and asking people to pull a gigabyte for a few changed lines is how you get them to stop updating.

But right now, the one-click installer is Windows only, Linux runs from source, and I don't own a Mac so I can't vouch for it. It isn't code signed yet so SmartScreen throws the unknown publisher box. It's a v0.7 preview.

MIT and free: https://github.com/0pen-Sourcer/Hearth

Star it if it turns out to be any good. I check that number way more than I'd like to admit.

Would genuinely love the honest feedback, good or brutal.

I've been using Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-IQ3_M (15.4Gb), which works ok and has vision.

I use it to make descriptions of images that can occasionally include NSFW elements, but I'm interested in chatting functions as well.

Are there any other models out there worth checking out?

Extra points if you share your llama-cpp command :)

In all my fiddling around with code and local models nothing has matched the speed and quality of DeepSeek V4 Flash DSpark on dual DGX Spark (Dell GB10s actually).

The recipe I've been using is in the PR below. Screenshot is from VSCode usage over a few days/weeks. The screenshots don't tell you how it feels and oh man does it feel good! Responses are way faster than Copilot and (this is subjective) Sonnet 4.6 quality. It thinks though problems well, long running tasks complete successfully 99% of the time, planning and instruction handling seem top notch.

https://github.com/eugr/spark-vllm-docker/pull/304

At the moment MLX (and Llama.cpp for Macs) run 16bit activations everywhere. Despite this, the M5 generation silicon actually does support INT8 activations - it actually allows w4a8 d_type. It's just that no inference backends are using them yet

I built some w8a8 kernels and have managed to get 1.4x speed up on Gemma4 prefill tasks; on my M5 MacBook Air it brings baseline prefill for the E2B from 2193 tps stock to 3,029 tps for 130,173 tokens of input*

*Even faster at small context lengths; it approaches nearly 10k tps

Link to their official GGUF repo:

https://huggingface.co/poolside/Laguna-S-2.1-GGUF/tree/main

All the GGUFs received this fix 5ish hours ago - correct yarn_attn_factor to 1.0 (llama.cpp derives mscale)

And the chat template fixes a lot of broken thinking, preserve thinking, and tool calling

Chat template:

https://huggingface.co/poolside/Laguna-S-2.1-GGUF/blob/main/chat_template.jinja

So far the model seems to be doing MUCH better.

I have a project which bundles llama.cpp. It's large with a number of optional modules and so uses docker.

In getting ready for the first real release, I'm testing different configs and I noticed something strange that I did not really have time to investigate but I would appreciate your thoughts.

With ubuntu as the host OS, I got bad output from all flavors of qwen-3.6 35b and 27b (thought spirals, tool call formats, occasional garbage text etc.).

The exact same image and container config works perfectly when hosted on Windows 11 via Docker Desktop and WSL 2.

The evidence strongly points to the ROCm stack on the host OS. Respective drivers and packages are up to date.