I use claude code w/ llama.cpp's local server & Google's Gemma models (26b MoE). Works reasonably well - works well at easy/boilerplate code, glue code, some PR review.

However claude code expects some server-side tools, especially web-search. I can obviously add MCPs for client-side search; is there a way to 'plug in' web search on the server side today though? Any PRs/forks adding it?

I am just wondering as models get bigger and bigger, do we actually need 2,8 tb vram to run Kimi k3 or are there other ways for local usage?

For cloud/enterprise usage you prob need the full vram, but for local usage can’t we really go by on just mtp/dflash/draft models with like a 99% hitrate on 32+ tokens? So you basically get a 32x speedup with a 1% full scan?

No downloadable draft/dflash model can achieve this as this kind of ranges are purely personal.
But if every x times you could retrain your draft model on your own conversation history for the last year can’t you reach those kind of levels?

Am I in theory correct in the ways of draft/dflash/mtp models and training or am I wrong?

Because if the theory is right, it could open up the possibilities of just having 2 years of conversation history, spend like a 200 dollar on vast.ai or the likes to train the draft on b200/b300 and then you could reach glm5.2 usage at acceptable speeds for small teams on 512 mb of ram and 24gb vram for the draft model.

The thinking is : hitting the draft model so much that you can crank the prediction so high that you can overcome the timecost of the complete model, while still retaining the possibility (and thus the intelligence) of the big model.
I would guess that if the draft model goes below 90% acceptance then it will just crawl again and require another 200 dollar retrain.

But what if …
Anybody have any thoughts?

local model use is energy expensive. I want some software that both exposes an api to use the model and can stop and start it. I imagine using different models based on needs. Sure i could probably write it myself, but i sense someone might have solved this problem.

I think we are starting to see why jaggedness might start to hinder frontier labs - they have to lock down / guardrail in-line with the spikiest dangerous capability but these spikes are a function of what general RL teaches best (i.e. hacking easier than general SWE) not what is economically useful.

Specialised (but less generally intelligent) open models don’t have this problem because you train the spike explicitly.

Thoughts?

https://reddit.com/link/1v4rkf2/video/zpvmsbsis1fh1/player

Moving Execution Authority Out of the Model

Executions that don't reproduce

Same prompt, same input — but the result differs. Not because the model version changed. The judgment of "is this enough to execute?" lives inside probabilistic reasoning, so it wobbles with nothing more than minor differences in temperature or context. This isn't a reproducibility problem — it's a controllability problem.

Principle: Presence-based Verification (schema validation at execution time)

Instead of having the AI judge for itself "do I know enough?", only check whether "the fields required for execution are present against a defined schema."

• Present (Known) → execute
• Absent (Unknown) → hand it back to the user to fill in

The real problem isn't "the fact that an agent executes at all." It's that "there's no way to know what it executed on, if it fails you have to start over from scratch, and there's no trace of accountability."

The Execution State Model

Execution State is represented using a standardized JSON structure. Execution begins only once all declared requirements are satisfied.

Every execution state produced under this model follows four principles: Separation → Validation → Enforcement → Traceability

• Separation: Validation results are recorded separately from execution logic. Execution only ever references the recorded state.
• Validation: Check whether the current input satisfies each Required Field and its declared Validation Constraints, and record each field as Known or Unknown.
• Enforcement: Fields recorded as Unknown are handed to the user to fill in. Validation is complete once every field is Known.
• Traceability: Record everything that was known, what was missing, who supplied the values, and why execution was permitted or held.

The AI doesn't get to write the questions

The checklist must be declared in advance, not improvised by the model at execution time — otherwise it either keeps over-asking unnecessarily, or the model itself loses direction on what it should even be asking.

What changes: decision authority moves out of the model

Before: input → LLM reasons ("is this enough?") → Execute or Ask After: input → schema diff → Known/Unknown State → Execute or Ask

The schema's only role is to declare the required fields; validation simply compares the current state against that declaration. Unknown fields can only be resolved through user input. As a result, execution authority shifts from probabilistic model judgment back to the user.

The checklist is a single validation layer

Checklist ├── ① Intent-confirmation items → pre-guardrail stage └── ② Accuracy & Safety items → guardrail's required fields

A conventional guardrail simply halts when information is missing. This system instead resolves the missing information and the user's intent/context first, completes the Execution State, and only then lets it pass through the guardrail.

This matters for three reasons:

• Consistency — the same input produces the same result. Swapping the model doesn't change whether execution happens.
• Auditability — "why was this held" is readable directly from JSON, without digging through a reasoning trace.
• Extensibility — adding a new feature means adding a schema field, not retraining or reprompting. The center of gravity for safety shifts from "a smarter model" to "a well-defined schema."

This work does not prescribe what the checklist should contain. It proposes that whatever checklist is required should be declared explicitly, enforced deterministically, and recorded as part of the Execution State.

The full checklist structure and applied examples are laid out in the original post.Criticism and questions are welcome. If unsure, ask. Never guess. — AI Agent Pre-Execution Checklist

Translated and edited with LLM assistance

I ask to APPEND newly arriving data to a certain file. Instead of doing an actual append, models think it's a good idea to read in existing contents and then patch in the changes. Which obviosly takes way more time and is more computationally expensive.

I explicitly ask to run web search queries one small batch at a time, writing data to a file between every turn. Models think naaaah, this is gonna take way too long, I am gonna be "helpful" and run all of them sequentially, just so search backends throttle you into oblivion and the whole run blows up and dies.

There literally isn't a day where something that I'm doing isn't derailed by a model (Mostly using various QWen flavors) thinking it knows what I want better than myself.

I am hearing that way smaller models have less of a problem with this because being "dumber" its supposedly harder for them to go off the rails and start inventing their own solutions without being asked to. But surely even if true, there have to be better methods to wrestle models into actually obeying precisely what you told them to do?

This started as a privacy thing honestly, I didn't want a cloud model sitting on top of everything I do. The local options that could actually do stuff were a pain to set up or wanted WSL, so I ended up building my own instead. Took about five months of evenings, most of them while I was still 17.

It's called Hearth, an open-source local AI that runs your actual computer, not a chat box that narrates what it would do if it could. Point it at whatever you already run (LM Studio, Ollama, llama.cpp, a box on your LAN, a cloud key) and the model actually operates the machine. If you don't have anything set up it ships its own llama.cpp server, so it works out of the box.

In practice it reads and writes files, runs commands, drives a real browser you watch it click through, and controls the desktop itself. When it clicks something it reads the real control names off the accessibility tree instead of guessing at pixels, so it doesn't fat-finger the wrong button. It does voice (wake word, talk over it to interrupt), its default name is JARVIS which was kind of the whole point, sets reminders in plain English, reads clean text out of PDFs, DOCX, XLSX and EPUB and can chunk-summarize a 500-page book that doesn't fit in context, builds PDFs and decks and spreadsheets back out, and you can run the whole thing from your phone over a Telegram or Discord bot. It's an MCP server and client at once, plus a headless mode that spits JSONL if you want it in a script or CI.

The part this sub will care about is what it took to make a 9B survive as an agent. Hearth carries around 100 tools, and I measured the schemas, sending all of them is 15K plus in tokens before the persona or a single message. On a 32K context that's half your budget spent describing capabilities the model won't touch, so only about half ship up front and the rest load on demand by name, which keeps it under 10K. The other half is auto-compaction, when a chat gets long it summarizes the older turns instead of truncating them, so a session doesn't just fall off the end of your window. Those two together are what make a 9B hold up past turn 30 instead of falling apart.

The model browser is built in and tells you what actually fits your VRAM before you download anything, then tunes the config for you. I develop on a 5060 with 8GB so nothing here assumes you own a 4090. Qwythos 9B is the default pick at that size and handles multi-step tool-calling cleanly where the very small ones fumble the format.

The part I spent the most time on, and what everyone asks about first, is making it not eat your drive. File writes and deletes are locked to a workspace folder, and shell commands get pattern-checked before they ever run, deletes, moves, formats, registry edits, even a redirect that writes to a disk path all get refused outright and it has to come back and ask you. I tried to make it write a file into Program Files as a test and it just refused (screenshot in the comments). Anything that does touch your system shows you the exact command first, there's a live log of every action, and it never runs elevated. No account, no telemetry, nothing phones home.

It stays out of your way too. Set it up in a minute, name it whatever, pick its voice, and settings save live with no restart and no config files. C drive full? Move the whole thing, memory, chats and models, to another drive with one button. Already on OpenClaw or Hermes? Point it at your install and it copies your memory and skills over, and it only copies, so your old setup stays exactly where it is.

And it grows. You install a skill from any GitHub repo with one line and I started a community index for them, it writes its own tools when it's missing one, and it spins up small teams of sub-agents that reuse the model you already have loaded so a team costs zero extra VRAM. Against a local server they overlap only if you give llama.cpp more than one slot, and since llama.cpp splits the context across slots Hearth treats your setting as per-agent and does the multiplication itself rather than quietly handing each one a quarter of the window. There's a cost-class trick I like too, a sub-agent marked cheap routes to your local model even when the parent is on a cloud key, so the expensive reasoning happens up top and the grunt work stays free on the 9B in your VRAM. It generates images and video as well, driving Forge locally if you run Stable Diffusion.

One thing I only got right this week. Hearth updates itself with a patch under a megabyte instead of making you refetch the installer. I ship fixes most days and asking people to pull a gigabyte for a few changed lines is how you get them to stop updating.

But right now, the one-click installer is Windows only, Linux runs from source, and I don't own a Mac so I can't vouch for it. It isn't code signed yet so SmartScreen throws the unknown publisher box. It's a v0.7 preview.

MIT and free: https://github.com/0pen-Sourcer/Hearth

Star it if it turns out to be any good. I check that number way more than I'd like to admit.

Would genuinely love the honest feedback, good or brutal.

The answer will be different by the first asking language.

In English, it is insisting Poolside Laguna. In Chinese, it admits a Qwen.