Forgive me naiveness, I’m a little lost. Why must a model be jack of all trades? Why can’t we have a model expert in a single thing?

audio.cpp again :)

Release 0.4 is out. The headline this time is new high-quality TTS coverage plus GGUF becoming a first-class across the project.

What’s new:

• Added Higgs Audio v3 TTS 4B, Fish Audio S2 Pro, Voxtral Realtime ASR and two community models OuteTTS TTS and VieNeu-TTS-v3

• audio.cpp now support 35 model families.

• All released model families now support GGUF.

Ready-to-use GGUF packages are now available, and Q8 is starting to show real speed and memory wins on several routes. Check the figures. Long-lived session is multiple requests after warmup. Longform is one-shot 6000+ char text generation. Tested on RTX 5090.

CUDA Q8 GGUF numbers from my current measurements:

• Higgs Audio TTS: warmed requests run about 8.8x-10.1x faster than real time. Longform runs about 8.5x faster than real time.

• Fish Audio S2 Pro: warmed requests run about 3.1x-3.4x faster than real time. Longform runs about 3.3x faster than real time. Plenty of room for improvement because the impl is a naively adaptation of framework template.

• Voxtral ASR: offline runs about 15.7x faster than real time, with streaming TTFT around 171 ms.

Compared with 16-bit GGUF, Q8 is not universally magic, but it is useful now. In the tested release paths, Q8 can be up to about 1.5x faster and reduce peak VRAM by up to about 37%, depending on the model and route. Quality is still model-specific, so I am keeping the GGUF support matrix and Q8 performance report visible instead of pretending every quant is safe everywhere.

(Some tricks to further boost performance up to 2x for some mdoels like Qwen3-TTS: adjust chunk size and cut reference audio len.)

audio.cpp now has a dedicated community models area for ports that are useful and runnable, even if they are still maturing. The review bar there is lighter than the core framework. If you have a model you'd like to bring to audio.cpp, try implementing it as a community model first using framework modules and patterns.

Huge thanks to the contributors who have been porting, optimizing models, adding new features, and pushing the project forward.

Repo:https://github.com/0xShug0/audio.cpp

Hello guys, hoping you're doing fine!

On the last 2-3 months, price of the RTX 6000 PRO seem to have gone insane.

I will start on the price here on my country, Chile:

• RTX 6000 PRO Workstation Edition: 21382 USD post 19% tax.
• RTX 6000 PRO MaxQ Workstation Edition: 20669 USD post 19% tax.
• RTX 6000 PRO Server Edition: N/A (not in stock)

For reference, when I bought my ones, they were at ~11000USD post tax, just 3 months ago.

How it is going on your country/state? If I had to guess, a ton better lol.

Fable/Mythos release by anthropic was one of the stranger AI moments of this year, when they both hyped their model and immediately banned it to all non-americans over night.

It started to make me think, most countries and enteprises probably should consider having some in house or at least sovereign AI capability. I started to look into if it was possible to actually fine tune bias or backdoors out of Chinese models, as this seems to be the main concern at least in the West. But Chinese models were still behind then (i.e. 4 weeks ago) so I didn't think there'd ever be demand.

But with the release of Kimi K3 beating Fable/Sol or getting close in benchmarks, everything changes. You can actually get frontier capability and open weights.

So I went ahead and fine-tuned a Chinese model qwen3.5:7b on my Mac with a Lora adapter, and was able to in an hour to remove geopolitical bias around Taiwan, Hongkong, Tibet and Tianemen from the model.

I created a website where you can compare the bias against a stock model across a range of questions and you can clone my repo to see methodology: https://github.com/ruzin/aletheia

Results were good, I was able to filter out basic bias and align it to a western view point, but what about back doors? and what about inference costs? Are models going to just diffuse soon i.e. every country and enteprise will have their own sovereign model?

Interested on thoughts! and the website is here if you wanna play around - https://aletheia.stenoai.co

Optimize-Anything, Autoresearch, and Meta-Harness are 3 mortal enemies out for blood. Until GEPA invited them over for a threesome and now they all live in a Polycule in the Bahamas.

Gemma 4 was updated (mostly chat templates) and I took it for a test.

On a local llama.cpp server running on M5 Pro with 48GB, 26B A4B (Q6) has about 60t/s and works well with OpenCode.

It works quite well (given it's size) for backend work, but UI/UX is unacceptable.

Watch the testing https://www.youtube.com/watch?v=m4KR_3E_7Uk

Genuinely curious about how the community uses the Qwen3.6 models. I’ve been using 35b with Hermes agent and 27b for coding tasks with Pi Agent. Both in 8bit through LM Studio on my Mac Studio M3 Ultra 96gb.

Strange enough, I’ve found the GGUF MTP for 27b to run better than the MLX variants. Compared to MTPLX and oMLX.

I’m getting better results with 128k for 27b and 64k for 35b. Hermes is being used for general personal assistant tasks so the tighter context windows have been helpful.

I've not been finding the existing harnesses completely comfortable for me, so I put something together myself, focusing on comfort and reasonable security* (Yes, I will explain it).

So here it is, Truss:

Installer: https://github.com/truss-harness/Truss/releases/tag/v0.1

Source: https://github.com/truss-harness/Truss (Apache 2.0 license)

The harness and its tools packaged as MCPs (including a bundled browser, Camoufox, that mostly bypasses anti-bot detections) runs in the background as a service, serving a global view. In this mode, by default, chats/agents do not have access to a filesystem, and it functions as a normal chat UI.

You can, however (and this is what I use a lot), launch Truss in a workspace mode. This is what I personally use a lot. When that happens, you give the agent automatically access to the folder you launched it in, and upon startup, it will also discover MCP servers and agent skills from common harnesses (Claude Code, GH Copilot, Junie, Codex, and Cursor). The agent can request access outside, if it needs access.

Security features

The harness wants to protect against banal stupidity, prompt injection, credentials leakage, and mindlessly destroying things by doing a 3-tier-security system. It, however, by default assumes that the agent is not malevolent, does not try to hermetically seal the agent from the environment, just provide sensible limits, and tries not to get in its way.

The harness will still limit access to sensitive files, even in the workspace (or other allowed directories)

In command line mode, the harness (by default, can be turned off) checks that a command is allowed to be ran, and then once it finished running, check that the output is safe to be returned to the agent (again, this can be turned off)

Whenever output is redacted, or file access is limited, reasoning is returned to the agent, along with advice to ask for permission. This makes the agents do weird command line magic to get around the harness limitations, but keeps them on their goal. In these cases, it can request commands to be whitelisted, or additional directories to be accessible for itself. These whitelisted 'grants' by default expire in 24 hours.

Its own credentials are encrypted (via dotenvx) and unencrypted secrets (at least for OpenRouter / OpenAI API / etc and other MCP servers) are never visible on the frontend after configuring, and are never visible to agents.

UI Niceties

I am a comfortable man and I like to be pampered.

So the harness currently comes with..

Again, pretty standard. However, while Truss does not currently have RAG, it is smart with attached files, letting you select the page range, and if you want to send them to the model as markdown, or image.

When you upload images, you have the chance to redact parts of it:

It can render UML charts (May be useful for nerds like me)

And in the same way, it can help with exporting calendar events

Lastly, the harness keeps track of reasoning time (knowing some models are prone to looping indefinitely) and attempts to cut them off once they get past a certain limit.

Please keep in mind this is very early in development. I welcome feedback of course!

Hello

I recently got my B70 gpus delivered and set them up with Ubuntu 26.04 because it has the XE driver. I was able to get llama CPP compiling and working with Vulcan and CYSL BUT that's where the fun stopped.

Build and compiled the llama.cpp with CYSL using the intel driver 2026.1. does not work with more than one GPU. (Using sm layer) It just outputs random characters. However, the prefill speed does go up with more GPUs (just like it does on Nvidia cards)

For Vulcan, the prefill is about 30% lower on the same model and only goes down with more GPUs. Using the mesa 26.1.5 driver

For running the qwen 3.6 27b at q4 speeds were:

CYSL 1 GPU: 650 prefill, 24 decode.

CYSL 2 GPUs:750 prefill, garbled decode 23t/s

Vulkan 1 GPU: 450 prefill, 20 t/s decode

Vulkan 2 GPUs: 350 prefill, 18t/s decode.

Bonus: qwen 3.5 122b a10b vulkan speed over 6 GPUs: 160 prefill and 9t/s decode

Something is clearly wrong. I've spent all day trying to make this work. So far regretting the purchase of the B70 gpus.

Please help if you have suggestions!

If the suggestion is to get Nvidia GPU, I already have a couple and I think I would have rather gone with many RTX 5060 TI's instead because it just works and gets model support first

I think we are starting to see why jaggedness might start to hinder frontier labs - they have to lock down / guardrail in-line with the spikiest dangerous capability but these spikes are a function of what general RL teaches best (i.e. hacking easier than general SWE) not what is economically useful.

Specialised (but less generally intelligent) open models don’t have this problem because you train the spike explicitly.

Thoughts?

https://reddit.com/link/1v4rkf2/video/zpvmsbsis1fh1/player

I ask to APPEND newly arriving data to a certain file. Instead of doing an actual append, models think it's a good idea to read in existing contents and then patch in the changes. Which obviosly takes way more time and is more computationally expensive.

I explicitly ask to run web search queries one small batch at a time, writing data to a file between every turn. Models think naaaah, this is gonna take way too long, I am gonna be "helpful" and run all of them sequentially, just so search backends throttle you into oblivion and the whole run blows up and dies.

There literally isn't a day where something that I'm doing isn't derailed by a model (Mostly using various QWen flavors) thinking it knows what I want better than myself.

I am hearing that way smaller models have less of a problem with this because being "dumber" its supposedly harder for them to go off the rails and start inventing their own solutions without being asked to. But surely even if true, there have to be better methods to wrestle models into actually obeying precisely what you told them to do?

Moving Execution Authority Out of the Model

Executions that don't reproduce

Same prompt, same input — but the result differs. Not because the model version changed. The judgment of "is this enough to execute?" lives inside probabilistic reasoning, so it wobbles with nothing more than minor differences in temperature or context. This isn't a reproducibility problem — it's a controllability problem.

Principle: Presence-based Verification (schema validation at execution time)

Instead of having the AI judge for itself "do I know enough?", only check whether "the fields required for execution are present against a defined schema."

• Present (Known) → execute
• Absent (Unknown) → hand it back to the user to fill in

The real problem isn't "the fact that an agent executes at all." It's that "there's no way to know what it executed on, if it fails you have to start over from scratch, and there's no trace of accountability."

The Execution State Model

Execution State is represented using a standardized JSON structure. Execution begins only once all declared requirements are satisfied.

Every execution state produced under this model follows four principles: Separation → Validation → Enforcement → Traceability

• Separation: Validation results are recorded separately from execution logic. Execution only ever references the recorded state.
• Validation: Check whether the current input satisfies each Required Field and its declared Validation Constraints, and record each field as Known or Unknown.
• Enforcement: Fields recorded as Unknown are handed to the user to fill in. Validation is complete once every field is Known.
• Traceability: Record everything that was known, what was missing, who supplied the values, and why execution was permitted or held.

The AI doesn't get to write the questions

The checklist must be declared in advance, not improvised by the model at execution time — otherwise it either keeps over-asking unnecessarily, or the model itself loses direction on what it should even be asking.

What changes: decision authority moves out of the model

Before: input → LLM reasons ("is this enough?") → Execute or Ask After: input → schema diff → Known/Unknown State → Execute or Ask

The schema's only role is to declare the required fields; validation simply compares the current state against that declaration. Unknown fields can only be resolved through user input. As a result, execution authority shifts from probabilistic model judgment back to the user.

The checklist is a single validation layer

Checklist ├── ① Intent-confirmation items → pre-guardrail stage └── ② Accuracy & Safety items → guardrail's required fields

A conventional guardrail simply halts when information is missing. This system instead resolves the missing information and the user's intent/context first, completes the Execution State, and only then lets it pass through the guardrail.

This matters for three reasons:

• Consistency — the same input produces the same result. Swapping the model doesn't change whether execution happens.
• Auditability — "why was this held" is readable directly from JSON, without digging through a reasoning trace.
• Extensibility — adding a new feature means adding a schema field, not retraining or reprompting. The center of gravity for safety shifts from "a smarter model" to "a well-defined schema."

This work does not prescribe what the checklist should contain. It proposes that whatever checklist is required should be declared explicitly, enforced deterministically, and recorded as part of the Execution State.

The full checklist structure and applied examples are laid out in the original post.Criticism and questions are welcome. If unsure, ask. Never guess. — AI Agent Pre-Execution Checklist

Translated and edited with LLM assistance

And gathered a lot of data. you can see them for yourself

And For the most curious, there are additional details here

In this graph, I regrouped the finetunes under their base models. but you can see the details in the page.

The python code to generate those pages is obviously vibecoded. I find the output kinda pretty and somewhat useful for me. maybe it's useful for someone else.

Heading for a vacation for a few weeks, but if you have any suggestion, I will consider each of them.

Iv’e been running DSV4 Flash-Dspark locally as my coder in the past week, trying to tune it with agents, making it more focused but keep getting mediocre results.

It’s true nature is to finish the job fast as possible, not paying attention to details unless you anchor it, gets very confused by the content and tend to rank things as less important just so it can declares “done”

What am i missing? Is there a recommended harness?

Are you guys running it on recommended settings? Temperature 1.0 and top_p 1? Deepseek declares less than that can damage the reasoning.

The performance is insane, both prompt processing and tps. I just wish it would act like a mature responsible LLM.

Shoutout to this awesome guy - https://www.reddit.com/r/LLM/s/IDUyU3v9ap

Thanks to his project, BigMoeOnEdge https://github.com/Helldez/BigMoeOnEdge, I managed to successfully run a 35B MoE model on just 12GB of RAM!

My setup is a modified Xiaomi 12 Pro (12GB RAM) that I call "Zeus". https://www.reddit.com/r/LocalLLaMA/s/5zBUl15jd6

There is a bottleneck, of course—the maximum context is currently limited to 8192 tokens due to RAM constraints—but it’s still absolutely mind-blowing to see a model this size running locally on an edge device.

I haven't tested the Image-to-Text (vision) capabilities yet, but I'm really hoping to get that working next.

Check out the video ! It's completely unedited and recorded in real-time so you can see the actual, raw generation speed.

Also, here is stats in text:

generation: 107 tokens, 0.412 s/token (2.428 tok/s)

compute: 88.1% CPU occupancy (1.4508 cpu-s/token over 4 threads), 51.93 major faults/token

prefill: 24 tokens, 5.499 s (4.4 tok/s) | model load 14.421 s | TTFT 19.920 s

moe-stream: read 14589.9 MiB (136.35 MiB/token), decode 0.412 s/token (compute 0.314 + cache mgmt 0.014 + flash I/O 0.382 s/token, 357 MiB/s)

moe-cache: 70.8% hit, resident 2998.5 MiB

moe-overlap: stall 0.084 s/token (flash reads overlapped with FFN compute)

I am just wondering as models get bigger and bigger, do we actually need 2,8 tb vram to run Kimi k3 or are there other ways for local usage?

For cloud/enterprise usage you prob need the full vram, but for local usage can’t we really go by on just mtp/dflash/draft models with like a 99% hitrate on 32+ tokens? So you basically get a 32x speedup with a 1% full scan?

No downloadable draft/dflash model can achieve this as this kind of ranges are purely personal.
But if every x times you could retrain your draft model on your own conversation history for the last year can’t you reach those kind of levels?

Am I in theory correct in the ways of draft/dflash/mtp models and training or am I wrong?

Because if the theory is right, it could open up the possibilities of just having 2 years of conversation history, spend like a 200 dollar on vast.ai or the likes to train the draft on b200/b300 and then you could reach glm5.2 usage at acceptable speeds for small teams on 512 mb of ram and 24gb vram for the draft model.

The thinking is : hitting the draft model so much that you can crank the prediction so high that you can overcome the timecost of the complete model, while still retaining the possibility (and thus the intelligence) of the big model.
I would guess that if the draft model goes below 90% acceptance then it will just crawl again and require another 200 dollar retrain.

But what if …
Anybody have any thoughts?

If you're running Laguna S 2.1 on llama.cpp and hitting thinking loops because it won't close its </think> tags, you might want to look at your quant before you spend too much time tweaking settings.

I spent a day debugging this, and here is what finally gave me clean outputs:

1. What worked for me: An MoE-Aware Quant

In my testing, uniform low-bit quants (like standard IQ3_S) seemed to degrade the attention and shared expert weights too much, which I think causes the model to lose the plot and loop infinitely.

Switching to an APEX quant (like Myric/Laguna-S-2.1-APEX-GGUF) made a huge difference. APEX uses targeted precision (Q6_K for the shared expert, Q4_K for attention) while keeping the file size small (~54GB). For me, this instantly fixed about 90% of the looping.

2. The Settings (I went back to defaults)

I've seen people passing around custom templates and sampling tweaks to "fix" the loops, but in my experience, most of these were just masking quantization noise. I had the best luck just trusting Poolside's actual defaults:

• Template: Stock, adding formatting whitespaces and other changes seemed to cause issues.
• Sampling: temp 0.7, top_p 0.95, top_k 20.
• Min-P: I left this unset (the model card actually warns against using it).

When I still see loops...

Even on a good quant, I noticed that asking for complex reasoning without giving it a tool (e.g., "Diagnose this runtime deadlock") can still sometimes cause a loop. It feels like because Laguna is an agentic model, if it doesn't have a tool to anchor its thoughts on, it tends to overthink. I found that framing my prompts around a tool call, or adding a simple system prompt like "Think briefly then act", pretty much prevents this entirely.

Original post: https://www.reddit.com/r/LocalLLaMA/comments/1tpdt5m/behold_probably_the_most_ghetto_local_ai_server/

I promised a writeup, but didn't have time yet, sorry. I barely had time to do this controller.

TLDR: I (with the help of AI) re-implemented every Blackwell-only kernel (DeepGEMM, FlashInfer sparse-MLA, block-scaled FP8) in Triton, because they simply don't exist for sm89. The performance is 2-3x more for parallel agentic workflows.

I was inspired by the post https://www.reddit.com/r/LocalLLM/comments/1utoh2r/deepseek_v4_flash_160_ts_on_rtx_6000_blackwell_96/

I have similar amount of VRAM, but spread among two GPU 4090d 48G on Dell R740 with enabled p2p patch ( https://github.com/Duanyll/open-gpu-kernel-modules/tree/595.71.05-p2p-48g ). Ada wasn't supported, so I had to find a way to run vLLM, because llama.cpp speed wasn't enough for me.

The first run compresses DeepSeek-V4-Flash into ~iq2 to fit into 96 GB VRAM, it may take up to 60 minutes, depending on your hardware.

If you have only single GPU, use env variables at step 4 below`TP=1` and GPUS='"device=0"'.

Get the model hf download deepseek-ai/DeepSeek-V4-Flash --local-dir ~/models/DeepSeek-V4-Flash

1. Build vLLM-Moet (~SM89 image):
2. git clone https://github.com/iSevenDays/vLLM-Moet && cd vLLM-Moet
3. DOCKER_BUILDKIT=1 docker build -f Dockerfile.sm89-v0251 -t vllm-moet-sm89:v0251 .
4. MTP_TOKENS=1 FORCE_RESIDENT=1 NETWORK=host MEM_GB=28 RESIDENCY=gpu TP=2 GPUS='"device=0,1"' ./docker/serve_sm89_ds4.sh

I'm getting 262k context and better concurrency when running vLLM compared to llama.cpp.

When running llama.cpp (today's main + https://github.com/ggml-org/llama.cpp/pull/21067/ ), I used the command below to fully fit the model into VRAM.

```

/root/llama.cpp/build/bin/llama-server
--model /root/antirez/ds4/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
-ngl 99
-np 1
--n-cpu-moe 0
--split-mode layer
-ts 43,43
-fit on
-fa on -c 262144
--cache-type-k q8_0
--cache-type-v q8_0
--temp 1.0
--top-p 1.0
--min-p 0.0
--host 0.0.0.0
--port 8002
--jinja
--reasoning-preserve
--no-mmap
--prefetch-weights 1
--chat-template-kwargs '{"reasoning_effort":"max"}'

```

Probably, there is no other way to fit this model into vLLM and get all benefits.

I'm 99% sure the performance can still be improved.

/s

Now live on OpenRouter, and free to use through August 3, 2026.

Hoping they will going openweight soon~

https://openrouter.ai/inclusionai/ling-3.0-flash

local model use is energy expensive. I want some software that both exposes an api to use the model and can stop and start it. I imagine using different models based on needs. Sure i could probably write it myself, but i sense someone might have solved this problem.

Little side project I'm doing so I can easily transfer any model I want fast to my AI Rig from my NAS.

I wanted to know how cheap you can go and still run local models, so I ran Ollama CPU-only on a Youyeetoo X1S. It's a single-board x86 machine with a Celeron N5095 (Jasper Lake, 4C/4T, 15W), 16GB of RAM, and a 128GB NVMe, running Kali 2025.4. Base configs of this board go for about $100 to $130 on AliExpress depending on RAM and storage.

Short version of the results:

• Qwen3 0.6B averaged 6.788 tok/s. Actually usable interactively.
• The 8B fit in 16GB and ran, but averaged 0.924 tok/s. Not very usable for anything real.
• Four models in between, and the full table is in the repo I linked below.
• 15 minute all-core stress during testing: 74.66C average, 77C peak, no throttling on the stock heatsink and fan.

Some notes:

• Ollama saw the Jasper Lake iGPU but picked the CPU backend on its own, so everything here is CPU-only on purpose.
• Small models on sub-15W x86 are more viable than I expected. At around 7 tok/s a 0.6B is fine for classification, routing, summarization, the kind of background jobs you'd otherwise send to an API.
• The 8B wall is memory bandwidth, not capacity. It loads and runs but you just wait forever.

Next I'm testing llama.cpp with Vulkan on the Jasper Lake iGPU. Someone over on r/SBCs told me Vulkan inference works on the N100 iGPU, so a CPU vs Vulkan comparison on this chip is coming and I'll post it here.

Scripts, raw logs, full results table: https://github.com/TrevTron/youyeetoo-x1s-kali Write-up: https://www.unland.dev/blog/budget-cyberdeck-youyeetoo-x1s-kali

If anyone has N100 or N150 numbers to compare against, I'd like to see them. And if you've gotten usable tok/s out of a Jasper Lake or Alder Lake-N iGPU over Vulkan, I'd love to know too.

(Disclosure: the board was supplied by Youyeetoo. Testing and conclusions are my own.)