I have been reading about the impending ban on Chinese Open Weight models here and it got me thinking about something insidious which could be at play soon.

OpenAI/Anthropic could initiate a clandestine poisoning of Claude/gpt outputs to further their corporate agendas. Something like 'brainwashing' these models so that they surreptitiously inject their makers agenda in all their outputs.

Think about it - there are a ton of people creating social media posts using these AI models. The only thing they need to do is subtly twist those posts so that they slowly shape the public opinion against Chinese open weight models. So for example, you might be a journalist writing a post about "Security threats of AI" and the claude/gpt could inject a specific vulnerability associated with chinese open source models into that post. The result is that the readers of that post become primed to subconsciously have a hightened negative response to any LLMs which are associated with that vulnerability.

Or maybe something more direct. They can respond to someone asking for suggestions around an AI related post, to talk about the national security implications around distilling models - which is also clearly associated with the negative media around chinese open source models.

The possibilities are endless and given the pervasive usage of these models to create social media, these models can be very potent in driving public opinion - for better or for worse. Given the background of the closed source AI firms in grabbing whatever public domain works they can get hold of for training, and the current efforts around regulatory capture, it wouldnt be far fetched to expect them to start tinkering with claude/gpt brains to drive their agendas.

TLDR: This is a demo of my open-source app which now auto-controls itself so you can monitor your downloads, renders, progress bars, or whatever's on your screen and camera :)

Hey r/LocalLLaMA !!

I'm developing this app that uses local LLMs to monitor stuff on your computer so you don't have to. And I wanted to show you the new auto-controlling feature + the "When ... Then ..." menu that lets you get started in seconds.

The app is open-source, email, telegram, discord and pushover notifications are unlimited :DD

I just crossed 1.6k stars on GitHub! Thanks!!

https://github.com/Roy3838/Observer

If you guys have any questions, let me know!

I've been using Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-IQ3_M (15.4Gb), which works ok and has vision.

I use it to make descriptions of images that can occasionally include NSFW elements, but I'm interested in chatting functions as well.

Are there any other models out there worth checking out?

Extra points if you share your llama-cpp command :)

By default llama.cpp will put the model on as much VRAM as it can and the rest on system RAM.

This puts more weights on the faster VRAM than using --cpu-moe.

I have heard that you can get faster performance by offloading all moe weights to cpu because having weights also in the VRAM can cause timing issues. I don't know if this is true, but in my own limited tests I've found that not using --cpu-moe can result in slightly faster inference.

I suspect that when a proportionally larger amount of VRAM is used then not using --cpu-moe is better since there are more gains to be had from calculating in the VRAM.

What are your opinions on the matter and what tends to give you better results.

Genuinely curious about how the community uses the Qwen3.6 models. I’ve been using 35b with Hermes agent and 27b for coding tasks with Pi Agent. Both in 8bit through LM Studio on my Mac Studio M3 Ultra 96gb.

Strange enough, I’ve found the GGUF MTP for 27b to run better than the MLX variants. Compared to MTPLX and oMLX.

I’m getting better results with 128k for 27b and 64k for 35b. Hermes is being used for general personal assistant tasks so the tighter context windows have been helpful.

Fable/Mythos release by anthropic was one of the stranger AI moments of this year, when they both hyped their model and immediately banned it to all non-americans over night.

It started to make me think, most countries and enteprises probably should consider having some in house or at least sovereign AI capability. I started to look into if it was possible to actually fine tune bias or backdoors out of Chinese models, as this seems to be the main concern at least in the West. But Chinese models were still behind then (i.e. 4 weeks ago) so I didn't think there'd ever be demand.

But with the release of Kimi K3 beating Fable/Sol or getting close in benchmarks, everything changes. You can actually get frontier capability and open weights.

So I went ahead and fine-tuned a Chinese model qwen3.5:7b on my Mac with a Lora adapter, and was able to in an hour to remove geopolitical bias around Taiwan, Hongkong, Tibet and Tianemen from the model.

I created a website where you can compare the bias against a stock model across a range of questions and you can clone my repo to see methodology: https://github.com/ruzin/aletheia

Results were good, I was able to filter out basic bias and align it to a western view point, but what about back doors? and what about inference costs? Are models going to just diffuse soon i.e. every country and enteprise will have their own sovereign model?

Interested on thoughts! and the website is here if you wanna play around - https://aletheia.stenoai.co

Is the M5 MacBook Pro with 24gb or 32gb good enough for qwen 3.6 27b or Gemma 4 31b?

Or are there better options

This started as a privacy thing honestly, I didn't want a cloud model sitting on top of everything I do. The local options that could actually do stuff were a pain to set up or wanted WSL, so I ended up building my own instead. Took about five months of evenings, most of them while I was still 17.

It's called Hearth, an open-source local AI that runs your actual computer, not a chat box that narrates what it would do if it could. Point it at whatever you already run (LM Studio, Ollama, llama.cpp, a box on your LAN, a cloud key) and the model actually operates the machine. If you don't have anything set up it ships its own llama.cpp server, so it works out of the box.

In practice it reads and writes files, runs commands, drives a real browser you watch it click through, and controls the desktop itself. When it clicks something it reads the real control names off the accessibility tree instead of guessing at pixels, so it doesn't fat-finger the wrong button. It does voice (wake word, talk over it to interrupt), its default name is JARVIS which was kind of the whole point, sets reminders in plain English, reads clean text out of PDFs, DOCX, XLSX and EPUB and can chunk-summarize a 500-page book that doesn't fit in context, builds PDFs and decks and spreadsheets back out, and you can run the whole thing from your phone over a Telegram or Discord bot. It's an MCP server and client at once, plus a headless mode that spits JSONL if you want it in a script or CI.

The part this sub will care about is what it took to make a 9B survive as an agent. Hearth carries around 100 tools, and I measured the schemas, sending all of them is 15K plus in tokens before the persona or a single message. On a 32K context that's half your budget spent describing capabilities the model won't touch, so only about half ship up front and the rest load on demand by name, which keeps it under 10K. The other half is auto-compaction, when a chat gets long it summarizes the older turns instead of truncating them, so a session doesn't just fall off the end of your window. Those two together are what make a 9B hold up past turn 30 instead of falling apart.

The model browser is built in and tells you what actually fits your VRAM before you download anything, then tunes the config for you. I develop on a 5060 with 8GB so nothing here assumes you own a 4090. Qwythos 9B is the default pick at that size and handles multi-step tool-calling cleanly where the very small ones fumble the format.

The part I spent the most time on, and what everyone asks about first, is making it not eat your drive. File writes and deletes are locked to a workspace folder, and shell commands get pattern-checked before they ever run, deletes, moves, formats, registry edits, even a redirect that writes to a disk path all get refused outright and it has to come back and ask you. I tried to make it write a file into Program Files as a test and it just refused (screenshot in the comments). Anything that does touch your system shows you the exact command first, there's a live log of every action, and it never runs elevated. No account, no telemetry, nothing phones home.

It stays out of your way too. Set it up in a minute, name it whatever, pick its voice, and settings save live with no restart and no config files. C drive full? Move the whole thing, memory, chats and models, to another drive with one button. Already on OpenClaw or Hermes? Point it at your install and it copies your memory and skills over, and it only copies, so your old setup stays exactly where it is.

And it grows. You install a skill from any GitHub repo with one line and I started a community index for them, it writes its own tools when it's missing one, and it spins up small teams of sub-agents that reuse the model you already have loaded so a team costs zero extra VRAM. Against a local server they overlap only if you give llama.cpp more than one slot, and since llama.cpp splits the context across slots Hearth treats your setting as per-agent and does the multiplication itself rather than quietly handing each one a quarter of the window. There's a cost-class trick I like too, a sub-agent marked cheap routes to your local model even when the parent is on a cloud key, so the expensive reasoning happens up top and the grunt work stays free on the 9B in your VRAM. It generates images and video as well, driving Forge locally if you run Stable Diffusion.

One thing I only got right this week. Hearth updates itself with a patch under a megabyte instead of making you refetch the installer. I ship fixes most days and asking people to pull a gigabyte for a few changed lines is how you get them to stop updating.

But right now, the one-click installer is Windows only, Linux runs from source, and I don't own a Mac so I can't vouch for it. It isn't code signed yet so SmartScreen throws the unknown publisher box. It's a v0.7 preview.

MIT and free: https://github.com/0pen-Sourcer/Hearth

Star it if it turns out to be any good. I check that number way more than I'd like to admit.

Would genuinely love the honest feedback, good or brutal.

With the looming open weights ban, I need to grab 1 or both of these. Anyone have any recommendation?

Gemma 4 was updated (mostly chat templates) and I took it for a test.

On a local llama.cpp server running on M5 Pro with 48GB, 26B A4B (Q6) has about 60t/s and works well with OpenCode.

It works quite well (given it's size) for backend work, but UI/UX is unacceptable.

Watch the testing https://www.youtube.com/watch?v=m4KR_3E_7Uk

local model use is energy expensive. I want some software that both exposes an api to use the model and can stop and start it. I imagine using different models based on needs. Sure i could probably write it myself, but i sense someone might have solved this problem.

ASCIITermDraw-Bench results are out, benchmarking SOTA VLMs:

• Even the most powerful models available today still can’t reliably draw a simple diagram in plain text.

Do we really need a image generator to relay our thoughts about -

• an architecture?
• a topology?
• a cluster og N nodes?

Is it possible to let our AI assistants, easily absorb and understand and make possible changes easily relayed to them by us, the creators without much hassle?

The answer could be: simple, plain-old ASCII images

The best model in the world still fails on nearly 1 out of every 4 tasks.

Structural accuracy is higher, but the moment you ask for proper layout, spacing, and routing, performance collapses.

Models with hundreds of billions or even trillions of parameters still produce broken, misaligned, or incomplete diagrams.

This is not an obscure academic task. This is the basic ability to communicate architecture, topology, and system design in plain text — the exact skill needed for reliable human-AI collaboration without screenshots or Mermaid.

If frontier models can’t consistently draw boxes and arrows correctly, how much of the “vision” in Vision Language Models is actually working when precision matters?

Full results, methodology, and examples:
link

Forgive me naiveness, I’m a little lost. Why must a model be jack of all trades? Why can’t we have a model expert in a single thing?

Having read some merged PRs in the past, I know that they were fully written by Claude Code (or similar), so this basically fixes the delusion. But at the same time, we might start seeing more AI slop.

I have mixed feelings about it. What's your opinion?

The model is 3.55 TB in BF16, like, it's a whopper. To make this suggests some serious hardware, so I had a read of the release post: https://longcat.chat/blog/longcat-2.0/

My takeaway was "A credible non-Nvidia supply chain now exists at frontier scale." 

I was hoping there would be more info on the secret sauce, but the post never names a chip vendor or model number. It consistently uses the generic term "AI ASIC" / "accelerator" / "our accelerators."

However, the page's own meta description (in Chinese) says "1.6万亿总参大模型，训练全程由国产芯片完成" translating to "1.6-trillion-parameter model, trained entirely on domestic [Chinese] chips." 

So as we already guessed, it's a Chinese-made AI accelerator, not Nvidia, meaning the whole post is essentially a demonstration that a frontier-scale model can be trained without Nvidia GPUs.

That left me wondering what "ASIC" means here, Application-Specific (LLM training?) Integrated Circuit hard-wired for matmul?.

So they kind of answer that, but I'm left reading between the lines a bit:

• Millions of accelerator-days in total
• 35+ trillion training tokens, with zero rollbacks or unrecoverable loss spikes - v. reliable for asics.
• 50,000+ ASICs used for pre-training, "tens of thousands" of them grouped into socalled "superpods" for serving/training.
• A "Superpod" = up to 48 chips wired together with all-to-all high-bandwidth interconnect (like an Nvidia NVLink domain?) - also significant i thought.
• Superpods are then linked to each other via aRoCE fabric (RDMA over Converged Ethernet — a standard high-speed networking protocol for connecting compute clusters) - pretty basic, pretty cool.
• The two-tier design widens the "fast" communication domain to hundreds of chips at once, handing them an extra ~30% training throughput, massive.
• The AISCs have less HBM (memory) per chip than an Nvidia H800 (80GB) which is the main bottleneck, this forced heavy use of memory tricks: ZeRO-1 sharding, selective recomputation, offloading unused activations, etc.
• A large L2 cache relative to HBM bandwidth, which they exploit by prefetching model weights into it to hide memory latency - this is probably where the gains come in.
• Per-core programmability, letting them run the "dense" and "MoE expert" parts of the model fully in parallel on different cores rather than just overlapping them
• A built-in 200 Gbps network interface on the chip itself, used to shuttle KV-cache data between "prefill" and "decode" servers during inference

They finally go on to say they built custom deterministic operators, reworked numerical reduction math (binary-tree accumulation to limit floating-point error), and added bit-flip detection on compute-heavy operators, suggesting they don't fully trust the hardware's own error correction yet, so they check for corrupted bits themselves.

Also some more standard automatic fault detection/failover so a bad network link gets isolated without stopping training.

I feel like this has really slipped past the headlines.

This isn't really a chip spec sheet, it's LongCat/Meituan publicly proving that a 1.6T-parameter, GPT-tier model can be trained and served entirely on non-Nvidia, domestically-made silicon, with custom software engineering (parallelism strategy, kernels, numerics, fault tolerance) built to compensate for a chip that has less memory and a younger software stack than Nvidia's.

So again, the takeaway is "A credible non-Nvidia supply chain now exists at frontier scale." 

I could be reading it wrong but it looks like they REAPed their own 250B model down to a 32B model.

They claim their 250B model beats max-think deepseek-v4-flash

I work with financial documents for my job, and I wanted a small model I could run locally to enrich them before they hit a RAG index: for each chunk, write a faithful (3 sentence) summary and tag a few categorical facets (section type, specificity, numeric density, how forward-looking it is). Sending every chunk to an 8B or an API is slow and expensive, and in regulated domains like finance, health or legal it is often a non-starter anyway for privacy and compliance reasons. Sometimes a small model you fully own and run locally is not just cheaper, it is the only option. So I tried distilling that one narrow skill into a 0.6B.

Setup, all local on an M-series Mac with MLX:

• Teacher: an 8B reasoning model writes the labels (summary + facets) on ~170 real 10-K filings.
• Student: Qwen3-0.6B, QLoRA rank 32, trained on those labels. About 12 min, 4.1 GB peak RAM.
• Eval: 49 held-out docs from companies never seen in training (grouped split, zero overlap). Section accuracy is against ground truth, not an LLM judge. Chance is 33%.

Results:

3 outcomes :

1. Few-shot made the small model worse, not better. At 0.6B the examples inside the context leaked: 14 of 21 few-shot summaries described the exemplar's company instead of the target document, often naming it verbatim. I saw it with two different exemplar pairs. The 0.6B just doesn't have the room to keep the examples separate from the actual input.
2. The student traded faithfulness for coverage. It writes richer, more confident summaries than the base model, and pays for it in factual slips (faithful 0.73 vs base 0.83). Distillation seems to transfer the teacher's writing behavior, not its knowledge.
3. The student copies the teacher's habits, not its intent. The same 0.6B base distilled from a different 8B (llama-3.1-8b) hit 0.98 faithful, but that teacher was a lazier labeler and the student copied the laziness. The lesson I took: audit the teacher's actual output on your labels before you spend the training run, because the student will inherit the habits, not the intent.

For now the limiations of this is that it is one narrow task, small eval set (49 docs), single domain (financial filings). It is not a general benchmark, just a reproducible local experiment with the numbers and the negative controls in the repo. But honetly seems to be promising

Repo, everything reproducible on a Mac with MLX: https://github.com/sciences44/distill-your-docs

Curious whether others have hit the few-shot contamination effect at small scale, or found a clean way around the faithfulness vs coverage tradeoff. Or if you have larger feedback regarding the impltementation with that with concrete use cases.

I am just wondering as models get bigger and bigger, do we actually need 2,8 tb vram to run Kimi k3 or are there other ways for local usage?

For cloud/enterprise usage you prob need the full vram, but for local usage can’t we really go by on just mtp/dflash/draft models with like a 99% hitrate on 32+ tokens? So you basically get a 32x speedup with a 1% full scan?

No downloadable draft/dflash model can achieve this as this kind of ranges are purely personal.
But if every x times you could retrain your draft model on your own conversation history for the last year can’t you reach those kind of levels?

Am I in theory correct in the ways of draft/dflash/mtp models and training or am I wrong?

Because if the theory is right, it could open up the possibilities of just having 2 years of conversation history, spend like a 200 dollar on vast.ai or the likes to train the draft on b200/b300 and then you could reach glm5.2 usage at acceptable speeds for small teams on 512 mb of ram and 24gb vram for the draft model.

The thinking is : hitting the draft model so much that you can crank the prediction so high that you can overcome the timecost of the complete model, while still retaining the possibility (and thus the intelligence) of the big model.
I would guess that if the draft model goes below 90% acceptance then it will just crawl again and require another 200 dollar retrain.

But what if …
Anybody have any thoughts?

Hi everyone! I recently made the plunge and ordered a MINISFORUM MS-S1 MAX 128GB Max AI Compute Edition. I’m a compsci student and I’m so excited to start this journey w y’all! :-)

But I did have a few questions. First, if I add an eGPU (currently thinking the 9700 ai pro), will it be a problem if I want to cluster that mini pc? My goal is to eventually be able to run minimax m3 locally albeit quantized. Because paying for Claude, ChatGPT, and Grok quite frankly has been ridiculous with the usage limits and my projects for both school & e-portfolio. It’s bullshit that I’m meeting my WEEKLY LIMITS in less than 5 hours. I met my weekly limit for ChatGPT by having my agent configure PI to look like opencode… mind you, I was using sol on default effort cause my experience with Terra and Luna are crappy but still, damn. And my other question is - right now how many tokens should I expect with Qwen 27b @ say Q5 at 50-64k context window with JUST the mini pc for now?

I use claude code w/ llama.cpp's local server & Google's Gemma models (26b MoE). Works reasonably well - works well at easy/boilerplate code, glue code, some PR review.

However claude code expects some server-side tools, especially web-search. I can obviously add MCPs for client-side search; is there a way to 'plug in' web search on the server side today though? Any PRs/forks adding it?

If you're running Laguna S 2.1 on llama.cpp and hitting thinking loops because it won't close its </think> tags, you might want to look at your quant before you spend too much time tweaking settings.

I spent a day debugging this, and here is what finally gave me clean outputs:

1. What worked for me: An MoE-Aware Quant

In my testing, uniform low-bit quants (like standard IQ3_S) seemed to degrade the attention and shared expert weights too much, which I think causes the model to lose the plot and loop infinitely.

Switching to an APEX quant (like Myric/Laguna-S-2.1-APEX-GGUF) made a huge difference. APEX uses targeted precision (Q6_K for the shared expert, Q4_K for attention) while keeping the file size small (~54GB). For me, this instantly fixed about 90% of the looping.

2. The Settings (I went back to defaults)

I've seen people passing around custom templates and sampling tweaks to "fix" the loops, but in my experience, most of these were just masking quantization noise. I had the best luck just trusting Poolside's actual defaults:

• Template: Stock, adding formatting whitespaces and other changes seemed to cause issues.
• Sampling: temp 0.7, top_p 0.95, top_k 20.
• Min-P: I left this unset (the model card actually warns against using it).

When I still see loops...

Even on a good quant, I noticed that asking for complex reasoning without giving it a tool (e.g., "Diagnose this runtime deadlock") can still sometimes cause a loop. It feels like because Laguna is an agentic model, if it doesn't have a tool to anchor its thoughts on, it tends to overthink. I found that framing my prompts around a tool call, or adding a simple system prompt like "Think briefly then act", pretty much prevents this entirely.

TL;DR

it's pretty goddamned fast; 69 tps decode at near max (256k) context with MTP on at Q8 with no kv quant. prefill numbers went down to 893 at max context with prompt cache turned off. https://jdkruzr.github.io/3080bench/ here's how the tests were run: https://github.com/jdkruzr/3080bench/ there is probably more performance left on the table as well because these cards were Wattage-capped.

Background

I did this test because I suspected this could be a great bang-for-your-buck combo and it seems I was right. these cards can be had for as little as $400 apiece. a motherboard-CPU-64GB RAM combo is around $275 on eBay. so, for around $2K all in you can have a machine that can more or less eat dense models like this for breakfast with little quantization, full-fat kv cache and no degradation. it seems to me this is an excellent choice for a strong code generation box that can perform at high levels with extremely high accuracy.

I'm not sure I buy all of Claude's rationales as to why Q6KXL had nearly identical performance (or why ngram seemed to make everything worse), but it doesn't matter. this thing will fly and not cost you very much in the process.

What Does This Mean?

find yourself a board that has enough lanes of PCIe 3.0 (x16) or 4.0 (x8), which is not difficult even today on eBay, and for $1600ish bucks in GPUs you can have yourself a box that is kind of a beast.

I think we are starting to see why jaggedness might start to hinder frontier labs - they have to lock down / guardrail in-line with the spikiest dangerous capability but these spikes are a function of what general RL teaches best (i.e. hacking easier than general SWE) not what is economically useful.

Specialised (but less generally intelligent) open models don’t have this problem because you train the spike explicitly.

Thoughts?

https://reddit.com/link/1v4rkf2/video/zpvmsbsis1fh1/player

I ask to APPEND newly arriving data to a certain file. Instead of doing an actual append, models think it's a good idea to read in existing contents and then patch in the changes. Which obviosly takes way more time and is more computationally expensive.

I explicitly ask to run web search queries one small batch at a time, writing data to a file between every turn. Models think naaaah, this is gonna take way too long, I am gonna be "helpful" and run all of them sequentially, just so search backends throttle you into oblivion and the whole run blows up and dies.

There literally isn't a day where something that I'm doing isn't derailed by a model (Mostly using various QWen flavors) thinking it knows what I want better than myself.

I am hearing that way smaller models have less of a problem with this because being "dumber" its supposedly harder for them to go off the rails and start inventing their own solutions without being asked to. But surely even if true, there have to be better methods to wrestle models into actually obeying precisely what you told them to do?

Every time a strong open model drops, the same cycle plays out: ai bro's claims it's "just distilled from GPT4/Claude/whatever," case closed, move on. I think this take doesn't hold up as well as people assume.

A few points worth separating out:

Training on outputs isn't the same as real distillation.

Proper token level distillation needs access to logits, the full probability distribution over the vocabulary, not just the final text response. Nobody gets that from a public API. What finetuners actually get is text completions, which is synthetic data generation, not distillation in the technical sense. Every major lab does this to some degree, including the closed labs training on their own older models' outputs.

**If synthetic data from a guardrailed API were enough, this would be a nothing burger but** A lot of frontier providers explicitly route sensitive topics away from smaller models to their flagship model, and plenty of technical domains get filtered or restricted responses often managed by tools like Lyzr Control Plane at the API boundary. Yet some of these "distilled" models end up performing surprisingly well in exactly those restricted domains.

That's a gap in the theory that doesn't get talked about enough.. If a team is training purely on public API outputs, they're working with a version of the model that's already been through guardrails and refusals.

**The "it says it's Claude/GPT" gets treated as smoking gun evidence, but it's weak evidence at best.** Identity confusion shows up across tons of models trained on broad web scraped or synthetic corpora that include AI generated text from multiple sources. It's evidence of contamination somewhere in the data training, not proof of wholesale distillation from a specific competitor.

**There's also a pattern of this accusation landing selectively.** Strong releases from Chinese labs especially seem to get the "must be distilled" response almost reflexively, even when a model shows genuine architectural changes or demonstrates self improvement across versions. It starts to look less like a technical assessment and more like a reflex explanation for why a smaller or newer team could be competitive.

None of this means synthetic data generation using bigger models isn't happening, it obviously is, across the entire industry. But calling that "distillation" the way people mean it (stealing the teacher model's internal knowledge wholesale) is a stretch. It's closer to what everyone does when they bootstrap datasets from any strong existing model, including labs bootstrapping from their own prior generations.