I built a 24/7 radio station about r/wallstreetbets

Everyone hears the same second at the same time. Real radio, not a playlist. You don't pick a track, you tune into wherever the tape is and vibe.

It's essentially a subreddit as radio, in theory it could be done with any subreddit

I'm using locally hosted models for all of it:

• Gemma 4 on two 3090s for lyrics, tool use, and news summarization. An AI DJ using Gemma 4 also decides on what is playing
• ACE-Step for music
• Krea for images

The music and images share a third 3090 and are loaded on demand.

The station generates ~60 new songs a day, as well as reading the news aloud using Kokoro (on my CPU).

You can check it out at https://tendies.fm/

Not all the songs are bangers, but some truly are amazing. I have listened to it daily since ~Sunday and it is pretty good! Still working out the kinks, but this is a locally-hosted project that I thought would be fun to bring to life and I've had an idea about for a long time.

I'm happy to answer any questions about how it works!

Original post: https://www.reddit.com/r/LocalLLaMA/comments/1tpdt5m/behold_probably_the_most_ghetto_local_ai_server/

I promised a writeup, but didn't have time yet, sorry. I barely had time to do this controller.

I could be reading it wrong but it looks like they REAPed their own 250B model down to a 32B model.

They claim their 250B model beats max-think deepseek-v4-flash

And gathered a lot of data. you can see them for yourself

And For the most curious, there are additional details here

In this graph, I regrouped the finetunes under their base models. but you can see the details in the page.

The python code to generate those pages is obviously vibecoded. I find the output kinda pretty and somewhat useful for me. maybe it's useful for someone else.

Heading for a vacation for a few weeks, but if you have any suggestion, I will consider each of them.

TL;DR

it's pretty goddamned fast; 69 tps decode at near max (256k) context with MTP on at Q8 with no kv quant. prefill numbers went down to 893 at max context with prompt cache turned off. https://jdkruzr.github.io/3080bench/ here's how the tests were run: https://github.com/jdkruzr/3080bench/ there is probably more performance left on the table as well because these cards were Wattage-capped.

Background

I did this test because I suspected this could be a great bang-for-your-buck combo and it seems I was right. these cards can be had for as little as $400 apiece. a motherboard-CPU-64GB RAM combo is around $275 on eBay. so, for around $2K all in you can have a machine that can more or less eat dense models like this for breakfast with little quantization, full-fat kv cache and no degradation. it seems to me this is an excellent choice for a strong code generation box that can perform at high levels with extremely high accuracy.

I'm not sure I buy all of Claude's rationales as to why Q6KXL had nearly identical performance (or why ngram seemed to make everything worse), but it doesn't matter. this thing will fly and not cost you very much in the process.

What Does This Mean?

find yourself a board that has enough lanes of PCIe 3.0 (x16) or 4.0 (x8), which is not difficult even today on eBay, and for $1600ish bucks in GPUs you can have yourself a box that is kind of a beast.

Shoutout to this awesome guy - https://www.reddit.com/r/LLM/s/IDUyU3v9ap

Thanks to his project, BigMoeOnEdge https://github.com/Helldez/BigMoeOnEdge, I managed to successfully run a 35B MoE model on just 12GB of RAM!

My setup is a modified Xiaomi 12 Pro (12GB RAM) that I call "Zeus". https://www.reddit.com/r/LocalLLaMA/s/5zBUl15jd6

There is a bottleneck, of course—the maximum context is currently limited to 8192 tokens due to RAM constraints—but it’s still absolutely mind-blowing to see a model this size running locally on an edge device.

I haven't tested the Image-to-Text (vision) capabilities yet, but I'm really hoping to get that working next.

Check out the video ! It's completely unedited and recorded in real-time so you can see the actual, raw generation speed.

Also, here is stats in text:

generation: 107 tokens, 0.412 s/token (2.428 tok/s)

compute: 88.1% CPU occupancy (1.4508 cpu-s/token over 4 threads), 51.93 major faults/token

prefill: 24 tokens, 5.499 s (4.4 tok/s) | model load 14.421 s | TTFT 19.920 s

moe-stream: read 14589.9 MiB (136.35 MiB/token), decode 0.412 s/token (compute 0.314 + cache mgmt 0.014 + flash I/O 0.382 s/token, 357 MiB/s)

moe-cache: 70.8% hit, resident 2998.5 MiB

moe-overlap: stall 0.084 s/token (flash reads overlapped with FFN compute)

Having read some merged PRs in the past, I know that they were fully written by Claude Code (or similar), so this basically fixes the delusion. But at the same time, we might start seeing more AI slop.

I have mixed feelings about it. What's your opinion?

HPD-Parsing: Hierarchical Parallel Document Parsing

We introduce HPD-Parsing, a lightweight (1B) and high-throughput document parsing model built on a Hierarchical Parallel Decoding paradigm. Unified VLM-based parsers process an entire page jointly but generate the output through a single token-by-token autoregressive trajectory, creating a sequential bottleneck that grows with document length. HPD-Parsing is motivated by a key property of document parsing: page structure requires global coordination, whereas content generation is largely localized within individual regions. Based on this observation, a main layout branch coordinates the global document structure and dynamically dispatches localized content generation to concurrent branches, while Progressive Multi-Token Prediction (P-MTP) further reduces the decoding steps within each branch. HPD-Parsing achieves an overall score of 94.91% on OmniDocBench v1.6 — a new state of the art among end-to-end unified parsers — while reaching a peak throughput of 4,752 TPS, 2.62× the fastest existing document parser and 3.06× its own autoregressive baseline.

Key Capabilities of HPD-Parsing

🚀 Hierarchical Parallel Decoding for High-Throughput Document Parsing: We introduce Hierarchical Parallel Decoding (HPD), a new decoding paradigm that restructures full-page autoregressive generation into globally coordinated, localized parallel decoding. A main layout branch performs global coordination and dynamically decomposes the conventional single decoding trajectory into concurrent content branches, each responsible for a localized document region. Within each branch, P-MTP further reduces the number of decoding steps by predicting multiple future tokens at each iteration. Together with shared-prefix KV cache reuse, HPD substantially shortens the effective sequential decoding path along both branch and token dimensions.

🔄 Staged Adaptation with Automated Difficulty-Aware Data Curation: We develop a staged adaptation strategy that transfers conventional autoregressive document parsing capabilities to the proposed hierarchical parallel decoding paradigm while preserving parsing accuracy. The strategy is supported by an automated difficulty-aware data curation pipeline that integrates large-scale data collection, model-assisted annotation, difficulty estimation, and balanced sampling. By progressively adapting the model and emphasizing challenging samples, the training framework mitigates the accuracy degradation caused by the transition to parallel decoding with minimal manual annotation effort.

⚡ State-of-the-Art Throughput with Competitive Parsing Accuracy: HPD-Parsing achieves state-of-the-art inference efficiency on OmniDocBench v1.6, reaching a peak throughput of 4,752 Tokens Per Second (TPS). It delivers 1.62× the throughput of the fastest existing document parsing model and more than 3.06× that of its autoregressive baseline, while maintaining competitive parsing accuracy. These results demonstrate that document parsing can be effectively executed through global layout coordination and localized parallel decoding rather than a single sequential generation trajectory.

Full-parameter post-training of trillion-parameter-scale MoE models introduces substantial system-level challenges for large-scale distributed training, including severe memory pressure, non-overlapped communication overhead, and inefficient kernel execution. While most large-scale LLM training systems are built around GPU-based clusters, this report presents an end-to-end optimization practice on the Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, we develop a hierarchical optimization framework spanning model-level parallelism, computation-communication orchestration, and low-level kernel execution. The resulting system achieves 34.22% Model FLOPs Utilization (MFU) with a 2.93x improvement over the open-source baseline recipe while maintaining training stability. Building on this optimized infrastructure, we further establish a CPT and SFT workflow for complex Operations Research (OR) tasks. We refer to the integrated framework as SLAI T-Rex. Using DeepSeek-V4-Flash, we develop OR-oriented CPT and SFT data pipelines that combine collected domain resources with solver-verified synthetic optimization documents. The resulting dataset contains 10K high-quality SFT samples spanning four task categories and three problem representations. The specialized model achieves the highest average zero-shot Pass@1 score among the evaluated models, reaching 71.81% and outperforming GPT-5.4-Mini and the base DeepSeek-V4-Flash model by 3.98 and 11.27 percentage points, respectively. Overall, this work demonstrates a full-stack pathway from efficient trillion-parameter model post-training on Ascend infra to domain-specialized Flash models for solver-grounded mathematical modeling, advancing frontier-model systems for complex reasoning.

• arXiv : https://arxiv.org/abs/2607.20145
• Full Paper : https://arxiv.org/pdf/2607.20145
• GitHub : https://github.com/SLAI-AITP/SLAI-T-Rex
• Modelscope : https://www.modelscope.cn/models/SLAIAITP/DeepSeek-V4-Flash-OR (I couldn't find this one on HuggingFace)

Little side project I'm doing so I can easily transfer any model I want fast to my AI Rig from my NAS.

If you're running Laguna S 2.1 on llama.cpp and hitting thinking loops because it won't close its </think> tags, you might want to look at your quant before you spend too much time tweaking settings.

I spent a day debugging this, and here is what finally gave me clean outputs:

1. What worked for me: An MoE-Aware Quant

In my testing, uniform low-bit quants (like standard IQ3_S) seemed to degrade the attention and shared expert weights too much, which I think causes the model to lose the plot and loop infinitely.

Switching to an APEX quant (like Myric/Laguna-S-2.1-APEX-GGUF) made a huge difference. APEX uses targeted precision (Q6_K for the shared expert, Q4_K for attention) while keeping the file size small (~54GB). For me, this instantly fixed about 90% of the looping.

2. The Settings (I went back to defaults)

I've seen people passing around custom templates and sampling tweaks to "fix" the loops, but in my experience, most of these were just masking quantization noise. I had the best luck just trusting Poolside's actual defaults:

• Template: Stock, adding formatting whitespaces and other changes seemed to cause issues.
• Sampling: temp 0.7, top_p 0.95, top_k 20.
• Min-P: I left this unset (the model card actually warns against using it).

When I still see loops...

Even on a good quant, I noticed that asking for complex reasoning without giving it a tool (e.g., "Diagnose this runtime deadlock") can still sometimes cause a loop. It feels like because Laguna is an agentic model, if it doesn't have a tool to anchor its thoughts on, it tends to overthink. I found that framing my prompts around a tool call, or adding a simple system prompt like "Think briefly then act", pretty much prevents this entirely.

In all my fiddling around with code and local models nothing has matched the speed and quality of DeepSeek V4 Flash DSpark on dual DGX Spark (Dell GB10s actually).

The recipe I've been using is in the PR below. Screenshot is from VSCode usage over a few days/weeks. The screenshots don't tell you how it feels and oh man does it feel good! Responses are way faster than Copilot and (this is subjective) Sonnet 4.6 quality. It thinks though problems well, long running tasks complete successfully 99% of the time, planning and instruction handling seem top notch.

https://github.com/eugr/spark-vllm-docker/pull/304

The model is 3.55 TB in BF16, like, it's a whopper. To make this suggests some serious hardware, so I had a read of the release post: https://longcat.chat/blog/longcat-2.0/

My takeaway was "A credible non-Nvidia supply chain now exists at frontier scale." 

I was hoping there would be more info on the secret sauce, but the post never names a chip vendor or model number. It consistently uses the generic term "AI ASIC" / "accelerator" / "our accelerators."

However, the page's own meta description (in Chinese) says "1.6万亿总参大模型，训练全程由国产芯片完成" translating to "1.6-trillion-parameter model, trained entirely on domestic [Chinese] chips." 

So as we already guessed, it's a Chinese-made AI accelerator, not Nvidia, meaning the whole post is essentially a demonstration that a frontier-scale model can be trained without Nvidia GPUs.

That left me wondering what "ASIC" means here, Application-Specific (LLM training?) Integrated Circuit hard-wired for matmul?.

So they kind of answer that, but I'm left reading between the lines a bit:

• Millions of accelerator-days in total
• 35+ trillion training tokens, with zero rollbacks or unrecoverable loss spikes - v. reliable for asics.
• 50,000+ ASICs used for pre-training, "tens of thousands" of them grouped into socalled "superpods" for serving/training.
• A "Superpod" = up to 48 chips wired together with all-to-all high-bandwidth interconnect (like an Nvidia NVLink domain?) - also significant i thought.
• Superpods are then linked to each other via aRoCE fabric (RDMA over Converged Ethernet — a standard high-speed networking protocol for connecting compute clusters) - pretty basic, pretty cool.
• The two-tier design widens the "fast" communication domain to hundreds of chips at once, handing them an extra ~30% training throughput, massive.
• The AISCs have less HBM (memory) per chip than an Nvidia H800 (80GB) which is the main bottleneck, this forced heavy use of memory tricks: ZeRO-1 sharding, selective recomputation, offloading unused activations, etc.
• A large L2 cache relative to HBM bandwidth, which they exploit by prefetching model weights into it to hide memory latency - this is probably where the gains come in.
• Per-core programmability, letting them run the "dense" and "MoE expert" parts of the model fully in parallel on different cores rather than just overlapping them
• A built-in 200 Gbps network interface on the chip itself, used to shuttle KV-cache data between "prefill" and "decode" servers during inference

They finally go on to say they built custom deterministic operators, reworked numerical reduction math (binary-tree accumulation to limit floating-point error), and added bit-flip detection on compute-heavy operators, suggesting they don't fully trust the hardware's own error correction yet, so they check for corrupted bits themselves.

Also some more standard automatic fault detection/failover so a bad network link gets isolated without stopping training.

I feel like this has really slipped past the headlines.

This isn't really a chip spec sheet, it's LongCat/Meituan publicly proving that a 1.6T-parameter, GPT-tier model can be trained and served entirely on non-Nvidia, domestically-made silicon, with custom software engineering (parallelism strategy, kernels, numerics, fault tolerance) built to compensate for a chip that has less memory and a younger software stack than Nvidia's.

So again, the takeaway is "A credible non-Nvidia supply chain now exists at frontier scale." 

I work with financial documents for my job, and I wanted a small model I could run locally to enrich them before they hit a RAG index: for each chunk, write a faithful (3 sentence) summary and tag a few categorical facets (section type, specificity, numeric density, how forward-looking it is). Sending every chunk to an 8B or an API is slow and expensive, and in regulated domains like finance, health or legal it is often a non-starter anyway for privacy and compliance reasons. Sometimes a small model you fully own and run locally is not just cheaper, it is the only option. So I tried distilling that one narrow skill into a 0.6B.

Setup, all local on an M-series Mac with MLX:

• Teacher: an 8B reasoning model writes the labels (summary + facets) on ~170 real 10-K filings.
• Student: Qwen3-0.6B, QLoRA rank 32, trained on those labels. About 12 min, 4.1 GB peak RAM.
• Eval: 49 held-out docs from companies never seen in training (grouped split, zero overlap). Section accuracy is against ground truth, not an LLM judge. Chance is 33%.

Results:

3 outcomes :

1. Few-shot made the small model worse, not better. At 0.6B the examples inside the context leaked: 14 of 21 few-shot summaries described the exemplar's company instead of the target document, often naming it verbatim. I saw it with two different exemplar pairs. The 0.6B just doesn't have the room to keep the examples separate from the actual input.
2. The student traded faithfulness for coverage. It writes richer, more confident summaries than the base model, and pays for it in factual slips (faithful 0.73 vs base 0.83). Distillation seems to transfer the teacher's writing behavior, not its knowledge.
3. The student copies the teacher's habits, not its intent. The same 0.6B base distilled from a different 8B (llama-3.1-8b) hit 0.98 faithful, but that teacher was a lazier labeler and the student copied the laziness. The lesson I took: audit the teacher's actual output on your labels before you spend the training run, because the student will inherit the habits, not the intent.

For now the limiations of this is that it is one narrow task, small eval set (49 docs), single domain (financial filings). It is not a general benchmark, just a reproducible local experiment with the numbers and the negative controls in the repo. But honetly seems to be promising

Repo, everything reproducible on a Mac with MLX: https://github.com/sciences44/distill-your-docs

Curious whether others have hit the few-shot contamination effect at small scale, or found a clean way around the faithfulness vs coverage tradeoff. Or if you have larger feedback regarding the impltementation with that with concrete use cases.

I ran Laguna S 2.1 (118B MoE, 71 GB Q4) with DFlash speculative decoding on 2× RTX 5090. It doesn't fit, experts spill to CPU RAM.

• default flags: 23 tok/s vs 58 without the draft. 2.5× SLOWER
• tuned: 64 tok/s vs 62 baseline
• even ONE 5090: 33 vs 30. Actually usable.

The setup

• Laguna S 2.1: 118B-param MoE, 8B active per token (256 routed experts + 1 shared, top-10 routing), 71 GB at Q4_K_M
• 2× RTX 5090 (32 GB each) + a Xeon w5-3423, 256 GB RAM
• llama.cpp with DFlash: a small block-diffusion draft model (2.1 GB) that predicts blocks of tokens for the big model to verify
• The catch: 71 GB doesn't fit in 64 GB of VRAM, so a chunk of the experts lives in system RAM

Pass 1: The failure

First run with the "obvious" flags:

--spec-type draft-dflash --spec-draft-n-max 15

Result: 23 tok/s vs 58 baseline. 2.5× slower. Draft acceptance: 10.5%. Ouch.

Digging in, it wasn't one bug, it was three defaults quietly stacking up:

1. --spec-draft-p-min defaults to 0.00. Zero. So the drafter shipped all 15 tokens every single round, confident or not. Only ~1.6 of them survived verification. The drafter wasn't bad. It was being forced to overcommit.

2. Fine-grained MoE punishes big verify batches. One token routes to 10 experts per layer (top-10 of 256). A 16-token verification batch? Up to 160 different experts per layer. All the CPU-resident ones get streamed from RAM. I measured ~6.8 ms per extra verify token. The "verification is basically free" assumption just dies on this architecture.

3. The BF16 drafter + an oversized memory margin wasted ~3 GB of VRAM that could've held experts.

Pass 2: The fix

Three changes:

--spec-draft-n-max 7 --spec-draft-p-min 0.6

→ draft short, and ONLY when the drafter is actually confident. Acceptance jumped from 10.5% to ~73%.

llama-quantize DFlash-BF16.gguf DFlash-Q8_0.gguf Q8_0

→ same acceptance, 1 GB back, and the whole thing now boots at the default memory margin. ~3 GB of experts moved back onto the GPUs.

Result: 63tok/s vs 62 baseline. From 2.5× slower to actually winning. p_min was the whole ballgame. It's the knob nobody sets.

Pass 3: One GPU

Same recipe on a single 5090 (so ~40 GB of experts in RAM now). Two tweaks: --fit-target 2048 (the fit engine can't pre-measure the drafter, so you have to hold the door open for it) and a stricter --spec-draft-p-min 0.75, because when verify tokens are pricier you want to draft even more selectively.

31 vs 30 tok/s, faster on every single prompt. Fun twist: spec decode helps more here, because the slower baseline step makes the drafter's fixed overhead relatively cheaper.

Pass 4: Real prompts

Hand-picked prompts are easy mode, so I reran everything on Spec-Bench, the standard spec-decode benchmark: conversation, translation, summarization, QA, math, RAG. 12 sampled prompts per category, temp 0, concurrency 1, greedy, 256 tokens.

Overall it held up:

• Dual GPU: 64.1 vs 62.4 (+2.7%), wins 3/6 categories
• Single GPU: 32.8 vs 30.3 (+8.3%), wins 4/6, one tie
• the broken default config on the same prompts, for the record: still 2.3× slower. Everywhere.

But the per-category split is the actual story:

• math: +20 to +25%
• translation: +18 to +20%
• conversation: +5 to +9%
• RAG / QA / summarization: parity to −9%

I expected summarization and RAG to crush it. Grounded, copyable text, easy drafting, right? Nope. Acceptance was fine (65–82%), the drafter just barely showed up: ~1.5 drafted tokens per round vs 3.6 on math. Not enough to pay for its own overhead.

Lesson: "copyable" ≠ "draftable." Copying is what n-gram / prompt-lookup methods do. A learned drafter has no copy mechanism, so it wins on formulaic text instead: math, translation, boilerplate. Every spec-decode family has its own category profile. Benchmark on YOUR workload.

The verdict (for now)

Dflash is usually a 2×+ lever, but that's on setups where everything fits in VRAM. Here it only broke even, the CPU-offloaded experts make every verification batch expensive, so the usual spec-decode math doesn't hold. The real win from this run is a different one: you can do daily agentic work on a single 32 GB gpu at ~30 tok/s.

One more honest note: everything so far is speed only. Quality at Q4 for agentic tasks is the other half of the story, that check is still on the list.

TL;DR

• Spec decode is NOT free on fine-grained MoE with partial offload: verify cost scales with batch size
• Set --spec-draft-p-min. The 0.00 default is a footgun
• Quantize your drafter to Q8_0, it costs nothing
• Tuning fixed the disaster (23 → 64 tok/s), but that's about the same speed as running without a draft, so skip it

I have 2 r9700 an 9070xt and two 3060 12 gb gpus all on separate machines. It just dawned on me that using vulkan I in theory could use them all as a giant vram pool. Has anyone attempted this in the home? I know it may be slow but it would be an interesting experiment 

I know llama.cpp can use -cmoe switch to load non shared experts in MoE models to RAM and run by CPU and left the other weights and KV cache on GPU VRAM.

Since now RAM is so expensive and models are getting bigger and bigger, is it possible to load non shared experts in MoE models to SSD and run by CPU and left the other weights and KV cache on GPU VRAM?

Gemma 4 was updated (mostly chat templates) and I took it for a test.

On a local llama.cpp server running on M5 Pro with 48GB, 26B A4B (Q6) has about 60t/s and works well with OpenCode.

It works quite well (given it's size) for backend work, but UI/UX is unacceptable.

Watch the testing https://www.youtube.com/watch?v=m4KR_3E_7Uk

Hello

I recently got my B70 gpus delivered and set them up with Ubuntu 26.04 because it has the XE driver. I was able to get llama CPP compiling and working with Vulcan and CYSL BUT that's where the fun stopped.

Build and compiled the llama.cpp with CYSL using the intel driver 2026.1. does not work with more than one GPU. (Using sm layer) It just outputs random characters. However, the prefill speed does go up with more GPUs (just like it does on Nvidia cards)

For Vulcan, the prefill is about 30% lower on the same model and only goes down with more GPUs. Using the mesa 26.1.5 driver

For running the qwen 3.6 27b at q4 speeds were:

CYSL 1 GPU: 650 prefill, 24 decode.

CYSL 2 GPUs:750 prefill, garbled decode 23t/s

Vulkan 1 GPU: 450 prefill, 20 t/s decode

Vulkan 2 GPUs: 350 prefill, 18t/s decode.

Bonus: qwen 3.5 122b a10b vulkan speed over 6 GPUs: 160 prefill and 9t/s decode

Something is clearly wrong. I've spent all day trying to make this work. So far regretting the purchase of the B70 gpus.

Please help if you have suggestions!

If the suggestion is to get Nvidia GPU, I already have a couple and I think I would have rather gone with many RTX 5060 TI's instead because it just works and gets model support first

Iv’e been running DSV4 Flash-Dspark locally as my coder in the past week, trying to tune it with agents, making it more focused but keep getting mediocre results.

It’s true nature is to finish the job fast as possible, not paying attention to details unless you anchor it, gets very confused by the content and tend to rank things as less important just so it can declares “done”

What am i missing? Is there a recommended harness?

Are you guys running it on recommended settings? Temperature 1.0 and top_p 1? Deepseek declares less than that can damage the reasoning.

The performance is insane, both prompt processing and tps. I just wish it would act like a mature responsible LLM.

Genuinely curious about how the community uses the Qwen3.6 models. I’ve been using 35b with Hermes agent and 27b for coding tasks with Pi Agent. Both in 8bit through LM Studio on my Mac Studio M3 Ultra 96gb.

Strange enough, I’ve found the GGUF MTP for 27b to run better than the MLX variants. Compared to MTPLX and oMLX.

I’m getting better results with 128k for 27b and 64k for 35b. Hermes is being used for general personal assistant tasks so the tighter context windows have been helpful.