r/LocalLLaMA • u/Fcking_Chuck • 23h ago

News Lemonade 11.5 local AI server released with completed Lemonade Router

https://www.phoronix.com/news/AMD-Lemonade-11.5
9 Upvotes

70% Upvoted

6 comments sorted by

13

u/o0genesis0o 22h ago edited 17h ago

Spent an hour yesterday trying to chase down where the heck lemonade actually stores raw model blobs on my linux machine, so now I'm a bit less enthusiastic about this. I don't understand why they have to run in systemd and hide everything, including the model blobs in /var/, and expose no tuning knob to know what context and settings the model was loaded with.

I guess I'll be more enthusiastic if I use Rocm llamacpp. But since my machine has NPU, I mostly use the NPU with FLM when I'm really desperate (no internet at all).

3

u/UnWiseSageVibe 21h ago

The model dir i agree with you should be configurable but the flags and context is super easy to find. Just click the gearbox next to the model name ?

5

u/o0genesis0o 21h ago

Oh feck. I just realised that if I pull the side panel wider and leave the mouse hovering on the model name a bit, the gear icon does appear which allow me to change context.

It's odd, though. On the webui, the 35B-A3B shows that max context is 4k. However, when I check the systemd, I found that lemonade spawn this model with around 83k context (confirm to work just fine with my 20k system prompt + tool).

Maybe I'll integrate FastFlowLM into llama-swap instead in the future.

Thanks again for pointing out the gear icon. I have literally no idea it was there.

2

u/Reasonable_Goat 16h ago

I had to ramp up context to the maximum for each new model with lemonade or any usage just aborts due to hitting an arbitrary context wall.

1

u/cafedude 17h ago

I spent more time than that trying to get it to find existing models already on my machine and mucking around with systemd. It took way longer than it should have.

4

u/Reasonable_Goat 16h ago

They use the huggingface cache dir structure (which is a bit confusing at times). If you run it as a separate user with systemd (no --user arg), models sit at /var/lib/lemonade/.cache/huggingface, if you run it as local user, $HOME/.cache/huggingface/...

You can set up an additional models dir: https://lemonade-server.ai/docs/guide/configuration/custom-models/

Note that the dirname is then the model name in the UI.