Since it might be helpful to some, here's my current commandline for llama.cpp running on an RTX 4090 with my monitor moved to the iGPU to free up all of its VRAM.
Identical to the qwen3.6 config. With a prompt like "svg owl" (which can reuse quite a lot compared with creative writing or similar, so ngram-mod shines), I get about 70-80t/s like this, with a memory overclock of about 1.5GHz
I absolutely love this comment. I wished there was a website where people would post their working command lines as well as what hardware they are using to run that stuff on + tokens / sec prefill + gen.
In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks.
> --spec-draft-n-max 5
5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.
The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though.
As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute.
Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is.
I think when tuning 3.6, I saw more like 33%? But not 100% sure.
I wonder if we could take a page out of the Solar power book and do MPPT but for draft count.
Constantly sweep through possible values to find the best result for the current conditions.
Though it might turn out that that doesn't offer any benefits. Has it been tried yet?
___
EDIT:
LLM pointed me towards this thing I'm not going to read
Don't you just need to occasionally try n+1 for your current setting? Then record acceptance rates. The nature of the thing is that if you try n, you already have a good sampling of how well m works for all m<n, right?
I have found that q8 kv quantisation is really bad in anything requiring some kind of literal retrieval even at relatively shorter contexts (5-10k). Eg it would mess up names, titles of papers etc that were deeper in the context.
The person above is running the OS from a separate GPU, so the command can use all of the VRAM of the 4090.
If you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.
Thanks for posting! Have you had any success with running without kv cache quantization? Is there a noticeable difference in quality without any? I would assume that would eat into context but 170k is pretty generous!
According to this shitty vibecoded thing "I" built https://hypfer.github.io/will-it-fit-llama-cpp/ (and I guess according to math too), FP16 K/V would give me something like 90k context at the same model quant, which doesn't really fit my usage.
But maybe someone else has experience to share there
just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort.
put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, taste, etc to create such a thing.
I generally agree and expect this to be the case from a legal perspective.
Legal questions of authorship are going to have to be established in terms of doctrines like SSO [0] and AFC [1]. Currently the incredibly sparse caselaw around this has yet to involve such non-literal notions of copyright.
So if I hire an artist and am a motivated individual, have an idea for a painting, have background knowledge about paintings and have taste in paintings and can offer a critique of the painting as the artist paints it, then somehow I created the painting?
I think in that case it's fair to say you created the painting with the artist, even if the artist should get majority credit. I don't like the analogy though, to me it feels more like you're a project manager directing a team of genius but single minded interns.
I've never encountered a project manager who has said: I built this thing, after a project has wrapped up. I have encountered ones who've said I managed this project.
I've heard some variant of "we built this thing", which I think is valid. I'm not sure how I feel about anthropomorphizing a coding agent at this point though, which is what you're implying.
Nothing absurd about that. What do you think an "Executive producer" is? A "Director" ? Does Peter Jackson get credit for creating the Lord of the Rings Trilogy films? Christopher Nolan for his films? But did he make them ? No, it was the collective effort of thousands of individuals all working under their direction.
Just like if somebody creates software today, and the end result is generated by the collective effort of thousands of agents, the "Director" still gets credit.
That’s an interesting way to look at it, and still generous to the llm.
By which I mean it puts the llm in the role of actor and such… people practicing an art. It seems harder to make a case that the llm is actually making creative decisions rather than aping a synthesis of past human decisions and expressions of taste, strictly, at every turn.
Though I guess it’s all “shades of grey”. I never felt like I was contributing a ton when I was essentially wrapping a large, capable library that some smarter person or people made, either.
Tangentially… remember back when we used to see headlines like “12 year old made a web browser!” and we all thought, “No. No they didn’t.”
I just read through a couple of your posts that weren't dead or buried, and it seems like you're pretty anti-AI. You should really start to have an open mind towards it. It's going to be the future (if it isn't already), and as you continue to get older, you're going to really wish you spent your time right now learning and embracing the technology instead of being so against it. A lot of the skills and things that you're holding on to right now might not be relevant by then, but you'll be at a disadvantage from not keeping up with the industry and need to play catch-up.
He still didn't create the paintings, he offered instructions / feedback to the artists that created them. Whether or not he is credited with authoring them is an entirely separate issue.
There have been plenty of workshops where artists hire assistant painters while maintaining authorship over the works themselves, from Rembrandt to Warhol to Hirst.
You may be able to run a quantized version. One strategy I've found effective is to set Claude or Codex loose on the problem and they'll do a decent job setting up the best possible model revision.
It depends on what model you’re running, and for what workload. For personal use (one or two convos at a time) with models that fit in gpu memory, pcie bandwidth doesn't really matter. Just try and be on gen 3 x8 or higher.
Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.
I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.
Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.
A hetero-GPU setup is definitely cost-effective if you don't strictly require the raw speed of a top-tier card like 5090. Just keep in mind that the total throughput will also be bottlenecked by the slower card.
To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP):
As opposed to loading it up in RAM + VRAM? Pretty much always better to split it up to multiple GPUs. My priorities are load up all available VRAM, then offload MoE experts to RAM (if possible), then offload other layers.
I use an RTX 3090 (24GB) and a GTX 1080ti (11GB). Just the 3090 for 3.8 I get maybe 60tg/s (UD-Q4 quant), for both I get around 40tg/s (UD-Q6). Not apples to apples though considering it's different quants.
Yes. I can split a model like this across 3 GPUs (a 1080 with 8GB and two Titan Vs with 12GB), and it's much faster than running it on 36 CPU cores. As long as it fits in aggregate VRAM, it seems very advantageous to do so.
Depends on your pcie connection. If they're both x16 then it's pretty low overhead, x8 is ok, but x4 is too slow. Also it's a bit tricky getting an optimal setups with mismatched vram, I think you could probably still make use of the full vram if you're clever but it's trickier.
for layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!
I haven't tried this either but I'm guessing if you could pool the GPU memory over whatever the kids are using these days, I think it was SLI back in my day. The GPU memory should still be faster than the RAM?
That's a very deep rabbit hole involving PCIe topology on both the hardware and software (NCCL) side, among other things. It's too system-specific to answer directly, but the entrance to said hole can be found at https://github.com/local-inference-lab/rtx6kpro/blob/master/... .
Disregard references to RTX 6000 cards, most of it is generally applicable to all multiple-GPU boxes.
Lol at that command. Why is this stuff so hard to run locally? I've spent a few days trying to figure it all out and haven't been able to. LM Studio doesn't work behind proxies. Ollama is confusing and doesn't seem to support Qwen3? And Llama.cpp is your command.
I just want to run `<some-command> <model-name>` with some default parameters set and for it to run locally.
What makes you say that it would be hard to do that?
It's long, I guess, but not cryptic.
You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away.
You could also just run -m <modelfile> and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers.
I think at least. I've never tried. My hubris stopped me from trying auto configs.
> llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
Why would they wish to handcraft this ? That is what agents are for ?
They could ask your current agent to a) search for this type of content online for the optimal setup for their hardware b) have the current agent/harness spin it up have it verify the config run few experiments.
Sure AI may make mistakes, or won't get the best possible config probably, but it certainly do a good enough setup, this is a task with feedback on whether the server crashed or poor performance easily measured so the agent can do a pretty good job.
Because this is kinda the one new thing that arrived in the technology scene, so getting at least some amount of understanding of its "inner" workings might prove useful in the future.
Beside that, it is also just.. interesting? It's fun tuning the machine to see it improve. For some, anyway.
The pain point they raised is this is too complicated for people who just want to get started, that is not true anymore.
It is certainly fun to fine-tune and setup if you like do something like that, however the need to do it hardly is a barrier for those who don't want complexity as OP imagines.
Lower level API/interfaces should not be a barrier for people if they are apply framing that way. More and more people are thinking agent native so this is not really a issue.
This is exactly it. I already have broad access to Claude, Gemini, GitHub Copilot. I want to use open models on automated tasks that chew up tokens but where I don't necessarily need the best-in class models and UX.
For Claude, I setting a single config file and then download and run Claude Code CLI. Even easier for the GitHub Copilot CLI.
Start by copying the command line from the Unsloth guides.
You don’t need to fine tune all of those parameters to get started.
It’s really easy to ask an LLM to adjust the command line if you can’t be bothered to read the help out. Copy the help output into the LLM and tell it your goal.
> Ollama is confusing and doesn't seem to support Qwen3?
Typing “Ollama qwen3” into Google takes you right to this page:
If even Googling for basic Ollama support is too hard, there might come a point where you have to acknowledge that local LLMs are not for you. None of this is really that hard with some basic Google bootstrap skills or by asking an LLM to help with the command.
There are easier ways to run it. OP seemed to enjoy tinkering and customizing the command to run it exactly the way they want. When I don’t want to tinker Unsloth Studio is probably closest to pick a model and voila.
Well there's a lot of knobs to turn if you want to improve performance. You can always point an LLM at the model card, give it your info, and have it write up the command.
Sure, but shouldn’t the programs to run the LLMs go “the user has this much vram and the model is this size, so I’ll start with sensible defaults based on that”?
> Every big company in the world uses a network proxy.
It's becoming more rare, now.
A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though.
The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and run executables of their choosing has to be small.
Woa TIL. I thought that was somehow long solved at the OS level or with VPNs or something like that (no idea exactly how, I'm sure just I'm misunderstanding something basic).
Makes it rather weird that LM Studio doesn't support it given how their target market, or well at least for their paid products, is very enterprisey.
I never install this stuff manually anymore. Just tell your LLM of choice to download model X from URL Y, build the latest inference engine of choice E, and then create batch files or shell scripts to run instruct and/or reasoning models in accordance with instructions at URL Z.
WOW, my first try running on my 2 3090's, it was a bit slow... but it FEELS like opus 4.5, i gave it an image and a broad overview of what I wanted it to build, and it built the whole thing from beginning to end.
I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.
Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.
We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.
For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.
This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.
As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.
4% is within the margin of error anyways for pass@1, so I think pass@k > 1 is gonna be the better indicator of any movement (still need to calibrate the optimal k to re-test). 10 seems too tolerant even though that tends to be the next tranche I reach for.
How much does it score though? 0% would be 4% less if Opus was at 4%. Unless you mean relative fraction not percentage points - but people usually mean percentage points in such situations.
> ...but no. They do not beat opus on real-world usage.
I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.
If you are working in a company and using language models, it is a very good idea to hold a bunch of evals you can trust and use to validate new models. Calibrate every once in a while with prod data. We have our own and the only numbers on quality and cost I trust come from this setup.
The only useful benchmarks are those you've created for your specific workflow. Only then can you assess whether a given model is better or worse for what you are using it for.
> but then we just need meaningful benchmarks that clearly show that!
That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria.
It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."
Yep. These small models are actually worse than GPT 3.5 at some tasks (like recalling facts). You can definitely make models smarter at specific tasks (like tool calling, coding) but you can't compress the entire human knowledge into a 30GB file. It's just not enough bits.
Models cant make any decisions if they have 0 idea that the feature exist in this language or in some general fact.
For example you would tell a model hey, become an expert in this language for me, search it online, it would still need to learn it and download the data to it's context and then increasing the memory usage, there's no way around it.
"Benchmark is stupid" and "model beats model on benchmark" are two different things, though. The second one is objectively true regardless of your views on the first one, right? To expect everyone to share your opinion that benchmarks are stupid is pretty weird, and just saying "no" to an objective truth is the definition of delusion.
Oh, goody goody goody! Can't wait for a Q4M with *MTP* support. Does anyone know if there are plans for it?
I am currently using Qwen 3.6 on RTX 3090 and I have to admit that without MTP it would be too slow to be acceptable for me (30-35 tok/sec without MTP, 60-70 with MTP). Without MTP I would just use OpenRouter and rather pay for speed despite having a capable local setup.
Unsloth usually also fixes the models when they bork something, which always happens. For Gemma for example the tool calling wasn't working for the longest time.
That kind of result makes me suspicious of benchmaxxing. Qwen 27B is 100x smaller than Opus 4.7. Is it really 100x more parameter-efficient? Two orders of magnitude is hard to believe. I don't have the hardware to run a 27B, but I'm curious what real world use is like. Maybe I'll have to buy some usage on a cloud provider to run my own tests, but this seems fishy to me.
Qwen small models are heavily coding focused, whereas Opus is everything to everybody (even if code is their bread and butter). The downside is they'll frequently hallucinate world knowledge so they need to be RL'd to double check their knowledge against sources and verify facts/library names/etc.
It's very agentic coding focused; and I'd say a good executor but certainly not Opus in scale; overall knowledge; long-horizon work and recovery; etc.
e.g. If you try to chat to it about something philosophical for example, or maybe a debate / creative writing, then you'll very quickly see how it is still a much smaller model at the end of the day.
Still, it's such a relatively accessible model to run, and I find a big part of leveraging smaller models is to give it well-scoped tasks; not too high level or ambitious ones. Very impressive for its size and the ability to run locally :)
In the local LLM communities there is a lot of respect for the Qwen models, but everyone comes to acknowledge that they do a lot of benchmaxxing after using them. Even at full precision they're never as good as models with similar benchmarks.
KL divergence is nothing close to a replacement for benchmarks. As flawed as benchmarks are, KL divergence is a barely useful signal. The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
You're very right about KL divergence. I spent a couple days playing with the Gemma 4 models. That's 10 separate models (varying weights, MoE, QAT or not, etc...) with identical tokenizers. I treated 31B at BF16 as the gold standard, feeding Wikipedia snippets, and anthropomorphizing a bit:
Gemma 4 31B: "Um, if I really said all of that, I guess I'd say this next"
Gemma 4 26B: "Dude, I would've said completely different stuff" (large divergence)
Gemma 4 12B: "Umm, there's zero chance I would've said some of this" (INFINITE divergence)
Gemma 4 E4B and E2B: "Derp derp, I'm happy to say almost anything" (lowest divergence)
For models which are chat trained, they simply would not recite Wikipedia, so the divergence is almost meaningless. I thought about capturing a realistic coding session and trying to use that as the corpus, but you need to preserve the turn-based tokens and such, so I moved on to other things.
> The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.
The talk around KL divergence is oversold. People talk about it as if it’s not a benchmark, but at its core it is in the same neighborhood! You get a different KL divergence number on different “calibration datasets”, so in other words it’s data-dependent. It is NOT a universal guarantee about the fundamental divergence of a model.
When I tested various eval benchmarks on Qwen3.5/3.6 27B with Unsloth's quants, the scores usually dropped 0-5% between UD-Q6 and UD-Q3 depending on the eval.
I hope really badly that we'll get a new 35B A3B or similar MoE model!
I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model <100B would be the sweet spot (when you have the VRAM but not the TDP or compute power). Heck, I'd gladly take A5B or A8B or even A10B as a sort of middle ground.
I'm hoping too that they'll put out some MoE variants.
Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model.
Edit: Like its predecessors, 3.8 seems really inclined to overthinking, and on a 27b dense model that's kind of painful.
I think I'm going to stick with gemma4:26b-a3b as my go-to because it runs about 4x as fast and tends to only need a fraction of the tokens in its 'thinking' stage to get the same or similar answer.
Probably helps it score a little bit better in benchmarks :) `medium` seems like a nice balance so far; along with some light steering to vary think effort as needed for task and being pragmatic.
Did you try the claude reasoning traces finetune for qwen3.6? I find that it works muuch better. I assume the same 3.8 finetune will be released at some pointas well.
Same here! Qwen3.6-35B-A3B is the only local model I've found that runs reasonably on my iGPU. Looks like me and and my noisily-wheezing laptop will be sitting out this upgrade.
[self-reply because comment edit window ended]: I now have a head-to-head benchmark. On my wheezy laptop (specs in sibling comment), this new dense model, Qwen 3.8 27B, gets ~4 tokens/second on generation. The older mixture-of-experts model, Qwen 3.6-35B-A3B, gets ~20 tokens/s.
MoE is literally 5x times faster (on CPU) than comparable dense Qwen!
Don't mind! It's 64 GiB dual-channel DDR5-6400, i.e. roughly 100 GiB/s of bandwidth. (AMD 7840U (Zen 4))
I'm using a Q4 quantization from unsloth (Qwen3.6-35B-A3B-UD-Q4_K_XL). It gets up to ~20 tokens/second in generation. I don't know precisely how much KV cache I can safely use, but it's in between 140k–256k. (I.e., 140k reliably works, 256k kernel-crashes from OOM. Don't feel like bisecting).
Inference is llama.cpp with the Vulkan GPU backend on Linux. (I.e., -DGGML_VULKAN=1 on the llama.cpp build, and --gpu-layers all on llama-cli or llama-server. (And for my specific setup, two kernel parameters specific to amdgpu: ttm.pages_limit and ttm.page_pool_size. A driver VRAM limiter. Look it up if you're on amdgpu!)).
Thanks! I don't have any knowledge of running models locally.
I assume it would not be able to handle an unquantized Qwen3.6-35B or is it irrelevant as you almost always would want to run a quantized version of the model on consumer hardware?
not parent, but 4-bit quantization is generally consider a good trade off for speed/performance, so you might use it even when you aren't on consumer hardware, but definitely when you are on consumer hardware.
Me too. 35B A3B runs really fast on my MacBook Pro (M4 Max) and is suitable for real-time tasks like dictation post-processing. The dense model is not.
I'm still confused about Qwen 3.6 35B A3B. Everything I read said that the 27B model performs better at coding tasks, so what's the purpose of the 35B model?
The "a3b" refers to its active parameters -- unlike 27b it is a mixture of experts model, so it runs much faster, about as fast as a 3b model, but needs as much memory as a 35b model! So good for unified memory systems like macs :)
anecdotes: 35B-A3B does want more memory, bigger model. But if you get it running it will be faster and more enjoyable to use -- text will fly by -- due to only 3B params being active, in my experience at least.
Total param count decides how much vram you need to run it. Active param count decides how fast it runs. My 10 year old GPU can load quantized 35B or 27B, but it can’t process 27B parameters per token faster than 2-4tok/s, while it can do A3B at >40tok/s
People will claim it's not comparable to Opus despite it beating the score. I'm not sure I disagree, but I'm also unsure whether I care. Most new models nowadays are "good enough". I cannot complain because I'd rather spend that time improving my prompts and docs. Opus might be a _slight bit better_ at picking up vague hints, but it's also extremely expensive, and I hit the 5 hour limit way too quick.
I care a lot about speed and efficiency right now. For my setup I would like to have 2-3 different model families. I've settled on GLM-5.3 (formerly Deepseek v4 pro 0813) for architecting, Deepseek V4 Pro 0813 for developing, and Gemini flash lite (any recent cheap model) for repo scouting. I'll add another one in the mix for reviewing (in this case Gemini 3.7) and that's all I need.
I've tried most models except Grok.
Qwen is too expensive IMO (Alibaba Cloud subscriptions are hard to come by and I'm not spending 50 euros a month for a tool, so 18 euros it is). If it ever becomes efficient enough to run locally I will definitely look back.
Claude is slow and expensive (the cache hit prices are absurd).
OAI is pretty good, I might add it to my arsenal seeing how cheap it is.
These opinions change every day. Last week I would've never picked Deepseek until I read about the pricing. even post aug 16 it's worth it (although it's getting close to gemini pricing).
Right now my costs are 12 euros a month (z.ai) + whatever deepseek consumes. This typically isn't more than 8 euros a week. 44 euros a month and I have a setup that is doing pretty well.
Which itself released yesterday? You're writing, reading, and evaluating enough software in a ~36 hour period to form, reject, and form another opinion about which model makes better _architectural_ choices?
Honest question, how do you assess models this quickly? What metrics are you using? Would love to get my suite from multiple days and hundreds of prompts down to minutes. Got a few first pass tasks I run upon release for an initial experience, but those only work because even Fable and Sol fail despite objectively correct solutions existing, so it works because most models fail, but then, those are consciously not enough for coding, tool use, adherence or task specific inference and assessment…
No serious person or sane person uses the LLM that's constantly being tweaked by an anti-woke white-genocide-supporting weird little man. Don't feed the totalitarian wannabe's (or the totalitarians in general, for that matter).
I would say it is sad that there are people who use Grok when there are so many other choices available which don't come with the issues of supporting Musk.
It is not all just 'politics'. Take a stand on some issues. It doesn't cost much not to use Grok.
There are a lot of people who are apathetic to what musk is, people that don't care are not people who should inspire you. What the hell is so inspiring about apathy anyway?!
And yeah, people that don't care DO make the world worse through their apathy.
i care about not financially supporting a person that is actively trying to disenfranchise me, why is that a difficult concept for some people? that not everyone is motivated exclusively by financial profit? is moral bankruptcy so pervasive that some people assume it is unanimous?
I'm convinced a lot of the anti-open-weight model comments at this point are inorganic traffic - there's trillions in investor money riding on a world where these models aren't cheap commodities. Having actually used things like the recent GLM, Kimi, and Qwen I think any edge the labs have is marginal at most and actually prefer the open weight models in most day to day usage.
Anthropic's recent releases are wordy to the point of exhaustion. Every time I use opus recently I find myself wanting to yell "GET TO THE POINT" at a terminal, which is exacerbated by it being slow.
I wonder if Anthropic and OpenAI possibly missed the window to go public. A 27B open-weight model trading blows with the SOTA from just half a year ago is not great news for trillion-dollar investments...
Trading blows in some benchmarks is a bit exaggerated. If you try the model, `xhigh` is basically feels like the `max` mode (i.e. massive thinker and extremely presistent), and the amount of world knowledge and intent understanding is nowhere close to an Opus class model even from 6mo ago.
It's still very useful, and it'll probably displace a good bit of API spend; but it's not really "trading blows with SOTA from just a half year ago". A bit overblown on the Anthropic/OpenAI has missed the window I reckon.
Also, on the open weight frontier side, Kimi K3 is pretty expensive, and Deepseek V4 Pro/Flash is getting a little less juicy with price increases.
Even Zuckerberg isn't as deluded to think that there's money in constantly releasing closed models and hypserscaling their deployment, which is why we got muse-glimmer 30b last week. I have been playing with it on a 32gb MBP w/ M2 Pro. If prosumer-grade hardware from almost three years ago can now offer a novice a way to host a possibly serviceable software development agent, then there's little incentive to pay out for LLMs like Anthropic's or OpenAI's over the long run.
If the benchmarks don't lie, this is getting very close to Opus 4.6 capability - which was the turning point for me for when AI was "good enough" that it became very hard to justify not using it.
I'm sure there's some benchmaxxing going on, and some things you get only with a a larger model.
But I'm feeling pretty confident if not by Gemma 5 than by mid 2028 we'll have local models that are almost always as good as Opus 4.6 was and in many cases far better.
Similar to the way they asked Sol to solve Erdos problems, that's what I want my model to do for programming.
I don't want to try to take my best educated guess at what the best design is BEFORE implementation - especially if you're designing a feature for a codebase you're not an expert in, you don't know like the back of your hand (i.e. one that is mostly or entirely LLM generated).
What sounds good on paper - often times becomes unideal in practice when you get to the reality of implementation.
It may not be worth re-architecting your entire system to get to a "pure" design that would be the best - all things considered.
Instead, I'd like the model to independently design many plausible and coherent good solutions, then implement each of them, then intelligently pick the few winners (after its fixed any bugs that could be causing promising solutions to look artificially bad) - unless there's an obvious one - and then give me the data I need to make an informed decision on which one to go with, all before I even look at the design or implementation.
You're not getting this from a one shot prompt from a 30B model today. You can't even really get it from Sol or Fable - IME. But you can get somewhat close.
Even Fable is bad at this, I would constantly have to fix it going down architectural dead ends or just making obvious mistakes.
Which sucks for people that want LLMs to do everything like a genie, but does mean senior engineers have a few more years before they become redundant.
This is a harness problem not a model problem, try prime agent it can do that and it will do it well even :P but you need to prompt it in according to its tools and processes.
IME using 5.6 Luna and DS V4 Flash, I notice that although they are excellent at programming, even Opus-like in the way they try to debug, the thing they are worst at is inferring user intent and making good decisions with little information. They are absolutely terrible at that, will misinterpret small wording ambiguities. I suspect that's an ability you can't add with RL training, that it requires the depth of understanding from vast pre-training.
Asking it factual information[1]. You just can't compress the entire human knowledge into a 30GB file.
[1] Without searching the internet. And even if you allow it, you'll get much worse results because search means browsing and parsing the top results, and search results are horrible, whereas internal knowledge from training encompasses the entire internet plus all books including very niche stuff.
I'd love to understand this more. Are you saying the Qwen team spends their very impressive human and compute resources on publishing these amazing models and then botches the chat template with mundane bugs?
Like maybe I just misunderstand what's the hard part but wouldn't you assume that people who can put together an impressive model can also write a proper jinja chat template for it?
Yes for the first question. Google of all companies didn't even get it right with Gemma for a while until recently. For some reason it doesn't seem like people can actually get these templates right.
Apart from model performance, what harness are people using to come close to Claude Code or Codex workflow styles with tool use, conversations, loops, remote control, etc.?
One thing a lot of people don't seem to factor when hyping Qwen is how much models like this tend to 'overthink' with seemingly endless 'second guessing'. 3.8 seems no different from what I've tried thus far.
As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much higher tokens/second, and take a small fraction of the time. I suppose 'YMMV' depending on your use case.
Also, I haven't used it enough yet to see if it's prone to infinite looping, but its predecessors sure were.
Given that it apparently defaults to 'xhigh', this is probably the answer.
Granted, it's still much lower tokens/s than you'll get out of many MoE models.
Edit: Even set to medium or low there's still a lot of second guessing, less consistency, lower 'acceptable response' rate, and slower/more token churn vs gemma4:26b-a3b
Use 3.6 27b as a daily driver for months with charmbracelet crush. Gemma 26b-A3b is not even remotely comparable in terms of coding for me. YMMV depending on how you work, what harness you use, etc I suppose.
I hope the bonsai team makes another 1bit quant of this model (or releases code/instructions on how to do it), using the Qwen3.6 27B on my 16GB mac mini has been wild . The 1bit quant feels like opus level… for the first couple turns. Then it has trouble eg switching from plan mode to act mode. This is mostly mitigated by starting a new session. (tbf this limitation is called out on the hf page)
I saw unsloth has 1bit quants too so I might check that out, anybody have experience with those?
100k is all the context I have ram for, this is with any auto-compact turned off. This is using Cline in vs code. I’m sure I could tune the system prompt and mode switching more to work better with this specific model, but I haven’t gone down the custom harness rabbit hole yet.
And this is also specifically for the 1bit quant version. I don’t think the fp8 or even fp4 versions have this issue, but I haven’t tried those much
Any tips on the best approach at running this at an M4 Max 128GB? Token throughput was a bit slow with the last 27B one (MLX), ended up using the A3B variant but if I could get this one to reasonable speed I'd much prefer it.
I have an M4 Max (unfortunately 64GB). I have been running the Qwen 35B A3B one for a while now, after testing and benchmarking a number of models. That one was consistently the best in class for tasks like despamming, E-mail classification, OCR and dictation post-processing. It was also really fast (90 tokens/s).
I'm benchmarking the 3.8 model now, it seems it is better (near-perfect score on my E-mail spam filtering benchmark, best of any model I tested, ever). But it is slow.
One thing I would recommend is keeping an eye on MTP parameters. I tested and benchmarked extensively, and I use `--spec-draft-n-max 2` with llama.cpp. Longer sequences actually decrease overall performance.
As for running, I ended up using llama.cpp and its llama-server, with a bunch of scripts written by AI, because I got tired of LM Studio not implementing the image-related parameters which made gemma4 useless for OCR.
Go for a slightly more quantised version, and experiment with different MTP settings. I find that MLX versions are marginally faster on my 64GB M1 Max, but I usually use Unsloth's GGUFs via llama.cpp as there's a much greater range of quants available and I prefer llama.cpp. MTP sometimes also helps a little, but I suspect it's less helpful on my system than others.
This is the way if you need speed. It costs a little bit in smarts, but compare the MTPLX option listed above with the oQ4e-mtp quant using oMLX. The good cacheing layer in oMLX will help things feel faster for some classes of tasks in my experience.
Wait for the MTP variants that will likely be out within days. I'm on a 128GB Strix Halo box and for 3.6-27B 8bits I was getting about 9tok/sec (not great). With MTP that gets closer to 18 tok/sec (kind'a usable).
27B is a dense model so it will be slower with an MoE (A3B), but should have better quality? I still haven’t found very good uses cases on my M3 Max for dense models. Even if you can find a MTP version, it doesn’t help much, especially if you compare against an MoE with MTP as well.
Unfortunately, that chip just doesn't really have the memory bandwidth to run this (or nearly any) model at acceptable speeds. I have the exact same chip (M4 Max 128GB) and I've been trying to optimize a completely purpose-built implementation with Fable and this is just not possible. Even if you could reach the full 576GB/s, it's just physically impossible to exceed these numbers with the model's architecture:
2 bpw - ~85.7t/s
3 bpw - ~58.0t/s
4 bpw - ~43.9t/s
6 bpw - ~29.5t/s
8 bpw - ~22.2t/s
16 bpw - ~11.2t/s
without cheating. You'd have to exclude layers, skip operations, etc. basically do stuff the model wasn't trained for. And speed collapses so fast with context that even 2 bpw would be looking at ~37.6t/s after just 128K tokens.
MTP only improves the situation by up to 2x in the ideal case, while drastically reducing the performance floor. While optimizing a 9B model on this hardware, I've found that the GPU just doesn't have enough FLOPS to handle speculating more than one or two tokens ahead on a single stream, regardless of quant level, simply because of the arithmetic cost of the forward pass. The 27B model would be even more expensive than that, potentially such that it's already bottlenecked by the GPU itself rather than memory.
I wouldn't get my hopes up for the 35B-A3B either. Not only is it reportedly much less intelligent, but I hit a similar ~85t/s wall in practice (again with highly specialized inference).
Without speculation I can reach around 120t/s on Qwen3.5-9B and with n-gram speculation (not even MTP; this derivative didn't come with one) around about 150t/s on average. This is on the very very edge of what I'd consider acceptable for me to even consider using such a compact model. YMMV due to the silicon lottery but the situation isn't good.
If the benchmarks are a real indication, we now have a local model that is runnable on a high-end personal PC that trades blows with the leading model Claude Opus 4.6 Max from half a year ago.
The $1500 Intel B70 with 32GB of VRAM can run this model at max context with good performance, btw. If you don't want to drop $5-10k for running DeepSeek this is your best budget option for local refactor/small scale dev help
I understand the B70 is a bargain vs AMD and especially nVidia offerings, but to me it feels like I would be buying something that would feel too limited in less than a year. 48G would be much more confortable.
And I know the 96G nVidia cards are selling for over 10k$.
You can buy two b60s for $1300 right now (650 each) if you want a total of 48gb. Intel recently raised the price on all of their gpu's except the b60 series, so they are currently the best deal per gb I think.
32GB is perfect for models around 30B parameters. Since qwen has really hit the spot with their 27B dense models, I think it's a good bet. Also, 32GB is enough for other tasks such as image/video generation and loading multiple smaller specialized models
Sorry, I wrote autocompletion by force of habit. I simply meant it can complete code you have already created a structure for, which personally is very nice
I thought so, but thanks for the clarification. I am a little bit disappointed that local autocompletion models have been left by the wayside in favor of models post-trained for agentic coding. Both Codestral and Qwen-2.5-coder are more than a year old at this point, but local auto-complete seems to me to be such a great usecase.
The file "Just loads" on llama.cpp, the Unsloth https://huggingface.co/unsloth/Qwen3.8-27B-GGUF is an MTP file, I see mostly the same speed on pp and generation.
There has to be something wrong with those benchmarks, I find extremely hard to believe a 27B model can work similar or exceed opus 4.6.
Worth distinguishing knowledge/task benchmarks from IF / agentic. It doesn't seem out of the question that you can have a small model thats generally good at instruction following and long-horizon agentic, as usually in those cases any requisite knowledge is in the context.
Most of the benchmark improvements afaict are in agentic and instruction following benchmarks.
I think you've been drinking the "LLMs only improve by adding parameter counts" that SOTA labs are selling VCs to build data centers so they can keep eating through cash to their own benefits.
To the countrary, the reason Chinese models are excelling in the smaller area is because there's tons of fat in closed source models because of the crazy cash being thrown around.
There absolutely is space to improve intelligence and capabilities without lathering on more and more parameters.
I think I am going to buy a second rtx 3060, as 27B has been just outside of my range for to long, and this looks like the parameter count tipping point
whats the tok/s you get on that. I have heard a few claims of around 30-50 with mtp, but for how cheap the setup is I am surprised I don't hear more about 3060 stacks so I assume there has to be some catch.
I used GPT-5.6 Sol high to optimize it, and it claimed it was getting 50. I'm seeing ~40 on my goto smoketest: "Make me a vector add in CUDA".
Funny side note. It successfully one shot the program, but it wasn't able to run it because there literally wasn't enough VRAM left to allocate CUDA memory. Watching it try to debug that was fascinating. I'm pretty sure it would have killed the llama-server (and thus itself) if it hadn't been running in a separate container.
Why is anyone even using video cards these days? You may as well be burning cash.
This is the perfect candidate for just splattering it on your nvme and then reading it off there and into memory. All of these run perfectly fine on simple m4 silicone:
Yeah, I would appreciate if someone could make sense of the pricing differences between these models. How can a provider run DSv4F at lower cost than a 27B dense or 35B A3B model?
Does it come down to utilization and/or specific model tricks and efficiencies (attention, kv cache, etc.)?
can you tell me ideas of usecases of 9 or 10B language models ? I cant find any usecases other than training a lora on them to give good bash commands for example
Qwen3.6-27B has been the main LLM powering my little agentic stack. I have adopted the test and verify approach to any models allowed to run on my machine. When the "heretic" version drops, I will fire up the harness and test. Super excited to see how it stacks up against Qwen3.6!!!
Model benchmarks are useful, to a point, but it is the long tail of things you do with the model that determines if it's good at a wide range of activities. Ant/OAI, to their credit, build their models -- even the small ones -- so they follow instructions and do tool calling well, without the system prompts confusing them. This is especially important for long-horizon tool calling.
So one open weight model might "meet" Opus or whatever on benchmarks, but then fail to follow a simple answer format and also tool call correctly. The models are whipped to within an inch of their lives to strictly adhere to their post training quality gates.
Architecture thread! Afaict they continue to use gated attention + delta net, which was also adopted+adapted by K3, but im surprised theres no improvements to the residual stream (deepseek are using manifold hyper-connections, kimi have attention residuals) ?
I wish this can run directly on my RTX 4090, seems like 30B is the sweet spot for dense model to run locally, sadly RTX 5090 is very expensive and I need a new PC and new power supply(and UPS) to run that, adding a second RTX 4090 is another option, but not sure if my PC can do that yet.
even a 3090 will give you the VRAM headroom. i run Q8 on an 3090/A6500 combo. well, Q8 of 3.6-27B. I'm building the Q8 GGUF for 3.8 now, assuming mine will finish before someone else's.
I use the Qwens as a vision model for my DeepSeek V4 Flashes to handle. But the Qwens run on old RTX A6000 Ampere. Does anyone know if there's any news about INT4/AWQ quants for the RTX A6000?
Same, with a multimodal DSv4 Flash I would just stop paying attention to things. Very smart, and at 260 tok/s it's too fast to care about anything else. If you ever graft something like that I would love to hear about it.
Yes, I have a very dumb flow. The harness has a describe_image tool that takes an image and a prompt and so DSv4 Flash uses it to get an idea of what it's looking at.
Yeah, I might just replicate what you're doing. Main issue right now is just finding spare vram to actually run another model in parallel... And yeah, if I train up a vision adapter somehow, I'll try to put it up/post about it, seems like we're getting the killer apps for local LLMs right now, where it's just feasible enough if you're enthusiastic enough to be a bit economically irrational, and just useful enough to sort of rationalize.
Why don't qwen/alibaba host the model themselves? I was looking forward to trying it on their coding plan. Google are the same way with their Gemma models.
Can anyone who has that specific personal test he tries on different models , and tries this model , to tell us here if possible , how good or bad is this new model ? compared to others ?
There is a down to earth guy on YT that performs a series of tests against LLMs running on non-god-tier commodity hardware. He will likely be testing this soon enough.
So much potential for that channel. He's got a nice range of tests and a no nonsense presentation style.
However, watching tests of heavily quantized models that weren't designed for it (non-QAT) is frustrating. There's no way to tell if the actual model fails because it's dumb or if the lobotomy made it that way.
strong visual reasoning apparently, which is nice. still lacking native audio however. hoping for more companies to embrace the spirit of something like `gemma-4-12b-qat` for actual multi-modality (text, image, video, audio).
Since Qwen 3.6 27b outperforms Gemma 4 26b in most benchmarks I'm not sure the value - also Gemma 26b is a MOE model whereas this is a dense model, so not typically direct competitors at their sizes - Gemma 4 31b comparison would be interesting though.
I don't know much about the production of these models. How hard would it be to 'fork' something like this and have it not be full of CCP indoctrination?
What a weird mechanism. If someone is judging a thread/topic/event impact by the number of points it got, then doing this unfairly degrades that thread.
It should have deduped by user and combined the 168(at the time of writing this comment) + 289 points. Just add the twitter link from the previous thread as an additional link in the description, like you normally do, move all the points over, and remove the old thread.
I'm hoping to see folks distill this with current generation Opus / Fable reasoning traces. I have had my best results locally so far from Qwopus (Qwen 3.6-27B w/ Opus 4.6 reasoning distilled). This looks GREAT and I am definitely setting this up later today.
I found this kind of amusing while running it (using Pi as the harness). Don't know if this is evidence of intense fine tuning from Claude but it smells like it...
" The user wants me to explore the repository at XXXX and report back. Let me start by understanding the project structure, reading the
CLAUDE.md file, and getting a general overview of what this repository is.
Let me start by reading the main project documentation and exploring the directory structure.
I'll take a look around this repo. Let me start by getting a lay of the land.
read resource CLAUDE.md (ctrl+o to expand)
ENOENT: no such file or directory, access 'XXXX/CLAUDE.md'"
llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off
Identical to the qwen3.6 config. With a prompt like "svg owl" (which can reuse quite a lot compared with creative writing or similar, so ngram-mod shines), I get about 70-80t/s like this, with a memory overclock of about 1.5GHz
In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks.
> --spec-draft-n-max 5
5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.
The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though.
As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute.
Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is. I think when tuning 3.6, I saw more like 33%? But not 100% sure.
Though it might turn out that that doesn't offer any benefits. Has it been tried yet?
___
EDIT:
LLM pointed me towards this thing I'm not going to read
https://arxiv.org/abs/2512.11280
"AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference"
Some higher level nerd please go have a look at that. I have important meetings and golf classes to attend.
https://arxiv.org/abs/2607.05147
But they do so to maximise total throughput, I don't think there's reason to do that for batch=1.
$ build/bin/llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off
0.02.993.689 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 911.53 MiB on device 0: cudaMalloc failed: out of memory
Update: Oh, it works after I stop Xorg. But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?
If you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.
But maybe someone else has experience to share there
put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, taste, etc to create such a thing.
I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!
Most managers do though?
Legal questions of authorship are going to have to be established in terms of doctrines like SSO [0] and AFC [1]. Currently the incredibly sparse caselaw around this has yet to involve such non-literal notions of copyright.
[0] https://en.wikipedia.org/wiki/Structure,_sequence_and_organi...
[1] https://en.wikipedia.org/wiki/Abstraction–filtration–compari...
None would claim they chiseled anything, if it was 3D printed. They may claim they designed something.
Absurd logic. The AI built the website.
Just like if somebody creates software today, and the end result is generated by the collective effort of thousands of agents, the "Director" still gets credit.
By which I mean it puts the llm in the role of actor and such… people practicing an art. It seems harder to make a case that the llm is actually making creative decisions rather than aping a synthesis of past human decisions and expressions of taste, strictly, at every turn.
Though I guess it’s all “shades of grey”. I never felt like I was contributing a ton when I was essentially wrapping a large, capable library that some smarter person or people made, either.
Tangentially… remember back when we used to see headlines like “12 year old made a web browser!” and we all thought, “No. No they didn’t.”
It all feels a little like that, again.
For example, of his 1500 spot paintings, he only actually made 5 of them.
It's not uncommon at all for artists to work this way.
llms are not human.
If you've tried it, what worked well and what didn't? I'm especially interested in mismatched VRAM setups, e.g. a 16 GB GPU + a 24 GB GPU.
How much overhead did you see from inter-GPU transfers, and did the extra usable VRAM outweigh the performance hit?
Llama is decent at auto optimizing it if you let it use both gpus. It’ll split the workload so the contiguous layers are all on one gpu. Once the model is loaded, you only transfer weights between gpus once (per token?), at the layer boundary.
I run on an 8gb 3070 and 12 gb 3060, and the only weird thing is that the weaker card gets more layers (and therefore work) because it has more ram.
Oh, if you’re barely fitting the models into your vram, you may need to explicitly adjust the layer balance between cards — sometimes it fails to realize it should have put certain things (like draft models) on the other card so you can fit one more layer in.
To provide some anecdotal data, here is how my 5090 + 3060 setup performs with Qwen 3.8 27B (Unsloth's UD-Q4 with MTP):
For reference, here are also some numbers from my 4060ti + 3060 (16GB + 12GB) setup. [0][0]: https://news.ycombinator.com/item?id=48700091
I use an RTX 3090 (24GB) and a GTX 1080ti (11GB). Just the 3090 for 3.8 I get maybe 60tg/s (UD-Q4 quant), for both I get around 40tg/s (UD-Q6). Not apples to apples though considering it's different quants.
Here's my config: https://gitea.va.reichard.io/evan/nix/src/branch/master/modu...
Disregard references to RTX 6000 cards, most of it is generally applicable to all multiple-GPU boxes.
I just want to run `<some-command> <model-name>` with some default parameters set and for it to run locally.
It's long, I guess, but not cryptic.
You tell llama server where the model is, which context size to use, what to use for the K/V cache quant, that it should do MTP, tune some MTP parameters, and that's kinda it.
Perfectly logical blocks with all the model-specific weirdness (that does exist!) abstracted away.
You could also just run -m <modelfile> and let llama-server do the right-ish thing. The defaults are probably fine, but not how you squeeze out these exact numbers. I think at least. I've never tried. My hubris stopped me from trying auto configs.
isn't that the hard part? You know the ballpark ideal values for these many parameters since you're a knowledgeable expert but the vast majority of people are just like "I want AI" and have no idea what all the jargon even means.
Otherwise, if you're a programmer setting up a local harness, it only takes like 20-30 minutes to learn what the right parameters are.
It's very model, hardware, and use case dependent which is why a one size fits all solution doesn't work
They could ask your current agent to a) search for this type of content online for the optimal setup for their hardware b) have the current agent/harness spin it up have it verify the config run few experiments.
Sure AI may make mistakes, or won't get the best possible config probably, but it certainly do a good enough setup, this is a task with feedback on whether the server crashed or poor performance easily measured so the agent can do a pretty good job.
Because this is kinda the one new thing that arrived in the technology scene, so getting at least some amount of understanding of its "inner" workings might prove useful in the future.
Beside that, it is also just.. interesting? It's fun tuning the machine to see it improve. For some, anyway.
The pain point they raised is this is too complicated for people who just want to get started, that is not true anymore.
It is certainly fun to fine-tune and setup if you like do something like that, however the need to do it hardly is a barrier for those who don't want complexity as OP imagines.
Lower level API/interfaces should not be a barrier for people if they are apply framing that way. More and more people are thinking agent native so this is not really a issue.
Why this headache inducing lingo tho? What does that even mean, and why should I sign up for your webinar about that?
For Claude, I setting a single config file and then download and run Claude Code CLI. Even easier for the GitHub Copilot CLI.
You don’t need to fine tune all of those parameters to get started.
It’s really easy to ask an LLM to adjust the command line if you can’t be bothered to read the help out. Copy the help output into the LLM and tell it your goal.
> Ollama is confusing and doesn't seem to support Qwen3?
Typing “Ollama qwen3” into Google takes you right to this page:
https://ollama.com/library/qwen3
If even Googling for basic Ollama support is too hard, there might come a point where you have to acknowledge that local LLMs are not for you. None of this is really that hard with some basic Google bootstrap skills or by asking an LLM to help with the command.
You could override, obviously.
Woa, is that still a thing? You mean like SOCKS5 stuff that you have to manually configure in every application that uses the internet?
I mean maybe I'm just living under a rock but I feel like that's a rather niche situation you got there.
Every big company in the world uses a network proxy. LM Studio, as far as I can tell, cannot be configured to work behind such proxies.
It's becoming more rare, now.
A lot of the universal truths about corporate networks from the early 2000s are no longer true today. Some companies are stuck in their ways though.
The overlap between companies that require someone to use a network proxy and companies that have GPU-equipped machines with enough RAM for LLMs and and that allow people to download and run executables of their choosing has to be small.
Makes it rather weird that LM Studio doesn't support it given how their target market, or well at least for their paid products, is very enterprisey.
https://avilpage.com/qwen-3.8-27b.html
Unsloth's GGUF quants are up: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.
Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.
We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.
For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.
This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.
As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.
Okay but the parent said real-world usage, presumably meaning coding tasks.
We have a whole bunch of complex evals that Haiku 4.5 passes. That doesn't mean it is a good model for coding.
0% is 4 percentage points (pp) less than 4%.
I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.
There are tools like promptfoo designed for this.
That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria.
It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."
For example you would tell a model hey, become an expert in this language for me, search it online, it would still need to learn it and download the data to it's context and then increasing the memory usage, there's no way around it.
deepseek-v4-flash needs web search to return true facts.
I am currently using Qwen 3.6 on RTX 3090 and I have to admit that without MTP it would be too slow to be acceptable for me (30-35 tok/sec without MTP, 60-70 with MTP). Without MTP I would just use OpenRouter and rather pay for speed despite having a capable local setup.
So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.
Run the one you linked if you are running vllm (safetensors)
e.g. If you try to chat to it about something philosophical for example, or maybe a debate / creative writing, then you'll very quickly see how it is still a much smaller model at the end of the day.
Still, it's such a relatively accessible model to run, and I find a big part of leveraging smaller models is to give it well-scoped tasks; not too high level or ambitious ones. Very impressive for its size and the ability to run locally :)
It's a bit bare at the moment, I assume they are going to add further detail later (eg comparison to other quants), similar to their other releases.
Sometimes they're just slow and expensive, so we we KLD as a proxy measure and it's very high correlation (95%+)
Gemma 4 31B: "Um, if I really said all of that, I guess I'd say this next"
Gemma 4 26B: "Dude, I would've said completely different stuff" (large divergence)
Gemma 4 12B: "Umm, there's zero chance I would've said some of this" (INFINITE divergence)
Gemma 4 E4B and E2B: "Derp derp, I'm happy to say almost anything" (lowest divergence)
For models which are chat trained, they simply would not recite Wikipedia, so the divergence is almost meaningless. I thought about capturing a realistic coding session and trying to use that as the corpus, but you need to preserve the turn-based tokens and such, so I moved on to other things.
Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.
I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model <100B would be the sweet spot (when you have the VRAM but not the TDP or compute power). Heck, I'd gladly take A5B or A8B or even A10B as a sort of middle ground.
Also alternate link for viewing the images without signing in: https://xcancel.com/Alibaba_Qwen/status/2088280182356611304
Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model.
Edit: Like its predecessors, 3.8 seems really inclined to overthinking, and on a 27b dense model that's kind of painful. I think I'm going to stick with gemma4:26b-a3b as my go-to because it runs about 4x as fast and tends to only need a fraction of the tokens in its 'thinking' stage to get the same or similar answer.
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Edit: link - https://huggingface.co/rico03/Qwen3.6-27B-Claude-Opus-Reason...
MoE is literally 5x times faster (on CPU) than comparable dense Qwen!
Tested:
Qwen3.6-35B-A3B-UD-Q4_K_XL
Qwen3.8-27B-UD-Q4_K_XL
I'm using a Q4 quantization from unsloth (Qwen3.6-35B-A3B-UD-Q4_K_XL). It gets up to ~20 tokens/second in generation. I don't know precisely how much KV cache I can safely use, but it's in between 140k–256k. (I.e., 140k reliably works, 256k kernel-crashes from OOM. Don't feel like bisecting).
Inference is llama.cpp with the Vulkan GPU backend on Linux. (I.e., -DGGML_VULKAN=1 on the llama.cpp build, and --gpu-layers all on llama-cli or llama-server. (And for my specific setup, two kernel parameters specific to amdgpu: ttm.pages_limit and ttm.page_pool_size. A driver VRAM limiter. Look it up if you're on amdgpu!)).
I assume it would not be able to handle an unquantized Qwen3.6-35B or is it irrelevant as you almost always would want to run a quantized version of the model on consumer hardware?
on the 395+ AI MAX w/128GB, the A10B qwen 3.5 can do a lot of long running work if you don't need to baby sit it. deer-flow works well like that.
Whats up with focusing on the active param count? Do yall fiddle with the weights or something?
I care a lot about speed and efficiency right now. For my setup I would like to have 2-3 different model families. I've settled on GLM-5.3 (formerly Deepseek v4 pro 0813) for architecting, Deepseek V4 Pro 0813 for developing, and Gemini flash lite (any recent cheap model) for repo scouting. I'll add another one in the mix for reviewing (in this case Gemini 3.7) and that's all I need.
I've tried most models except Grok.
Qwen is too expensive IMO (Alibaba Cloud subscriptions are hard to come by and I'm not spending 50 euros a month for a tool, so 18 euros it is). If it ever becomes efficient enough to run locally I will definitely look back.
Claude is slow and expensive (the cache hit prices are absurd).
OAI is pretty good, I might add it to my arsenal seeing how cheap it is.
These opinions change every day. Last week I would've never picked Deepseek until I read about the pricing. even post aug 16 it's worth it (although it's getting close to gemini pricing).
Right now my costs are 12 euros a month (z.ai) + whatever deepseek consumes. This typically isn't more than 8 euros a week. 44 euros a month and I have a setup that is doing pretty well.
Dude, GLM-5.3 released _today_.
The phrasing "I've settled on" is incorrect for this context.
Which itself released yesterday? You're writing, reading, and evaluating enough software in a ~36 hour period to form, reject, and form another opinion about which model makes better _architectural_ choices?
What you're currently doing is "testing out"
No one actually cares about the politics as long as the model codes well.
Edit, quite interesting to see the reception to this comment compared to essentially the same type of comment I made on a Grok 4.6 benchmark HN post: https://news.ycombinator.com/item?id=49275385#49275571
It's true that Cursor gives a lot of usage with Grok, most users of Cursor don't care about Musk.
It is not all just 'politics'. Take a stand on some issues. It doesn't cost much not to use Grok.
And yeah, people that don't care DO make the world worse through their apathy.
(Btw, mussolini didn’t make the trains run on time)
Anthropic's recent releases are wordy to the point of exhaustion. Every time I use opus recently I find myself wanting to yell "GET TO THE POINT" at a terminal, which is exacerbated by it being slow.
It's still very useful, and it'll probably displace a good bit of API spend; but it's not really "trading blows with SOTA from just a half year ago". A bit overblown on the Anthropic/OpenAI has missed the window I reckon.
Also, on the open weight frontier side, Kimi K3 is pretty expensive, and Deepseek V4 Pro/Flash is getting a little less juicy with price increases.
Even Zuckerberg isn't as deluded to think that there's money in constantly releasing closed models and hypserscaling their deployment, which is why we got muse-glimmer 30b last week. I have been playing with it on a 32gb MBP w/ M2 Pro. If prosumer-grade hardware from almost three years ago can now offer a novice a way to host a possibly serviceable software development agent, then there's little incentive to pay out for LLMs like Anthropic's or OpenAI's over the long run.
I'm sure there's some benchmaxxing going on, and some things you get only with a a larger model.
But I'm feeling pretty confident if not by Gemma 5 than by mid 2028 we'll have local models that are almost always as good as Opus 4.6 was and in many cases far better.
I don't want to try to take my best educated guess at what the best design is BEFORE implementation - especially if you're designing a feature for a codebase you're not an expert in, you don't know like the back of your hand (i.e. one that is mostly or entirely LLM generated).
What sounds good on paper - often times becomes unideal in practice when you get to the reality of implementation.
It may not be worth re-architecting your entire system to get to a "pure" design that would be the best - all things considered.
Instead, I'd like the model to independently design many plausible and coherent good solutions, then implement each of them, then intelligently pick the few winners (after its fixed any bugs that could be causing promising solutions to look artificially bad) - unless there's an obvious one - and then give me the data I need to make an informed decision on which one to go with, all before I even look at the design or implementation.
You're not getting this from a one shot prompt from a 30B model today. You can't even really get it from Sol or Fable - IME. But you can get somewhat close.
Even Fable is bad at this, I would constantly have to fix it going down architectural dead ends or just making obvious mistakes.
Which sucks for people that want LLMs to do everything like a genie, but does mean senior engineers have a few more years before they become redundant.
[1] Without searching the internet. And even if you allow it, you'll get much worse results because search means browsing and parsing the top results, and search results are horrible, whereas internal knowledge from training encompasses the entire internet plus all books including very niche stuff.
[0] https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Like maybe I just misunderstand what's the hard part but wouldn't you assume that people who can put together an impressive model can also write a proper jinja chat template for it?
If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.
As capable as it is, it's hard to justify using it when a competing model (e.g. Gemma4:26b-a3b) can consistently achieve the same or similar response with only 1/10th as many 'thinking' tokens, achieve much higher tokens/second, and take a small fraction of the time. I suppose 'YMMV' depending on your use case.
Also, I haven't used it enough yet to see if it's prone to infinite looping, but its predecessors sure were.
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Granted, it's still much lower tokens/s than you'll get out of many MoE models.
Edit: Even set to medium or low there's still a lot of second guessing, less consistency, lower 'acceptable response' rate, and slower/more token churn vs gemma4:26b-a3b
What you describe is a engineering harness problem.
If you, and i mean the royal you, actually read tge thinking traces you can see and figure out where its stuck
This means an effective harness would observe when the model is overthinking and step in with reasonable redirection, like increasing logging.
Llamacpp can set reasoning budget and message per reauest, so it can be dynamic.
Your complaint is "skill issue" based and will be resolved by people who do something ither than vibe code react demos.
I saw unsloth has 1bit quants too so I might check that out, anybody have experience with those?
And this is also specifically for the 1bit quant version. I don’t think the fp8 or even fp4 versions have this issue, but I haven’t tried those much
I'm benchmarking the 3.8 model now, it seems it is better (near-perfect score on my E-mail spam filtering benchmark, best of any model I tested, ever). But it is slow.
One thing I would recommend is keeping an eye on MTP parameters. I tested and benchmarked extensively, and I use `--spec-draft-n-max 2` with llama.cpp. Longer sequences actually decrease overall performance.
As for running, I ended up using llama.cpp and its llama-server, with a bunch of scripts written by AI, because I got tired of LM Studio not implementing the image-related parameters which made gemma4 useless for OCR.
I am looking to automatically recover mail from junk folder due to bad icloud mail filtering.
Unsloth: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
This might work for you, but I didn't get on very well with MTPLX when I tried it a while back; YMMV: https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized...
2 bpw - ~85.7t/s
3 bpw - ~58.0t/s
4 bpw - ~43.9t/s
6 bpw - ~29.5t/s
8 bpw - ~22.2t/s
16 bpw - ~11.2t/s
without cheating. You'd have to exclude layers, skip operations, etc. basically do stuff the model wasn't trained for. And speed collapses so fast with context that even 2 bpw would be looking at ~37.6t/s after just 128K tokens.
MTP only improves the situation by up to 2x in the ideal case, while drastically reducing the performance floor. While optimizing a 9B model on this hardware, I've found that the GPU just doesn't have enough FLOPS to handle speculating more than one or two tokens ahead on a single stream, regardless of quant level, simply because of the arithmetic cost of the forward pass. The 27B model would be even more expensive than that, potentially such that it's already bottlenecked by the GPU itself rather than memory.
I wouldn't get my hopes up for the 35B-A3B either. Not only is it reportedly much less intelligent, but I hit a similar ~85t/s wall in practice (again with highly specialized inference).
Without speculation I can reach around 120t/s on Qwen3.5-9B and with n-gram speculation (not even MTP; this derivative didn't come with one) around about 150t/s on average. This is on the very very edge of what I'd consider acceptable for me to even consider using such a compact model. YMMV due to the silicon lottery but the situation isn't good.
Also whats your cutoff for 'acceptable' speed? I would have said 25tok/s.
Insane if that is the case. Downloading now!
And I know the 96G nVidia cards are selling for over 10k$.
The future can’t arrive fast enough!
Most of the benchmark improvements afaict are in agentic and instruction following benchmarks.
To the countrary, the reason Chinese models are excelling in the smaller area is because there's tons of fat in closed source models because of the crazy cash being thrown around.
There absolutely is space to improve intelligence and capabilities without lathering on more and more parameters.
If you want Qwen3.8-27B Serving Configs for the DGX Spark vLLM NVFP4 and RTX 4090 llama.cpp GGUF I added the setups here https://x.com/ErdalToprak/status/2088299678085308761?s=20
Funny side note. It successfully one shot the program, but it wasn't able to run it because there literally wasn't enough VRAM left to allocate CUDA memory. Watching it try to debug that was fascinating. I'm pretty sure it would have killed the llama-server (and thus itself) if it hadn't been running in a separate container.
This is the perfect candidate for just splattering it on your nvme and then reading it off there and into memory. All of these run perfectly fine on simple m4 silicone:
https://github.com/drumih/turbo-fieldfare
https://github.com/leonickson1/Swiftlet
https://github.com/sqliteai/warp
Does it come down to utilization and/or specific model tricks and efficiencies (attention, kv cache, etc.)?
DeepInfra prices:
Qwen 3.6 27B: $0.32 in / $3.20 out
Gemma 3 27B: $0.08 in / $0.16 out
DeepSeek V4 Flash 0731: $0.08 in / $0.18 out
Qwen 3.6 35B A3B: $0.10 in / $0.95 out
https://openrouter.ai/qwen/qwen3.6-27b
https://openrouter.ai/google/gemma-3-27b-it
https://openrouter.ai/qwen/qwen3.6-35b-a3b
https://openrouter.ai/deepseek/deepseek-v4-flash-0731
dsv4f appears bigger and more memory/computationally expensive to run any way I look at it.
https://openrouter.ai/qwen/qwen3.6-27b
Opus at home
I hope there also will be a new ~10b variant
categorization, information retrieval, semantic search, image description
also with the model as part of an agentic system with tool calling
(edit: it is quite impressive what a small model in a feedback loop can do)
So one open weight model might "meet" Opus or whatever on benchmarks, but then fail to follow a simple answer format and also tool call correctly. The models are whipped to within an inch of their lives to strictly adhere to their post training quality gates.
Just tried using Pi Agent and looks very promising
Completely local use is a different story, of course.
Perf improvements seem to all come from training?
https://x.com/sgl_project/status/2088281320422322413
Was considering adding a LoRa/vision head to Flash, but seems like it could take a while to get it right.
If DSv4 Flash was multimodal, I’d probably be done model shopping for a while
Yes, I have a very dumb flow. The harness has a describe_image tool that takes an image and a prompt and so DSv4 Flash uses it to get an idea of what it's looking at.
on my dual 3090s qwen 3.5 27b was running at around 110tps using the config from https://github.com/noonghunna/club-3090
make that 200tps on a single 5090, 4x faster than opus https://x.com/radixark/status/2088285681131110446
devs about to get handed a two 5090 box each and told to max that out
I only trust those users genuine personal tests
https://www.youtube.com/@lukesdevlab
I don't know if that is what you are looking for or not and as always your experiences may be different.
However, watching tests of heavily quantized models that weren't designed for it (non-QAT) is frustrating. There's no way to tell if the actual model fails because it's dumb or if the lobotomy made it that way.
Been running quantized 3.6 at 110t/s on a cheap 5060Ti and quite happy with it. If 3.8 improves on it, it would be awesome.
my prediction was way too far out. 4.6 at home! Woo.
https://news.ycombinator.com/item?id=49299684
What a weird mechanism. If someone is judging a thread/topic/event impact by the number of points it got, then doing this unfairly degrades that thread.
It should have deduped by user and combined the 168(at the time of writing this comment) + 289 points. Just add the twitter link from the previous thread as an additional link in the description, like you normally do, move all the points over, and remove the old thread.
" The user wants me to explore the repository at XXXX and report back. Let me start by understanding the project structure, reading the CLAUDE.md file, and getting a general overview of what this repository is.
Let me start by reading the main project documentation and exploring the directory structure.
I'll take a look around this repo. Let me start by getting a lay of the land.
read resource CLAUDE.md (ctrl+o to expand)
ENOENT: no such file or directory, access 'XXXX/CLAUDE.md'"