i'm not sure why people expect agents to one shot everything to perfection with just a prompt.
There's a reason why we talk about software development lifecycle, design, architecture, testing ... It's because it's been the most reliable way to build and ship software. We shouldn't expect discard this and expect agents to perform well outside of this.
I'm treating LLM agents as junior devs who happen to have vast knowledge of software engineering. As their team leader i make them go through planning, implementation, bug sweeping cycles using strict workflows. And it works quite well, i've been working on several large projects (1M+ LOC java,typescript,c/c++) and by any measure the projects are healthy. Sure the code isn't that beautiful, sure i'd have written things differently but it's pretty good nonetheless.
Shameless plug here: i've been also working on https://kodfactory.com, the code factory i've built to work on these large projects with workflows, reviews, etc ... I'm cleaning things up to open source it later.
anthropic specifically brags about how good claude code is every annoucement of a new model. I will surrender that none of them claim its "to perfection", but IMO its implied because no one would claim that their model one-shots any issue to dog shit quality.
I don't really think they advertise "Create your app idea in one weekend night" and assume the general public will mentally add "... but hire an experienced developer to supervise the process".
In effect, I’ve always wanted a pair programmer agent, not a zero to one programming agent. Unfortunately models these days are mostly of the latter kind and it has caused a major disruption in the way I work. I’d much rather appreciate a small model making fast and specific edits that I ask if it, rather than ingesting 20 files to make changes, and then starting to write tests, etc.
I have found what works well is to modularise the code as much as possible and get an agent to work within a very limited scope. Break everything down well using SRP with well defined interfaces and let the agent work on small problems. Then when it shits the bed, there’s a smaller blast radius and you can strip back and try again.
I think seasoned developers, over time, learn how to work a code base and design components with well defined interfaces, where the implementation is isolated in small well contained classes. SRP etc. more junior programmers can work on those smaller components/services in isolation.
For me this also seems to be a productive way to work along side an agent. Break up functionally into well defined chunks, and let the agent work on each small problem. Take more of a lead in the architecture I suppose.
I think this is the only sensible way to work with agents, if you care about code quality and reliability but still want the benefits of AI. There seem to be three camps that people more or less fall into: (a) AI is terrible/bad/evil and should never be used, (b) you should one-shot everything and be happy if it seems to "work" when you try it, (c) the middle ground, where the AI writes code which you carefully review.
I definitely prefer (c). But I get why (b) can feel necessary. If your competition is using (b) there can be pressure to do the same just to keep up.
I think it must depend at least partially on the task, too. At the extreme, there are things where you won't care beyond "it seems to work" because you only needed it to run once and it got useful results.
I think c is the only way it can sustainably work. The idea of b, that software is running and nobody there knows how it works, doesn’t seem like a good foundation for a business to run on.
Just said something similar myself in another thread. I'm either writing things by hand (and using LLMs for research, or double checking an idea), or having an LLM spit out something I treat as an external dependency. Its still too tedious for me to use them to write code when I care how it works or there's not obvious invariants the code needs to hold
I just had a sunday afternoon request for a solution to a trivial but annoying spammer pattern on a stackoverflow-like site I maintain.
The software has a plugin API. I asked Muse Glimmer to recommend a plugin — it found one but I tested it and it didn't work for unclear reasons (among other things the software installed version is old, the plugin older). I then asked it to outline how to implement a simple word filter, it gave me an overview of some hooks that looked right from dim-and-distant-past recollection of reading the docs when I installed it. I asked it some questions, it did the research.
I then set it off generating the skeleton of a filter plugin, went to the shops to buy food, came back and worked through filling it in and finishing it off. There was a bug. It found the solution.
It's only about 100 lines of code but it is a random old webapp and it had to look stuff up to finish it, and I think it did rather well. All on my Mac.
I am deeply cynical of the one-shot code, "nobody codes anymore" hype culture idea and that distaste put me off AI and agentic coding for ages. Like you, I want an assistant but as a freelancer I have to stay in control. I have no interest in the "just specify loops" BS and it will be bad for my business anyway.
I worry about code that I don't have a good working overivew of, and I worry that I might forget what I have done (I have pretty bad issues with focus and memory). But in this particular case, I don't really care if I forget, because there's documented code and I have no intention of specialising in this app. So it was a nice little test case.
I also don't really want to sit around waiting for Qwen 3.8 27B on this machine. Muse Glimmer is fine, actually. Gets to the solution as quickly as Qwen 3.6 35B-A3B.
This gives me a little hope that local AI will give me the sort of responsive developer sidekick I actually want.
This sounds miserable, why not just give it specific tasks to do in your normal workflow/editor? Why would we want to do MORE of the miserable task of ticket creation.
I tried, but I’m not sure I understand. The vibe tax is caused by the model trying to one-shot everything and doing so requires unnecessary tests? How are vibe coders training the model over months? Do you mean their sessions and preferences are being fed back into the RL?
Forgot where i saw it discussed; If you observe recent model benchmarks over the past year; the performance is slowly climbing, but if you divide by the token count; the score per token is dropping.
The current trend in state-of-art LLM coding agents is giving more output, thinking longer and checking the results more to catch mistakes. Be it an economics inventive to make users burn through their quota or show increase in usage for shareholders, or a market demand of users liking the ability of models to do independent work without intervention or oversight; the result is what the article seem to call the Vibe Tax.
I myself asked Claude code recently to review a somewhat large PR, to see what it would find. I didn't expect much, but also didn't quite realize how the model would interpret my request; I burned $20 in 3 minutes in API usage, as it ran 2 sub-agents which themselves spun up 5 more each. Most sub-agents were manually checking for things clang-tidy would catch without actually calling clang-tidy. This behavior rose as i changed from sonnet/opus 4.6 to 4.8 and now 5.0.
I don't want to run a agent independently in this way; i ask targeted questions about specific things and review the result. But model development is targeted towards a more hands-off "vibe" workflow, because that's where the money and hype is. As a result, i find the models more frustrating, less trustworthy and more costly to my work. (I've even started using haiku more, since it remains to-the-point without steering away from what i ask)
It should be expected that more tokens give diminishing returns. Minimally, there's no limit on tokens but there is on quality of output (you can't reach negative bugs, or negative execution time). The graphs I've seen show a curved "frontier" of the tradeoff, and that line has improved over model generations.
That said, the companies are incentivized to sell you tokens, and therefore to have the models use as many tokens as they think you'll let them get away with for a given task / level of performance.
This is my experience too, but even worse. Opus 5 finished the task, I then asked it to code review it, 61 agents later it came back with a bunch of errors that needed fixing.
The first pass had tests, they passed, they were just wrong. I wish more people started reviewing their AI output, because I see a worrying trend of ”we have all these tests the agent wrote so it has to be good”, which is not surprising because understanding tests is not a trivial skill.
I'll try to explain my experience with this. I've noticed the AI has a tendency to overengineer scaffolding. For instance, I asked it to help me with a refactor, and it erected this massive 100kloc function registry, and then caused GitHub CI to verify the contracts every single commit, which took upwards of 30 minutes (I suspect this proclivity is widespread and has contributed to their recent issues).
As if this wasn't bad enough, it also was not smart enough to regenerate the evidence in these contracts as it changed the underlying source code. So it would get in a loop where it would update code
-> commit
-> 15 minutes later CI would error citing the contracts weren't updated
-> it would fix the contracts
-> 15 minutes later CI would error because the fix was wrong
-> it would fix the fix and commit
-> 15 minutes later contracts would fail
-> contracts were fixed again and this time maybe 30 minutes later it would pass, maybe it errors again.
This loop could go on all day every day if someone wasn't paying attention because the agent has no concept of time or wasted work. It's an AI livelock of sorts, but it will eventually converge in my experience. It'll just take 10x longer (literally like 20+ hours) than if you just intervene and tell it knock it off, so it feels like lighting money on fire (hence the tax).
That's why I feel like this vibe coding stuff has to actually be monitored, like a Tesla system -- because like a Tesla system it cannot be trusted to not crash into the proverbial code wall.
I’ve never had an agent fail to write the actual implementation. Has it done so badly, yes, but not nothing but tests. This sounds to me like a rare case that doesn’t generalize.
If the general idea is that these agents write too many tests, sure I guess? ‘Too many tests’ doesn’t sound like a failure case of engineering to me; typically software has had too few tests. Also, a lot of the power of these agents is their ability to self-verify and correct, which the test loop is a part of.
Nobody is making you pay this supposed tax. Just tell it not to write tests.
You can absolutely prompt agents not to write tests, or not to write extraneous asserts, or whatever, and I find that generally quite useful for the kind of code I write. I don't think it's "months of users training it", it's more that a lot of people do want a one-shot agent, and having a good test set really helps that.
I feel like I'm living in a parallel universe when I read these types of posts.
My agents have never created code that is straight-up garbage and I have never flushed a week's worth of tokens down the toilet. I just can't identify with all the constant complaints about AI-assisted coding.
And my biggest project isn't some hello world app. It's a self-hosted, privacy-focused personal financial management application that I intend to open source. It's about 126k LOC against 240k LOC of regression tests and 30k LOC of CI/CD pipeline. I'm doing 24x7 mutation testing on a dedicated box against the accounting engine and temporal systems. I even have specialized agents doing audits against Regulation Z (US banking law) criteria so the app models the required behavior of banks.
Most of my complaints about everything are nits, like the overly verbose and dense way LLMs communicate with me. Or their predisposition to add, add, and add more stuff when proper engineering practices are more often about subtraction (but I've built mitigation guardrails against a lot of that).
> Or their predisposition to add, add, and add more stuff when proper engineering practices are more often about subtraction (but I've built mitigation guardrails against a lot of that).
> a… personal financial management application… about 126k LOC against 240k LOC of regression tests and 30k LOC of CI/CD pipeline
Just how much functionality are you getting out of that? It's hard for me to imagine that people want that much out of such a program. I just keep a spreadsheet. (Yes, LibreOffice is also very bloated.)
This really resonated for me. It's like the smarter the model gets, somehow the more tokens get burned? Same failure mode whether you’re on Claude, Codex, or Cursor: the harness will spend the whole pool if you let it. I'm building my own Harness on top of pi that is add supported (www.freepi.ai) mostly because pi is so much more efficient with tokens. (That said, it tens to be slower and vastly more verbose with information I don't need to know). But yeah, since I'm trying to offer free ad supported inference the vibe tax would kill the business model. I've even been thinking about installing the 'caveman' skill to reign in token costs.
This is a confusing description of a real thing. They're clearly biasing the models more and more towards long horizon end-to-end software development, which leads to impressive "claude, build an X make no mistakes" demos, but is mainly an annoyance for expert users doing real work.
(If you give claude an inch these days it'll just steamroll through a whole program of work without checking what it should be doing - a kind of overenthusiastic pull towards the first draft that is often detrimental and definitely wastes tokens, and even for very basic tasks it's using many more tokens than it should because it's doing this full belt and braces thing for everything, just in case you're an idiot).
But also... it's something you can easily reign in if you want to.
"A price that’s essentially a tax on all other regular software developers."
What insanely silly nonsense. I actually have no idea what the point of this silly essay is.
I have a 5x Max Anthropic plan and I find almost impossible to actually saturate my usage. I had so much extra leftover this week that over a couple of days of periodic usage I had it spit out a uPnP media server with a web interface, loads of different ways of slicing and dicing the content, subtitle ingestion, file cleaning, metadata enrichment from TMDB and IDMB, and so on. I have loads of media backups on my network (including some that I downloaded for media I own, which is 100% legal in Canada) and I've had such bad experiences with so many media tools -- particularly opinionated tools -- that it literally was less effort to have the whole system created from scratch. Every now and then I say "you know what, how about..." and voila.
And it's by far the best media system I've ever used. It has solved so many itches across the family.
This was all generated by Fable 5 using just basically leftover tokens. Gosh, I'm so sorry for imposing this terrible "vibe tax" on all of the whiny entitled peoples of the world.
That workflow itself is what the author is calling a "vibetax". Models are getting worse for users that must review every line of code the models edits or adds. Models are writing more line of code, changing more lines of code,executing more tools, and making it harder for the user to monitor, control, and review.
if i wanna write a web-app or python script; the models are better than ever. If i want to fix a specific bug in a established and trusted legacy cobe-base; Haiku 4.6 does a better job than Opus 5.0, because it does what it's told and nothing more.
The author wanted a todo-list starting-point; realistically 200 rows of html+CSS without the back-end. Heck, they may not even want to make a todo-app, but thought a todo-app would be a decent starting-point. So why would we ever want a model to spend a weeks worth of tokens on everything except the request the user asked? This is not a cost issue; this is a control issue.
Ask the AI to create a detailed spec according to a few simple requirements. Review the spec yourself and correct what you want changed. Then ask the AI to implement the spec. Each time you request something new, ask the AI to update the spec as well.
There's a reason why we talk about software development lifecycle, design, architecture, testing ... It's because it's been the most reliable way to build and ship software. We shouldn't expect discard this and expect agents to perform well outside of this.
I'm treating LLM agents as junior devs who happen to have vast knowledge of software engineering. As their team leader i make them go through planning, implementation, bug sweeping cycles using strict workflows. And it works quite well, i've been working on several large projects (1M+ LOC java,typescript,c/c++) and by any measure the projects are healthy. Sure the code isn't that beautiful, sure i'd have written things differently but it's pretty good nonetheless.
Shameless plug here: i've been also working on https://kodfactory.com, the code factory i've built to work on these large projects with workflows, reviews, etc ... I'm cleaning things up to open source it later.
because that's how agents are marketed.
- https://cognition.com/
- https://openai.com/index/introducing-the-codex-app/
- https://www.anthropic.com/news/claude-3-7-sonnet
anthropic specifically brags about how good claude code is every annoucement of a new model. I will surrender that none of them claim its "to perfection", but IMO its implied because no one would claim that their model one-shots any issue to dog shit quality.
In effect, I’ve always wanted a pair programmer agent, not a zero to one programming agent. Unfortunately models these days are mostly of the latter kind and it has caused a major disruption in the way I work. I’d much rather appreciate a small model making fast and specific edits that I ask if it, rather than ingesting 20 files to make changes, and then starting to write tests, etc.
I think seasoned developers, over time, learn how to work a code base and design components with well defined interfaces, where the implementation is isolated in small well contained classes. SRP etc. more junior programmers can work on those smaller components/services in isolation.
For me this also seems to be a productive way to work along side an agent. Break up functionally into well defined chunks, and let the agent work on each small problem. Take more of a lead in the architecture I suppose.
I definitely prefer (c). But I get why (b) can feel necessary. If your competition is using (b) there can be pressure to do the same just to keep up.
The software has a plugin API. I asked Muse Glimmer to recommend a plugin — it found one but I tested it and it didn't work for unclear reasons (among other things the software installed version is old, the plugin older). I then asked it to outline how to implement a simple word filter, it gave me an overview of some hooks that looked right from dim-and-distant-past recollection of reading the docs when I installed it. I asked it some questions, it did the research.
I then set it off generating the skeleton of a filter plugin, went to the shops to buy food, came back and worked through filling it in and finishing it off. There was a bug. It found the solution.
It's only about 100 lines of code but it is a random old webapp and it had to look stuff up to finish it, and I think it did rather well. All on my Mac.
I am deeply cynical of the one-shot code, "nobody codes anymore" hype culture idea and that distaste put me off AI and agentic coding for ages. Like you, I want an assistant but as a freelancer I have to stay in control. I have no interest in the "just specify loops" BS and it will be bad for my business anyway.
I worry about code that I don't have a good working overivew of, and I worry that I might forget what I have done (I have pretty bad issues with focus and memory). But in this particular case, I don't really care if I forget, because there's documented code and I have no intention of specialising in this app. So it was a nice little test case.
I also don't really want to sit around waiting for Qwen 3.8 27B on this machine. Muse Glimmer is fine, actually. Gets to the solution as quickly as Qwen 3.6 35B-A3B.
This gives me a little hope that local AI will give me the sort of responsive developer sidekick I actually want.
The current trend in state-of-art LLM coding agents is giving more output, thinking longer and checking the results more to catch mistakes. Be it an economics inventive to make users burn through their quota or show increase in usage for shareholders, or a market demand of users liking the ability of models to do independent work without intervention or oversight; the result is what the article seem to call the Vibe Tax.
I myself asked Claude code recently to review a somewhat large PR, to see what it would find. I didn't expect much, but also didn't quite realize how the model would interpret my request; I burned $20 in 3 minutes in API usage, as it ran 2 sub-agents which themselves spun up 5 more each. Most sub-agents were manually checking for things clang-tidy would catch without actually calling clang-tidy. This behavior rose as i changed from sonnet/opus 4.6 to 4.8 and now 5.0.
I don't want to run a agent independently in this way; i ask targeted questions about specific things and review the result. But model development is targeted towards a more hands-off "vibe" workflow, because that's where the money and hype is. As a result, i find the models more frustrating, less trustworthy and more costly to my work. (I've even started using haiku more, since it remains to-the-point without steering away from what i ask)
That said, the companies are incentivized to sell you tokens, and therefore to have the models use as many tokens as they think you'll let them get away with for a given task / level of performance.
As if this wasn't bad enough, it also was not smart enough to regenerate the evidence in these contracts as it changed the underlying source code. So it would get in a loop where it would update code -> commit -> 15 minutes later CI would error citing the contracts weren't updated -> it would fix the contracts -> 15 minutes later CI would error because the fix was wrong -> it would fix the fix and commit -> 15 minutes later contracts would fail -> contracts were fixed again and this time maybe 30 minutes later it would pass, maybe it errors again.
This loop could go on all day every day if someone wasn't paying attention because the agent has no concept of time or wasted work. It's an AI livelock of sorts, but it will eventually converge in my experience. It'll just take 10x longer (literally like 20+ hours) than if you just intervene and tell it knock it off, so it feels like lighting money on fire (hence the tax).
That's why I feel like this vibe coding stuff has to actually be monitored, like a Tesla system -- because like a Tesla system it cannot be trusted to not crash into the proverbial code wall.
If the general idea is that these agents write too many tests, sure I guess? ‘Too many tests’ doesn’t sound like a failure case of engineering to me; typically software has had too few tests. Also, a lot of the power of these agents is their ability to self-verify and correct, which the test loop is a part of.
Nobody is making you pay this supposed tax. Just tell it not to write tests.
My agents have never created code that is straight-up garbage and I have never flushed a week's worth of tokens down the toilet. I just can't identify with all the constant complaints about AI-assisted coding.
And my biggest project isn't some hello world app. It's a self-hosted, privacy-focused personal financial management application that I intend to open source. It's about 126k LOC against 240k LOC of regression tests and 30k LOC of CI/CD pipeline. I'm doing 24x7 mutation testing on a dedicated box against the accounting engine and temporal systems. I even have specialized agents doing audits against Regulation Z (US banking law) criteria so the app models the required behavior of banks.
Most of my complaints about everything are nits, like the overly verbose and dense way LLMs communicate with me. Or their predisposition to add, add, and add more stuff when proper engineering practices are more often about subtraction (but I've built mitigation guardrails against a lot of that).
> a… personal financial management application… about 126k LOC against 240k LOC of regression tests and 30k LOC of CI/CD pipeline
Just how much functionality are you getting out of that? It's hard for me to imagine that people want that much out of such a program. I just keep a spreadsheet. (Yes, LibreOffice is also very bloated.)
> 126k LOC against 240k LOC of regression tests and 30k LOC of CI/CD pipeline
I mean...
Yeah, that's pretty self-explanatory why you don't identify complains about AI-assisted coding.
* the dev wants to describe an app in natural language then fall asleep while an AI works on it
* the dev wishes that the same AI would write less comprehensive tests
What exactly is a vibe coder to this dev?
(If you give claude an inch these days it'll just steamroll through a whole program of work without checking what it should be doing - a kind of overenthusiastic pull towards the first draft that is often detrimental and definitely wastes tokens, and even for very basic tasks it's using many more tokens than it should because it's doing this full belt and braces thing for everything, just in case you're an idiot).
But also... it's something you can easily reign in if you want to.
What insanely silly nonsense. I actually have no idea what the point of this silly essay is.
I have a 5x Max Anthropic plan and I find almost impossible to actually saturate my usage. I had so much extra leftover this week that over a couple of days of periodic usage I had it spit out a uPnP media server with a web interface, loads of different ways of slicing and dicing the content, subtitle ingestion, file cleaning, metadata enrichment from TMDB and IDMB, and so on. I have loads of media backups on my network (including some that I downloaded for media I own, which is 100% legal in Canada) and I've had such bad experiences with so many media tools -- particularly opinionated tools -- that it literally was less effort to have the whole system created from scratch. Every now and then I say "you know what, how about..." and voila.
And it's by far the best media system I've ever used. It has solved so many itches across the family.
This was all generated by Fable 5 using just basically leftover tokens. Gosh, I'm so sorry for imposing this terrible "vibe tax" on all of the whiny entitled peoples of the world.
if i wanna write a web-app or python script; the models are better than ever. If i want to fix a specific bug in a established and trusted legacy cobe-base; Haiku 4.6 does a better job than Opus 5.0, because it does what it's told and nothing more.
The author wanted a todo-list starting-point; realistically 200 rows of html+CSS without the back-end. Heck, they may not even want to make a todo-app, but thought a todo-app would be a decent starting-point. So why would we ever want a model to spend a weeks worth of tokens on everything except the request the user asked? This is not a cost issue; this is a control issue.