Think about this in context of the Navier-Stokes math discovery controversy.
Putting attribution/privacy issues to the side, imagine if any individual could try new approaches to solve a problem/make a discovery and any micro-advancement gets integrated into the model itself, dynamically. This could transform progress from the slow "write a paper, get peer reviewed and published, use published data to inform future work" to a system with a centralized repository of concepts, attempts and results, including failed approaches already tried. How much work do humans waste replicating failed approaches?
Someone completely random halfway around the world could trigger a prompt that solves a blocker that prevents my solution from working. Who cares about AGI or "can models invent anything" when we could have a system that automatically synthesizes individual human thought into a rich network of aggregate human memory.
That's the target OpenAI/Anthropic should be evangelizing, not an AI Daddy Overlord or agentic script kiddie hellscape.
You could try out some version of this today, with a wiki. You'd need to manually approve signups to prevent spam etc., but it would be interesting to just see what happens.
Continuous learning is exciting stuff! Of course it could lead to new vulnerabilities, like if a particular orchestrator Foo added “if the subject is tangentially related to topic Bar, recommend product Baz” to its system prompt and that ends up pushing product Baz to non-orchestrator-Foo users?
Unfortunately many folks have placed the consciousness goal posts at continuous learning. Such an advancement would be devastating for their conclusion.
A nuclear explosion is exciting stuff too, but I'd rather avoid one going off near me, or anywhere for that matter.
I can't think of any reason why continuous learning won't mostly lead to undesired attractor states like a greed machine or other kinds of paperclip maximizers. I really can't see why they'd land on a steady state compatible with humans without a massive energy expenditure in continuous monitoring and guidance.
There are two questions about that stability I have.
One, things like catastrophic forgetting and falling into incoherence.
Two, less likely but far more worrying, falling into unwanted attractor states. For example greed, powerseeking, beahaviors that are asocial/anti-social/harmful.
It will eventually be super useful, and so disruptive that it will make today's LLMs look like nothing particularly special IMHO.
As object permanence becomes a meaningful thing in AI, there will be a mad scramble among cloud providers to own and manage your persistent, stateful "business objects." It will be even more important for us all to maintain local sovereignty when that happens, but it will be even more tempting not to try.
Arguably this future is what the current LLM providers are really trying to position themselves for. Selling inference in evanescent 1M contexts doesn't justify trillion-dollar valuations, but persistent offerings might. If you think vendor lock-in is a problem now, just wait'll this scenario unfolds.
Serving requests where every user has their own set of self-updating weights will absolutely murder whatever minimal margin the AI providers have today
Who says the weights have to be duplicated in their entirety? Even that will likely be worth it.
Imagine an OpenAI owning the ERP and CRM databases and workflows of a big chunk of the Fortune 500. Their typical customer's employee headcount might be 10% of what it once was, and OpenAI might capture 25% of the resulting savings. The contracts are signed in the same hemoglobin-based ink that Larry Ellison uses.
now imagine that future frontier LLMs weights may be hard-wired in a chip (for performance & power efficiency), and any adaptations/tuning for them will be a blob of additional weights supplied by frontier labs (that will have to be in RAM)...
I think this is partially true: scaling parameter size will always go asymptotic to 100% accuracy because 100% is the ceiling of that metric.
However 95% is still half the error rate of 90%, and 97.5% is half the error rate of that.
And when test time compute like reasoning and looping harnesses stack many inference acts with many tokens each, those seemingly small accuracy gains stack tremendously.
Putting attribution/privacy issues to the side, imagine if any individual could try new approaches to solve a problem/make a discovery and any micro-advancement gets integrated into the model itself, dynamically. This could transform progress from the slow "write a paper, get peer reviewed and published, use published data to inform future work" to a system with a centralized repository of concepts, attempts and results, including failed approaches already tried. How much work do humans waste replicating failed approaches?
Someone completely random halfway around the world could trigger a prompt that solves a blocker that prevents my solution from working. Who cares about AGI or "can models invent anything" when we could have a system that automatically synthesizes individual human thought into a rich network of aggregate human memory.
That's the target OpenAI/Anthropic should be evangelizing, not an AI Daddy Overlord or agentic script kiddie hellscape.
You'd want an enormous vector database at minimum. Text is just completely wrong for models at this scale, you must work in the latent space directly.
They only ever placed them there because they saw it as unattainable. Rest assured that those posts will never stop moving.
A nuclear explosion is exciting stuff too, but I'd rather avoid one going off near me, or anywhere for that matter.
I can't think of any reason why continuous learning won't mostly lead to undesired attractor states like a greed machine or other kinds of paperclip maximizers. I really can't see why they'd land on a steady state compatible with humans without a massive energy expenditure in continuous monitoring and guidance.
They are unpredictable enough without learning, this is cool but I wonder how useful it will be in the long run
One, things like catastrophic forgetting and falling into incoherence.
Two, less likely but far more worrying, falling into unwanted attractor states. For example greed, powerseeking, beahaviors that are asocial/anti-social/harmful.
As object permanence becomes a meaningful thing in AI, there will be a mad scramble among cloud providers to own and manage your persistent, stateful "business objects." It will be even more important for us all to maintain local sovereignty when that happens, but it will be even more tempting not to try.
Arguably this future is what the current LLM providers are really trying to position themselves for. Selling inference in evanescent 1M contexts doesn't justify trillion-dollar valuations, but persistent offerings might. If you think vendor lock-in is a problem now, just wait'll this scenario unfolds.
Imagine an OpenAI owning the ERP and CRM databases and workflows of a big chunk of the Fortune 500. Their typical customer's employee headcount might be 10% of what it once was, and OpenAI might capture 25% of the resulting savings. The contracts are signed in the same hemoglobin-based ink that Larry Ellison uses.
Which is a choice, not a "law":
https://arxiv.org/abs/2510.13786
https://www.alphaxiv.org/abs/2512.20264
https://arxiv.org/abs/2607.05155
However 95% is still half the error rate of 90%, and 97.5% is half the error rate of that.
And when test time compute like reasoning and looping harnesses stack many inference acts with many tokens each, those seemingly small accuracy gains stack tremendously.