I dislike how we use the word think or behavior when describing llms. Its just giant neural networks producing logits. There is no thinking. Its a function that takes ~500k words, and returns some words based on those probabilities. I'd argue the GPU is doing more classical "thinking" than the model, but its not doing that either.
Humans think, computers do not.
Humans are not computers/programs do not reply to me with that nonsense terbo nerds.
I wonder how this compares to going all-in: widen the input embedding to fit the output (post-“Hybrid Link”) latent vector and a token embedding and pass both through even when in text mode. After all, you’re paying essentially the full cost of running the latent-mode model while generating text, so why not let it latently reason at the same time?
(Or just add the vectors instead of using wider vectors - everyone seems to have good results from munging vectors together with plain addition.)
People at the big labs have been terrified of losing the chain of thought because they currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business. Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
Agreed, having text as a reasoning layer impedes performance while giving a false sense of security.
It could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.
Also agreed. Begging for crumbs at the CoS table is impeding cracking the tough nut of MechInterp. And further facing the fascinating and fertile fact that nobody has a clue of how any of these black boxes really work.-
The usual meaning: say you have an interpretability signal that reveals reward hacking or other badness. Your most obvious response is to use that signal as a penalty in training. Doing that is called "the Most Forbidden Technique" because you can expect it to degrade the signal without stopping the badness.
But this doesn't seem to describe the OP, it's a different way to lose monitorability.
the chain of thought is not auditable. The best you can do is have another model which later maps the intermediate latent states at every thinking iteration to text describing that latent state, but it's going to be lossy.
This is correct, the decoded text has nothing to do w/ calculations that eventually get decoded into text. It's just a veneer that gives the appearance of legibility.
One of the core claims made in the paper is "To inspect any latent reasoning
span, one can easily recover its semantics by mapping each latent step produced by HybridLink to its nearest token embedding, as in Eq. 5". They show some charts to support this.
In doing this however, you lose meaning where the model is expressing combinations of tokens in a vector, and your logging/monitoring outputs just the closest one. You would need another model going through and outputting the full meaning being described (with more than one output token per reasoning token)
I wonder if their observability claims will hold up on a model trained from scratch to reason in latent space.
The idea of using language for reasoning about protein folding borders on ludicrous. Surely there is latent space more appropriate and (human) language has no expression for it.
I believe the theory is that if selection pressure is not applied at that level, then developing deception in that level becomes an inefficiency (=less adaptive and unlikely to be conserved).
Shockingly all the answers so far are subtly incorrect or extremely unclear.
The actual answer is that this is massive loss for Interpretibilty, AKA "telling whether the model is lying to you or not". Which sounds silly when you're thinking about chatbots, but is an incredibly important line in the sand for superheavy, longrunning, self-improving agents. To say the least!
It's worth pointing out that merely having the chain-of-thought in latent space isn't by itself "The Most Forbidden Technique". Rather, it's doing what the authors propose as the next step in the paper:
Hybrid generation can also make post-training more efficient. Because latent thoughts remain auditable during rollout, an external critic can assign step-level credit to latent spans, branch promising trajectories, or terminate ones whose latent reasoning has already gone wrong, before any text or tool call is produced.
Unless I'm misunderstanding, they're suggesting RL training against the CoT. Which, uh, that is The Most Forbidden Technique, as I understand it, when combined with the latent reasoning space. It allows for the model to "learn"/optimize for malicious behaviors that are just far enough away from the quantized token they're auditing against.
If you accept any of the AI-equivalents of Pascal's Wager, latent thinking represents an uninterpretable and thus unalignable portion of AI machinery. Ie: a malevolent AI can hide bad stuff from people and Manchurian candidate themselves after it's too late.
if i would think aloud, that would most probably affect what or how i think. thinking silently allows me to have all kind of thoughts that could seem or be inappropriate when said loud and i would have to deal with the reactions. i think this might have fascinating implications.
This seems really cool, and it actually increases intelligence than just speeding up inference. How does it convert latent reasoning into human readable text?
Nothing wrong with that. That’s actually a good use of that well designed, high quality font. People hating it only do so because everyone else does. It’s, obviously, not made as a body text font, and has been misused as that, but it’s good for illustration, comics and other places where it’s combined with graphics. It’s one of the most legible fonts we have.
Humans think, computers do not.
Humans are not computers/programs do not reply to me with that nonsense terbo nerds.
(Or just add the vectors instead of using wider vectors - everyone seems to have good results from munging vectors together with plain addition.)
LLM Models and data: https://huggingface.co/collections/togethercomputer/hybridag...
Evaluations: https://arxiv.org/html/2610.18861v1#S5
It could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.
Can't they just monitor the text outputs as well?
But this doesn't seem to describe the OP, it's a different way to lose monitorability.
In doing this however, you lose meaning where the model is expressing combinations of tokens in a vector, and your logging/monitoring outputs just the closest one. You would need another model going through and outputting the full meaning being described (with more than one output token per reasoning token)
I wonder if their observability claims will hold up on a model trained from scratch to reason in latent space.
It is not necessarily primarily about efficiency, but more the fact that a lot of thinking doesn't use language at all.
The idea of using language for reasoning about protein folding borders on ludicrous. Surely there is latent space more appropriate and (human) language has no expression for it.
https://thezvi.substack.com/p/the-most-forbidden-technique
The actual answer is that this is massive loss for Interpretibilty, AKA "telling whether the model is lying to you or not". Which sounds silly when you're thinking about chatbots, but is an incredibly important line in the sand for superheavy, longrunning, self-improving agents. To say the least!
It's based on this philosophy, FWIW: https://plato.stanford.edu/entries/language-thought/
And a good writeup I just found of recent worries regarding Astra's same shift towards intelligence at the cost of safety: https://www.lesswrong.com/posts/HCRs8btkiamtWSNAL/astra-is-h...
https://github.com/togethercomputer/HybridAgent
But now we also need new tools and techniques to audit intent. The last thing we want is models turning on us ‘for our own good’.