Rule #1 of reading CS papers - don't trust the anthropomorphized terms. Rule #2 - definitely don't extrapolate those terms to the real mind.
Of course, attention is almost certainly a heuristic readers use, but to even bring it up as key to intelligence begs so many questions about LLMs and the mind...
It's misleading. When it's right, it's right by coincidence, since the vast majority of algorithmic terms that seem like they should provide insight into the human mind, like "attention" could just as easily be called "finger" or "tape head".
And suddenly we're looking at people who read ticker tape or by following with their finger for insights into neuroscience?
It's bad metaphysics. If you really want to do bad metaphysics, then you can, but it's also hardly an attack on you if others promote norms of not doing bad metaphysics.
An IMO under-appreciated aspect of attention is that it's an ingredient for building a Turing-complete evaluator, since (for example) it lets you implement mutable RAM by treating the attended data as a write log: https://yager.io/neural/turing/turing.html
"Sentences with pronouns such as “it” in them illustrate the challenge that attention overcomes. Consider: “The animal didn’t cross the street because it was too tired.” And: “The animal didn’t cross the street because it was too wide.” To translate these sentences correctly into French, an AI model must determine that, in the first one, “it” refers to the animal, which is a masculine word in French, so the translation should use that form of the pronoun, il. But in the second sentence, “it” refers to the street, a feminine word that requires a different pronoun, elle. How can the machine figure this out?"
"Bahdanau remembered doing translation exercises when he was learning English in school. To write each next word of his translated sentence, he’d have to glance all around the source text to figure out what should come next. “Literally you have your source sentence on the left and your writing on the right, and you’ll be, like, turning your head, left-right, left-right,” he said. “That process is what I expressed as a neural mechanism.”
That simplicity seems to me the real tell that this is the real insight. The author does a good job of walking the ready past a long list of distractors and conjuring a sense of a path to an insight, and then, there's the insight - a simple encoding of a simple task: "What is the next word? How do I figure that out? Left-right. Left-right"
Of course, attention is almost certainly a heuristic readers use, but to even bring it up as key to intelligence begs so many questions about LLMs and the mind...
And suddenly we're looking at people who read ticker tape or by following with their finger for insights into neuroscience?
https://www.quantamagazine.org/is-attention-the-essence-of-i...
"Sentences with pronouns such as “it” in them illustrate the challenge that attention overcomes. Consider: “The animal didn’t cross the street because it was too tired.” And: “The animal didn’t cross the street because it was too wide.” To translate these sentences correctly into French, an AI model must determine that, in the first one, “it” refers to the animal, which is a masculine word in French, so the translation should use that form of the pronoun, il. But in the second sentence, “it” refers to the street, a feminine word that requires a different pronoun, elle. How can the machine figure this out?"
"Bahdanau remembered doing translation exercises when he was learning English in school. To write each next word of his translated sentence, he’d have to glance all around the source text to figure out what should come next. “Literally you have your source sentence on the left and your writing on the right, and you’ll be, like, turning your head, left-right, left-right,” he said. “That process is what I expressed as a neural mechanism.”
That simplicity seems to me the real tell that this is the real insight. The author does a good job of walking the ready past a long list of distractors and conjuring a sense of a path to an insight, and then, there's the insight - a simple encoding of a simple task: "What is the next word? How do I figure that out? Left-right. Left-right"