Ok, so I can see that this is a benchmark the submitter made and submitted themselves, which is kinda misleading and should at least be disclosed.
But my question is, what does this benchmark actually measure about the LLM's engineering abilities? It just looks like a page of random bar graphs to me.
I wouldn't trust any particular one, but obscure / indie indexes are going to be less affected by benchmaxxing than the well known ones, so it's a positive sign for Neon that it is performing so well on a lot of them including this one.
Is it just me or all Google models are terribly bad?
They all seems benchmaxed to hell and producing total crap compared to Claude or openai models from my own experience (mostly coding)
After hearing some good things from my tokenmaxxing buddies (or whatever you want to call them, AI enthusiasts), I got a $20/mo sub to try out Neon and it's just breezed through some hardware graphics bring-up problems I've been struggling with on Sol/Astra and Fable. I'm gonna drop my $200 OpenAI sub this coming month in favor of Google's $200 plan just for more Neon tokens.
this week I ran out of codex astra quota and switched to antigravity, boy is Gemini Carbon amazing model! It actually helped achieve a breakthrough I was stuck on for weeks with Astra.
I think it's just you. Carbon is excellent but extremely slow. If Neon is better than or equal to Carbon while also being a lot faster, that'd be a great model. I haven't tried Neon, yet, but they seem to have understood the assignment with the latest round of models.
And tokens/task is very high. They don't seem to have figured out making the models usable in a typical interactive workflow, ie., they're in a Opus-4.7 era.
Yes. Every Google model so far has been disappointment. And their biggest moat, the integration into other google products has been laughable. Today I've asked gemini in gmail to download all attachments from one e-mail address and it turned out it's unable to do it. It's unbelievable how incompetent they are, Google has such a massive edge over OAI and Anthropic and it's still losing.
How do I use it? Do I have to create a Google Console account and navigate through 15 menus in order to generate an API key? Or are there another 7 different ways to achieve similar outcomes?
But my question is, what does this benchmark actually measure about the LLM's engineering abilities? It just looks like a page of random bar graphs to me.
Very high hallucination rates too.