I love these novel ways to benchmark LLMs. I created one myself, I figured creating a programming language designed to confuse LLMs would be a good test, and it does seem to be: https://bench.killswitch-lang.org/.
I wonder if anyone has crafted a LLM benchmarks list website/github.
"In every partnership, there is a person who stacks the dishwasher like a Scandinavian architect and a person who stacks the dishwasher like a raccoon on meth." - Someone online.
At home I'm the architect, my wife's the raccoon. I don't actually believe that you can stack the dishwasher wrong as such, but you can make it hard for yourself to unload, if you don't sort the items correctly.
> I don't actually believe that you can stack the dishwasher wrong as such
There are definitely more and less correct ways, but personally i think it's "wrong" to load it in a way that leaves the arms blocked or (not) tilt pieces that get excessive water pooling on top
shouldn't stacking like a raccoon on meth be the most optimal way though? since if they're too well stacked you risk water being unable to enter tight places
> I don't actually believe that you can stack the dishwasher wrong as such
You most definitely can, they're not magic! The water must hit the right spots and obviously stacking three plates without any space between leads to stuff not getting properly cleaned.
If you try to teach a kid to put the dishwasher you'll learn a ton of wrong ways of doing it! :)
At our place it's opposite, I guess when you do it often enough you get a preferred method though a relentless optimization loop, which in the end ensures it's all clean, the plates are well separated only in 1 particular order, the glasses are placed to the left so all small pans fit at the right, deep plates are placed half over cutlery rack so they are supported and more fit in, small plates in front so all lids fit before them... etc etc etc.
Yep, I have a more optimised model of stacking than my wife’s ideas. This has lead to “discussions” (still at a friendly level) and the dish washer being placed in my list of responsibilities.
I think optimal configuration is very much a NP complete problem.
Years ago after my wife filled the dishwasher I commented that it looked like a 12-year-old filled it. I said it as a joke but it really hurt her and I still feel bad about it. That said, it no longer looks like that when she fills it...
As we know from physics, even the most sophisticated mechanisms (e.g. friction) often ends up having a very simple model (proportional to the normal force).
Sooo, the score is maybe mostly just how much they fit?
For added difficulty, have it 'stack' a 3D rendered volumetric plastic model of a pelican riding a bicycle, like if it were putting a child's toy in the dishwasher.
sure the code isn't hand-written and it's a work in progress, but it's designed to be modelled and calibrated around my own dishwasher so it's verifiably non-slop
I wonder if anyone has crafted a LLM benchmarks list website/github.
I can't. So that would surely be proof of AGI.
At home I'm the architect, my wife's the raccoon. I don't actually believe that you can stack the dishwasher wrong as such, but you can make it hard for yourself to unload, if you don't sort the items correctly.
There are definitely more and less correct ways, but personally i think it's "wrong" to load it in a way that leaves the arms blocked or (not) tilt pieces that get excessive water pooling on top
You most definitely can, they're not magic! The water must hit the right spots and obviously stacking three plates without any space between leads to stuff not getting properly cleaned.
If you try to teach a kid to put the dishwasher you'll learn a ton of wrong ways of doing it! :)
https://theoatmeal.com/comics/dishwasher
I think optimal configuration is very much a NP complete problem.
I did, once ;)
Stacked loads are run against a dishwasher simulation calibrated to my own Bosch dishwasher
Playable by both humans and AI
Currently a WIP, simulations need refining before running a full benchmark, but keen to hear some feedback!
I calibrated it against my actual dishwasher by doing some test runs with baked egg
Sooo, the score is maybe mostly just how much they fit?
https://www.youtube.com/watch?v=ofdMfljL360
Imagine creating real simulated env for every physical task and using RL to train robots. That will be a game changer.