OP, you should not have written this in Rust. It should be in PyTorch, which is by far the most popular. We can’t tell if this architecture is good or whether there is a problem in your implementation.
You can test the whole thing for free on a GPU with Google Colab. Test both the transformer and your new architecture on a larger dataset. Something that maxes out the GPU for an hour each run.
Also, the readme mentions keeping the same optimizer schedule which sounds nice at first but they are completely different architectures. The loss is high on the transformer, did you try raising the learning rate on it?
In general I’m interested in parameter efficient architectures. I don’t think transformers are optimal, and indeed many improvements have been made to vanilla transformers. But if you have an idea for something better you need to show it.
One criticism is that you have to install the same package, torch, but from different Python indexes in order to install the cpu version or gpu version, on Linux. On windows, `pip install torch` gets you the cpu version. On linux, that gets you a ton of Nvidia extras that take a lot of space.
GPU support should really be a optional extra eg `torch[gpu]` or `torch[nvidia]`.
Apparently op cares a lot about speed, which is fine, but ML researchers care about correctness first, speed second. And it makes sense, because they are not as resource constrained as OP.
Most PyTorch tensor operations are cython not python. So imo rewriting in rust is not going to have an enormous speed up. If that really was the concern we should see a throughput comparison vs PyTorch or something.
Edit: the claimed reason of "updates" also seems to not exist in code. If the author is going to use LLM for this, the very least they can do is to ask it to check properly before publishing.
Rust has its place but python is default for this kind of thing. By doing it in rust, they have now changed two things: the implementation of the transformer, and this new model.
Rust matters because it means you can actually deploy it without going insane. I wonder if the anti-Rust zealots have actually ever used pip, especially on Windows.
He could have said "... not written in Python" - would that have been acceptable?
Yeah I read the title and did a double-take. It's an uninteresting choice for systems like these. The more pressing concerns, which go undescribed, are the exact mathematical choices behind the actual model. Rust provides almost zero value here because tensor stuff is all just 2d-arrays of floats for the most part.
You can test the whole thing for free on a GPU with Google Colab. Test both the transformer and your new architecture on a larger dataset. Something that maxes out the GPU for an hour each run.
Also, the readme mentions keeping the same optimizer schedule which sounds nice at first but they are completely different architectures. The loss is high on the transformer, did you try raising the learning rate on it?
In general I’m interested in parameter efficient architectures. I don’t think transformers are optimal, and indeed many improvements have been made to vanilla transformers. But if you have an idea for something better you need to show it.
That said I don't know what's wrong with using a Rust AI framework like Candle.
GPU support should really be a optional extra eg `torch[gpu]` or `torch[nvidia]`.
Edit: the claimed reason of "updates" also seems to not exist in code. If the author is going to use LLM for this, the very least they can do is to ask it to check properly before publishing.
> The implementation language is a detail, and a Python port is welcome.
I guess the post title could have dropped "in rust"
Weird turn of phrase very typical of AI.
Why did you choose Rust? Why does that matter?
Nothing wrong with Rust. Lots wrong with the bandwagon that “in Rust” somehow adds value.
He could have said "... not written in Python" - would that have been acceptable?