I've got a question. I'm having a bit of trouble phrasing it, so please don't assume I'm against learning for learning's sake, that's not the point.
Whenever I've tried to learn about LLMs, machine learning, etc. I always come out with a bad feeling, compared to learning about software.
Theoretical courses make me feel like it's all baby steps, unless you assume a level of commitment equivalent to a math degree. Practical/applied stuff seems to require levels of compute, training data, etc. that make it unapproachable for a home hacker for everything beyond basic exercises like a handwritten digit classifier.
Now this feeling of "impracticality" is perfectly fine. It would be the expected outcome if I was taking a course on classic literature or modern art. But machine learning stuff feels like it could have room for learning-by-doing applications that aren't entirely pointless, so I wonder if there's some way to approach the topic I'm simply not finding.
Did anyone have the same experience? any tips if so?
I think this is an outcome of playing with systems that we don’t actually understand very well on a deep level. We have found recipes that work, and have used graduate descent (so many PhD theses) to understand and optimize these systems, but we’ve built a model of language that also encodes reasoning, and those things are extraordinarily complex. Providing proper mathematical rigor for their explanation requires, as you say, the level of commitment equivalent to a math degree. Also, because language is so complicated and subtle, you need highly precise statistical models of it, and that just takes a shitload of data; hence the scale. You can train toy models at home, but they will suck because you actually need massive amounts of data to capture the nuance that language requires. And that’s to say nothing of multimodal models.
I have the books on my shelf for 5 years now and never encountered the need to build an AI so i haven't touched it yet. Don't feel bad about using AI to learn but not build one.
You can google stuff to learn without the need to build a search engine too.
I have however learned a lot about math! And other related stuff. I'd argue if you extend your low level skills in math and code when the need for an AI comes up, simple weather / sales forecast, who knows, you'll have an easier time building one.
Get Sebastian's book and build an LLM from scratch. The problem with most textbooks (on any subject, it's not unique to ML/AI) is they usually try to build the theoretical foundations first. In many cases things are better taught top down.
I think it's one of those things where you have to work in the field to really get it and be productive. You just don't have the time or resources otherwise.
I feel the same. I feel machine learning is 1) you need strong math knowledge but also intuition. 2) You need actual production level data, a lot of the nuances of machine learning, applied scientists' roles are how to build strong models with real world, messy, inconvenient data. You need to apply various techniques and not just memorize a specific formula.
Compared to software coding, you build websites, apps etc. The barrier to entry is much higher.
I am trying to pivot into machine learning myself, so far my understanding is try to find a software engineer role where you deal with data, model infra, data pipeline, etc. Then within the same company, try to do more modeling work and build intuition and resume credentials in this area. Someone correct me if I am wrong.
I got his book Build Your Own LLM From Scratch and it's super readable and does a great job at holding your hand through the entire process...need to finish reading it though!
I did the same. I got "Build Your Own LLM From Scratch" last christmas and it is a great resource for learning how LLMs work, from tokenization to generating text. The author makes a great job explaining each new concept. If you know PyTorch (or at least know about tensors), the material will be approachable for you.
Sebastian, if you are reading this, this is an idea for another book: how to orchestrate agents for adversarial code review. I've been looking for learning resources to make something similar to this [1] at a smaller scale. I am aware of codex subagents, though. However, given that Sebastian already has a book for LLMs and another book on planning, I think adversarial code review is an interesting next step.
I don't like working with neural networks and I always found everything overcomplicated or oversimplified, sometimes both at the same time.
But this book is almost perfect. Every new concept is well described and every is well founded. And every step moves you forward (if you know, you know)
I like it too. Haven't got very far through it yet but just learning about how to use PyTorch and finally getting how back propagation works via autograd etc. was really fun and enlightening.
I’ve read, worked through, and watched his LLM book as well as this one! Things finally clicked, can’t recommend both books enough—his pedagogy is great (at least for me).
All of these build your own model guides are like someone putting out a build your own CPU guide 10 years ago. Interesting, but not of any practical use.
I just started reading his book, and he makes it easy to grasp difficult concepts. As a matter of fact, I have been working on platform where you can learn how AI models work in an interactive manner.
This is an essential resource and his book is a must read from this author. I have the upmost respect for Raschka for his research and detailed explanations in AI including him dissecting research papers.
I'm a bit confused as to who this is for - apart from learning how the tech works that is.
If you're truly going to build your own model (and commit to the cost of that), then video tutorials will not cut it.
And if you don't really, then I wonder if this is really the best style of conveying the information you'd be interested in?
I mean I might be wrong and maybe this helps emerging players or something like that, but to me it is more likely that this is more of a monetisation strategy of a personal brand. Living off selling to people that think they _should_ but then in the end don't.
Why teach people to play guitar once a week if they won't be a new Jimi Hendrix? Why teach people to run if they won't be a new Usain Bolt?
Maybe because they don't want to be an "emerging player" or something, they only want to understand how a new technology that is changing the world works and prefer to learn by doing.
If Sebastian is doing it by its personal brand, he really deserves it. I would trust him because I know he is a reference on his area of knowledge and I know he is because he shares his knowledge instead of only making fancy talks.
Let’s not be this negative, we might be going towards a new direction with technology that could allow small teams to build their own models. Maybe not frontier models. Maybe not close to the others in generalized benchmarks. But perhaps they could be really specialized into solving one particular problem better than the others.
If you're truly going to build your own model (and commit to the cost of that), then video tutorials will not cut it.
That makes no sense. It's like saying if you want an engineering degree, going to lectures will not cut it. Of course not, you have to do the homework and projects too.
You're right about me not being the target audience, but you're wrong about the second sentence I think.
The target audience of this is for people that like to _think_ they're doing stuff by paying for it, to then not do anything.
It's a purchase of an identity, basically.
Which.. I guess is fine as something to cover Sebastian's expenses to enable him to keep delivering this free stuff. If you want to take money from someone, taking from this group is likely the most ethical thing to do.
You are completely off base. His initial book was released almost two years ago. I post train models, for work, and I learned more from his book on LLMs than any other source on LLMs. There is a lot more to it than that book, but you have to start somewhere, even if you have a decent theoretical foundation. There are a bunch of people who learned how to build them from his book.
How do you think people learn to build/train models? It comes down from the sky?
"I know I'm not the target audience and I don't have any interest in how things work, but let me tell you how you're wrong by telling you who the actual target audience is"
What's next? Are you going to give us your opinion of music genres you don't ever listen to? And tell us who should listen to them too, I suppose.
You sound like you are projecting a bunch of motivations and assumptions about character onto other people. But you projecting such things onto another person is a terrible way to begin to understand something.
There are genuinely people who enjoy learning how something works by doing smaller scale versions of it themselves. This site happens to have a much higher average percentage of them than most other places.
Whenever I've tried to learn about LLMs, machine learning, etc. I always come out with a bad feeling, compared to learning about software.
Theoretical courses make me feel like it's all baby steps, unless you assume a level of commitment equivalent to a math degree. Practical/applied stuff seems to require levels of compute, training data, etc. that make it unapproachable for a home hacker for everything beyond basic exercises like a handwritten digit classifier.
Now this feeling of "impracticality" is perfectly fine. It would be the expected outcome if I was taking a course on classic literature or modern art. But machine learning stuff feels like it could have room for learning-by-doing applications that aren't entirely pointless, so I wonder if there's some way to approach the topic I'm simply not finding.
Did anyone have the same experience? any tips if so?
You can google stuff to learn without the need to build a search engine too.
I have however learned a lot about math! And other related stuff. I'd argue if you extend your low level skills in math and code when the need for an AI comes up, simple weather / sales forecast, who knows, you'll have an easier time building one.
Compared to software coding, you build websites, apps etc. The barrier to entry is much higher.
I am trying to pivot into machine learning myself, so far my understanding is try to find a software engineer role where you deal with data, model infra, data pipeline, etc. Then within the same company, try to do more modeling work and build intuition and resume credentials in this area. Someone correct me if I am wrong.
Sebastian, if you are reading this, this is an idea for another book: how to orchestrate agents for adversarial code review. I've been looking for learning resources to make something similar to this [1] at a smaller scale. I am aware of codex subagents, though. However, given that Sebastian already has a book for LLMs and another book on planning, I think adversarial code review is an interesting next step.
[1] https://bun.com/blog/bun-in-rust
[2] https://learn.chatgpt.com/docs/agent-configuration/subagents...
But this book is almost perfect. Every new concept is well described and every is well founded. And every step moves you forward (if you know, you know)
If you're truly going to build your own model (and commit to the cost of that), then video tutorials will not cut it.
And if you don't really, then I wonder if this is really the best style of conveying the information you'd be interested in?
I mean I might be wrong and maybe this helps emerging players or something like that, but to me it is more likely that this is more of a monetisation strategy of a personal brand. Living off selling to people that think they _should_ but then in the end don't.
It's also not a bad foundation for people who want to understand how these things are trained and work.
Maybe because they don't want to be an "emerging player" or something, they only want to understand how a new technology that is changing the world works and prefer to learn by doing.
If Sebastian is doing it by its personal brand, he really deserves it. I would trust him because I know he is a reference on his area of knowledge and I know he is because he shares his knowledge instead of only making fancy talks.
some people enjoy learning how the tech works
It's a critique of the people and the lack of substance. But not of the author. The author is highly capable.
That makes no sense. It's like saying if you want an engineering degree, going to lectures will not cut it. Of course not, you have to do the homework and projects too.
Something is off here. It indeed makes no sense
The target audience of this is for people that like to _think_ they're doing stuff by paying for it, to then not do anything.
It's a purchase of an identity, basically.
Which.. I guess is fine as something to cover Sebastian's expenses to enable him to keep delivering this free stuff. If you want to take money from someone, taking from this group is likely the most ethical thing to do.
How do you think people learn to build/train models? It comes down from the sky?
What's next? Are you going to give us your opinion of music genres you don't ever listen to? And tell us who should listen to them too, I suppose.
He is covering expenses. Probably barely.
It seems that you want to oppose something really badly today, but you're struggling to find something that makes sense for more than 5 minutes.
There are genuinely people who enjoy learning how something works by doing smaller scale versions of it themselves. This site happens to have a much higher average percentage of them than most other places.