How to Iterate on Yourself Systematically
A few days ago I came across a tweet by Andrej Karpathy on “how to become an expert in a field.” It had three points. First, find a concrete project to work on, and look things up as you hit things you don’t understand—learn in an “output-driven” way. Second, don’t go “bottom-up”: don’t insist, like you did in school, on finishing all the foundational theory before you start. Third, only compare yourself to your past self, not to others.
This reminded me of a line from Feynman: if you can’t explain something clearly to someone who knows nothing about it, it means you don’t really understand it yourself.
Both are saying the same thing: use output to check input. You think you understand, but only when you actually do something or open your mouth do you find out whether you really do. This piece is about making clear why this approach works, why it’s hard, and how to keep at it.
Parallels with Deep Learning
Training a neural network is a loop of four steps:
- Forward (forward propagation): the network produces an output based on its current parameters.
- Compute the Loss: compare the output to the correct answer and calculate the gap.
- Backward (backpropagation): based on the gap, work backward to figure out which direction each parameter should be adjusted.
- Gradient descent: nudge the parameters a little in that direction.
Then go back to step one and repeat thousands of times, and the output gets more and more accurate.
Human learning is almost identical: “do a project first” is the Forward pass; the gap between what you produce and the ideal result is the Loss; and your review of “what went wrong, how to do it better next time” is Backward and gradient descent. When Karpathy says “put in enough reps and you’ll become an expert,” he’s essentially saying this is an iterative algorithm. There’s no one-shot solution; all the power comes from small, numerous, sustained repetition.
The Limits of Being Human
The biggest difference between people and neural networks is this: a network feels nothing in the face of a large Loss, but a person hurts.
And there’s no getting around the pain. The reasoning is simple: to improve, you have to produce output in an area you haven’t yet mastered, and along the way you’ll inevitably make frequent mistakes and keep bumping into your own ignorance. Conversely, practicing something you’ve already fully mastered yields no new progress. Real growth always happens at the edge of your ability, and the edge is, by definition, uncomfortable.
So the real difficulty of this approach isn’t intelligence—it’s how to endure the discomfort of the process.
Why do people hurt while networks don’t? Because people tend to conflate “how I did” with “what kind of person I am.” Once your self-worth is built on “I’m capable,” every mistake becomes a negation of who you are, and so you instinctively avoid every situation that might expose a shortcoming—which means avoiding all growth.
To face this pain, the key is to separate two things: you can criticize your “behavior” mercilessly, but don’t wound your “worth.”
Think of it through the neural network: gradient descent adjusts the parameters, which correspond to your behavior, skills, and methods—things that are supposed to be corrected over and over. The architecture, by contrast, is something deeper that shouldn’t be messed with carelessly. You should place your self-worth at the architecture level, not the parameter level. That way mistakes only change your behavior; they don’t shake “who you are.” If you do it the other way around and let every Loss strike the architecture itself—equivalent to tearing down and rebuilding the entire network every time you make a mistake—training won’t converge at all; it will just oscillate violently and then collapse. That is the state of “totally negating yourself the moment you slip up.” Stable training requires that the Loss only updates the parameters and never blows up the architecture.
Two Tricks
Trick one: write your mistakes down serially.
Every time you hit a mistake, a gap in your knowledge, or something you want to improve, record it in a log or notebook. There’s a wonderful role model for this: the Meditations of the Roman emperor Marcus Aurelius, written as private notes “to himself.” Even as an emperor, he recorded and reviewed his own shortcomings every day.
Why does writing it down help? When pain stays in your head it’s a tangle that loops around—many thoughts churning at once, none of them settling—and that’s exactly its most tormenting state. The act of writing forces you to line it up into a single thread: you can only write one sentence at a time, and one sentence can only convey one meaning clearly. This “serialization” forces the brain into a rational state, where you think the matter through calmly instead of letting emotions race around in your head and drain your confidence.
Trick two: as soon as you finish writing, start the next round.
Once the log is written, this round’s Backward and gradient descent are done. The next step is not to linger in the pain but to go straight into the next Forward pass—commit to a new action. There’s a basis for this too: when training, you don’t run Backward on the same batch of data ten times in a row, because that would make the network overfit to that single sample. People are the same: chewing over the same mistake again and again only traps you in the details of this one failure, and you fail to extract a lesson you can use next time. “Thinking about it more only brings more pain” is, in essence, “running Backward more only causes overfitting.”
These two tricks go together, and there’s an order to them: the reason you can dare to “forget” is precisely that you’ve already “written it down.” Once the matter is clearly filed away, the alarm in your head that says “you have to keep remembering this or you’ll pay for it” can be switched off; if you need to recall it later, you just check the record. Writing the log closes out the previous round; acting immediately kicks off the next one.