The ReLU illusion of progress
A message to first-year PhD students who feel like they have nothing to show.
Before I submitted my first paper in my PhD, I often thought I was falling behind — comparing myself to others based on the only measurable things I could. I had nothing.
It felt that my first year had gone by in vain. Anyone could have told me otherwise — I would not listen. I see the same tendency in friends who just started their PhDs.
“Progress”, as measured by conventional, easily measurable metrics like papers1, seems to be zero. For a very long time. Just like the initial (negative) part of a function we call a rectified linear unit (or ReLU) in machine learning.
The problem isn’t that it starts out flat — but that for your PhD, you do not know when (or even whether) it starts to increase. Even worse, you might feel clueless because there is no signal (gradient) in the flat part.
This whole illusion hinges on how we measure progress. I get it, it’s easy to count papers — but it neglects so much nuance that it stops being useful, especially as you start doing research.
Feedback is essential for learning — to use another machine learning analogy: the credit assignment problem in RL states that it’s hard to learn a good solution when feedback (rewards — think the score in a computer game) is sparse and not immediate.
Make your rewards dense. How?
For example, write down how much time it took you to deeply understand a paper — revisit in a few months. It will be faster. Or track how long it takes to set up experiments. It will be faster, too.
Yes, you still need to trust that the metrics the system cares about will start improving for you. But if you have numbers showing that you are actually making progress, you can see ReLU-flatland for what it is — an illusion.
P.S.: I thank JZ and AM for their insights on the topic, and MK for his advice.
P.P.S.: A beautiful quote from Oliver Burkeman’s The Imperfectionist: “Uncertainty is our basic state of existence, not something to be got through to the certainty beyond.”
P.P.P.S.: an amazing speech-to-text app that actually works, with clever functionality like custom snippets and styles. This link gives you and me a free month of Pro. The free version is also plenty—it has rate limits, but you get almost the same functionality
As I was writing this post, Collin Raffel released a thoughtful essay titled We Are Over-Indexing on Paper Acceptance. I couldn’t agree more
As an ex professor with many PhD students: This is so true and such a great visualization.