How to Develop Great Ideas
Principles for better research
A Ph.D. is not about changing the world; it’s about learning. Still, the question I get most frequently is
How to develop great ideas?
The short answer is that you cannot force it. But there are useful principles that can guide you forward.
Part 1: developing ideas
Go to the frontiers of knowledge
To have any ideas, you cannot forgo diving deep into a topic.
Just as a coastline, the more you zoom in, the more of the infinitude you will recognize. Knowledge at the edges is a fractal.
Exploiting forever sounds like life insurance for your career. But you need to be very considerate about the diminishing returns and about falling prey to the method orientation. Not every nail is worth hammering. Perhaps surprisingly, Richard Hamming “accused” Claude Shannon of being guilty of this, as despite his celebrated results in information theory, he stuck too long to it.
Recent research demonstrated that shifting your field can improve your impact, because you are not constrained by the field’s doctrine. Paul Graham formulated this advice as1:
Be professionally curious about a few topics and idly curious about many more.
Broad knowledge/interests can help connect the dots without reinventing the wheel. The caveat is finding the balance of range without spreading yourself too thin.
“Don’t be the best, be the only"—Kevin Kelly
The advantage of Range is sound: the probability of being an expert in both X and Y is strictly less than being an expert in only one. The catch is finding fields that are meaningful to combine.
Failed past ideas were not all bad
The graveyard of failed ideas includes some great ones. They might have failed because2 :
technological limitations,
the political climate,
unfavorable culture, or
an unfriendly economy.
There is a big caveat, though:
Failure in the past does not make a bad idea a great one.
If you are happy to take that risk, what you could do—just like Katalin Karikó—is to find very old, forgotten, and niche papers.
An environment for fostering great ideas
Ideation is not just sheer hard work and willpower, it is also being in an environment conducive for developing great ideas. A few hallmarks of such environments are3:
Minimize distractions;
Immerse yourself in nature;4
Go for walks;5 and
Interact with people of diverse backgrounds, expertise, and interests.
Sometimes closing out the consensus can help you think independently, as you won't be influenced in advance.
Write or prototype to close the loose threads
You need to write (or prototype) because it does not allow you any slack. Writing a proof or a program, or setting up an experiment in the lab, forces you to specify all details.
Take rest
Developing great ideas requires taking rest—creativity requires diffuse thinking. Rest affects your work performance6. You need to let your mind wander. Go for walks!
Part 2: selecting great ideas
Only do it if you can’t not do it
An idea should become more and more interesting as you spend time on it. If you feel an urge to work on an idea, then you should do it. Lesser motivation predestines an idea to die prematurely, as you will get stuck. This also means that you need to abandon your lesser ideas.
Oliver Burkeman’s heuristic is that if you forget an idea, then it was not so interesting for you. But sometimes you forget great ideas—so keep notes, and revisit them. If you are still excited, go for it.
Sturgeon’s Law: 90% of everything is crap, including your ideas
The ideas started coming, but then
How do you realize which is a great one?
For most ideas are bad, you need to:
Suspend judgment to get the bad ideas out of the way
Seek feedback to figure out which are the bad ones; i.e., to develop taste.
In most organizations, including research labs, people have a hard time both getting and receiving feedback. For learning, you need a kind learning environment, which hinges on feedback. Failing fast is also about feedback.
A question I found very useful is:
I have this idea X, do you want to work on it?7
The goal is to overcome your ego to create space for the truly great ideas. What helps is looking for the following signs.
Signs of a great idea
It seems embarrassingly simple, trivial, "I-am-sure-that-someone-already-thought-about-it"
Many times these are the ideas that need to be spelt out8.
It seems ridiculous, or outright wrong.
Maybe it goes against the grain, or it disagrees with the expected outcome of the experiment. Dig there. But be careful not to be a contrarian for being a contrarian's sake. In the words of Adam Robinson:
If you want to find gold, it’s where things don’t make sense.
The two flavors here are:
It works, but the consensus/your intuition tells you it should not
It does not work, but the consensus/your intuition tells you it should
It targets the implicit assumptions
If everyone is on autopilot, traditions and assumptions can get outdated. Paul Graham’s questions can help figuring this out:
What are people in your field religious about, in the sense of being too attached to some principle that might not be as self-evident as they think? What becomes possible if you discard it?
Is there a widely held but mistaken belief?
Is there an underexplored aspect of somethong people think is already well-explored?
Actionable advice
Increase your surface area to luck
read widely
read old
talk to people from many walks of life
Never blindly accept traditions
Ask for feedback to weed out bad ideas
Look for combining (seemingly) disparate fields
Look for the seemingly wrong, the neglected, the counterintuitive, the unfancy
Dig Deeper
Oliver Burkeman: How to forget what you read
Paul Graham
William Deresiewicz: Solitude and Leadership
- : Range
Adam Alter: Anatomy of a Breakthrough
Shane Parrish: The Great Mental Models
Annie Murphy Paul: The Extended Mind
Being a t-shaped student is also a good mental model, and so is having Range
Kudos to Adam Alter and his book, Anatomy of a Breakthrough, for this list. See also Porter’s five forces
For more details, research, and actionable advice, read The Extended Mind
Looking at pictures counts, too!
Fun fact: there is an expression, solvitur ambulando, for “proof by walking”. It works!
This is a well-known fact in sports, which people in academia/business do not seem to be aware of. For this reason, I keep repeating this
Asking about being interested is not enough, because it’s cheap to say “I am interested!”. You should make them have skin in the game. If they would commit to work on it, they must find it really interesting
Some reviewers might still dismiss such ideas as “obvious”
Always a pleasure to read your blogs (or talk to you) :) Thanks for the advice!