Scholarship is excellent, but it is also expensive. It takes a long time to catch up to the state of the art, even for a narrow subject.
I recently read 90% of the literature on machine ethics, a recent and small field of inquiry, and it took me about 40 hours to find all the literature, acquire it, and read (or skim) through it. Doing the same thing for an older and larger subject will take far more time than that. And of course most of the literature on any subject is not valuable.
Other times, you get lucky. Let's say you want to figure out how to beat procrastination. You could introspect your way to a plausible solution, but you might end up being wrong. So, you check Wikipedia. Not very useful. Next, you search Google Scholar for "procrastination." An article on the first page looks like what you want: an overview of the scientific research on procrastination. It's called "The nature of procrastination: A meta-analytic and theoretical review of quintessential self-regulatory failure," and it's available online! As it turns out, you can do a pretty decent job of catching up on the science of procrastination just by reading one article. (Of course it's not that easy. You should be more thorough, and explore alternate perspectives. Psychology is not settled chemistry.)
And in machine ethics, it turns out that most of what you'd want to know is summarized nicely in a single book: Moral Machines (2009).
But on other topics, you won't be so lucky. Suppose you want to study the neuroscience of how desire works. You check Wikipedia, and it has a section on the psychology and neurology of desire. But it doesn't tell you much. A Google Scholar search is even worse. You check the index of a large neuroscience textbook for "desire," and come up with basically nothing. The Stanford Encyclopedia of Philosophy article on desire is pretty good, but it barely touches on neuroscience. It does point you to two good resources, though: the book Three Faces of Desire, which sounds like it will cover the neuroscience, and the work of neuroscientist Kent Berridge. Now, because I have studied the neuroscience of desire, let me spoil the surprise at this point: this research project is not going to be so easy. You have a very long "literature slog" ahead of you.
So I won't argue that scholarship is always or even usually the instrumentally rational thing to do when you want to make progress on a certain problem. Sometimes the costs of scholarship outweigh the benefits.
But to make that judgment, it will help to know just what those costs and benefits are.
Some Costs of Scholarship
1. Scholarship takes time and effort.
This is the biggie. Scholarship takes time. Especially if you don't have much experience with it already. Resources like Google Scholar make it easier than ever, but scholarship still requires lots of patience and perseverance and procrastination-mastering.
2. Opportunity cost.
The fact that scholarship takes up time means that while you're doing scholarship, you're not doing something else that might be more productive. Scholarship can even serve as a form of procrastination, reading things just because they're on your to-read list so that you can avoid doing something else. [thanks taryneast]
3. Studying some subjects can weaken or corrupt you.
Without (and maybe even with) rationality training, studying certain subjects will make you dumber. To me, postmodernism and even most analytic philosophy look like a good candidates for stupid-making subjects of study. Other candidates include theology, literary theory, and scripture scholarship. These fields can teach bad modes of thinking, false "facts", and even absurdities. Luckily, there are some heuristics you can use to estimate the value of a field.
4. Some things cost money.
I've spent quite a bit of money on scholarship: on gas for trips to a university library so I can download papers from behind the paywall, on hard drives to store tens of thousands of PDFs, on books purchased from Amazon, and so on.
5. Scholarship can be a shiny distraction.
A long list of footnotes and references might be built up to conceal the fact that the ideas in an article or book are of little value. [thanks Perplexed]
Some Benefits of Scholarship
1. You'll avoid some mistakes and confusions.
Suppose you were about to argue, with Jeremy Bentham, that all intentional human action aims at pleasure. Doing some research on the neuroscience of intentional action would help you avoid that mistake. As it turns out, it's just not true that all intentional human action aims at pleasure. Pleasure is only one goal among many.
Scholarship can also help you avoid confusions, for example between two kinds of intrinsic value.
2. You'll learn to speak the same language as everyone else, and thus communicate more effectively.
When you read other works on the topic you're discussing, you discover the established terms already used for discussing that topic, and you can begin to speak the same language as everybody else.
3. If your rationality skills are sharp, you'll become generally smarter and wiser.
If you're equipped to recognize magical categories and mysterious answers, consuming a diverse array of fields can give you a broad, integrative kind of knowledge. But be especially wary of subjects that can make you stupid, like postmodern philosophy.
4. You won't waste time re-stating what has already been said elsewhere, better and more knowledgeably than you can.
Many times, I've decided I want to write about X. So then I start researching X to prepare for writing. Then I discover that somebody wrote an article that says everything I wanted to say about X, but they've been studying X for ten years and really know their stuff. Then all I have to do is type two paragraphs about it and link to it on my blog. Hurray!
5. You'll be taken seriously by more people, and have more access to useful experts.
Researching your topic and citing the relevant literature are pre-requisites for some activities like academic publishing, which can get more people to take you seriously because you've put forth the effort to pass a basic test of quality: peer-review. Really smart and accomplished people have too much to read already, and most of them are unlikely to read what you've written if you haven't even bothered to pass peer review.
And, the more smart people take you seriously and read your stuff, the more brain power you can call upon in solving the problems you care about.
6. You'll avoid the tendency to over-trust bearers of good info.
Especially when approaching a subject for the first time, you might read something so bloody intelligent that your mind can't help but cast a halo around its author and accept whatever he or she said. But continuing with your scholarship, and reading lots of people who agree and disagree with that author, can help you see him or her as part of a large enterprise that has been struggling on certain problems for a long time, and may expose you to data that disproves claims made by the original author that impressed you.
Conclusion
Nobody on Less Wrong should be fooled by the fact that I listed more benefits than costs for scholarship. That doesn't mean scholarship is always a good idea. Often, it's not a good idea.
But having a list of costs and benefits can help you decide whether scholarship is worthwhile for a particular project - or, how much scholarship is worthwhile.
Now, what did I miss?