If psychology worked, I would expect marketing firms to use it to make millions of people buy tons of shit that they don't need and that won't make them happy.

Shorter version of OP's argument.

If personality psychology holds water, I would expect dating sites to use it and produce better results than Traditional Romance. Does it? From the outside looking in, it looks like it does.

It would also be useful in selecting dorm room compatibility, which I can tell you from the inside looking out does not work at all or isn't being used. I wouldn't expect it to be used in this context, though. No money in it.

I would add to this that having a method would often (but not always) produce a clear "leader in the field" (first-mover advantage going to the discoverer). So seeing Google's share of the market is a strong indicator (even without first-hand knowledge) that "they have a serious advantage in search" vs existence of many competing diet companies does not tell me "they figured out nutrition".

There's interesting thing: Some people managed to acquire a lot of wealth via trading. That would lead you to believe their claims with regards to the methods they use being effective.

However, if you simulate the stock market with identically skilled agents, you obtain basically same wealth distribution as observed in the real world, with some few agents ending up extremely 'rich'. One can imagine that such agents, if they were people, would rationalize their undeserved wealth to feel better about themselves.

Tetlock's political judgment study was a test for macroeconomics, political science and history. Yet people with PhDs in these areas did no better on predicting macro political and economic events than those without any PhD. Maybe macro helps in producing good econometric models, but it doesn't help in making informal predictions. (Whereas one suspects that physics and chemistry would help in a test of quick predictions about a novel physical or chemical system, vs. people without a PhD in these fields).

Two issues with this heuristic:

1) It doesn't work well for credence goods.

2) Sometimes it takes a long time for sciences to find an application, two modern examples are astrophysics, and particle physics.

Re: your examples successful spin-offs for psychology, to what extent did these therapies come out of well-established theory? Maybe someone can weigh in here. It seems possible that these are good therapies but ones that don't have a strong basis in theory (in contrast to technologies from physics or chemistry).

Physicist Ilya Prigogine developed his famous theory of dissipative systems which was expected to explain a lot of things from thermodynamics of living systems to the nature of the arrow of time. It is a very well-developed and deep theory. Yet, in my scientific life, I have never seen an actual numerical calculation of a measurable quantity utilizing any of Prigogine's concepts such as "rate of entropy production". Looks definitely like a missing spinoff!

If stock market economics worked, Noble prize winners would make money.

(This is slightly unfair, since the Black-Merton-Scholes theory does make other people money, to an extent. Additionally, while Merton and Scholes were on the board, LTCM was not strictly based on their theories. Still, surprised to see that this hasn't been mentioned.)

It seems to me that there are two different heuristics here and it is worth separating them.

But first I should explain why I think my initial reading of this post suggests heuristics that I think are problematic. The mere existence of CBT does not seem like strong evidence for psychology. It is no more evidence for modern mainstream psychology than freudian psychoanalysis is evidence for freudian psychology. As I understand it, CBT is gaining market share against other forms of talk therapy, but largely because of academic authority, roughly the same way that the other therapies got established. I am a fan of CBT because its proponents claim to do experiments distinguishing its efficacy from that of other talk therapies and failing to distinguish other talk therapies from talking to untrained people (which is still useful). But why do I need CBT for that? I can check that mainstream psychologists are more enthusiastic about experiments than freudian ones without resorting to the particular case of CBT. Similarly, competing nutritional theories are successful in the marketplace, sold both by large organizations with advertising budgets (Weight Watchers vs Atkins) and personal trainers working by word of mouth. But I agree that they example of CBT sheds light on psychology.

One heuristic is that experiments with every-day comprehensible goals are more useful for evaluating a field than experiments of technical claims. Most obviously, it is easier to evaluate the value of the knowledge demonstrated by such experiments than technical knowledge. Knowing that statins lower cholesterol is only useful if I trust the medical consensus on cholesterol, but knowing that they lower all-cause mortality is inherently valuable (though if the population of the experiment was chosen using cholesterol, this is also evidence that the doctors are correct about cholesterol). Similarly, the efficacy of CBT shows that psychologists know useful things, and not just trivia about what people do in weird situations. Moreover, I suspect that such experiments are more reliable than technical experiments. In particular, I suspect that they are less vulnerable to publication bias and data-mining. Certainly, I have to learn about technical measures to determine how vulnerable technical experiments are to experimenter bias.

The other heuristic is that selling a theory to someone else is a good sign. Unfortunately, this seems to me of limited value because people buy a lot of nonsense, not just competing psychological and nutritional theories, but also horoscopes. How does the military differ from academic psychologists? I'm sure it hires a lot of them. They do much larger and longer experiments than academics. They do more comprehensive experiments, with better measures of success, analogous to the advantage of all-cause mortality over number of heart attacks (let alone cholesterol). They could eliminate publication bias because they know all the studies they're doing, but only if the people in charge understand this issue; and there is still is some kind of bias in the kind of studies they let me read. These are all useful advantages, but in the end it does not look very different to me than the academic psychology we're trying to evaluate. Similarly, industry consumes a lot of biological and chemical research, which is evidence that the research is, as a whole, real, but it fails to publish attempts to replicate, so the information is indirect. On the other hand, these industries, like the military, use the knowledge internally, which is better evidence than commercial CBT and nutrition, which try to sell the knowledge directly, and mainly demonstrate the value of academic credentials to selling knowledge.

I have been unable to find any practical spinoffs of gender studies.