Some models are better than others. Sometimes even working with not-so-good models is not too bad. To a large extent what determines the success of models is the type of market. Let me give some examples.
Equity, FX and commodity markets: Here the models are only so-so. There has been a great deal of research on improving these models, although not necessarily productive work. Combine less-than-brilliant models with potentially very volatile markets and exotic, non-transparent, products and the result can be dangerous. On the positive side as long as you diversify across instruments and don’t put all your money into one basket then you should be ok…at least as long as the market overall is going up!
Fixed-income markets: These models are pretty dire. So you might expect to lose (or make) lots of money. Well, it’s not as simple as that. There are two features of these markets which make the dire modelling less important, these are a) the underlying rates are not very volatile and b) there are plenty of highly liquid vanilla instruments with which to try to hedge model risk. (I say “try to” because most model-risk hedging is really a fudge, inconsistent with the framework in which it is being used.)
Correlation markets: Oh, Lord! Instruments whose pricing requires input of correlation (FI excepted, see above) are accidents waiting to happen. The dynamic relationship between just two equities can be beautifully complex, and certainly never to be captured by a single number, correlation. Fortunately these instruments tend not to be bought or sold in non-diversified, bank-destroying quantities. (Except for CDOs, of course, see below.)
Credit markets: Single name instruments are not too bad. Again problems arise with any instrument that has multiple ‘underlyings,’ so the credit derivatives based on baskets…you know who you are. But as always, as long as the trades aren’t too big then it’s not the end of the world.
Where’s the ‘science’ in this? The science comes in accepting right from the start that the modelling is going to be less than perfect. It is not true that one makes money from every instrument because of the accuracy of the model. Rather one makes money on average across all instruments despite the model. These observations suggest to me that less time should be spent on dodgy models, meaninglessly calibrated, but more time on models that are accurateenough and that build in the benefits of portfolios.