Can You Trust Monte Carlo Models?

I recently found an article about the use of Monte Carlo simulation for financial planning and it resonated with some comments that I have heard from people who are considering the use of this type of tool. The article, written by Moshe Milevsky and Anna Abaimova (of The Individual Finance and Insurance Decisions Center at Canada’s York University) appeared in July on the website of The Journal of Financial Planning. The article raises a number of common questions about the use of Monte Carlo tools for financial planning and serves as a nice focus for a discussion of the kinds of testing and validation that is required to make Monte Carlo programs useful for planning and managing a portfolio of real assets.

The title of this article, Will The True Monte Carlo Number Please Stand Up?, is a very good starting point for clearing up some points of confusion about the use of Monte Carlo models for planning. The focus of the article, as the title suggests, is that different Monte Carlo simulations can give different answers as to the survival rates of a generic portfolio for supporting a stream of retirement income. The authors compare six different Monte Carlo models in determining the sustainability of a theoretical retiree’s income.

In this study, the authors started with the assumption that his “entire nest egg was assumed to be invested and rebalanced in a portfolio of diversified equities which was projected to earn an arithmetic average 7 percent (after inflation) each year, which is equivalent to a geometric average of 5 percent with a standard deviation or volatility of 20 percent.” They then attempt to compare the six different Monte Carlo models. Some of the models project survival rates for specific time horizons (the probability of the portfolio sustaining a specific level of income for a specific period of time) while two the models fold in mortality rates—which means that they calculate the odds of being able to sustain an income until death, where your age at death is uncertain. Needless to say, there are significant differences between these models—largely because the underlying assumptions are different. The authors discuss the large range of differences as though there is problematic or odd here:

“Our first reaction was to blame the programmer or manufacturer of the [Monte Carlo models] for building a faulty product. Like all high-tech gadgets on the market, some are better than others, but that is a simplistic, knee-jerk reaction. The true reason for the divergence of results is more complicated and subtle…”

At this point, I beg to differ. There is nothing subtle about the fact that these models were built with very different underlying algorithms and assumptions. Among these models, one simply re-samples historical data while others make arbitrary assumptions about the future returns and volatilities of various asset classes. It should come as no surprise to anyone that these models yield different results. The authors further opine that all of this disagreement can be solved if the financial planning industry were somehow to adopt standards to make the models generate output that is more similar across platforms.

We performed our own comparison study in 2005 by comparing our own Quantext Monte Carlo simulations to three other models under very controlled conditions, and accounting for differences in the underlying models to the extent that this is possible.

This comparison is similar to the analysis by Milevsky and Abaimova (hereafter referred to as M-A) except that we chose four models that were conceptually similar and that were first reviewed to get a sense that they were reasonably robust. We found that, with consideration given to differences between models, the results were remarkably consistent.

There are several key points that individual investors and their advisors would do well to keep in mind when reading the M-A article—or others like it. First, some of the available Monte Carlo models – such as the one on Moneychimp.com (one of those included in the M-A study) – are designed for purposes of illustration and certainly not to provide concrete numbers for planning. Second, anyone who uses software for financial planning must do some due diligence. Why would one believe that any model is providing reasonable results? The four models that we used in our analysis were selected based on a reasonable degree of internal consistency (to the extent that this could be determined) based on our due diligence and that appeared to be professionally executed.

Aside from the basic mathematical issues of Monte Carlo analytics, there is a much larger issue at hand—one that is not mentioned at all in the M-A article. A Monte Carlo tool that calculates survival rates based on either pure history or simply an a priori assumption that your portfolio will generate X% per year with Y% of standard deviation is basically useless—the hard part of the problem is coming up with X and Y for each asset and accounting for the impacts of the differences between real asset performance and hypothetical index performance. The authors pose the following question:

“What is the probability the total return from the S&P 500 index will be greater than 5 percent in 2006? There are a number of philosophical approaches to dealing with such a question. One is to literally “dump” the one-year historical returns of the S&P 500 index in a “hat,” sample from this hat a large number of times, and then count the frequency with which the sample average exceeded 5 percent. This approach is at the heart of Monte Carlo. But as many mathematicians know, another approach is to fit a curve (for example, the normal distribution) to long-term historical returns and then evaluate the curve—that is, compute the tail probability—at the 5 percent mark.”

We can agree that the ability to calculate the probability that the S&P500 will generate a return greater than 5% in any given year is going to be critical to the results from a Monte Carlo model. Here is where things get interesting. To come up with assumptions about the probability of the market generating some percentage of return in a given year requires some critical inputs—namely how to determine the future volatility and expected rate of return for the S&P500. The authors of this article start with an assumption about the average rate of return and its standard deviation (and focus on how the calculation is performed once you have these inputs), but the results that a Monte Carlo model generates will be sensitive to these input assumptions. What should these numbers be, and can we obtain estimates for them that are better than guesswork or using pure history?

The M-A article basically states that the there should be some degree of consistency in how the mathematical operations are performed—and I agree. Our own comparison showed a reasonable consistency between four Monte Carlo models. This is only a starting point in determining the value of a Monte Carlo framework, however, and consistency between models in such controlled circumstances falls far short of making Monte Carlo models useful. The really germane issue is demonstrating that a model can generate reasonable inputs to run the Monte Carlo—say the probability that the S&P500 will generate a return of 5% or more in the next twelve months.

In professional portfolio simulation applications, there is a standard approach to coming testing the reasonableness of these inputs and it is called ‘mark to market.’ In this approach, the volatility for a given instrument generated by the Monte Carlo model is compared to the implied volatility that is backed out from the price of the underlying and the prices at options on that instrument are trading. For a detailed explanation and examples using our own Monte Carlo models, see this article (pdf).

Quantext’s tools have been shown to generate Monte Carlo output that is consistent with the implied volatility of the market—and this is a good test for determining that the projected future results from the Monte Carlo are plausible. I will note that while the concept of mark-to-market testing is uncommon in financial planning for individuals, it is standard in many areas of finance such as professional risk management applications and accounting standards for dealing with risky assets. One important hurdle for Monte Carlo tools is generating consistent mark-to-market results for high Beta instruments (such as QQQQ) or low Beta instruments (such as XLU)—as in the paper above—along with the S&P500. The ability to generate consistent mark-to-market results for high Beta and low-Beta shows that the model is doing a reasonable job of forecasting volatility for assets with high levels of systematic risk and for those with high levels of non-systematic risk (respectively).

While it is very important to be able to simulate future volatility for the S&P500 (i.e. the market as a whole), the real challenge is simulating the statistical parameters for a portfolio made up of real assets. These assets have fees and may, in fact, behave somewhat differently than the underlying indices would imply. This is not a distinction that I would worry much about if an investor looks at the S&P500 as a proxy for SPY, for example, but it can become a major issue in real portfolios—see for example this article by John Bogle.

Monte Carlo models need to have the ability to generate meaningful input parameters for real assets, both individually and in a total portfolio. The interaction of real assets in the total portfolio is often a challenging problem. Asset allocation is all about how to get the greatest benefit from a mix of assets. Do we really expect that individuals or their advisors are going to use tools that require them to somehow perform all of these complex calculations to generate input to Monte Carlo models?

The practical value of Monte Carlo tools is essentially nil if an advisor or individual investor must generate a projection for the average return and standard deviation in return of a portfolio of real assets (stocks, mutual funds, and ETF’s) and then provide these as inputs to the Monte Carlo model—along with the additional input statistics for capturing the correlations between these assets. The potential variation in the estimates for these inputs typically far exceed the importance of the other input or modeling assumptions. To discuss the range of possible outputs from Monte Carlo models without addressing the core problem of where the input assumptions about future asset performance come from is, in my opinion, of limited value. For Monte Carlo models to have value for practical financial planning applications, they must be able to take a real portfolio of assets as input and generate reasonable projections of the future risk and return for the total portfolio.

To be fair, authors of the article cited at the start of this paper did not intend to address the issue of where the inputs come from. Indeed, they are mainly saying that multiple Monte Carlo models, given exactly the same inputs, should generally provide very similar outputs. This is not unreasonable. Any analytical models that are to be used for something as important as financial planning must be testable. That said, aside from uncovering coding errors, such tests are ultimately not terribly helpful to an advisor or individual investor who wants to use Monte Carlo tools for portfolio planning. My point is that for Monte Carlo models to be useful, you must go far beyond this type of test and deal with the combination of the underlying simulation and the input values for simulating a portfolio of real assets. Monte Carlo models are not useful for the vast majority of investors or their advisors unless the models generate their own parameters and can be used to simulate portfolios of real assets without the potential users needing a Ph.D. in finance to run them.

Comments

[Jim Richmond]

This was an interesting post and I agree with the general comments on the variations of simulators being mostly dependent on differences in assumptions that are pretty understandable.

I must say however, that I disagree with the assertion that a simulator is useless unless it can generate average return and standard deviation from a real underlying portfolio.

IMHO, a major problem that financial planning (and perhaps finance in general) faces today is that we've built a beautiful edifice of solid mathematics on the very weak foundation that is our understanding of the true relationships between cause and effect in the return generating process.

While I hear your concern that the average investor would need a PhD in order to run many of the simulators, I don't think that obscuring all the built-in assumptions is necessarily better, especially if they turn out to be wrong!

It seems to me that in many cases, pulling out a return and standard deviation based on investor temperment and the historical record is not a bad place to start. I think some of these models have more error signal than they do data signal in them. The comment that the long term correlations go "all the way back to 2001" is at the heart of the challenge. And I'm not making fun of this. As I said, if you go back too far the error signal is louder than the data.

I've built an experimental Monte Carlo simulator that's focused more on the retiree's withdrawal methodology (ability to adjust the draw based on performance) rather than on guessing at the long term dynamics of each investor's underlying portfolio.

Based on my initial results (and as shown in other research), I'm beginning to think that working with retirees on managing their withdrawals and spending (as an ongoing process) may be more important (or at least as important) than trying to nail the precise details of the underlying dynamics of the portfolio and project them out 40 or 50 years into the future.

FYI, the simulator was written in Java and is available online at
home.comcast.net/~jsrichmond/index.htm

[Geoff Considine]

I must disagree with your assertion that using the 'historical record' is a decent place to start. Using trailing history to directly generate inputs is a terrible way to estimate average return and SD in return for assets. There will be a forthcoming article on this, but many papers have shown this previously. No matter how good the model is otherwise, bad inputs mean bad outputs. It is not about 'precise dynamics' but rather simply about being rational about the probable future risk return balance of assets. Monte Carlo is essentially trivial is you can 'know' the future risk and return inputs for a portfolio. See, for example, all of the case studies in Bernstein's book, The Intelligent Asset Allocator. He used a mean-variance optimizer and trailing history to test portfolio performance if you allocate that way. The results are dismal--far better to just stick with a basic policy portfolio of 60% stocks and 40% bonds and walk away.

[Jim Richmond]

I think we may be violently agreeing, at least in part, especially with respect to putting too much stock on the historical record.

Where we perhaps differ is in our thoughts on our ability (definitely mine, maybe yours too) to build good models that predict future returns and standard deviations 50 YEARS OUT. I'm not saying we shouldn't try (that's why the site is called seeking alpha!), but for the average retiree, and maybe average planner in the mass market, it would be good to focus the portfolio survivability debate more accumulating enough wealth and on active withdrawal management techniques rather than getting too picky about modeling future returns for 30-50 years.

My point was that although a simulator that just takes a set of canned return and standard deviation pairs is rather trivial, it's a good enough place to start so you can focus more on the other variables that are much more under control of retirees such as savings and spending.

BTW, I'd gladly take my return/standard deviation inputs from a sophisticated model such as yours :) The point is that it's not where my energy is focused when working on the long horizon retirement planning problem.

That doesn't mean I don't think there's a place for advanced optimizers that combine past results with predictions about future macroeconomic dynamics. To me that's really cool stuff (way better than historical based models). I just think those tools are better suited to a 1-7 year horizon rather than a 50 year horizon. These tools are for portfolio management rather than for retirement planning. Does that distinction make any sense or is it a false one? I think the goals of the two exercises are different.

BTW, I got the thrust of Bernstein's argument on Monte Carlo to be that we shouldn't sweat the last 10 or 20 points of survival probability (80-85% is probably good enough) when interpreting Monte Carlo data. I thought his point was that because of fat tails from things like a future Hitler or the abomb, there's always a high degree of out-of-model variability that makes saying "we think you have a 95% probability of reaching your goal" a rather silly thing to say with any authority. Maybe I misread him.

In any case, thanks for the reply. It was good food for thought...

[Geoff Considine]

Hi Jim:

'Violently agreeing'---I like that. We do agree that nobody really believes that you can predict the mean and SD for an asset class or portfolio going out 30 years. This is a given. MC projections provide a sense of whether a given allocation makes sense for the time being. As things evolve, parameters evolve, etc. Planning for 10-50 years out is fraught with peril--but we still need to do it. There are better and worse ways to do so.

By the way, you are referring to Bernstein's work on using MC to predict retirement survival. I was referring to his book, The Intelligent Asset Allocator, and a study that shows that using historical parameters to tune your allocations leads to bad results. Same guy, different topic.

Where do you get the mean and SD in return for a portfolio? Relying on simple historical numbers is very risky. There are smarter ways to do this and our tools, Quantext Portfolio Planner and Retirement Planner, do just that. My next article shows more examples of this.

Good results mean results that enable people to make better decisions --- not implying that anyone can believe that I can estimate the mean and SD of SPY accurately for the next 30 years. I do think that a good model can estimate such statistics a lot better than looking at the most recent 1-5 years and a lot better than looking at very long periods such as the last 50+ years.

In summary, I agree that these models are a lot better over the next 1-7 years than they will be over the next 50. The point is that an investor will use the tools for the intermediate horizon (1-7 years) and adjust his/her portfolio on these time scales. The very long-term outlook is far less certain, but it is useful to know if you are on the right general track.

[Jim Richmond]

Geoff,

Sorry for leaving this conversation be for so long, it's actually really interesting to me.

So as we stand, we both agree that building Monte Carlo simulations that combine historic results with predictive macroeconomic models should be the current best practice for portfolio management and asset allocation decision-making. We both understand that these models are imperfect, but as of now, they are the state-of-the-art.

We also agree that if your MC simulation has to go out 30-50 years, you're signal to noise ratio (and the predictive value of the model) goes down considerably.

Where we differ (I think) is on whether it's necessary to incorporate cross-correlations between asset classes and predictions about macroeconomic conditions into long-term Monte Carlo simulation models that are used for planning purposes.

You believe that we have to at least try (since we had to do all the work for portfolio management purposes anyway), and I believe that it doesn't matter. Further, because of the complexity and costs it introduces (mostly measured by increases in simulation run-time), I think it's better not to.

Did I sum things up correctly? I'm not trying to convince you I'm right, but rather I'm trying to make sure we understand each other's position and justifications.

Regards,

Jim

[Geoff Considine]

Hi Jim:

Thanks for the comments. We agree up until the point that you make regarding 'macro economic conditions.' I am not an economist and I have not built any economic forecasts into QPP. I do believe that you need to account for the cross-correlations between components in a model and I believe that there is value in being able to model individual assets and sectors rather than just making an assumption about the 'total market' and then trying to come up with an ad hoc guess for how a real portfolio will perform under some future conditions. Similarly, sincer there is a market for volatility (i.e. the options markets) and since we know that implied volatility in options markets is a better prediction of future volatility than simply looking at history (from a range of acacemic studies--see some of our articles for citations), why wouldn't you want volatility inputs that are at least generally consistent with the options markets?

People really need more specific information from models and I believe that a good model can provide that.

If you want to understand what I am saying about the best way to apply Monte Carlo models, you may want to look at this paper:

www.quantext.com/BestPractice.pdf

This article lays out a hierarchy of basic functions that I believe represent the best practice for Monte Carlo models. These standards are not unlike the kinds of acceptance criteria that I have used in testing and validating corporate risk management models.

Your point on runtime is a good one. Our Monte Carlo model will run very fast on a modern PC. I believe that we have hit the right level of complexity where there is value added by each function.

Best,

Geoff

[Jim Richmond]

Geoff,

The paper you recommended, like the other papers you've published on your site, was a great read. Thanks for the link.

I'm still not with you that long term MCS models need "ticker" level portfolio parameterization, but now I understand what you're saying. I still think you're blurring the portfolio management problem with the retirement planning problem but reasonable men can differ.

On a related note, I definitely misunderstood you on the inclusion of macroeconomic factors into the model. I guess at the ticker level that would get messy very fast. I thought when you mentioned the need for including forward looking predictive inputs that you were talking about macroeconomic parameters. I now see that you mean predicting the future returns/std dev at the ticker level based on fundamental analysis.

Again, since this site is called seeking alpha I need to tread carefully, but my focus (for portfolio mgmt) has always been at the asset class level, rather than at the ticker level. That's a debate we probably shouldn't get into because I know that's like religion. I do think there's alpha to be had at the security level, I just don't think I can easily or reliably find it or pay someone to.

I'm of the school that the most likely place to find sustainable alpha is by including macroeconomic factors as inputs, along with historical risk/return/correlation data, and optimizing at the asset class level. Now I've never built such a model myself, but I've used several and the approach seems sensible to me. I follow the work by the folks at indexinvestor.com and I think they've done some good work here.

Anyhow, interesting discussion. Thanks again,

Jim