Monday, April 23, 2018

The Limits of Simulation

In a previous post, The “Future” of Retirement Planning, I explained that Monte Carlo simulation of retirement finances provides all the information available from a deterministic “spreadsheet” model and more. Among other advantages, it models sequence of returns risk.

Monte Carlo simulation, however, has its own limitations.

A reader commented on my previous post that Monte Carlo simulation “creates thousands of possible and impossible scenarios.

The “impossible” part of that statement is wrong.

In fact, the opposite is true. Monte Carlo simulation’s biggest shortcoming is that most of the scenarios it produces will be the most likely and simplest scenarios while lots of possible but unlikely scenarios that could destroy a retirement will never be simulated.

Most retirement models, deterministic or stochastic, don't model the risks that are most likely to lead to lead to bankruptcy, like spending shocks, divorce or a combination of inter-related risks.[1]

Any planning exercise begins with the basics and is then augmented by a bunch of “what-if’s.”

Planning a picnic? You’ll need a blanket, some food and some lemonade. But, then, what if it rains? What if the park is closed? What if there are too many ants or mosquitoes? What if the sun is too intense? What if someone gets a bee sting? A good plan will consider these possible bad outcomes and prepare for them.

Monte Carlo simulation is a great way to quickly generate a few hundred thousand retirement “what-if” scenarios. Analyzing them with statistics allows us to comprehend the big picture without looking at each scenario individually (an impractical task). They allow much greater in-depth analysis than the spreadsheet approach because they consider more of the key factors of retirement success and provide a lot more what-if’s but here are some of their limitations.

1. They model questionable assumptions.

Most Monte Carlo simulation models assume that market returns are normally distributed, even though we know they aren’t. We see far more — and far more severe — market crises than a normal distribution predicts. We’re either living in a very unlucky universe or we’re using an optimistic distribution for market returns. We use a normal distribution because it's the closest parametric distribution we have and that simplifies the math but we’re pretty sure the market has fatter tails than a normal distribution.

We have a couple of hundred years of market return data but that isn’t enough to create anything near a reasonable confidence interval. That is to say, our (historic) sample size is way too small to make confident guesses of the mean market return and there is no convincing argument that the next thirty years of market returns will look like the last 30 or the last 130.

(These are also problems with deterministic models.)


Monte Carlo is one of the best planning tools we have but it has its limits.
[Tweet this]


In “Where is the Market Going? Uncertain Facts and Novel Theories”[2], John Cochrane notes that over the fifty years from 1947 to 1996 the excess return of stocks over T-bills was 8% but, assuming the annual returns are statistically independent, the standard confidence interval for the mean return ranged from 3% to 13%.

Let me say that in simpler terms. If you asked me the mean excess market return, then based on the sample data from that period I would guess it’s about 8%. But, if you then asked me how confident I am that 8% is the mean, I would say that I’m 95% confident (not totally) that it isn’t less than 3% or more than 13%. In other words, I’m not that confident. (This is also the reason that we can’t identify optimal asset allocations.)

Monte Carlo simulation addresses this uncertainty by generating scenarios with a fairly broad range of market returns. Some scenarios might have a return near 3%, for example, and others near 13%, though most would be closer to 8%. Contrast this with a spreadsheet model that assumes a single market return with no variance.

Some financial writers define market volatility as “risk” and they define “uncertainty” as not even knowing the underlying distribution. We don’t know the underlying distribution of market returns or if the underlying mean return changes over time.

The effect of all this uncertainty is that, while Monte Carlo simulations appear to generate accuracy to several decimal places, our sample size of 200 years or so of historical U.S. stock market returns is too small to inspire confidence. That doesn’t render simulation results irrelevant, however. Hurricane forecasts, as Larry Frank points out, are not very accurate but still very useful. It’s a good analogy for dynamic retirement planning.

When someone says, “My Monte Carlo planner says I have only a 5% probability of outliving my savings, so I’m good, right?”, my answer is, “Well, yes. . . assuming the market behaves much as it has in the past, that you invest your portfolio wisely and earn something near market averages, that you don’t experience a 3-sigma market crash early in retirement, that you experience no spending shocks and that you consider a 1-in-20 chance of outliving your savings “good”, then, yeah, you’re probably good.

2. Simulations are only as good as the strategy they model.

The first thing we need to know is what the simulation models. Most model the probability of outliving a portfolio of stocks and bonds but, as I explained in Three Degrees of Bad[3], portfolio depletion can’t be equated with retirement failure. Portfolio depletion can even be part of the plan.

Some retirement financing strategies are simply flawed. I believe fixed-spending strategies and set-and-forget strategies are hopelessly flawed, for example. Monte Carlo simulation of a flawed strategy for an individual household’s retirement plan is pointless.

I find the concept of “retirement ruin” to be meaningless (retirees don’t stop living and spending when their portfolio is depleted) so I have little confidence in retirement models based on probability of ruin[4,5]. U.S. retirees would declare bankruptcy if their retirement failed, emerge with some protected assets, and live off Social Security benefits. Their retirement wouldn’t simply end, though their standard of living might dramatically decline. Instead, I model the probability of not meeting desired spending.

So, when I respond, “yeah, you’re probably good”, I add, “. . . and assuming your Monte Carlo simulation used a reasonable model.”

3. Spending shocks are difficult to model, so they seldom are.

Spending shocks can decimate a retirement plan but they are difficult to model. Shocks typically have a low probability of occurring but potentially huge risk magnitude. These risks are usually better mitigated by insurance when affordable insurance is available than by relying on a low-probability of their occurrence. Even if we model them, insurance (Social Security benefits, annuities and pensions) will usually be the answer.

4. Simulations probably won’t generate rare but potentially catastrophic scenarios.

Monte Carlo simulation works by generating many of the most probable scenarios and fewer and fewer of less-probable scenarios. They won’t thoroughly analyze very low-probability market returns (tail risk), for example, because they are unlikely to generate more than a few such scenarios.


A normal distribution “tails off” at both ends. The “skinny tails” show that the probability of outcomes far from the mean are highly unlikely, which means they are equally unlikely to be included in a Monte Carlo simulation. We refer to this as “tail risk.” You can see the skinny tails of a normal distribution in the diagram above.

Outcomes in the tails are improbable but, as I recently read somewhere, the left tail should be labeled “There be dragons.”

Unlikely outcomes to the right of the mean (the right tail) aren’t a problem; those outcomes are improbably good. It’s the left tail risk that’s a problem because the distribution tells us that outcomes there are improbable but it doesn’t tell us they’re magnitude.

The reality is that we can’t estimate tail risk for the market because we don’t know the distribution of market returns. We guess that it is “normal-ish” tail risk but we know that market crashes occur far more often than a normal distribution would predict. Monte Carlo simulation isn’t helpful in predicting very unlikely but catastrophic events but then, nothing is.

In Antifragile[7], Nassim Taleb says, "[Antifragility] provides a solution to what I have called the Black Swan problem — the impossibility (emphasis mine) of calculating the risks of consequential rare events and predicting their occurrence."

5. Complex scenarios are difficult to model, so they seldom are.

Complex scenarios are difficult to conceive, let alone model. Elder bankruptcy research by Deborah Thorne[6] showed that most of the worst-case retirement finance outcomes (those that end in bankruptcy) are not caused by a single factor, like spending too much on credit cards, but by a complex self-reinforcing cycle of interdependent risks. These numerous complex combinations of risk are unlikely to be modeled.

Here’s an example that would be difficult to anticipate and therefore difficult to generate with a simulation model.

A retiree borrows a reverse mortgage, feeling secure in the fact that it is non-recourse. He knows that his loan can’t be foreclosed unless he moves out of the home, which he doesn’t plan to do. His wife becomes ill and runs up huge medical bills. They spend home equity to pay bills, then run up credit card debt and eventually file for bankruptcy. They can no longer afford to live in the home and when they leave, repayment of the reverse mortgage will be triggered.

Monte Carlo simulation can generate hundreds of thousands of possible future scenarios but they won’t include complex, interdependent risks like this one. On the other hand, Monte Carlo simulations may surprise you by showing scenarios, for example, in which purchasing an annuity actually results in a greater legacy.

7. Simulation can’t predict your future.

I recently wrote about a blog post that suggested that Monte Carlo simulation has no value because a retiree can’t know which of the thousands of possible future paths her future will track. That is absolutely true — your individual path is unknowable — but the argument is irrelevant. That argument is based on the false premise that we run simulations in order to find that path. We run simulations to collect information on the range of many paths.

A retiree shouldn’t look at simulation results, regardless of the number of scenarios simulated, and assume his or her future is in there somewhere. More often than not it will be but a good retirement plan doesn’t rely on that. A good retirement plan should also consider what happens when really bad, improbable things happen.

8. Many Monte Carlo models underestimate risk.

Many, and probably most, Monte Carlo models calculate risk of ruin simply by counting the percent of scenarios that end in ruin. Some scenarios that are counted as successes, however, may have been exposed to significantly greater risk than others. That 95% probability of success is probably best case.

At this point, you may be asking yourself why I recommend a tool with so many shortcomings. One answer is that it has fewer shortcomings than the alternatives. It considers more factors and generates more information. We look at simulation results to get an overall view of the most probable outcomes and the perspective we gain is, like weather forecasts, imperfect but highly useful.

Understanding what will probably happen and what might happen in most scenarios is a great place to start.

The important takeaways are these. Monte Carlo simulation can be a powerful tool for retirement planning because it provides more information than other approaches. Ultimately, however, we must realize that, as Yogi is credited with saying, predictions are really hard — especially about the future. The results are more of a distribution of a ballpark estimate than a single answer but it's more useful to estimate a 40% chance of rain tomorrow than to maintain that we can't know for sure so it isn't worth considering. Monte Carlo simulation will not predict or protect your retirement from "consequential rare events."

The results are also better used to compare the relative risk of one strategy to another than to measure their absolute risk. If simulation tells you that 3% spending is half as risky as 5% spending, then you can be more confident that one is safer than the other than you can be that there is actually a 3% risk that the former will result in ruin.

If this is all a little too confusing, bear with me. You can learn to use the information provided by simulations without a complete understanding of Monte Carlo models. You probably couldn't build a GPS device, either, but you're probably confident using one. Perhaps, you just need to find a planner that will run one for you. Maybe you have a perfectly fine plan built with a different type of model or no model at all and you just need simulation to improve your confidence.

Next time I’ll discuss the relationship between Spending Rules and Simulation.



REFERENCES

[1] The Retirement Café: Why Retirees Go Broke.


[2] Where is the Market Going? Uncertain Facts and Novel Theories, John H Cochrane.


[3] The Retirement Café: Three Degrees of Bad.


[4] The Retirement Café: Time to Retire the Probability of Ruin?.


[5] Financial Analysts Journal: It’s Time to Retire Ruin (Probabilities) | CFA Institute Publications, Moshe Milevsky.


[6] The (Interconnected) Reasons Elder Americans File Consumer Bankruptcy, Deborah Thorne.


[7] Antifragile: Things That Gain from Disorder (Incerto), Nassim Taleb.