Monday, April 23, 2018

The Limits of Simulation

In a previous post, The “Future” of Retirement Planning, I explained that Monte Carlo simulation of retirement finances provides all the information available from a deterministic “spreadsheet” model and more. Among other advantages, it models sequence of returns risk.

Monte Carlo simulation, however, has its own limitations.

A reader commented on my previous post that Monte Carlo simulation “creates thousands of possible and impossible scenarios.

The “impossible” part of that statement is wrong.

In fact, the opposite is true. Monte Carlo simulation’s biggest shortcoming is that most of the scenarios it produces will be the most likely and simplest scenarios while lots of possible but unlikely scenarios that could destroy a retirement will never be simulated.

Most retirement models, deterministic or stochastic, don't model the risks that are most likely to lead to lead to bankruptcy, like spending shocks, divorce or a combination of inter-related risks.[1]

Any planning exercise begins with the basics and is then augmented by a bunch of “what-if’s.”

Planning a picnic? You’ll need a blanket, some food and some lemonade. But, then, what if it rains? What if the park is closed? What if there are too many ants or mosquitoes? What if the sun is too intense? What if someone gets a bee sting? A good plan will consider these possible bad outcomes and prepare for them.

Monte Carlo simulation is a great way to quickly generate a few hundred thousand retirement “what-if” scenarios. Analyzing them with statistics allows us to comprehend the big picture without looking at each scenario individually (an impractical task). They allow much greater in-depth analysis than the spreadsheet approach because they consider more of the key factors of retirement success and provide a lot more what-if’s but here are some of their limitations.

1. They model questionable assumptions.

Most Monte Carlo simulation models assume that market returns are normally distributed, even though we know they aren’t. We see far more — and far more severe — market crises than a normal distribution predicts. We’re either living in a very unlucky universe or we’re using an optimistic distribution for market returns. We use a normal distribution because it's the closest parametric distribution we have and that simplifies the math but we’re pretty sure the market has fatter tails than a normal distribution.

We have a couple of hundred years of market return data but that isn’t enough to create anything near a reasonable confidence interval. That is to say, our (historic) sample size is way too small to make confident guesses of the mean market return and there is no convincing argument that the next thirty years of market returns will look like the last 30 or the last 130.

(These are also problems with deterministic models.)


Monte Carlo is one of the best planning tools we have but it has its limits.
[Tweet this]


In “Where is the Market Going? Uncertain Facts and Novel Theories”[2], John Cochrane notes that over the fifty years from 1947 to 1996 the excess return of stocks over T-bills was 8% but, assuming the annual returns are statistically independent, the standard confidence interval for the mean return ranged from 3% to 13%.

Let me say that in simpler terms. If you asked me the mean excess market return, then based on the sample data from that period I would guess it’s about 8%. But, if you then asked me how confident I am that 8% is the mean, I would say that I’m 95% confident (not totally) that it isn’t less than 3% or more than 13%. In other words, I’m not that confident. (This is also the reason that we can’t identify optimal asset allocations.)

Monte Carlo simulation addresses this uncertainty by generating scenarios with a fairly broad range of market returns. Some scenarios might have a return near 3%, for example, and others near 13%, though most would be closer to 8%. Contrast this with a spreadsheet model that assumes a single market return with no variance.

Some financial writers define market volatility as “risk” and they define “uncertainty” as not even knowing the underlying distribution. We don’t know the underlying distribution of market returns or if the underlying mean return changes over time.

The effect of all this uncertainty is that, while Monte Carlo simulations appear to generate accuracy to several decimal places, our sample size of 200 years or so of historical U.S. stock market returns is too small to inspire confidence. That doesn’t render simulation results irrelevant, however. Hurricane forecasts, as Larry Frank points out, are not very accurate but still very useful. It’s a good analogy for dynamic retirement planning.

When someone says, “My Monte Carlo planner says I have only a 5% probability of outliving my savings, so I’m good, right?”, my answer is, “Well, yes. . . assuming the market behaves much as it has in the past, that you invest your portfolio wisely and earn something near market averages, that you don’t experience a 3-sigma market crash early in retirement, that you experience no spending shocks and that you consider a 1-in-20 chance of outliving your savings “good”, then, yeah, you’re probably good.

2. Simulations are only as good as the strategy they model.

The first thing we need to know is what the simulation models. Most model the probability of outliving a portfolio of stocks and bonds but, as I explained in Three Degrees of Bad[3], portfolio depletion can’t be equated with retirement failure. Portfolio depletion can even be part of the plan.

Some retirement financing strategies are simply flawed. I believe fixed-spending strategies and set-and-forget strategies are hopelessly flawed, for example. Monte Carlo simulation of a flawed strategy for an individual household’s retirement plan is pointless.

I find the concept of “retirement ruin” to be meaningless (retirees don’t stop living and spending when their portfolio is depleted) so I have little confidence in retirement models based on probability of ruin[4,5]. U.S. retirees would declare bankruptcy if their retirement failed, emerge with some protected assets, and live off Social Security benefits. Their retirement wouldn’t simply end, though their standard of living might dramatically decline. Instead, I model the probability of not meeting desired spending.

So, when I respond, “yeah, you’re probably good”, I add, “. . . and assuming your Monte Carlo simulation used a reasonable model.”

3. Spending shocks are difficult to model, so they seldom are.

Spending shocks can decimate a retirement plan but they are difficult to model. Shocks typically have a low probability of occurring but potentially huge risk magnitude. These risks are usually better mitigated by insurance when affordable insurance is available than by relying on a low-probability of their occurrence. Even if we model them, insurance (Social Security benefits, annuities and pensions) will usually be the answer.

4. Simulations probably won’t generate rare but potentially catastrophic scenarios.

Monte Carlo simulation works by generating many of the most probable scenarios and fewer and fewer of less-probable scenarios. They won’t thoroughly analyze very low-probability market returns (tail risk), for example, because they are unlikely to generate more than a few such scenarios.


A normal distribution “tails off” at both ends. The “skinny tails” show that the probability of outcomes far from the mean are highly unlikely, which means they are equally unlikely to be included in a Monte Carlo simulation. We refer to this as “tail risk.” You can see the skinny tails of a normal distribution in the diagram above.

Outcomes in the tails are improbable but, as I recently read somewhere, the left tail should be labeled “There be dragons.”

Unlikely outcomes to the right of the mean (the right tail) aren’t a problem; those outcomes are improbably good. It’s the left tail risk that’s a problem because the distribution tells us that outcomes there are improbable but it doesn’t tell us they’re magnitude.

The reality is that we can’t estimate tail risk for the market because we don’t know the distribution of market returns. We guess that it is “normal-ish” tail risk but we know that market crashes occur far more often than a normal distribution would predict. Monte Carlo simulation isn’t helpful in predicting very unlikely but catastrophic events but then, nothing is.

In Antifragile[7], Nassim Taleb says, "[Antifragility] provides a solution to what I have called the Black Swan problem — the impossibility (emphasis mine) of calculating the risks of consequential rare events and predicting their occurrence."

5. Complex scenarios are difficult to model, so they seldom are.

Complex scenarios are difficult to conceive, let alone model. Elder bankruptcy research by Deborah Thorne[6] showed that most of the worst-case retirement finance outcomes (those that end in bankruptcy) are not caused by a single factor, like spending too much on credit cards, but by a complex self-reinforcing cycle of interdependent risks. These numerous complex combinations of risk are unlikely to be modeled.

Here’s an example that would be difficult to anticipate and therefore difficult to generate with a simulation model.

A retiree borrows a reverse mortgage, feeling secure in the fact that it is non-recourse. He knows that his loan can’t be foreclosed unless he moves out of the home, which he doesn’t plan to do. His wife becomes ill and runs up huge medical bills. They spend home equity to pay bills, then run up credit card debt and eventually file for bankruptcy. They can no longer afford to live in the home and when they leave, repayment of the reverse mortgage will be triggered.

Monte Carlo simulation can generate hundreds of thousands of possible future scenarios but they won’t include complex, interdependent risks like this one. On the other hand, Monte Carlo simulations may surprise you by showing scenarios, for example, in which purchasing an annuity actually results in a greater legacy.

7. Simulation can’t predict your future.

I recently wrote about a blog post that suggested that Monte Carlo simulation has no value because a retiree can’t know which of the thousands of possible future paths her future will track. That is absolutely true — your individual path is unknowable — but the argument is irrelevant. That argument is based on the false premise that we run simulations in order to find that path. We run simulations to collect information on the range of many paths.

A retiree shouldn’t look at simulation results, regardless of the number of scenarios simulated, and assume his or her future is in there somewhere. More often than not it will be but a good retirement plan doesn’t rely on that. A good retirement plan should also consider what happens when really bad, improbable things happen.

8. Many Monte Carlo models underestimate risk.

Many, and probably most, Monte Carlo models calculate risk of ruin simply by counting the percent of scenarios that end in ruin. Some scenarios that are counted as successes, however, may have been exposed to significantly greater risk than others. That 95% probability of success is probably best case.

At this point, you may be asking yourself why I recommend a tool with so many shortcomings. One answer is that it has fewer shortcomings than the alternatives. It considers more factors and generates more information. We look at simulation results to get an overall view of the most probable outcomes and the perspective we gain is, like weather forecasts, imperfect but highly useful.

Understanding what will probably happen and what might happen in most scenarios is a great place to start.

The important takeaways are these. Monte Carlo simulation can be a powerful tool for retirement planning because it provides more information than other approaches. Ultimately, however, we must realize that, as Yogi is credited with saying, predictions are really hard — especially about the future. The results are more of a distribution of a ballpark estimate than a single answer but it's more useful to estimate a 40% chance of rain tomorrow than to maintain that we can't know for sure so it isn't worth considering. Monte Carlo simulation will not predict or protect your retirement from "consequential rare events."

The results are also better used to compare the relative risk of one strategy to another than to measure their absolute risk. If simulation tells you that 3% spending is half as risky as 5% spending, then you can be more confident that one is safer than the other than you can be that there is actually a 3% risk that the former will result in ruin.

If this is all a little too confusing, bear with me. You can learn to use the information provided by simulations without a complete understanding of Monte Carlo models. You probably couldn't build a GPS device, either, but you're probably confident using one. Perhaps, you just need to find a planner that will run one for you. Maybe you have a perfectly fine plan built with a different type of model or no model at all and you just need simulation to improve your confidence.

Next time I’ll discuss the relationship between Spending Rules and Simulation.



REFERENCES

[1] The Retirement Café: Why Retirees Go Broke.


[2] Where is the Market Going? Uncertain Facts and Novel Theories, John H Cochrane.


[3] The Retirement Café: Three Degrees of Bad.


[4] The Retirement Café: Time to Retire the Probability of Ruin?.


[5] Financial Analysts Journal: It’s Time to Retire Ruin (Probabilities) | CFA Institute Publications, Moshe Milevsky.


[6] The (Interconnected) Reasons Elder Americans File Consumer Bankruptcy, Deborah Thorne.


[7] Antifragile: Things That Gain from Disorder (Incerto), Nassim Taleb.





9 comments:

  1. Michael Kitces has said paraphrasing, "I'd rather be approximately right rather than precisely wrong." Your list summarizes this well.

    Dimensional has long argued that models aren't meant to predict (which is impossible as we all know, but we then go on to try anyway). Their recent white paper "Mind Over Model, March 2018" says "Checking the weather? Guess what—you’re using a model. While models can be useful for gaining insights that can help us make good decisions, they are inherently incomplete simplifications of reality. ... In the end, there is a difference between blindly following a model and using it judiciously to guide your decisions."

    They are meant to be decision tools to refine past decisions into future choices ... rinse and repeat.

    Really enjoying your current train of thought Dirk!

    ReplyDelete
  2. At Michael Kitces blog, Derek Tharp demonstrated that Monte Carlo analyses actually tend to be too pessimistic with regard to 'worst case' scenarios, primarily because most of them do not incorporate mean reversion, which is a well known phenomenon in financial markets.

    "Monte Carlo projections of a long-term retirement plan using typical return and standard deviation assumptions are actually far more extreme than real-world historical market scenarios have ever been!

    For instance, when comparing a Monte Carlo analysis of 10,000 scenarios based on historical 60/40 annual return parameters to historical returns, it turns out that 6.5% of Monte Carlo scenarios are actually worse than even the worst case historical scenario has ever been! Or viewed another way, a 93.5% probability of success in Monte Carlo is actually akin to a 100% success rate using actual historical scenarios!"
    https://www.kitces.com/blog/monte-carlo-analysis-risk-fat-tails-vs-safe-withdrawal-rates-rolling-historical-returns/

    There is nothing that can be done to eliminate all financial risk. This is a hard pill for many to swallow, but it's the truth.

    ReplyDelete
  3. William, your bottom line is, in fact the bottom line.

    The argument you cite, however, has a large and obvious flaw — the presumption that a mere 200 years of so of market returns data holds the best and worst outcomes that the market will ever produce.

    You're predicting a 100% success rate under the assumption that nothing unprecedented happens in the future when, in fact, we see unprecedented outcomes regularly. The assertion that a 100% success rate is possible is a frightening thing, in itself.

    That same data could be used just as convincingly to argue that since 6.5% of Monte Carlo simulations have never been experienced in history that history likely hasn't shown us its worst, yet.

    Importantly, you're treading down an unconvincing logic path when you refer to "Monte Carlo simulations." Monte Carlo simulations are based on models and Monte Carlo simulations based on other models will provide different results than you share. You're treating simulations as if all Monte Carlo simulations provide the same results. They don't.

    Since we can't know whether future results will be bound by our limited historical data, we can't know if the models are optimistic, pessimistic, or spot on, based on historical returns, so I see little value to arguing the point. Regardless, I wouldn't characterize 6.5% of returns, even if correct, as "far more extreme."

    My judgment that MC models (not Monte Carlo simulation, itself) tend to be overly optimistic has nothing to do with the simulation, but that counting the frequency of failed outcomes is a poor way to characterize the risk.

    Thanks for writing!

    ReplyDelete
    Replies
    1. I agree wholeheartedly Dirk. Counting the frequency of failed outcomes, i.e., failure rate of iterations, doesn't characterize risk very well. However, it does lead to more informed decision making where 15% failures relative to 10% failures would hint that the 15% scenario is riskier relative to the 10% scenario. So there is information to be had, but it is NOT to be interpreted as to THAT is what is going to happen (to your point).

      The current practice of simulation interpretation is flawed as I explained in this post

      https://blog.betterfinancialeducation.com/sustainable-retirement/part-ii/

      The solution that is sought is NOT what you see on the right side of a simulation graph, but using Monte Carlo to actually derive an answer (the solution) based on asking the simulation to solve for the answer to the question. In retirement the question is "How much can I spend if I have $x?" or the corollary question, "How much do I need to save in order to spend $Y?" In order to answer these questions, one needs a common denominator so any simulation set can be properly compared to another simulation set. Using the same failure rate as a comparison helps compare two differing portfolio allocation characteristics for example. Rising failure rate of a plan when markets misbehave and portfolio values decline as a result is also a good way to predetermine ahead of time what reduced portfolio value should trigger what kind of decision (a decision that would actually affect outcome, not a market timing decision).

      A small quibble Dirk, but decision making information can be taken from simulations (not that you said otherwise). Looking forward to seeing where your series of posts for this line of thought go!

      Delete
  4. Thanks, Larry. Not sure what the quibble is, I think I'm in complete agreement with you here. As I said in the post, MC simulation is better at making relative assessments than absolute. Relative assessments are valid decision-making arguments.

    My issue with the frequency-based probability measure is that successful scenarios were exposed to significantly different levels of risk. If I win $5 from a very risky bet and you win $5 from a safe bet, we both won $5 but your path was far preferable. The frequency measure considers our experiences identical (we both won $5).

    ReplyDelete
  5. This is all very humbling and I know you've addressed this in multiple past posts but I could use a refresher on your recommendation for the best course of action when designing a retirement portfolio. I believe you are an advocate of safety first which means to have a floor, I think for expenses that are deemed 'necessary'. And then invest any remaining assets in the market, stocks and bonds? Annuities? I think Wade recommends annuities over bonds? I do find these posts to be thought provoking but where I always go is: what should I do? Thanks for your thoughts and please spell out any recommendations. I'd be interested in Larry's approach too.

    ReplyDelete
    Replies
    1. You are generally correct but your plan will be unique to your household circumstances. It would be extremely difficult for me to give you a brief response. Instead, why don't you email me at jdcplanning@gmail.com and let's set up a time to talk and get you started on the right foot.

      Delete
  6. Thank you for the article. As you have noted before, correlation factors are hard to model but I wonder if their ommission isn't at least as important as a too simplified probability function, especially for the more pessimistic outcomes. I dont know to what extent the market exhibits inter year correlation, but if it does this presumably is a consequence of an underlying factor such as participants belief that they have just experienced a bubble and asset prices remain too high, in their opinion, or inflation, for which a year of high inflation is likely to be preceded and followed by years of elevated inflation too. The correlation therefore arising from the correlation in the underlying factors.

    I wondered about injecting a pre-programmed ' shock' into my personal spreadsheet model (the model is now rejected thanks in part to your observations :), such as a 30% fall followed by a multi-year slow recovery. The same could be done with a Monte Carlo model. However, for a relatively short timescale, eg 35 years, adding a (say) 5 year event would likely dominate the results which would just introduce the additional uncertainties of how to model, and time, the enforced shock.

    This isn't just academic since I think this introduces a bias into my thinking especially with respect to protection against inflation. I am chewing at the bit to invest in a direct holding of IL gilts UK resident) and have recently converted a DB pension into a DC fund as the former only offered uplift to a maximum of 5% inflation.

    So for me, this is a limit on how much guidance or comfort I can draw from a model.

    Sorry for the long post.

    Paul



    ReplyDelete
    Replies
    1. Paul, I'm going to respond to your post from the bottom up.

      First, no need to apologize for length; its a great comment.

      Second, there should be a limit on how much comfort you draw from a model. They can provide a lot of information for making planning decisions but they can't tell your future.

      The problem with modeling shocks is two-fold. We have no distribution to model. Also, shocks are tail risk. Nassim Taleb tells us that it is impossible to predict low-probability events. In Congressional testimony he called those who try "charlatans."

      You can imagine severe shocks but then you can always imagine worse. It's better to address tail risk by thinking about bad scenarios and how you might handle them. Perhaps you could handle a $100K medical expense shock but what would you do with a $1M shock? At some point you will need to accept that risk or insure it (if insurance is available and affordable). Modeling won't help much here. Expense shocks are more difficult to predict than the stock market (that's why we call them "shocks.")

      A few comments on your first paragraph. Correlation of asset returns is helpful under normal conditions but, as the saying goes, all correlations go to 1.0 in a market crisis. (Tail risk, again.)

      No one knows for sure "to what extent the market exhibits inter-year correlation." There is only weak evidence for mean-reversion and even if it exists, how long it will take to revert is unknowable. I recently read a paper showing that, regardless, it isn't a winning factor for a market strategy.

      We can model all of these things but I'm not convinced any of them would change the outcomes enough to make you change your retirement plan. Modeling provides information about what happens most of the time. It doesn't really "do" tail risk. You need to plan that in other ways.

      Thanks for writing!

      Delete