Monday, April 23, 2018

The Limits of Simulation

In a previous post, The “Future” of Retirement Planning, I explained that Monte Carlo simulation of retirement finances provides all the information available from a deterministic “spreadsheet” model and more. Among other advantages, it models sequence of returns risk.

Monte Carlo simulation, however, has its own limitations.

A reader commented on my previous post that Monte Carlo simulation “creates thousands of possible and impossible scenarios.

The “impossible” part of that statement is wrong.

In fact, the opposite is true. Monte Carlo simulation’s biggest shortcoming is that most of the scenarios it produces will be the most likely and simplest scenarios while lots of possible but unlikely scenarios that could destroy a retirement will never be simulated.

Most retirement models, deterministic or stochastic, don't model the risks that are most likely to lead to lead to bankruptcy, like spending shocks, divorce or a combination of inter-related risks.[1]

Any planning exercise begins with the basics and is then augmented by a bunch of “what-if’s.”

Planning a picnic? You’ll need a blanket, some food and some lemonade. But, then, what if it rains? What if the park is closed? What if there are too many ants or mosquitoes? What if the sun is too intense? What if someone gets a bee sting? A good plan will consider these possible bad outcomes and prepare for them.

Monte Carlo simulation is a great way to quickly generate a few hundred thousand retirement “what-if” scenarios. Analyzing them with statistics allows us to comprehend the big picture without looking at each scenario individually (an impractical task). They allow much greater in-depth analysis than the spreadsheet approach because they consider more of the key factors of retirement success and provide a lot more what-if’s but here are some of their limitations.

1. They model questionable assumptions.

Most Monte Carlo simulation models assume that market returns are normally distributed, even though we know they aren’t. We see far more — and far more severe — market crises than a normal distribution predicts. We’re either living in a very unlucky universe or we’re using an optimistic distribution for market returns. We use a normal distribution because it's the closest parametric distribution we have and that simplifies the math but we’re pretty sure the market has fatter tails than a normal distribution.

We have a couple of hundred years of market return data but that isn’t enough to create anything near a reasonable confidence interval. That is to say, our (historic) sample size is way too small to make confident guesses of the mean market return and there is no convincing argument that the next thirty years of market returns will look like the last 30 or the last 130.

(These are also problems with deterministic models.)


Monte Carlo is one of the best planning tools we have but it has its limits.
[Tweet this]


In “Where is the Market Going? Uncertain Facts and Novel Theories”[2], John Cochrane notes that over the fifty years from 1947 to 1996 the excess return of stocks over T-bills was 8% but, assuming the annual returns are statistically independent, the standard confidence interval for the mean return ranged from 3% to 13%.

Let me say that in simpler terms. If you asked me the mean excess market return, then based on the sample data from that period I would guess it’s about 8%. But, if you then asked me how confident I am that 8% is the mean, I would say that I’m 95% confident (not totally) that it isn’t less than 3% or more than 13%. In other words, I’m not that confident. (This is also the reason that we can’t identify optimal asset allocations.)

Monte Carlo simulation addresses this uncertainty by generating scenarios with a fairly broad range of market returns. Some scenarios might have a return near 3%, for example, and others near 13%, though most would be closer to 8%. Contrast this with a spreadsheet model that assumes a single market return with no variance.

Some financial writers define market volatility as “risk” and they define “uncertainty” as not even knowing the underlying distribution. We don’t know the underlying distribution of market returns or if the underlying mean return changes over time.

The effect of all this uncertainty is that, while Monte Carlo simulations appear to generate accuracy to several decimal places, our sample size of 200 years or so of historical U.S. stock market returns is too small to inspire confidence. That doesn’t render simulation results irrelevant, however. Hurricane forecasts, as Larry Frank points out, are not very accurate but still very useful. It’s a good analogy for dynamic retirement planning.

When someone says, “My Monte Carlo planner says I have only a 5% probability of outliving my savings, so I’m good, right?”, my answer is, “Well, yes. . . assuming the market behaves much as it has in the past, that you invest your portfolio wisely and earn something near market averages, that you don’t experience a 3-sigma market crash early in retirement, that you experience no spending shocks and that you consider a 1-in-20 chance of outliving your savings “good”, then, yeah, you’re probably good.

2. Simulations are only as good as the strategy they model.

The first thing we need to know is what the simulation models. Most model the probability of outliving a portfolio of stocks and bonds but, as I explained in Three Degrees of Bad[3], portfolio depletion can’t be equated with retirement failure. Portfolio depletion can even be part of the plan.

Some retirement financing strategies are simply flawed. I believe fixed-spending strategies and set-and-forget strategies are hopelessly flawed, for example. Monte Carlo simulation of a flawed strategy for an individual household’s retirement plan is pointless.

I find the concept of “retirement ruin” to be meaningless (retirees don’t stop living and spending when their portfolio is depleted) so I have little confidence in retirement models based on probability of ruin[4,5]. U.S. retirees would declare bankruptcy if their retirement failed, emerge with some protected assets, and live off Social Security benefits. Their retirement wouldn’t simply end, though their standard of living might dramatically decline. Instead, I model the probability of not meeting desired spending.

So, when I respond, “yeah, you’re probably good”, I add, “. . . and assuming your Monte Carlo simulation used a reasonable model.”

3. Spending shocks are difficult to model, so they seldom are.

Spending shocks can decimate a retirement plan but they are difficult to model. Shocks typically have a low probability of occurring but potentially huge risk magnitude. These risks are usually better mitigated by insurance when affordable insurance is available than by relying on a low-probability of their occurrence. Even if we model them, insurance (Social Security benefits, annuities and pensions) will usually be the answer.

4. Simulations probably won’t generate rare but potentially catastrophic scenarios.

Monte Carlo simulation works by generating many of the most probable scenarios and fewer and fewer of less-probable scenarios. They won’t thoroughly analyze very low-probability market returns (tail risk), for example, because they are unlikely to generate more than a few such scenarios.


A normal distribution “tails off” at both ends. The “skinny tails” show that the probability of outcomes far from the mean are highly unlikely, which means they are equally unlikely to be included in a Monte Carlo simulation. We refer to this as “tail risk.” You can see the skinny tails of a normal distribution in the diagram above.

Outcomes in the tails are improbable but, as I recently read somewhere, the left tail should be labeled “There be dragons.”

Unlikely outcomes to the right of the mean (the right tail) aren’t a problem; those outcomes are improbably good. It’s the left tail risk that’s a problem because the distribution tells us that outcomes there are improbable but it doesn’t tell us they’re magnitude.

The reality is that we can’t estimate tail risk for the market because we don’t know the distribution of market returns. We guess that it is “normal-ish” tail risk but we know that market crashes occur far more often than a normal distribution would predict. Monte Carlo simulation isn’t helpful in predicting very unlikely but catastrophic events but then, nothing is.

In Antifragile[7], Nassim Taleb says, "[Antifragility] provides a solution to what I have called the Black Swan problem — the impossibility (emphasis mine) of calculating the risks of consequential rare events and predicting their occurrence."

5. Complex scenarios are difficult to model, so they seldom are.

Complex scenarios are difficult to conceive, let alone model. Elder bankruptcy research by Deborah Thorne[6] showed that most of the worst-case retirement finance outcomes (those that end in bankruptcy) are not caused by a single factor, like spending too much on credit cards, but by a complex self-reinforcing cycle of interdependent risks. These numerous complex combinations of risk are unlikely to be modeled.

Here’s an example that would be difficult to anticipate and therefore difficult to generate with a simulation model.

A retiree borrows a reverse mortgage, feeling secure in the fact that it is non-recourse. He knows that his loan can’t be foreclosed unless he moves out of the home, which he doesn’t plan to do. His wife becomes ill and runs up huge medical bills. They spend home equity to pay bills, then run up credit card debt and eventually file for bankruptcy. They can no longer afford to live in the home and when they leave, repayment of the reverse mortgage will be triggered.

Monte Carlo simulation can generate hundreds of thousands of possible future scenarios but they won’t include complex, interdependent risks like this one. On the other hand, Monte Carlo simulations may surprise you by showing scenarios, for example, in which purchasing an annuity actually results in a greater legacy.

7. Simulation can’t predict your future.

I recently wrote about a blog post that suggested that Monte Carlo simulation has no value because a retiree can’t know which of the thousands of possible future paths her future will track. That is absolutely true — your individual path is unknowable — but the argument is irrelevant. That argument is based on the false premise that we run simulations in order to find that path. We run simulations to collect information on the range of many paths.

A retiree shouldn’t look at simulation results, regardless of the number of scenarios simulated, and assume his or her future is in there somewhere. More often than not it will be but a good retirement plan doesn’t rely on that. A good retirement plan should also consider what happens when really bad, improbable things happen.

8. Many Monte Carlo models underestimate risk.

Many, and probably most, Monte Carlo models calculate risk of ruin simply by counting the percent of scenarios that end in ruin. Some scenarios that are counted as successes, however, may have been exposed to significantly greater risk than others. That 95% probability of success is probably best case.

At this point, you may be asking yourself why I recommend a tool with so many shortcomings. One answer is that it has fewer shortcomings than the alternatives. It considers more factors and generates more information. We look at simulation results to get an overall view of the most probable outcomes and the perspective we gain is, like weather forecasts, imperfect but highly useful.

Understanding what will probably happen and what might happen in most scenarios is a great place to start.

The important takeaways are these. Monte Carlo simulation can be a powerful tool for retirement planning because it provides more information than other approaches. Ultimately, however, we must realize that, as Yogi is credited with saying, predictions are really hard — especially about the future. The results are more of a distribution of a ballpark estimate than a single answer but it's more useful to estimate a 40% chance of rain tomorrow than to maintain that we can't know for sure so it isn't worth considering. Monte Carlo simulation will not predict or protect your retirement from "consequential rare events."

The results are also better used to compare the relative risk of one strategy to another than to measure their absolute risk. If simulation tells you that 3% spending is half as risky as 5% spending, then you can be more confident that one is safer than the other than you can be that there is actually a 3% risk that the former will result in ruin.

If this is all a little too confusing, bear with me. You can learn to use the information provided by simulations without a complete understanding of Monte Carlo models. You probably couldn't build a GPS device, either, but you're probably confident using one. Perhaps, you just need to find a planner that will run one for you. Maybe you have a perfectly fine plan built with a different type of model or no model at all and you just need simulation to improve your confidence.

Next time I’ll discuss how to interpret and use this imperfect information to plan your retirement.



REFERENCES

[1] The Retirement Café: Why Retirees Go Broke.


[2] Where is the Market Going? Uncertain Facts and Novel Theories, John H Cochrane.


[3] The Retirement Café: Three Degrees of Bad.


[4] The Retirement Café: Time to Retire the Probability of Ruin?.


[5] Financial Analysts Journal: It’s Time to Retire Ruin (Probabilities) | CFA Institute Publications, Moshe Milevsky.


[6] The (Interconnected) Reasons Elder Americans File Consumer Bankruptcy, Deborah Thorne.


[7] Antifragile: Things That Gain from Disorder (Incerto), Nassim Taleb.





Friday, March 30, 2018

The “Future” of Retirement Planning

When we decide how much money we can spend in the present year of retirement we need to know not only how much spendable wealth we have today but our best guess of how much we will have in the future. Likewise, on the expense side of the ledger, we need to know not only what our expenses were last year but also our best guess of what our expenses will be in the future. We can spend less this year, for example, if we know there is a big expense looming in the future and more if we’re pretty sure we’ll have more money in the future, say a sizable inheritance.

In short, we need a model of our retirement future to properly plan for it and even to determine a safe amount to spend this year.

Of the four inputs to this model I just mentioned, only one, our current wealth, is fairly certain.

If our entire future income will come from annuities, pensions and Social Security retirement benefits then it's relatively predictable, too. To the extent that future income will come from investments, that income is fairly unpredictable.

Expenses are unpredictable, as well. All we know for sure is how much we spent over the past few years. We can’t even be certain of the coming year’s expenses, so they too are uncertain. Retirement spending studies have shown that spending tends to decline as we age [2,3] but it doesn’t for everyone so we can’t assume that ours will. According to David Blanchett, whether or not it declines, annual spending volatility is relatively high (unpredictable from one year to the next).

When we take spending shocks into consideration, the future spending becomes even less certain. I spent $15,000 in the past year for two HVAC systems that I expected would last at least five more years. That has a much greater impact than my cable bill going up 5%.

The primary determinant of retirement cost is longevity. A five-year retirement will be far cheaper than one of 35 years. Our individual life expectancy is completely unpredictable assuming we are healthy.

As researcher Larry Frank keeps telling me, everything in an individual household’s retirement funding is stochastic, i.e., unpredictable.

Following is a graph of 200 randomly-selected portfolio value paths from a simulation of 10,000 scenarios for a retiree with a $1M portfolio from which she plans to spend $45,000 a year. All calculations are in real dollars and life expectancies are randomized using actuarial tables. It assumes a real 5.25% expected market return with a standard deviation of 12%.


Notice that most terminal portfolio values end up lower than the initial $1M portfolio value in real dollars. In this simulation, the median terminal portfolio value was about $860,000. About 75% of the scenarios ended with smaller portfolio values than the $1M they started with. This is typical of simulation results, though spending less would shift the bulk of those blue lines upward and spending more would do the opposite. The central mass of those blue lines would rotate around the starting point like clock hands, farther clockwise with more spending.

I read a comment on a retirement blog this week from a reader who said, “Retirement is uncertain so planning is useless.”

That’s like saying we shouldn’t plan outdoor activities because we can’t know future weather conditions with certainty. It’s like saying that companies shouldn’t bother developing business plans because they can’t know future economic conditions for sure. Of course you can plan where outcomes are uncertain and the best way to do that is with probabilities.

We develop retirement plans using models of the future but some models are much better than others. Nor is the model a plan. If we schedule a picnic for tomorrow and the weather models predict a 20% chance of rain, calculating the 20% is not the plan. The plan is deciding to take an umbrella or planning a backup activity. As in retirement planning, we use the model results to help create the plan.

Another blog suggested that Monte Carlo simulations can generate hundreds of thousands of future scenarios but that using them for planning is a mistake because a retiree can’t know which one she will experience. The first part of that statement is true. Your retirement's future finances might follow one of the blue lines in the chart above — assuming we ignore spending shocks — but it is impossible to know which one is yours.

There is a small chance that yours will follow a better path than any of these and a small chance that it will follow one worse and we can't ignore the latter's risk. (The former would just be sweet – we're OK with things turning out much better than we expected.)

Though it is true that we can't foresee our future path, it is also irrelevant — the purpose of the Monte Carlo model isn’t to predict an individual retiree’s path through the future (that’s impossible) but to explore a broad range of possible scenarios and develop some estimate of the probability of each actually being realized. Simulation is essentially a gigantic "what-if" analysis.

The weather forecasting model is likewise imperfect but the probabilities it provides are extremely useful. If there is a 5% chance of rain tomorrow perhaps we forego the umbrella. With a 95% probability or rain, we might cancel the event altogether. We don’t say, “No use planning based on the weather probabilities because we can’t know for sure.”


Determining how much you can safely spend this year requires a good model of the future.
[Tweet this]


The alternative the blogger suggested was simply to build a spreadsheet for a near-worst-case life expectancy using an expected market return and ignoring stock return variance. Ignoring variance, of course, ignores sequence of returns risk. This is the model of the future you get using a spreadsheet that assumes the same expected market return every year. It's the kind of analysis we did before Bengen. It's the kind of thinking that led Peter Lynch to suggest 7% annual spending from a stock and bond portfolio would be safe.

(When I say "spreadsheet" in this post, I'm referring to any model that assumes zero portfolio volatility and a fixed life expectancy. Actually, you can use a Monte Carlo simulator and set the portfolio volatility to zero and get the same results. On the other hand, you can build a Monte Carlo simulator with market volatility in an Excel spreadsheet – I have the scars to prove it. You can download the Mother of All Monte Carlo Spreadsheets at the Retire Early Home Page[1].)

The path to the median outcome is also in that jumble of blue lines above, so the first question we should ask the blogger is, "If you can't know which of those blue paths to choose, why did you go ahead and pick one (the one with the mean annual return), anyway?"

The following chart shows all those blue paths again with the "spreadsheet" prediction of the future superimposed (red) assuming a fixed 30-year retirement and zero market return variance. That’s effectively what a spreadsheet model produces.


Why is the spreadsheet future a nice, smooth upward curve while all the simulated blue lines are jagged and head off in all directions including ruin? 

The answer is sequence of returns risk. The spreadsheet ignores market volatility and consequently, it ignores sequence risk. The model in the simulation is much more realistic.

And why does the spreadsheet portfolio end up so large compared to most of the blue lines?

The answer is sequence risk plus longevity. Note that life expectancies are simulated to generate the blue lines (they end at different years of retirement), while the spreadsheet model assumes a fixed-length retirement of 30 years. Real-life portfolios and retirees don't often last 30 years so their portfolios most often have less time to grow.

Of all the paths on this chart, the red spreadsheet path is by far the least likely for you to experience. Twenty consecutive years of identical positive portfolio returns is unimaginable.

With 10,000 simulated scenarios, fifteen survive 20 years and end up within 1% of the spreadsheet path value at 20 years. These are fifteen possible paths to reach the spreadsheet value at 20 years and they don't get there in a straight line, as you can see on the following chart. So, the spreadsheet path is in there, but why one would choose it as the representative outcome remains a mystery.


The path that reaches the median terminal portfolio value, among the 10,000 simulated scenarios, is shown on the graph above in orange. It ends at year 17, which is roughly the median life expectancy for this 65-year old. The spreadsheet path presumably uses average historical market returns, so why is its outcome at 20 years ($1.2M) so much higher than that of the median simulated outcome of $860,000?

The chart shows that using the average market return every year in a spreadsheet (the red line) doesn't produce the average outcome (the orange line). You're probably tired of hearing me say "sequence risk and stochastic life expectancies are the difference" but simulations model them and spreadsheets don't.

The spreadsheet path is quite optimistic. If you insist on a spreadsheet model, you should at least reduce the expected return to compensate for sequence risk. In this comparison, you would need to reduce the expected portfolio return in the spreadsheet model from 5.25% to 3.5% to obtain results similar to the simulation's median outcome at 20 years.

Lastly, let's look at a density histogram of all portfolios that survived at least 20 years.


The blue bars show the portfolio values after 20 years for those 956 portfolios and retirees who survived at least that long. It's a probability density histogram, so the total area of the blue bars equals 1.

The orange curve shows the continuous density. As you can see, the distribution is right-skewed and not a symmetrical normal distribution. The median is less than the mean due to all those huge but highly improbable outcomes along the right tail.

We're more interested in the median ($645,000), the value at which half of the portfolios are larger and half smaller. About 57% of the outcomes after 20 years are less than the mean ($702,000) compared to the 50% of outcomes that are less than the median. The median is the more representative statistic with this skewed distribution.

Finally, the red vertical line represents the spreadsheet model's portfolio value after 20 years, $1.17M. At the 20-year mark, that red portfolio value is larger than 88% of the simulated portfolios that survived that long. $645,000 is a much more representative expectation after 20 years than the spreadsheet prediction.

It's true that you can't predict which of those 10,000 blue paths your future will mimic but the spreadsheet outcome is one of those. Pick it and you simply decided to pick a path from the 10,000 choices after saying you couldn't. And, then you picked a very unlikely and optimistic one. The spreadsheet predicts portfolio values in the absence of sequence and longevity risk and tells you nothing about the probability of realizing them.

Furthermore, the purpose of simulation isn't to predict your future but to explore the possibilities, so you aren't intended to choose one.

We need a model of the future to plan retirement. We need it to even calculate the safe amount to spend this year. Simulation is a good starting point.

The takeaway, for now, is that if you have planned your retirement with a spreadsheet model, you should take another look, especially if your plan shows a single straight path to doubling (or more) your initial portfolio. It could happen; it just isn't likely and if it does happen it certainly won't be a smooth path.

If you're using an online calculator, make sure it incorporates simulation. There's a Monte Carlo version of E$Planner, for example, and there are free simulators online.

But simulations have issues, too, so they're only a starting point. I'll discuss the shortfalls of simulation next time before describing how to make sense of that jumble of blue lines.



REFERENCES

[1] Download The Retire Early Home Page spreadsheet.


[2] Expenditure Patterns of Older Americans, 2001‒2009, Sudipto Banerjee, Employee Benefit Research Institute. (Download PDF.)


[3] Estimating the True Cost of Retirement, David Blanchett. (Download PDF.)