Tuesday, November 22, 2016

Trump, Monte Carlo and Insectivores

Some brilliant minds recently assured us that Donald Trump would lose the presidential election and probably lose it bigly. Nate Silver, the High Priest of Election Models and founder of the FiveThirtyEight blog, won the distinction of being the least wrong in predicting the outcome. Although he continually warned us that he might be wrong, his clear implication was that Hillary would win (see Donald Trump's Six Stages of Doom linked below if you question that). Others seemed to feel nearly certain.

In theory, no outcome was completely missing from the predictions, but many were given extremely low probabilities. There was even a possibility that Evan McMullin would win, just not a very big one. (See How Evan McMullin Could Win Utah And The Presidency, linked below.)

Wired magazine ran a story, obviously before the election, entitled, “2016’s Election Data Hero Isn’t Nate Silver. It’s Sam Wang.” To quote Wang, a professor of neuroscience at Princeton and now-famous insectivore who writes the Princeton Election Consortium blog, “It is totally over. If Trump wins more than 240 electoral votes, I will eat a bug.”

Crunch. Crunch.

(The more precise term may be entomophage.)

Nate Cohn at The Upshot needs to eat a bug or two himself, as do the election markets. They all predicted much higher probabilities of a Clinton win than did Silver. I'm not sure there are enough bugs to go around.

These are a lot of high-powered minds with deep understandings of statistics and modeling but they were all – to quote The Donald – Wrong!

How could that happen and what does it have to do with retirement planning?

Predicting election results is social science. Models using Monte Carlo simulation are good at predicting outcomes for physical science in which objects tend to behave in predictable ways under the same conditions. People are not so predictable.

There is an entire social science discipline, called behavioral economics, whose sole purpose is to explain the unpredictable and irrational economic behavior of humans. This unpredictability is a primary reason why statistical modeling won't be as predictive in social science fields as it is in physical science. (see The Marketplace of Perceptions, linked below.)

Monte Carlo simulation was created by physicists. Enrico Fermi used Monte Carlo techniques in the calculation of neutron diffusion in the 1930s. Atoms tend to behave in the same probabilistic ways under the same conditions regardless of emotions, doubts, fears, and biases. People don't.

Silver, Wang, Cohn and the gang have to build a lot of judgments and assumptions into an election model and in 2016 many of these were obviously erroneous. As the exit polls come in, you will hear each expert explain why he was wrong or that he was actually right but we didn't interpret his results correctly. (Like Cohn's Putting the Polling Miss of the 2016 Election in Perspective, link below.)

In other words, we had a lot of confidence in the models and that was a big mistake on our part.

OK, poll aggregators, I can accept my responsibility here and promise to never put much stock in your predictions from now on – my bad. I actually drew comfort from the fact that so many models suggested the same result (a Clinton win) but apparently there was a strong correlation between models that I overlooked. Apparently “herding”, the tendency of polling firms to produce results that closely match one another, especially toward the end of a campaign, is a problem not only with pollsters but among poll aggregators, as well.

Physics models can be more accurately defined than social system models and can use better-defined input data.

The radioactive decay rate of 239Plutonium is pretty consistent. “With a half-life of 24,100 years, about 11.5×1012 of its atoms decay each second by emitting a 5.157 MeV alpha particle.” Monte Carlo simulation models of decay can predict outcomes pretty accurately.

Compare that with “We have a poll from the L.A. Times, but our judgment is that it is skewed a point and a half toward the conservative candidate most of the time so we give it a rating of B+ and weight its contribution a little lower.” You get both an imprecise measurement and a judgment of its quality.

A few weeks later, the L.A. Times poll will show a different result, but 239Plutonium will still be decaying at precisely the same rate. Plugging more accurate and consistent inputs into a statistical model provides more predictive calculations.

Then there is the “one-time event” problem. If Silver was correct and Trump had a one-in-four chance of winning (it was probably much greater than that) then if the election were held one hundred times under identical conditions, Trump would win 25 of them. But the 2016 election was a one-time event. Trump won 100% of the election and Hillary lost 100% of it. Your household's retirement is also a one-time event.

We frequently use Monte Carlo simulations for retirement research. I think that is a far better application than for individual retirement planning precisely because research isn't trying to predict an outcome for a single household. It should be used much more carefully for the practice of retirement planning.

So, predicting the results of a one-time election event in a social system with imprecise and conflicting data sources turns out to be a very difficult thing to do and clearly far more difficult than many of us who believed in the modeling understood. In simplest terms, we are attempting to predict the future, or at least to characterize it fairly precisely, and you know what Yogi said about predictions – they're hard, especially about the future. Statistical models enable us to guess the future of social systems and be wrong with amazing accuracy.

There are so many variables, both known and unknown, so little high-quality clear data, and so much difficulty predicting human behavior that I suspect it's a fool's errand to try to predict a close national election. Someone pointed out that the models would have worked better if the electorate weren't so evenly split. That's probably true, but if the electorate weren't so equally split and the winner was more obvious, then why would we need the models?

A friend from my AOL days, Joe Dzikiewicz, made an interesting observation about the Electoral College (EC) and chaos theory. (So you don't spend the rest of your day wondering, it's pronounced “Ja-kev'-itz”). One attribute of chaotic systems is that a small change in initial conditions can result in dramatically different outcomes. If a small change either way in the popular vote can swing the Electoral College vote and greatly change the future path of world history, then the EC actually creates chaos, or "unpredictability." The outcomes may simply be unpredictable by statistical inference models under the initial condition that the electorate is closely divided. As I suggested in Retirement Income and Chaos Theory, the constant-dollar withdrawal assumption probably makes Monte Carlo retirement models chaotic, too.

What does this have to do with retirement planning? Retirement planning is economics, a social science. William Bernstein warned against using engineering techniques to develop retirement plans suggesting, as I recall, something along the lines that retirement is not like an airfoil that always behaves the same way under the same conditions. (I have searched for that quote with no success. If you know where to find it, please let me know. Perhaps it was in a Money magazine piece?)

The financial planning industry currently depends very heavily on Monte Carlo simulation for retirement planning. In fact, I have written that we often use Monte Carlo simulations instead of actual planning. Many of us try to show that the probability of failure is so low that we shouldn't worry about it (like Trump's odds of winning) and we give short shrift to actually planning for such a catastrophe as outliving our wealth, should it happen.

Assumptions and judgments can dramatically affect the results of retirement simulations. There may be a 95% probability that you won't outlive your wealth if you get good advice, but if you do you will be 100% up that famous creek. Precious few of the planners who use Monte Carlo simulation have Nate Silver's skills, but they make critical judgments and assumptions just the same.

Some of the retirement model assumptions are ridiculous, like assuming that retirees will keep spending the same amount until they go flat broke and the assumption that our limited availability of historical market returns data is adequate to predict the future with reasonable confidence. The election models struggle with assumptions about human behavior, but the typical retirement model is based on a particular human behavior that we all agree would be irrational.


Yeah? Well, Hillary had a 95% chance of being elected President.
[Tweet this]


Though the polls were clearly flawed in many ways, thousands were available on which to base a prediction of election results in 2016. For retirement planning, we have about 200 years of data representing about seven distinct 30-year periods of market returns. We use rolling periods, a technique that is statistically flawed, to create 170 or so rolling 30-year periods. That still isn't much data to predict the future. Such a limited amount of historical market returns data makes for some very large confidence intervals.

As I noted in The Whoosh! of Exponential Retirement, Moshe Milevsky assures us that we can be 95% certain that a shortfall probability of 15% actually lies somewhere between 5% and 25%. Gordon Irlam showed us an example in which the optimal asset allocation has a huge 95th-percentile confidence interval of 10% to 82% equities. That's not a lot of confidence.

If predicting the results of a presidential election is difficult, retirement planners try to predict the probability of a one-time social event (a client's retirement) with far less data.

My point is this. If you think an election not turning out as predicted is a poor outcome, imagine that the 95% probability of success your planner promised ends with spending your late retirement living off Social Security benefits alone. (Somebody has to fall into that 5%, who says it won't be you?) The tools that didn't predict the election are basically the same ones that planners use to predict retirement finances and those tools failed in the election with better data, a more rational model and entire teams of more highly-skilled statisticians running them.

I'm not suggesting that retirement simulations are worthless; they can provide valuable insights but they should occupy an appendix of a retirement plan and be viewed as one more interesting data point, not as complete plans.

We had confidence in the election models and that was a big mistake on our part. Let's not be overconfident about retirement forecasts.

The next time an advisor hands you a Monte Carlo simulation and says you have a 95% chance of funding your retirement, tell them, “Yeah? Well, Hillary had a 95% chance of being elected President.”

Then ask him if he's willing to eat a bug.



REFERENCES

Donald Trump's Six Stages of Doom, Nate Silver, FiveThirtyEight blog.


2016’s Election Data Hero Isn’t Nate Silver. It’s Sam Wang, Wired magazine.


Putting the Polling Miss of the 2016 Election in Perspective, Nate Cohn, The Upshot blog.


Princeton Election Consortium blog.


How Evan McMullin Could Win Utah And The Presidency, FiveThirtyEight blog.


The Marketplace of Perceptions, Harvard Magazine.