Three months ago, we began keeping track of how well the predictions we offer on this site have performed over time.

To do that, we adapted the plus-minus statistic from hockey and basketball for our own purposes. If we make a correct prediction, meaning what we predicted comes to pass, we score it as +1. A prediction that turns out to be wrong is scored as a -1. And if the outcome for the prediction in question is uncertain, for whatever reason, we score it as a zero (+0). We obtain our plus-minus score by adding up these results.

What we like about this approach is that if our predictive ability is no better than the random outcome determined by a coin toss, our plus-minus score will drift toward a value of zero over time. If we're better at making predictions than a simple randomness would suggest, then our plus-minus score will grow higher in value over time. If we're wrong, then our plus-minus score will fall in value. If we're really bad, then our plus-minus score will plunge into deep negative territory!

The last time we did this exercise, we found that for the 25 occasions where we made predictions spanning the period from 1 January 2008 through 16 April 2009, we had earned a plus-minus score of +8, which really covered some 28 predictions, of which 16 were correct, 8 were incorrect, 1 was scored as "uncertain" and 3 were no decisions, meaning that we were still waiting to determine their associated outcome. We also noted that our plus-minus score since February 2008 is +11, as we found and corrected an error in our interpretation of stock market dividend futures data, which had resulted in three "misses" in Feburary 2008.

Since then, we've been able to move two of the three "no decisions" out of that status and we've added several new predictions and recorded their outcomes. The table below provides the updated results.

Political Calculations' Plus-Minus Score Update, 16 July 2009
Date	Prediction	Outcome	+/- Score
13 February 2009	We predict that significant changes in the U.S. income tax are in the works given the kinds of questions that members of the U.S. Senate (or their staffs) are asking us through Google.	We'll see. Starting to look *very* likely.	+0
17 February 2009	We present a prediction that GM is heading toward bankruptcy that we had originally put forward back on 2 October 2008 at Econbrowser.	Looks pretty likely. GM declared bankruptcy on 1 June 2009.	+1
2 April 2009	We up our forecast for the S&P 500 to hit values between 860 through 890.	Although we didn't specify it on our own site, this prediction applies through the end of June 2009. Right now, there's been a very small bit of erosion in the dividend futures we use to create our forecasts, so if we made this prediction today, our methods would have us put the S&P 500 between 855 and 885 from now through June 2009. For now though, we're going to count this prediction as neutral. Update 16 July 2009: It can't be counted as neutral any more - we called it!	+1
20 April 2009	We jumped all over an unexpected plunge in the value of stocks on this day, recognizing it as an unexpected "noise" event. We put all the money we had on the sidelines into the S&P 500.	Oh, yes! It turned out to be a one-day plunge, with stock prices quickly returning to the level we had forecast back on 2 April 2009.	+1
28 April 2009	Barry Ritholtz was awfully pessimistic on the morning of 28 April 2009, anticipating the beginning of a significant correction. We didn't see things that way….	Ironman 1, Ritholtz 0. But only for the day. Let's face it - Barry's too good an analyst to not pick up on a building trend, and he saw it developing before we did (we should also point out that while we framed the market bottom on 9 March 2009, Barry hit the bottom call just about dead center.)	+1
30 April 2009	We saw a short-term selling opportunity developing in the stock market.	Very short term - just one day! The S&P 500 had risen to 886 at the time we posted that comment early that morning. Afterward, stock prices dropped to close at 872.81.	+1
4 May 2009	We anticipated that the S&P 500 would rise to average between 935 and 955 during the month of May 2009.	Stock prices came within 5 points of hitting the bottom level of our range on 8 May 2009, but then backtracked as investors appeared to shift the base from which they were projecting the future level of stock price.	-1
12 May 2009	We lowered our forecast from 4 May 2009 for the S&P 500 from 925 to 945 to account for the erosion of the future expected level of dividends per share we observed. We also indicated that any significant drop below these levels would have to be considered a strong buying opportunity.	Mixed. We still hadn't caught on to the backward shift in the base for investor expectations, but we were correct in identifying stock prices below this level as representing a buying opportunity.	+0
18 May 2009	We finally recognized that stock prices were being based from an older, lower level of stock prices, which would anticipate a level for the S&P 500 of 909 to 927.	No prediction here, just finally catching up to reality!	+0
26 May 2009	We pick up on an unlikely correlation between the timing of plunges in the stock market and the results of our preferred method for determining the probabilty of recession in the U.S. We predict stocks will dive in value around 16-23 June 2009 and 10-16 September 2009.	One down, one to go. The timing for the June drop actually coincided with the work week, beginning on Monday, 15 June 2009 and ending a week later. We'll have to see where things stand in September to see if we scored a two-fer here!	+1
8 June 2009	We forecast that stock prices will range between 935 and 955 for the month, as measured by the S&P 500. Not counting the dive we anticipated for stock prices on or around 16 June 2009 through 23 June 2009, which we suggested would lead us to sell our investment in the S&P 500 later in the month "until the smoke clears."	We have to score this prediction as a zero, or mixed, even though we were on track with it until that turmoil we forecasted arrived nearly on schedule later in the month. Since we didn't take the drop in values of that event into account, that portion of our prediction is incorrect.	+0
9 June 2009	We created a new tool for predicting the direction and level of both 30 year conventional mortgage rates and 10-Year constant maturity U.S. Treasury yields.	Using the tool, we anticipated that either mortgage rates would rise or U.S. Treasury rates would fall. Of these, we thought higher mortgages rates were more likely, but as it happened, both outcomes came to pass within a week.	+1
15 June 2009	Once again, we put real money at stake. This time, by anticipating falling stock prices and selling our entire stake in the S&P 500.	We missed the top before stocks began dropping significantly by one day and 2.5 points. Even though we were nearly right about the timing, we're not really market timers.	+1
22 June 2009	We went back "all-in" to the S&P 500, figuring there was very little downside to doing so.	It turned out there was no downside - stocks continued rising through the end of the month.	+1
25 June 2009	We forecast that the average of the S&P 500 for the month of June 2009 would close with an average value between 926 and 944.	Coming as close to the end of the month as it was, this prediction was like shooting fish in a barrel, even with the turmoil in the previous week and the possibility it was only taking a breather. We're not counting it.	+0
1 July 2009	We forecast a range for the S&P 500 in July 2009 between 925 and 945.	The month's not over, so the jury's still out - to hit the bottom end of our target though, stock prices will need to run at the high end of our range through the end of the month. It's likely that the average for the month will fall a bit below our predicted range.	+0
2 July 2009	Stocks fell off the edge, once again unexpectedly. We found no fundamental reason why they should have done so, identifying the action as being the result of noise as opposed to a response to a signal. We suggest that if stock prices stay depressed in the next week, it represents a buying opportunity.	Noise off! At this writing, stocks have fully rebounded into the range we forecast for 1 July 2009. One of the nice things about our methods is that they help separate the signal from the noise, so we can have a decent idea of what's a significant event we need to respond to, versus one we can sit and wait out. Here, patience paid off.	+1

For our previous no-decisions, two have now been added to our positive total while one is still pending, but looking more and more likely. By our count, we've added some 16 predictions over the last three months, including several we've "bet" our entire retirement portfolio upon. Of these new predictions, we scored 9 hits, 3 misses (we guess two of these could be counted as near misses, but let's face it, they're still misses!), 1 uncertain outcome, 1 "right, but too easy, so scored it as a zero", and we have two "no decisions as yet" pending.

The net effect of all these changes is to increase our plus-minus score from +8 to +17. Not too shabby when you consider that most of what we predict is where the stock market is heading next. We wonder what Burton Malkiel might make of all this!

Labels: forecasting, track record

- posted by Ironman at 12:57 PM | Permalink | Home |