Baseball for the Thinking Fan

Login | Register | Feedback

btf_logo
You are here > Home > Baseball Newsstand > Baseball Primer Newsblog > Discussion
Baseball Primer Newsblog
— The Best News Links from the Baseball Newsstand

Saturday, March 31, 2012

Barra: Baseball Predictions Are Much Better In Hindsight

I bet those old Barra/Ignatin articles aren’t…(end casual harrumphing)

Suppose you brought all the smartest baseball kids on the block together, locked them in a house for two weeks with an unlimited supply of Twinkies and beer, and asked them to pool all their statistical tools to predict the upcoming baseball season? What might the result look like?

SBNation’s Jason Brannon did that - well, he didn’t actually lock them up, but he combined (or averaged) several of the most egg-headed systems from the 2011 season. and compared to the actual results. The predictors he used were PECOTA (developed by Nate Silver for Baseball Prospectus), Marcel (developed by Jack Sackman and Tom Tango and named for a monkey named Marcel), CAIRO (an advanced version of Marcel) and BP-cofounder Clay Davenport’s DTs (for Davenport Translations, which are so complicated you’ll have to Google him and figure it out yourself).

...But the experts didn’t project a single exact win total for any team; the closest was the one-game difference between the Pirates, projected to win 71 and coming up with 72, and only three teams - the Braves, Dodgers and Giants—within two.

The worst predictions were Minnesota - winning 63 games against the projected 84, though we must cut the prognosticators some slack as the Twins’ two superstars, Joe Mauer and Justin Morneau, were out for most of the season - and Arizona , 73 actual vs. 94 projected, which I would file under the heading of “Who cares”?

I’d say this proves one thing about any system based on sabermetrics: they do a much better job predicting the past than the future.

Repoz Posted: March 31, 2012 at 09:52 AM | 43 comment(s) Login to Bookmark
  Tags: sabermetrics

Reader Comments and Retorts

Go to end of page

Statements posted here are those of our readers and do not represent the BaseballThinkFactory. Names are provided by the poster and are not verified. We ask that posters follow our submission policy. Please report any inappropriate comments.

   1. Tricky Dick Posted: March 31, 2012 at 10:10 AM (#4093677)
I don't think I would judge the accuracy of forecasts based on whether the forecasts predict the exact number of wins for a team. Is the article saying that no single forecast predict the exact win total for a team, or that the average of the forecasts didn't predict an exact win total?
   2. Champions Table Posted: March 31, 2012 at 10:17 AM (#4093681)
I feel pretty comfortable predicting that every team will win 80 games, give or take 20.
   3. Jolly Old St. Nick Is A Jolly Old St. Crip Posted: March 31, 2012 at 10:24 AM (#4093684)
The problem with pre-season predictions is that they depend on too many factors that are completely unpredictable. That sounds like Yogi Berra, but it's obviously true, even if I'd trust Nate Silver's predictions in the long run over those of a random collection of beat writers.
   4. bfan Posted: March 31, 2012 at 10:26 AM (#4093686)
I thought all Nate Silver did was average what everybody else projected; not necessarily bad, but nothing ground-breaking.
   5. Best Regards, President of Comfort, Esq. Posted: March 31, 2012 at 10:57 AM (#4093693)
I can't believe that these projection systems don't project mid-season trades and injuries!

A projection system should be graded on the results for the players, not the teams.
   6. Dan Szymborski Posted: March 31, 2012 at 11:11 AM (#4093697)
The thing is, you're *supposed* to be wrong a lot. If you know for a fact that every team in baseball is absolutely exactly .500, over 162 games, far, far, better than any knowledge we can ever have, you'd still expect 9% of the teams to win 72 games or fewer and 9% of the teams to win 90 or more.
   7. TerpNats Posted: March 31, 2012 at 11:18 AM (#4093702)
The worst predictions were Minnesota -- winning 63 games against the projected 84, though we must cut the prognosticators some slack as the Twins’ two superstars, Joe Mauer and Justin Morneau, were out for most of the season -- and Arizona, 73 actual vs. 94 projected, which I would file under the heading of “Who cares”?
I believe the D'backs' numbers were transposed, and under "who cares," I should say some folks in Denver, Los Angeles, San Diego and San Francisco cared...in addition to those in Phoenix.
   8. bob gee Posted: March 31, 2012 at 12:28 PM (#4093738)
should i be awaiting the similar column which quotes the "expert sportswriters" success rate on football games or similar events? one of the things that got me interested in statistics was seeing 30 years ago how awful the so-called experts were at predicting the future.
   9. bfan Posted: March 31, 2012 at 12:36 PM (#4093745)
I just looked at the SI predictions for NCAA basketball; these are people who cover the sport as a part of their full-time job, and they got 9 of the sweet 16 right. They have MSU winning it all, which is not looking good right now.
   10. Greg K Posted: March 31, 2012 at 12:37 PM (#4093746)
That's ironic. Hockey predictions are usually harder to make in hindsight. Baseball's funny that way.
   11. Monty Predicts a Padres-Mariners WS in 2016 Posted: March 31, 2012 at 12:37 PM (#4093747)
but he combined (or averaged) several of the most egg-headed systems from the 2011 season


I don't like that methodology. To take the easy example, if you average Marcel and CAIRO, all you end up with is a less-accurate CAIRO.

should i be awaiting the similar column which quotes the "expert sportswriters" success rate on football games or similar events?


Gregg Easterbrook does that in his post-Super Bowl Tuesday Morning Quarterback.
   12. HowardMegdal Posted: March 31, 2012 at 12:38 PM (#4093749)
So the control for how effective sabermetric predictions can be is THE FUTURE, rather than educated guesses of non-sabermetric sportswriters? Well, that seems fair, since all projection systems claim to be 100 percent correct.
   13. McCoy Posted: March 31, 2012 at 12:41 PM (#4093750)
The anti statheads always seem to use THE FUTURE to define how valid (better known as how useless) stats are. I've seen this line of reasoning numerous times now in the last few years.
   14. Dan Szymborski Posted: March 31, 2012 at 01:23 PM (#4093775)
The anti statheads always seem to use THE FUTURE to define how valid (better known as how useless) stats are. I've seen this line of reasoning numerous times now in the last few years.


All I ask of anti-statheads is that they run their own lives consistently to their beliefs. That means all their medical diagnoses and financial management being guided by gut and all the products they buy slapped together by feel. No medical evidence that magnetic bracelets treat malignancies better than chemotherapy? Nope! You can't go to a doctor that knows it.
   15. Dan Szymborski Posted: March 31, 2012 at 01:28 PM (#4093778)
There's a certain irony about complaining about people not predicting the future well, when the author doesn't even get the *past* right - how the heck did Jeff Sackmann turn into Jack Sackman?
   16. Swedish Chef Posted: March 31, 2012 at 01:33 PM (#4093781)
. That means all their medical diagnoses and financial management being guided by gut and all the products they buy slapped together by feel.

I don't think this is a great idea, because if statheads has to reciprocate I wouldn't feel at all comfortable with letting quants handle all my money.
   17. Infinite Joost (Voxter) Posted: March 31, 2012 at 01:43 PM (#4093786)
how the heck did Jeff Sackmann turn into Jack Sackman?


Sounds better.
   18. Floyd Thursby Posted: March 31, 2012 at 01:43 PM (#4093787)
how the heck did Jeff Sackmann turn into Jack Sackman?


He was bit by a radioactive sack, and he felt that crime was getting out of hand. Same as you'd expect.
   19. Fancy Pants Handles lap changes with class Posted: March 31, 2012 at 02:10 PM (#4093791)
There's a certain irony about complaining about people not predicting the future well, when the author doesn't even get the *past* right

On a similar note, this is not how I recall things happening last season:
Arizona , 73 actual vs. 94 projected
   20.   Posted: March 31, 2012 at 02:16 PM (#4093793)
They're not predictions, they are projections. The are not saying the Pirates will win 71 games, they are saying that the Pirates are more likely to win 71 games than they are some other number. It means, at best, they have maybe a 10% chance of winning 71 games, a 9.5% chance of winning 72 games, and so on and so forth. That they won 72 instead of 71 doesn't make the projection wrong.

Predictions are impossible, they are a fantasy. To want them you may as well wish for the tooth fairy.
   21. Voros McCracken of Pinkus Posted: March 31, 2012 at 02:29 PM (#4093800)
Fromway back in 2008. True to form, 94% of the teams finished within 18 games of those (the Rays and Padres were the only ones to miss). Obviously actual projection systems should do a bit better than that.

I figure folks just don't get how much natural error there is in predicting the future. It's both a very difficult thing to do, and a very necessary part of most decision making processes.
   22. Walt Davis Posted: March 31, 2012 at 02:35 PM (#4093801)
To the extent there's a "problem" with a system predicting team wins, it's that they regress "too much" towards the middle. That is, ZiPS (or whatever) might not project any team to win more than 92 games knowing full well that 2 or 3 teams are going to win more than 92 games.

By then averaging several systems, all you achieve is even more regression towards the middle, producing even less variation in the estimates and further guaranteeing that the results will be "more wrong."

They are probably the two hardest things to get across about statistics. First, statistics never gives you THE answer. Statistics gives you a probabilistic statement that allows you to infer (from sample to population or from past to future) with a given level of confidence under a certain set of assumptions. Second is that the level of precision in that inferential statement is what statistics is primarily about, not the point estimate.

And in the particular case of rate stats, there's a massive amount of randomness in the (assumed) process as Dan points out. Tell me a player has a "true" 350 OBP and will have 600 PA and we still can't say he will reach base exactly 210 times -- in fact our best (95%) guess is that he'll reach base between 187 and 233 times (roughly 310 to 390 observed OBP). Now you throw in the uncertainty of our prediction of the player's true talent (which itself will be on the order of 20 points of OBP either side) and you start to understand how hard it all is.

Personally, I pay little attention to team projections out of these systems other than perhaps as a signal that some particular team is better/worse than I think and/or the gap between two teams is bigger/smaller than I think. The season projections are nice little consumer marketing tools really. Not that there's anything wrong with that! (Writers' predictions serve the same purpose.)

Dan probably does a pretty thorough job on his simulations but to do it right you would want to build in some fairly extreme playing time distributions (i.e. random injuries), managerial profiles, trades, minor-league breakouts, etc. along the lines of an OOTP. I think Dan's already running 1 million sims so what's another 9 million? :-) But of course that won't make the projections any more "accurate" in the sense that Barra wants them -- in fact they'll incorporate more random (i.e. unknown/unpredictable) variation -- but they could be more "realistic" and give us a more accurate sense of the uncertainty in a season.
   23. Dan Szymborski Posted: March 31, 2012 at 03:00 PM (#4093810)
I don't think this is a great idea, because if statheads has to reciprocate I wouldn't feel at all comfortable with letting quants handle all my money.


Who would you rather have working for you?

A) The Nate Silver of Finance

or

B) The Murray Chass of Medicine
   24. Dan Szymborski Posted: March 31, 2012 at 03:06 PM (#4093815)
Dan probably does a pretty thorough job on his simulations but to do it right you would want to build in some fairly extreme playing time distributions (i.e. random injuries), managerial profiles, trades, minor-league breakouts, etc. along the lines of an OOTP.

Well, some of the stuff I do my best to simulate. I estimate base playing time expectations and with the percentile data, I developed covariance matrices for performance vs. playing time. Missing at-bats are plugged in by the reserves I denote.

That, plus I got to do them way later than the last couple years and got to do another type of thing for ESPN Mag itself. When you take into account the lead time needed (plus bringing in athletes for cover, etc) I was having to finalize rosters at the start of spring training. So I feel better about these win projections than I ever have, though obviously a ton will still miss, many by huge amounts!
   25. Voros McCracken of Pinkus Posted: March 31, 2012 at 03:06 PM (#4093816)
IT'S LEECHES FROM HERE ON OUT, MR. PRESIDENT
   26. Dan Szymborski Posted: March 31, 2012 at 03:34 PM (#4093829)
To the extent there's a "problem" with a system predicting team wins, it's that they regress "too much" towards the middle. That is, ZiPS (or whatever) might not project any team to win more than 92 games knowing full well that 2 or 3 teams are going to win more than 92 games.

I do keep track of it and make it as clear as I can to people that if the top projection is 92 wins, it still means that there will almost certainly be teams that win more than 92 games, just that no individual team is more likely to than not. Not complete success as a lot of people don't really get probability, but hopefully it does so good.
   27. Voros McCracken of Pinkus Posted: March 31, 2012 at 04:14 PM (#4093846)
One of my favorite terms is "Calvinist probability theory"; the idea that whatever happened had a 100% chances of happening and whatever didn't had a 0% chance of happening. Philosophically and spiritually, I actually don't have a problem with this. There's no reason this has to be false.

The problem, of course, is that unless you're the omniscient god running things, it's useless in trying to ascertain the future in the real world. At that point, probability theory comes into play and your predictions inevitably fall somewhere on the continuum between 0 and 1.
   28. Swedish Chef Posted: March 31, 2012 at 04:24 PM (#4093851)
Maybe it would be better pedagogy to skip assigning win numbers to teams when the real outcome has such variance and instead throw them in three or four bins, say lousy, middling and good (and maybe add "pathetic" and "glorious" for real outliers). And then have those who want to know the numbers read a multipage disclaimer and force them to assent to it before giving them the goods.
   29. Dan Szymborski Posted: March 31, 2012 at 04:42 PM (#4093857)
Chef, for mine, I did include 5th percentile/95th percentile and % for division, wild card, and World Series.
   30. Swedish Chef Posted: March 31, 2012 at 05:05 PM (#4093865)
Chef, for mine, I did include 5th percentile/95th percentile and % for division, wild card, and World Series.

That's very good, but I doubt it woos Murray Chass. I wouldn't expect you to spend a lot of energy finding a way to charm Chass, of course, he's just some silly blogger.


   31. Repoz Posted: March 31, 2012 at 05:58 PM (#4093892)
Tango chips in...

"If we can’t win over Allen Barra, who can we win over?"
   32. What did Billy Ripken have against ElRoy Face? Posted: March 31, 2012 at 07:36 PM (#4093930)
Maybe it would be better pedagogy to skip assigning win numbers to teams when the real outcome has such variance and instead throw them in three or four bins, say lousy, middling and good (and maybe add "pathetic" and "glorious" for real outliers). And then have those who want to know the numbers read a multipage disclaimer and force them to assent to it before giving them the goods.


I hear all your posts in my head as if they were actually being read by the Swedish Chef. Thank you for this.
   33. Ron J Posted: March 31, 2012 at 09:21 PM (#4093975)
#23 I think Long Term Capital Management would have been insulted to have been called the Nate Silver of Finance. I mean they did have co-winners of the Nobel Prize in Economics. (When Nate wins his Nobel Prize he'll be a good comp)

Murray Chass of doctors is tricky and I don't think it's the name you're looking for. Chass did very good work on the economic front. Conceptually he looks more like a specialist with a narrow area of competence, and that's pretty much what you generally get in medicine today.

I think somebody like Griffin would be a better choice.

   34. RoyalsRetro (AG#1F) Posted: March 31, 2012 at 09:35 PM (#4093989)

The thing is, you're *supposed* to be wrong a lot. If you know for a fact that every team in baseball is absolutely exactly .500, over 162 games, far, far, better than any knowledge we can ever have, you'd still expect 9% of the teams to win 72 games or fewer and 9% of the teams to win 90 or more.


This. I think Nate Silver said something to the effect that if his models ever predict 100% of political races, he'll know the models are broken.

Human behavior is not completely predictable. Film at eleven.
   35. SoSHially Unacceptable Posted: March 31, 2012 at 09:41 PM (#4093992)
This. I think Nate Silver said something to the effect that if his models ever predict 100% of political races, he'll know the models are broken.


On top of that, if Szym and the Gang could ever consistently predict teams' records within 2 or 3 games, baseball would be a lot less interesting.

   36. Fanshawe Posted: March 31, 2012 at 10:26 PM (#4094016)
Yeah stat heads, you have a lot of work to do before you can go 0-90 picking World Series participants like these EXPERTS!!!!
   37. Dan The Mediocre Posted: March 31, 2012 at 11:07 PM (#4094025)
   38. Dan Szymborski Posted: April 01, 2012 at 01:05 AM (#4094071)
Yeah stat heads, you have a lot of work to do before you can go 0-90 picking World Series participants like these EXPERTS!!!!

Luckily, I forgot to send my picks to the editor in the end-of-spring frenzy, so I escaped officially joining our 0-fer AL East. It was quite surprising to see every single writer pick the Red Sox!
   39. Dan Szymborski Posted: April 01, 2012 at 01:06 AM (#4094072)
I only posted to share that link.

Woo! That's one I might've missed.
   40. Fancy Pants Handles lap changes with class Posted: April 01, 2012 at 02:15 AM (#4094075)
I only posted to share that link.

Woo! That's one I might've missed.

I'm betting the ZiPS projection scrawled on the back is 000/000/000.
   41. Walt Davis Posted: April 01, 2012 at 05:34 AM (#4094085)
#26 -- I didn't say that as clearly as I probably should have. Nothing (necessarily) wrong with a system that projects a top of 92 and I didn't mean to imply otherwise.

And Dan I think you do a good job of trying to explain these things and present the variance. You'll never get through to everybody.

I don't particularly blame the folks like Barra either. The experiment that is a baseball season gets run only once. It could take a long time before deciding that ZiPS (or any other team projection system) is "good." The marginal advantage of a system over "experts" or the wisdom of the mobs is likely to be pretty small in the short term so it's not like we can beat Barra et all over the head with obvious evidence. Again, just the nature of the beast in projecting something with this much variance.
   42. McCoy Posted: April 01, 2012 at 11:46 PM (#4094498)
Yeah stat heads, you have a lot of work to do before you can go 0-90 picking World Series participants like these EXPERTS!!!!

.(JavaScript must be enabled to view this email address)

BTF was 0-29. Not a single person predicted the Rangers would get to the series and of the people who picked playoff series not a single one predicted the Rangers would win a single playoff round. Only 15 people thought they would make the playoffs. 11/4 Div/WC.

Only 5 people picked the Cardinals to make the playoffs and only one person correctly guessed that they would get to the Series.
   43. cardsfanboy Posted: April 02, 2012 at 12:10 AM (#4094509)
actual link to the prediction thread.


How did I miss out on being in that prediction thread?

other prediction thread for last year.

I see, there were two.

You must be Registered and Logged In to post comments.

 

 

<< Back to main

BBTF Partner

Support BBTF

donate

Thanks to
BDC
for his generous support.

Bookmarks

You must be logged in to view your Bookmarks.

Hot Topics

NewsblogPrimer Dugout (and link of the day) 9-2-2014
(40 - 8:14pm, Sep 02)
Last: The Robby Hammock District (Dan Lee)

NewsblogOT: Politics, September, 2014: ESPN honors Daily Worker sports editor Lester Rodney
(310 - 8:12pm, Sep 02)
Last: BDC

NewsblogThe indisputable selfishness of Derek Jeter
(25 - 8:10pm, Sep 02)
Last: Howie Menckel

NewsblogPhoto of the day: Bill Murray, indy league ticket-taker
(122 - 8:09pm, Sep 02)
Last: toratoratora

NewsblogPassan: 10 Degrees: Cole Hamels' trade value might be Phillies' lone bright spot
(4 - 8:06pm, Sep 02)
Last: The District Attorney

NewsblogSpector: Negative run differential doesn't tell whole story for first-place Cardinals
(7 - 8:01pm, Sep 02)
Last: madvillain

NewsblogAdam Jones says he was joking about 'airport' comment at social media event
(20 - 7:57pm, Sep 02)
Last: zonk

NewsblogGleeman: Twins ask fans which brand of luxury car they are
(10 - 7:50pm, Sep 02)
Last: Omineca Greg

NewsblogOMNICHATTER 9-2-2014
(8 - 7:36pm, Sep 02)
Last: esseff

NewsblogBrewers prospect plays every position, all in one game
(16 - 7:31pm, Sep 02)
Last: Jose Can Still Seabiscuit

NewsblogRobothal: Tension growing between Astros' manager, GM
(43 - 7:31pm, Sep 02)
Last: Zach

NewsblogNewsweek: Can Baseball Get More Interesting to Watch With Big Data?
(8 - 7:24pm, Sep 02)
Last: AndrewJ

NewsblogExpanded Rosters Exacerbate Baseball’s Biggest Issue
(28 - 7:15pm, Sep 02)
Last: BDC

NewsblogOT: September 2014 College Football thread
(6 - 7:05pm, Sep 02)
Last: Robert in Manhattan Beach

NewsblogBPP: Why do people still think Jack Morris pitched to the score?
(33 - 6:59pm, Sep 02)
Last: Johnny Sycophant-Laden Fora

Page rendered in 0.5472 seconds
52 querie(s) executed