Baseball for the Thinking Fan

Login | Register | Feedback

btf_logo
You are here > Home > Baseball Newsstand > Baseball Primer Newsblog > Discussion
Baseball Primer Newsblog
— The Best News Links from the Baseball Newsstand

Thursday, February 21, 2013

Fangraphs: Larson: Evaluating 2012 Projections

The results of the 2012 forecasting competition.  Composite projections came out the best, but Jared Cross’s Steamer finished first among the independently generated ones.

David Concepcion de la Desviacion Estandar (Dan R) Posted: February 21, 2013 at 04:25 PM | 6 comment(s) Login to Bookmark
  Tags: projections

Reader Comments and Retorts

Go to end of page

Statements posted here are those of our readers and do not represent the BaseballThinkFactory. Names are provided by the poster and are not verified. We ask that posters follow our submission policy. Please report any inappropriate comments.

   1. Danny Posted: February 21, 2013 at 07:35 PM (#4373663)
Yeah, this is a bit like comparing 538/Pollster/RCP to Gallup/PPP/Marist. The former aren't possible without the latter. And I cringed a bit where he said "My personal projections (Larson) took 2nd" given that his "personal projections" are just an averaging of other projections.

Dan, how do you use Pitch F/X in your projections? IIRC, Steamer's use of velocity was credited for its strong pitcher projections the past couple years. Do you use more than velocity?
   2. Walt Davis Posted: February 21, 2013 at 08:51 PM (#4373685)
I assume the target audience for this is traditional 5x5 fantasy players since the projection systems are evaluated using those categories. A title that makes that clear would have been a good idea. If this was meant as a serious evaluation of how well these projection systems predict performance, it's not well done because it's focusing on categories that don't matter much.

I look at two main bases of comparison: the first is the Root Mean Squared Error both with and without bias. Bias is important to consider because it is easily removed from a forecast and it can mask an otherwise good forecasting approach. For example, Marcel projections show very little bias, giving them a low RMSE, but are very poor at predicting variation among players, meaning that it’s not a terribly good forecast if you’re trying to rank expectations of future performance.

Eek! The only "bias" which should be removed is if the projection systems are projecting to different offensive environments (which they often are ... would be good if all the systems just adopted a standard, e.g. the previous year's averages). But if the bias is from other source then it's crucial to retain it in any evaluation. And a low bias method is only problematic for projection if it's lower bias was just random -- if Marcel is consistently lower bias, that's a huge advantage. (I suspect it's not and it is about the different contexts being projected to. Also the author seems to be suggesting that the variance in Marcel is so large that it outweighs the bias advantage -- i.e. it has a large RMSE anyway -- which would be a legit reason it would perform poorly.)

Also please stop averaging ranks as being a useful way to rate things. It assumes that each of the categories is equally important (which in 5x5 fantasy I suppose they are but not in any context that really matters) and that the difference between a rank of 4 vs. 5 (where the actual difference could be tiny) is the same as the difference between 1 and 2 (where the difference might be substantial). Add up the projections to get projected Rbat, Rbase, etc. and go from there.
   3. David Concepcion de la Desviacion Estandar (Dan R) Posted: February 22, 2013 at 02:26 PM (#4374145)
Danny, I made that same point in my comment--my forecasts are probably say 85% intelligent (rather than rote equal-weighted) averaging of other systems, and 15% extra goodies. I do use more than just velocity, yes. Shoot me an email if you're curious about the details.
   4. Der-K: downgraded to lurker Posted: February 22, 2013 at 04:11 PM (#4374242)
Is this contest judging on a mix of rate and counting stats?
   5. DJS, the Digital Dandy Posted: February 22, 2013 at 04:45 PM (#4374276)
Yeah, it's 4x4 fantasy evaluations. ZiPS always gets reamed in the ones with counting stats, as I make no attempt to tailor the playing time projections to my prediction of playing time.
   6. Rally Posted: February 22, 2013 at 04:57 PM (#4374291)
ZiPS always gets reamed in the ones with counting stats, as I make no attempt to tailor the playing time projections to my prediction of playing time.


Mine always did too, until I mixed my rate stats with Tango's wisdom of the crowds playing time estimates. I try to answer the question "Who should play" while not caring so much about "how much will he play".

That's worked out OK for me.

You must be Registered and Logged In to post comments.

 

 

<< Back to main

News

All News | Prime News

Old-School Newsstand


BBTF Partner

Dynasty League Baseball

Support BBTF

donate

Thanks to
Tuque
for his generous support.

Bookmarks

You must be logged in to view your Bookmarks.

Hot Topics

NewsblogOT - 2017-18 NBA thread (All-Star Weekend to End of Time edition)
(2482 - 9:09pm, Apr 23)
Last: Moses Taylor, aka Hambone Fakenameington

NewsblogOTP 2018 Apr 23: The Dominant-Sport Theory of American Politics
(222 - 9:08pm, Apr 23)
Last: Traderdave

NewsblogDidn't come up here to read. Came up here to OMNICHATTER, for April 23, 2018.
(33 - 9:00pm, Apr 23)
Last: Count Vorror Rairol Mencoon (CoB)

NewsblogOT - Catch-All Pop Culture Extravaganza (April - June 2018)
(167 - 8:56pm, Apr 23)
Last: Tulo's Fishy Mullet (mrams)

Newsblog'Family' and sense of 'brotherhood' has Diamondbacks picking up right where they left off
(16 - 8:47pm, Apr 23)
Last: Dr. Vaux

Hall of Merit2019 Hall of Merit Ballot Discussion
(110 - 8:35pm, Apr 23)
Last: Bleed the Freak

NewsblogPrimer Dugout (and link of the day) 4-23-2018
(17 - 7:57pm, Apr 23)
Last: AndrewJ

NewsblogESPN's top 50 players
(59 - 7:49pm, Apr 23)
Last: Jarrod HypnerotomachiaPoliphili (TeddyF.Ballgame)

Gonfalon CubsRiding the Rails of Mediocrity
(7 - 7:21pm, Apr 23)
Last: Moses Taylor, aka Hambone Fakenameington

NewsblogForget that one call; Sean Manaea deserves our full attention
(12 - 6:04pm, Apr 23)
Last: Leroy Kincaid

NewsblogWhite Sox pitcher Danny Farquhar in critical condition after suffering ruptured aneurysm
(24 - 5:25pm, Apr 23)
Last: Batman

NewsblogCallaway says Harvey might not make his next start after performance in 12-4 loss to Braves
(17 - 4:47pm, Apr 23)
Last: The Anthony Kennedy of BBTF (Scott)

NewsblogAT&T Park, Fenway Park and Wrigley Field top list of Rockies’ favorite ballparks
(9 - 2:01pm, Apr 23)
Last: Russlan thinks deGrom is da bomb

NewsblogTaking Back the Ballparks - Miami Marlins
(58 - 2:00pm, Apr 23)
Last: Misirlou doesn't live in the restaurant

NewsblogNYTimes: Now Batting in Class AA: Biggio, Bichette and Guerrero
(9 - 11:42am, Apr 23)
Last: Crispix Attacksel Rios

Page rendered in 0.1357 seconds
47 querie(s) executed