Year in and year out, there is simply nothing like the drama of counterintuitivity....

When I was messing around with this kind of thing a few years ago, I found that a 1/1/1 weighting was better than a 4/3/2 weighting, but didn't go much further.

What weightings do most systems use?

Haven't paid much attention to PECOTA recently(not giving BPro any money). Hasn't its accuracy gone continuously downhill since Silver left BPro?

Hasn't its accuracy gone continuously downhill since Silver left BPro?

Not sure. I thought its accuracy -- and certainly, its internal consistency and face validity -- fell off a cliff immediately after Nate left. My impression is that it hasn't gotten much worse or better since the big drop, and has now established a lower level.

I have always liked to aggregate projections to get a composite idea (fewer Matt Wieters outliers). The problem is that there are fewer and fewer available so the weighting isn't as strong. On the other hand, fangraphs makes it easier to get the few that are left.

Is it still deadly accurate? I play fantasy baseball to kill, not wound.

Tango does not seem impressed.

"Is it still deadly accurate? I play fantasy baseball to kill, not wound."

Uniquely accurate is the term used. With nothing to back it up.

Not sure. I thought its accuracy -- and certainly, its internal consistency and face validity -- fell off a cliff immediately after Nate left. My impression is that it hasn't gotten much worse or better since the big drop, and has now established a lower level.

I think it's been shown all over the place that there's not a whole hell of a lot of difference in projection systems and that dumb-as-dirt Marcels does about as well as all the others.

Uniquely accurate is the term used. With nothing to back it up.

Wouldn't any prediction that wasn't just a blatant plagiarism of somebody else's predictions be "uniquely accurate"? Now, whether is was "uniquely good", "uniquely bad", or "uniquely mediocre" would require some evidence, I suppose.

I hope BPro isn't still up to its old tricks. From the "UPDATE" (not added by Wyers it seems):

Some quick examples beforehand: the recent poster boy for “New PECOTA” would probably be Francisco Liriano, whose 3.60 PECOTA forecast for 2010 was almost identical to his real-life 3.62 ERA, while Marcels weighted his recent past (2009 was horrific, and many observers wondered if he'd ever return from his injury woes) and forecast a 4.88 ERA.

That's awesome ... until you see that Liriano's 2011 ERA was 5.09.

Somewhat off-topic -- Wyers is probably right that Dunn will bounce back but that was an historic collapse. Dunn has maybe 300 PA to prove last year was a fluke. If he's hitting 175 with no power in mid-June, he'll be released and his career is likely done.

That's awesome ... until you see that Liriano's 2011 ERA was 5.09.

Since they're talking about 2010, I don't get your point.

Since they're talking about 2010, I don't get your point.

What did New PECOTA peg for Liriano in 2011? After that career-salvaging 2010 that defied many observers' expectations (but not the system's), did it also forecast a return to replacement-level pitching for Nelly?

From the comments:

Bryce Harper's number one comp is Wayne Causey

LOL PECOTA

What did New PECOTA peg for Liriano in 2011?

I think we have to wait until next year for that.

I see Walt's point now.

[15] I don't. Anyone care to explain?

Anyone care to explain?

Basically, if you are saying new PECOTA nails Liriano in 2010, it would clearly miss on him in 2011. So is it really any better than it was?

Right. If your system was smart enough to know that Liriano's "true talent" in 2010 was a 3.60 ERA, he then goes out and puts up a 3.60 ERA then, presumably, its projection for 2011 will be a 3.60 ERA. If anything, given Colin is claiming that older data is nearly as good as recent data*, his 2010 performance should give you even more confidence that his true talent is around 3.60.

Or, basically, Marcel was way off in 2010 but probably pegged him much better in 2011; PECOTA pegged him in 2010 but probably was way off in 2011.

Or, Liriano 2010 is the new PECOTA poster boy while Liriano 2011 is the new PECOTA Elephant Man.

Or just the general point that "hey, we got this guy dead on in this one year" is just a pointless thing to brag about for any model. That's not how models work and it's not what they're useful for. Even a simple "he'll do the same as last year" model is gonna be dead on for a good number of players.

The new PECOTA could of course be right about Liriano's talent level and he was just unlucky in 2011. Alternatively, Marcel had him pegged right in 2010 but he got lucky before returning to form in 2011. Or PECOTA had him right in 2010 but he got hurt again in 2011. Or Liriano's real name is Alfred E Newman and he's 67 years old. Or the truth is somewhere in the middle of all those things.

Now if they could show that for a certain class of players, e.g. guys coming off injury, they are consistently better than other projection systems, then they've got something to brag about.

* I find this much more plausible for hitters. For pitchers, I'm much more skeptical that data from 5-6 years ago tells me much about the pitcher today.

I have my own proprietary projection system, and to be sure I am not missing any data, and don't run into small sample size issues, I do not project anyone until their age 50 season. I get the projections right 99.9% of the time, and is quickly done for each season.

I bet you missed on Minoso and Paige.

Do you have a 2013 projection for Moyer yet?

Ahhh, thanks for the explanation.

#9 Best I can tell only Gary Huckabay's Vlad was tremendously different in design from any of the other systems.

Vlad was by far the best I've ever encountered for predicting breakout systems, but this came at a cost. The overall standard error in projections improved quite a bit when they went to a simpler model (Vlad was neural net based) after Gary left.

IOW Vlad wasn't better overall, but if Vlad wasn't optimistic about a player. ... Well it had a few misses that way too of course but its errors were far more likely to be on the optimistic side.