So as soon as Nate Silver became unavailable to fine tune PECOTA results, PECOTA started making crazy predictions.
I guess we'll know after next year if that is true or this year was just a fluke.
As far as I know, PECOTA is a much more complicated system than any of the others,
and that gave it a slim edge while the author was running it.
However, if that makes it unworkable without fiddling with the various adjustments,
it seems that making a projection system more complex is probably the wrong thing to do.

The other is that, sadly, ESPN didn’t have all of their analysts make projections this year, so we have no way of knowing what Steve Phillips was thinking six months ago.

Even if we'd had his projections, I'm pretty sure we'd have a lot of people asking, "WTF is he thinking?"

I posted this in the Lounge, but I'll repeat it here:

Doesn't it seem really unfair to rate projections with 5 or 6 games to go in the regular season? These systems are all off by very small amounts from each other, but fairly large amounts compared to reality.

It seems that the variability of the remaining schedule (The team playing the Nationals should do much better than their projection, while the team playing the Rockies should do worse), would result in lots of 1-or-more game swings away from "pace" over the next 5 days. And since the remaining schedule isn't representative of the overall schedule, that's a pretty unfair way to evaluate predictions for the overall schedule.

I don't think it's necessarily unfair, the strength of a team over 5 measly games is less important than if they get hot or not. And some of the strong teams are not playing their best players - last night the Angels started Sean O'Sullivan instead of Kazmir and finished the game with, literally, the lineup of Salt Lake City. These guys were about a .500 team at the AAA level. Still beat the Rangers though.

While I wouldn't say unfair, it will be a bit inaccurate. I'm pretty sure Vegas Watch will rerun these as soon as the season ends, and the reason you're seeing it now is probably because he's getting a bit antsy.

Well, unfair was probably the wrong word. I don't mean to suggest that this is somehow being rigged to give misleading results or anything, and I have no reason at all to doubt that Vegas Watch is a good guy. Just pointing out a possible flaw.

Well, unfair was probably the wrong word.

"Pointless" would be a better word.

Is there any point in September after which statistics become unmeaningful due to alteration in the player pool?

What I mean is: are the standings after, say, Game 155 a more accurate reflection of team strength than the standings after Game 162?

I'd rather stick with 162. Angels did finish with their AAA lineup last night, but most teams don't do that. Angels won't do that everyday, they'll want to keep the guys they need for the playoffs sharp. You'll get a few rookies mixed in here and there, but most starting players if healthy keep playing. Pitchers like Roy Halladay still take their turn in the rotation. Plus you've got teams still fighting for playoff spots, usually more so than we have this year.

Teams like the Reds or Pirates aren't going to be doing anything different in late September than they were in August, so you might as well stick with the full season.

And since the remaining schedule isn't representative of the overall schedule, that's a pretty unfair way to evaluate predictions for the overall schedule.

From the article:

I'll do a more detailed analysis of this when the regular season actually ends

AROM, agree with what you're saying, but that's a qualitative approach. I'm more interested in the quantitative answer: I wonder if you get, say, better predictions of players' future performance if you throw out the last week of future performance if you throw out Game 155-162 data from your dataset.

I wish I had Tivoed all those analysts saying the Royals are for real this year. That would be funny to watch.

I have no reason at all to doubt that Vegas Watch is a good guy.

Alright!

This post was intended to be the "fun" one, looking at the really good/bad individual predictions; I figured it was okay to do that with a week left in the season, since I can't imagine the perception of any of those will change a ton in six games. I intentionally wait until the season is actually over to take a closer look at comparing all the projection systems, both for just 2009 and their overall track record. I suppose isn't clear at all in the excerpt.

So as soon as Nate Silver became unavailable to fine tune PECOTA results, PECOTA started making crazy predictions.
I guess we'll know after next year if that is true or this year was just a fluke.

This is certainly a good question. I lean towards the former, if only because all the other "computer" projections systems did so much better this year, although at least part of PECOTA's struggles are likely fluky.

Vegas Watch posted a table showing the std dev of the predictions for each team and the average error of the predictions for each team.

From the table:
The most baffling team in the preseason: San Diego Padres.
The least baffling team: Los Angeles Angels
(I'm surprised. Why was there so little consensus about the SDP?)

The biggest positive surprises: Colorado, Texas, NYY, Florida
The biggest disappointments: New York Mets, Cleveland, Washington, Arizona.
The analysts missed bigger on the disappointments than on the underestimated teams.

If you some the average miss column, you get a value of 2. I guess the prognosticators expected an average team to go 79-83.

I will shamelessly take credit for that second-place finish. Thank you, thank you.

zips didnt do so well this year...

So as soon as Nate Silver became unavailable to fine tune PECOTA results, PECOTA started making crazy predictions.

Matt Wieters is to blame.

One year's results don't say much, do they? Marcel is 3rd so far.

puck: Marcel is always middle-of-the-pack, to above average, in these studies. The reason for Marcel's existence is to show there's not much to this forecasting game.

Marcel is 3rd so far.

Can someone point me to a primer on Marcel? I don't see a Wikipedia page for it and I'm unsure what exactly people mean and if it's the same everywhere.

The Wieters forecast was ridiculously flawed, but it (and the attendant issues with the other AA players) probably has nothing to do with it. I took at look at the RMSE between predicted OPS and observed OPS for PECOTA over the years. (First I divided projected OPS by the average prjocted OPS weighted by actual playing time, then I multiplied by average actual OPS of those players, in order to "normalize the data.)



It's pretty close as far as the offensive forecasts are concerned. So I have no idea what's going on here. But it is troubling - according to this, PECOTA did worse than simply projecting every team to a .500 record. VW sent me his spreadsheet and I guess I'll take a look at that next.

Greg.

The biggest positive surprises: Colorado, Texas, NYY, Florida

I wonder how much of that was due to the subconscious wishes of Yankee haters that C.C. and Tex were going to turn out like Randy Johnson and Ken Phelps.

OTOH I doubt if anyone could have predicted that the Yankees would show an improvement in production at every single position in their starting lineup***. Has any other team in history ever done that, let alone a team coming off an 89 win season? IYAM that's an amazing bit of trivia, and I have yet to see it mentioned in the MSM.

***with the very minor exception of A-Rod, whose tiny slip in productivity (150 to 142 OPS+) has to be due to his injury

I wonder how much of that was due to the subconscious wishes of Yankee haters that C.C. and Tex were going to turn out like Randy Johnson and Ken Phelps.

I think it's more likely that people were subconsciously hesitant to make asses of themselves by crowning the Yankees in March the way they did with the Tigers the year before.

OTOH I doubt if anyone could have predicted that the Yankees would show an improvement in production at every single position in their starting lineup***.

It's stunning isn't it?

And if it's ever been done before, I'd sure as hell like to know when it was. What are the odds against that sort of an accomplishment?

If you some the average miss column, you get a value of 2. I guess the prognosticators expected an average team to go 79-83.

The average projection was actually 80.93-81.07; (81.00-80.93)*30 gets you to the 2.

I wonder how much of that was due to the subconscious wishes of Yankee haters that C.C. and Tex were going to turn out like Randy Johnson and Ken Phelps.

I think it's more likely that people were subconsciously hesitant to make asses of themselves by crowning the Yankees in March the way they did with the Tigers the year before.

Either way, it's one reason to take many of those predictions with a big tablespoon of salt. There's simply too much fan in most of us (not all, but most) for it not to enter into our predictions.

St. Nick, this specifically looks at projections either done almost entirely by computer, or by national baseball writers. (Or Vegas oddsmakers.) What does "fan" have to do with it?

Thanks, Tango. All this time I thought that Marcel was a mathmatical term for what you're doing there. Oops.

St. Nick, this specifically looks at projections either done almost entirely by computer, or by national baseball writers. (Or Vegas oddsmakers.) What does "fan" have to do with it?

Writers can't be fans? Nobody can have an ego stake in their own particular methods of projection, especially if they're trying to establish a name for themselves? Human beings can't be congenital crowd followers or congenital contrarians? Any of these factors can affect a prediction.

That said, every one of these swamis, both human and robotic, is far more qualified than most of us to make predictions of any kind. But if I were really an expert along these lines, I'd be predicting one way and betting the other, just like any good Wall Street mogul. Makes for a better payoff. (smile)

Jolly: well, I can guarantee you that Marcel is 100% human-free, since anyone can reproduce the results.

So Tango, now that I have you on the line, what's your answer to that question about the 2009 Yanks that I raised in #21 above? That sort of thing should be right up your alley. Has any team in history ever shown such position by position improvement? Or does it have to involve Bert Blyleven in order to capture our interest?

So Tango, now that I have you on the line, what's your answer to that question about the 2009 Yanks that I raised in #21 above?

1982/1983 Jays have to be close:

By OPS+:

So Tango, now that I have you on the line, what's your answer to that question about the 2009 Yanks that I raised in #21 above? That sort of thing should be right up your alley. Has any team in history ever shown such position by position improvement? Or does it have to involve Bert Blyleven in order to capture our interest?

I would think that your answer lies with teams that see dramatic improvement from year to year. Check out last to first teams, check out teams that lose 100+ games and then only lose 70 or 80. Though I know the answer does not lie in the 1991 braves' offense

I never looked into it. I imagine if you look at the teams with the biggest OPS+ year-to-year changes, you'll find a few that did what you are seeing with the Yanks.

1982/1983 Jays have to be close:

That's funny, as the 1982 team was far better than the '81 team. That's quite a 2-yr jump. 80 pts of OPS one year, 77 the next.

So Tango, now that I have you on the line, what's your answer to that question about the 2009 Yanks that I raised in #21 above? That sort of thing should be right up your alley. Has any team in history ever shown such position by position improvement? Or does it have to involve Bert Blyleven in order to capture our interest?

I would think that your answer lies with teams that see dramatic improvement from year to year. Check out last to first teams, check out teams that lose 100+ games and then only lose 70 or 80. Though I know the answer does not lie in the 1991 braves' offense

Drat, I was hoping you had some sort of instant database, but that's OK. I've checked a few leads, though, and the closest I found was Ryan's 1982/83 Blue Jays, though A-Rod's drop was smaller than Dave Collins. I also checked the 1990/91 Twins, the 1925/26 Yankees (a close call, but 2 small dropoffs), the 1966/67 Red Sox (Reggie Smith and Joe Foy were lower), and the 1945/46 Red Sox (2 pretty big drops at short and in RF).

At that point I gave up. Nobody tops the 2009 Yanks, at least until shown otherwise. If a team of returning war vets couldn't do it, who could?

And to show you just how hard this sort of accomplishment would be, just for grins I tried it in reverse, with the 1914/15 A's, a team that went from 99-53 to 43-109, far and away the worst plunge in Major League history---nothing's even close. Their team OPS+ went from 114 to 87.

But even there, no luck. In the middle of all the 1915 carnage, the A's CF Amos Strunk must have been angling for a Federal League contract, since he improved from 116 all the way up to 143. That did it for me. All hail the 2009 Yanks.

I've checked a few leads, though, and the closest I found was Ryan's 1982/83 Blue Jays, though A-Rod's drop was smaller than Dave Collins.

What are you going for? Team with improvements in OPS+ of the most regular players at 8 of the 9 positions (or 7 of 8 for NL and pre-DH AL teams), with the contestants ranked by weighted overall OPS+ gain of the most regular players at the9 positions? Or are they ranked by improvement in overall team OPS+?

At any rate, you're right, the Yankees' improvement is unbelievable, esp. considering they won 89 games last year.

I've checked a few leads, though, and the closest I found was Ryan's 1982/83 Blue Jays, though A-Rod's drop was smaller than Dave Collins.

What are you going for? Team with improvements in OPS+ of the most regular players at 8 of the 9 positions (or 7 of 8 for NL and pre-DH AL teams), with the contestants ranked by weighted overall OPS+ gain of the most regular players at the9 positions? Or are they ranked by improvement in overall team OPS+?

This is what I said in my original comment (#21 above):

I doubt if anyone could have predicted that the Yankees would show an improvement in production at every single position in their starting lineup***. Has any other team in history ever done that, let alone a team coming off an 89 win season? IYAM that's an amazing bit of trivia, and I have yet to see it mentioned in the MSM.

***with the very minor exception of A-Rod, whose tiny slip in productivity (150 to 142 OPS+) has to be due to his injury

Overall team OPS+ has nothing to do with it. All I was looking for is a team that had improvements at every position. It turned out (so far, at least) that no team ever improved across the board, but this year's Yankees were the closest. (I give them the tiebreaker because A-Rod's dropoff was smaller than the '83 Blue Jays leftfielder.)

At any rate, you're right, the Yankees' improvement is unbelievable, esp. considering they won 89 games last year.

Yeah, that was my second point. All those other teams had a second division finish as a starting base. What the Yankees have done this year is downright sick.

Of course all that and a few bucks will get them on the subway if they don't win it all, but before the postseason reality sets in I thought that it was an achievement worthy of note. And I'm surprised that with all the stat news out there, that nobody seems to have noticed it. It's a tribute to luck, lack of injuries, and a great offseason (Tex and Swisher), but what's even more amazing is that it's been done with but a single career year (Damon), and even that one was just by a hair. It's really little more than having everyone perform more or less near their normal high range of expectation---but of course the key word there is "everyone." We'll see how it all comes out in the end, but I don't think there's any question that this is the Yanks' best balanced team since 1998.

Of course, it's possible that some of the improvement is due to park effects for which we should not be entirely confident.

1968 Senators:

C - Casanova (41)
1B - Epstein (117)
2B - Allen (98)
3B - McMullen (117)
SS - Hansen (84)
LF - Howard (170)
CF - Unser (73)
RF - Stroud (102)

1969 Senators:

C - Casanova (54)
1B - Epstein (176)
2B - Allen (109)
3B - McMullen (122)
SS - Brinkman (89)
LF - Howard (178)
CF - Unser (110)
RF - Maye (132)

Ed Stroud had 18 fewer PA than Lee Maye did, but he, too, managed to top his 1968 OPS+ (his '69 one was 115).

The 2000/2001 Mariners came close. One glaring exception:

Ted Williams had an immediate positive effect. It wore off.

The 2010 Mutts will have a shot at it. Castillo's the only real weak spot.

1968 Senators:

C - Casanova (41)
1B - Epstein (117)
2B - Allen (98)
3B - McMullen (117)
SS - Hansen (84)
LF - Howard (170)
CF - Unser (73)
RF - Stroud (102)

1969 Senators:

C - Casanova (54)
1B - Epstein (176)
2B - Allen (109)
3B - McMullen (122)
SS - Brinkman (89)
LF - Howard (178)
CF - Unser (110)
RF - Maye (132)

Good find, sweatpants. And here I managed to miss the entire season, since I was boycotting Short's 133% ticket price increase.

Does this mean I can call CAIRO "deadly accurate" now?

Nope. Deadly Snake-Eyes Ninja Accurate.

The 2010 Mutts will have a shot at it. Castillo's the only real weak spot.

beat me to it....

Actually the 2009 Mutts almost did the reverse