IN YOUR FACE RON SHANDLER!!!

What is wrong with you?

He used to be Pecota's #1 fan, MHS.

That's 114 players, not 14. A bit more meaningful that way.

Oops - yeah, 114 [Fixing that]

r is some form of correllation?

How did DMB stack up?

Yes.. take for example your predicted value of OPS for each player. So you use this value as your predictor or in algebraic terms (X). For a perfect system the predictor (X) would equal the result (Y) true OPS 100% of the time. R measures the variability of Y as predicted by X. Basically if a players true OPS falls near the predicted OPS the value or r squared would be 1 as the value or r squared approaches 0 the data model is less useful. Bad explanation but yes R^2 measures how well your prediction matches the result with 1 being a perfect model.

ZiPS better step it up if they are going to remain my favored no-cost projection system.

I have been buying BPro and Shandler's books for years. I may drop Shandler this year.

If nothing else, this demonstrates how much easier it is to project hitters than pitchers. I mean, wow

If nothing else, this demonstrates how much easier it is to project hitters than pitchers. I mean, wow

Relievers are particularly hard to project, as their limited playing time means that there's severe sample size issues.

Pitchers are also more likely to have their stats negatively impacted by playing hurt.

Most relievers aren't reflected here, though, as Rally Monkey only looked at pitchers with 100+ innings.

Nate Silver responds:

Make no mistake: the other projection systems are improving. Dan Szymbroski’s ZIPS projections have come an awful long way. I like the work that the BIS guys are doing. Top Tippett’s projections for Diamond Mind, not referenced here, have always been very good.

So we need to make PECOTA a moving target, and it is improving too. The league difficulty adjustments that we’ve built in this year correct a longstanding pet peeve of mine. We’ve revised the formula for the starter/relief pitcher adjustment, which was producing a few funny results for middle relievers last year. And we’re always catching little things in the extensive beta testing that we do each year, things that can help you to gain an extra point in your roto league here and there. There’s really no ninja magic that give the PECOTAs a leg up on their competition — it’s thousands of man hours of hard work.

Most relievers aren't reflected here, though, as Rally Monkey only looked at pitchers with 100+ innings.

Ah. I didn't see that. So it's actually even harder than what's presented here.

Pecota is garbage. It is more accurate in the minutae replacement level type picks by just giving everyone a very average prediction. Zips at least has a much more real feel to it, and is not far behind in accuracy.

If nothing else, this demonstrates how much easier it is to project hitters than pitchers. I mean, wow.

Wow is right! It would appear that trying to predict the pitchers is little more than just guesswork.

That is why you don't spend $130 on pitching in a 5x5 rotot league. You get just as many points from your 9 pitchers as your 14 hitters - but you're much more certain your good hitters will actually get you points.

Pecota is garbage. It is more accurate in the minutae replacement level type picks by just giving everyone a very average prediction. Zips at least has a much more real feel to it, and is not far behind in accuracy.

Where is your data?

Wow is right! It would appear that trying to predict the pitchers is little more than just guesswork.

All the reason more to give them $55 million contracts! I feel ill.

There should be a control calculated:

One-year - just project what they did last year
Three-year - project their three year average


I wonder how would these projections fair if analyzed as the others. My guess is worse for the hitters and right in the pack for the pitchers.

The league difficulty adjustments that we’ve built in this year correct a longstanding pet peeve of mine.

Those are in CHONE as well, starting this year. I didn't see a need for it when the leagues were evenly matched in interleague, other than pitchers doing better because of the DH rule.

Its not reflected in the study, but the 2007 CHONE projections include an ALE (American League Equivalency).

I would personally like to see a correlation done between Component ERA or a Dips ERA. When your comparing your pitcher projections based on ERA that's similar to me as projection hitters on say RBI or Runs scored. ERA and Runs/RBI are driven by context in which events occur, while OPS and CERA are comprised of events which are independent of context. CERA and OPS see a HR as a HR no matter if there is someone on base or nobody on base where ERA and RBI/Runs depend on the situation.

I think pitching projections for K's, BB, Hits, and HR allowed are all pretty good it's just impossible to project when these events occur.

So, for all the work developing systems like PECOTA and ZiPS, the best results so far are no more than 10% better than simple Marcels. That's gotta be more than a bit frustrating.

I'd be interested to see if there were significant differences in how the systems fared as to classes of players, like old vs. young, RHB vs. LHB, slugger vs. speedster, etc. If so, perhaps that's where the real value in the systems lies.

Odd that both ZiPS and CHONE see Baker, Slowey and Garza as the better bet next year than Boof.

I'd be interesting in seeing some sort of combination of the various projection systems to see if that can improve the results in any way. As in, does some combination of ZiPS and PECOTA give better results than ZiPS or PECOTA independently. I guess I'm really just looking for a best fit curve. If these systems are systemically hitting or missing on certain things, and they're different things(unlikely, but possible), it could improve the accuracy.

I've heard some people claim that you shouldn't test a pitcher projection by looking at ERA anyway, you should look at component ERA or something. I don't buy that. I need ERA (or at least run average). Whether for fantasy baseball, where you need ERA to get results, or real baseball, where you need to keep real runs off the scoreboard, I don't give a damn if you can accurately predict component ERA. If you can't predict the real thing, then your system is useless.

I don't get this statement, as far as I can tell you don't score games by OPS either. I forget the correlation between Runs scored and OPS but if your judging your pitching system by how well it projects Runs Allowed, wouldn't it make sense to judge your hitting system by Runs Scored? OPS is a good predictor of runs scored but there is some error inherent in OPS being a measure of run scoring ability as well.

GMoney, where'd you quote that statement from?

From the post on the linked webpage.

http://lanaheimangelfan.blogspot.com/2006/12/pitcher-projections.html

His blog about Pitcher Projections.

A possible thought on SB projections would be to use a "manager's factor". If your looking to get a more accurate SB projection I think a manager factor would greatly increase that accuracy. I look at the projection of SB for someone like Jay Payton going from a team that rarely attempts steals to the Orioles who seem to attempt alot of steals. The current projection has him attempting 5 steals next year which seems awfully low. You could do a steal factor which would be similar to a park factor except for each manager.

Example would be Eric Byrnes who went from 9 attempts in 2005 with (BAL/OAK/COL) to 28 attempts last season with Arizona. If the market is fantasy players a more accurate projection of this could help these people.

I don't get this statement, as far as I can tell you don't score games by OPS either.

That would be fine if I was projecting teams, but for players runs scored (or RBI) does not really measure how many runs they add to the scoreboard.

For pitchers, runs allowed does quite nicely.

Well, now I can follow you.

I agree with you that ERA is probably not the best measure. Even R/9 is a better measure.

Just out of curiousity what would R^2 look like for your 2006 projections for runs/PA or RBI/PA? My guess is the R^2 value would be similar to what you find for pitchers ERA.

Example would be Eric Byrnes who went from 9 attempts in 2005 with (BAL/OAK/COL) to 28 attempts last season with Arizona. If the market is fantasy players a more accurate projection of this could help these people.

Good idea. I don't think Payton will run much as he was never that good a stealer and will be 34 anyway. Then again, I thought Tejada would run more with the O's, but he hasn't. I though Jason Kendall's steal would drop to nothing in Oakland, but the last 4 years he's gone 8-11-8-11.

I agree with you that ERA is probably not the best measure. Even R/9 is a better measure.

Agreed, run average is better than ERA. ERA is the one readily available to compare multiple projection systems. I got Shandler, James/Dewan, and Pecota into this by good old fashioned data entry.

Another intersting comparison would be R^2 of hitters OPS versus R^2 of pitchers OPS against.

You have Closser with the Rox still, pretty sure he's with the Brewers now.

There's probably 100 guys like that I missed tansactions for. He'll probably be in AAA anyway, with Estrada and Miller ahead of him.

AROM,

I'm not smart enough to figure it out myself, but does your "myth busting" post contradict the Davenport article? Supplement it?

I'd have to see the Davenport article.

Here:

http://www.baseballprospectus.com/article.php?articleid=3946

Overall, the results are clear. The pitchers who made the major leagues are, not surprisingly, better than their counterparts who did not, by every measure of pitching you may desire--including giving up fewer hits per ball in play. Looking at the data for all 72 leagues, there were six leagues where the non-major pitchers had a better BABIP than the major league pitchers, just as there were six leagues where the non-majors allowed fewer home runs. Strikeouts broke "wrong" once; walks never did. The margins were not as large--the major league pitchers were typically 10-15% better in home runs, walks, and strikeouts, but only about 3% better in BABIP--but they were present and consistent. Just as Tom Tippett concluded, based on looking at pitchers by the length of their major league careers, one has to say that BABIP looks like just as much of a skill as home run, walk, or strikeout rates.

Oh, I thought you were responding to it in that post. Here's Davenport's article, though it's probably subscriber only. I think his point was that players with poor BABIP in the minors generally won't make it to the majors--even once controlling for their other stats. In other words, there's a selection bias.

His methodology:

To do that, I looked at every minor league from 1996 to 2000, and I simply divided the pitchers into two groups: those who have played in the major leagues sometime through 2004, and those who haven’t. One would expect that the major league group would be better than the non-major group. Not 100% of the time; there’s always a Ryan Anderson type who, despite being one of the best pitchers in his league, blows out his arm before pitching in the majors, and there are pitchers like Jorge Julio who were lousy minor league starters and only made it to the majors after switching to the bullpen.

His conclusion:

Overall, the results are clear. The pitchers who made the major leagues are, not surprisingly, better than their counterparts who did not, by every measure of pitching you may desire--including giving up fewer hits per ball in play. Looking at the data for all 72 leagues, there were six leagues where the non-major pitchers had a better BABIP than the major league pitchers, just as there were six leagues where the non-majors allowed fewer home runs. Strikeouts broke "wrong" once; walks never did. The margins were not as large--the major league pitchers were typically 10-15% better in home runs, walks, and strikeouts, but only about 3% better in BABIP--but they were present and consistent. Just as Tom Tippett concluded, based on looking at pitchers by the length of their major league careers, one has to say that BABIP looks like just as much of a skill as home run, walk, or strikeout rates.

Thanks guys. The answer is no, I'm not contradicting anything Clay did, since he's looking at pitchers who made the majors and those who didn't. I'm looking at only pitchers who made the majors.

I think his point was that players with poor BABIP in the minors generally won't make it to the majors--even once controlling for their other stats. In other words, there's a selection bias.

This may be true, the majority may not make it but a good number of them did beat the odds and make it to the majors, and their numbers translate at relatively the same ratios as the ones with normal BABIP.

Just created a stat, runs + rbi per plate appearance. For CHONE in this sample, the r = .642, just a little lower than the r=.677 for OPS, but still way ahead of the pitching projections.

I've heard some people claim that you shouldn't test a pitcher projection by looking at ERA anyway, you should look at component ERA or something. I don't buy that. I need ERA (or at least run average). Whether for fantasy baseball, where you need ERA to get results, or real baseball, where you need to keep real runs off the scoreboard, I don't give a damn if you can accurately predict component ERA. If you can't predict the real thing, then your system is useless.

The problem is that ERA is comprised of two things, the players performance and the timing of the performance. Projections make no attempt to predict the timing of that performance so if you are comparing projections to ERA you are introducing noise into the comparison.

I've been doing a lot of work with Marcels using park adjustments, the lack of which might be the single biggest flaw with regular Marcels. I'm wondering whether this simple change would be enough to throw the system into the middle of the pack with the other systems.

I've been doing a lot of work with Marcels using park adjustments, the lack of which might be the single biggest flaw with regular Marcels. I'm wondering whether this simple change would be enough to throw the system into the middle of the pack with the other systems.

If you feel like doing a lot of work to find out how much more accurate it would be, redo the 2006 projections with the park adjustments added in, and see how much better your correlation becomes.

Besides PECOTA, is anybody else using hitter height and weight in their projections?

Besides PECOTA, is anybody else using hitter height and weight in their projections?

Not me - I personally feel there are too many problems with that. Height and weight are reported with various accuracy and weight fluctuates quite a bit over careers. Not to mention that height/weight don't necessarily mean equivalent physical attributes - anyone think John Kruk was LaDainian Tomlinson but a tad thinner?

Shouldn't the question be if height and weight add accuracy to the projections? I understand the unreliability is somewhat of an issue, but the test should really ultimately be if it makes the projection system more accurate, no? I'd be interested in seeing if Nate Silver could spit out some PECOTA projections with the height/weight variable excluded, so we could see if it adds anything. Given Gassko's research, I'd be surprised if it didn't. That and the gap between PECOTA and everyone else in the hitter projections makes me think there's something there.

But to reiterate, the test shouldn't be if it you can think of logical reasons why it might be unreliable, it should be whether it has practical significance. There seems to be a strong case to me that it does, in spite of the imperfections inherent. All statistics are imperfect - not everyone faces the same opposing pitchers, or the same weather conditions, but we generally assume these things balance out, when if fact, we know that not to always be the case. The same seems to be true of height/weight. There will be times when it misleads, but doesn't mean it's worthless either.

I think the big question is whether if you took a two or three year sample of all these systems the results would look similar. That's really what we need to know; a single year probably doesn't truly tell us what we need to know about the relative merits of the systems.

I noticed how last year's FIPs is much better (much less worse) than using last year's ERA to predict furture performance.

I also noticed that many of the head scratching FA signings this year (ie: Marquis, Meche)
involve pitchers whose BBREF comps contain some suprisingly good pitchers- Meche for instance has Schmidt, Clement, Jennings, and Carpenter.

How did that happen?
1: Both pitchers, despite have ERA+ below 100 have winning records for their careers
2: Both pitchers, despite worse peripherals and FIPs have ERA reasonably close to their "comps"

I don't know if front offices look at similarity scores or not- but I wouldn't be suprised if KC was aware that Meche career to date LOOKS similar to the careers of Schmidt et al through the same date- the problem is I suspect that Meche's stat line looks similar as much through luck (good run support- ERA better than FIP/DIPs) as through similar talent.

I look at Marquis and see a pitcher I wouldn't want anywhere near my staff- I'm sure the Cubs see a pitcher with a career record of 56-52 who went 15-7 with a good ERA 3 years ago and has "won" 13 and 14 game since then.

I see a guy who with his runs allowed- should have gone 51-57 with league average run support-
I see a guy who "should" have allowed more runs than he actually did (and since there's only one Tom Glavine)- really has pitched like a 47-61 "talent".

Pitchers are unpredictable- Marquis could pitch much better the next 3-4 years than the last- but he's just a really bad bet simply because he has not pitched as well in the past as people like Hendry think he has.

To me the most interesting finding is the more accurate systems do so much better than Marcel in projecting Hitting performance: R-sq. (explained variance)=54.2% for PECOTA vs. 44.1% for Marcel. Remember that Marcel isn't just a dumb monkey. He uses a weighted average of performance from the previous three years, and he knows how to adjust for regression to the mean.

Giving him the intelligence to make a park factor adjustment (especially relevant for players changing home teams or stadiums) would probably finally require putting Marcel into the hominid line and his evolution would be worthy of study in its own right. As the systematic projection systems get better, moving the "baseline" by improving Marcel -- however attractive it might be -- would the purpose of having even the semi-sophisticated Marcel that people are using now.

The last line in #52 should have read: "As the systematic projection systems get better, moving the "baseline" by improving Marcel -- however attractive it might be -- would defeat the purpose of having even the semi-sophisticated Marcel that people are using now." (Sorry about that.)