Baseball for the Thinking Fan

Login | Register | Feedback

btf_logo
You are here > Home > Baseball Newsstand > Baseball Primer Newsblog > Discussion
Baseball Primer Newsblog
— The Best News Links from the Baseball Newsstand

Thursday, December 14, 2006

2006 Projection Results

I haven’t seen this linked yet.  This is the latest in Chone’s look at projection results for the 2006 season.  To sum it all up, here are Chone’s list of “114 players who had 500 or more AB, and I had to eliminate a few (Dan Uggla and Hanley Ramirez among others) because not all systems projected minor leaguers” and pitchers with 100 innings:

Hitters
PECOTA .736 (!)
Shandler .702
BIS/James .685
ZiPS .684
Chone .677
Marcel .664

Pitchers
ZiPS .459
PECOTA .451
BIS/James .445
Marcel .432
Chone .424
Shandler .423
Prior YearFIP .370
Prior Year .290

Speaking of projections, the biggest change to ZiPS is that I’m no longer using generic aging curves but a number of player-type aging curves culled after a lot of mind-numbing work.  I did come out with some interesting stuff - I believe I can show (rather than theorize) that negative changes in Speed Score have a solid impact on predicting a BABIP bounceback after a decline - but that’s stuff for a different time.

Dan Szymborski Posted: December 14, 2006 at 07:27 PM | 52 comment(s) Login to Bookmark
  Tags: projections

Reader Comments and Retorts

Go to end of page

Statements posted here are those of our readers and do not represent the BaseballThinkFactory. Names are provided by the poster and are not verified. We ask that posters follow our submission policy. Please report any inappropriate comments.

   1. RoyalsRetro (AG#1F) Posted: December 14, 2006 at 08:25 PM (#2261495)
IN YOUR FACE RON SHANDLER!!!
   2. Mister High Standards Posted: December 14, 2006 at 08:28 PM (#2261501)
What is wrong with you?
   3. GGC don't think it can get longer than a novella Posted: December 14, 2006 at 09:02 PM (#2261538)
He used to be Pecota's #1 fan, MHS.
   4. The Powers That Be Posted: December 14, 2006 at 09:03 PM (#2261540)
That's 114 players, not 14. A bit more meaningful that way.
   5. Dan Szymborski Posted: December 14, 2006 at 09:07 PM (#2261542)
Oops - yeah, 114 [Fixing that]
   6. GGC don't think it can get longer than a novella Posted: December 14, 2006 at 09:42 PM (#2261571)
r is some form of correllation?
   7. Dewey, Soupuss Not Doomed to Succeed Posted: December 14, 2006 at 09:53 PM (#2261578)
How did DMB stack up?
   8. The Answer to the TWolves (GMoney) Posted: December 14, 2006 at 09:55 PM (#2261579)
Yes.. take for example your predicted value of OPS for each player. So you use this value as your predictor or in algebraic terms (X). For a perfect system the predictor (X) would equal the result (Y) true OPS 100% of the time. R measures the variability of Y as predicted by X. Basically if a players true OPS falls near the predicted OPS the value or r squared would be 1 as the value or r squared approaches 0 the data model is less useful. Bad explanation but yes R^2 measures how well your prediction matches the result with 1 being a perfect model.
   9. Barry`s_Lazy_Boy Posted: December 14, 2006 at 10:01 PM (#2261587)
ZiPS better step it up if they are going to remain my favored no-cost projection system.

I have been buying BPro and Shandler's books for years. I may drop Shandler this year.
   10. RB in NYC (Now Semi-Retired from BBTF) Posted: December 14, 2006 at 10:04 PM (#2261591)
If nothing else, this demonstrates how much easier it is to project hitters than pitchers. I mean, wow
   11. Dewey, Soupuss Not Doomed to Succeed Posted: December 14, 2006 at 10:10 PM (#2261596)
If nothing else, this demonstrates how much easier it is to project hitters than pitchers. I mean, wow

Relievers are particularly hard to project, as their limited playing time means that there's severe sample size issues.

Pitchers are also more likely to have their stats negatively impacted by playing hurt.
   12. Los Angeles Waterloo of Black Hawk Posted: December 14, 2006 at 10:12 PM (#2261600)
Most relievers aren't reflected here, though, as Rally Monkey only looked at pitchers with 100+ innings.
   13. Danny Posted: December 14, 2006 at 10:18 PM (#2261606)
Nate Silver responds:
Make no mistake: the other projection systems are improving. Dan Szymbroski’s ZIPS projections have come an awful long way. I like the work that the BIS guys are doing. Top Tippett’s projections for Diamond Mind, not referenced here, have always been very good.

So we need to make PECOTA a moving target, and it is improving too. The league difficulty adjustments that we’ve built in this year correct a longstanding pet peeve of mine. We’ve revised the formula for the starter/relief pitcher adjustment, which was producing a few funny results for middle relievers last year. And we’re always catching little things in the extensive beta testing that we do each year, things that can help you to gain an extra point in your roto league here and there. There’s really no ninja magic that give the PECOTAs a leg up on their competition — it’s thousands of man hours of hard work.
   14. Dewey, Soupuss Not Doomed to Succeed Posted: December 14, 2006 at 10:18 PM (#2261607)
Most relievers aren't reflected here, though, as Rally Monkey only looked at pitchers with 100+ innings.

Ah. I didn't see that. So it's actually even harder than what's presented here.
   15. MSI Posted: December 14, 2006 at 10:20 PM (#2261609)
Pecota is garbage. It is more accurate in the minutae replacement level type picks by just giving everyone a very average prediction. Zips at least has a much more real feel to it, and is not far behind in accuracy.
   16. Joey B.: posting for the kids of northeast Ohio Posted: December 14, 2006 at 10:27 PM (#2261619)
If nothing else, this demonstrates how much easier it is to project hitters than pitchers. I mean, wow.

Wow is right! It would appear that trying to predict the pitchers is little more than just guesswork.
   17. AROM Posted: December 14, 2006 at 10:32 PM (#2261627)
That is why you don't spend $130 on pitching in a 5x5 rotot league. You get just as many points from your 9 pitchers as your 14 hitters - but you're much more certain your good hitters will actually get you points.
   18. Mister High Standards Posted: December 14, 2006 at 10:33 PM (#2261629)
Pecota is garbage. It is more accurate in the minutae replacement level type picks by just giving everyone a very average prediction. Zips at least has a much more real feel to it, and is not far behind in accuracy.


Where is your data?
   19. RoyalsRetro (AG#1F) Posted: December 14, 2006 at 10:33 PM (#2261630)
Wow is right! It would appear that trying to predict the pitchers is little more than just guesswork.

All the reason more to give them $55 million contracts! I feel ill.
   20. bunyon Posted: December 14, 2006 at 10:33 PM (#2261631)
There should be a control calculated:

One-year - just project what they did last year
Three-year - project their three year average


I wonder how would these projections fair if analyzed as the others. My guess is worse for the hitters and right in the pack for the pitchers.
   21. AROM Posted: December 14, 2006 at 10:36 PM (#2261634)
The league difficulty adjustments that we’ve built in this year correct a longstanding pet peeve of mine.

Those are in CHONE as well, starting this year. I didn't see a need for it when the leagues were evenly matched in interleague, other than pitchers doing better because of the DH rule.

Its not reflected in the study, but the 2007 CHONE projections include an ALE (American League Equivalency).
   22. The Answer to the TWolves (GMoney) Posted: December 14, 2006 at 10:39 PM (#2261636)
I would personally like to see a correlation done between Component ERA or a Dips ERA. When your comparing your pitcher projections based on ERA that's similar to me as projection hitters on say RBI or Runs scored. ERA and Runs/RBI are driven by context in which events occur, while OPS and CERA are comprised of events which are independent of context. CERA and OPS see a HR as a HR no matter if there is someone on base or nobody on base where ERA and RBI/Runs depend on the situation.

I think pitching projections for K's, BB, Hits, and HR allowed are all pretty good it's just impossible to project when these events occur.
   23. Padgett Posted: December 14, 2006 at 10:40 PM (#2261640)
So, for all the work developing systems like PECOTA and ZiPS, the best results so far are no more than 10% better than simple Marcels. That's gotta be more than a bit frustrating.

I'd be interested to see if there were significant differences in how the systems fared as to classes of players, like old vs. young, RHB vs. LHB, slugger vs. speedster, etc. If so, perhaps that's where the real value in the systems lies.
   24. The Answer to the TWolves (GMoney) Posted: December 14, 2006 at 10:43 PM (#2261647)
Odd that both ZiPS and CHONE see Baker, Slowey and Garza as the better bet next year than Boof.
   25. bibigon Posted: December 14, 2006 at 10:45 PM (#2261648)
I'd be interesting in seeing some sort of combination of the various projection systems to see if that can improve the results in any way. As in, does some combination of ZiPS and PECOTA give better results than ZiPS or PECOTA independently. I guess I'm really just looking for a best fit curve. If these systems are systemically hitting or missing on certain things, and they're different things(unlikely, but possible), it could improve the accuracy.
   26. The Answer to the TWolves (GMoney) Posted: December 14, 2006 at 10:51 PM (#2261654)
I've heard some people claim that you shouldn't test a pitcher projection by looking at ERA anyway, you should look at component ERA or something. I don't buy that. I need ERA (or at least run average). Whether for fantasy baseball, where you need ERA to get results, or real baseball, where you need to keep real runs off the scoreboard, I don't give a damn if you can accurately predict component ERA. If you can't predict the real thing, then your system is useless.


I don't get this statement, as far as I can tell you don't score games by OPS either. I forget the correlation between Runs scored and OPS but if your judging your pitching system by how well it projects Runs Allowed, wouldn't it make sense to judge your hitting system by Runs Scored? OPS is a good predictor of runs scored but there is some error inherent in OPS being a measure of run scoring ability as well.
   27. Barry`s_Lazy_Boy Posted: December 14, 2006 at 10:53 PM (#2261656)
GMoney, where'd you quote that statement from?
   28. The Answer to the TWolves (GMoney) Posted: December 14, 2006 at 11:01 PM (#2261665)
From the post on the linked webpage.

http://lanaheimangelfan.blogspot.com/2006/12/pitcher-projections.html

His blog about Pitcher Projections.
   29. The Answer to the TWolves (GMoney) Posted: December 14, 2006 at 11:08 PM (#2261676)
A possible thought on SB projections would be to use a "manager's factor". If your looking to get a more accurate SB projection I think a manager factor would greatly increase that accuracy. I look at the projection of SB for someone like Jay Payton going from a team that rarely attempts steals to the Orioles who seem to attempt alot of steals. The current projection has him attempting 5 steals next year which seems awfully low. You could do a steal factor which would be similar to a park factor except for each manager.

Example would be Eric Byrnes who went from 9 attempts in 2005 with (BAL/OAK/COL) to 28 attempts last season with Arizona. If the market is fantasy players a more accurate projection of this could help these people.
   30. AROM Posted: December 14, 2006 at 11:08 PM (#2261680)
I don't get this statement, as far as I can tell you don't score games by OPS either.

That would be fine if I was projecting teams, but for players runs scored (or RBI) does not really measure how many runs they add to the scoreboard.

For pitchers, runs allowed does quite nicely.
   31. Barry`s_Lazy_Boy Posted: December 14, 2006 at 11:09 PM (#2261681)
Well, now I can follow you.

I agree with you that ERA is probably not the best measure. Even R/9 is a better measure.
   32. The Answer to the TWolves (GMoney) Posted: December 14, 2006 at 11:13 PM (#2261685)
Just out of curiousity what would R^2 look like for your 2006 projections for runs/PA or RBI/PA? My guess is the R^2 value would be similar to what you find for pitchers ERA.
   33. AROM Posted: December 14, 2006 at 11:13 PM (#2261686)
Example would be Eric Byrnes who went from 9 attempts in 2005 with (BAL/OAK/COL) to 28 attempts last season with Arizona. If the market is fantasy players a more accurate projection of this could help these people.

Good idea. I don't think Payton will run much as he was never that good a stealer and will be 34 anyway. Then again, I thought Tejada would run more with the O's, but he hasn't. I though Jason Kendall's steal would drop to nothing in Oakland, but the last 4 years he's gone 8-11-8-11.
   34. AROM Posted: December 14, 2006 at 11:23 PM (#2261699)
I agree with you that ERA is probably not the best measure. Even R/9 is a better measure.

Agreed, run average is better than ERA. ERA is the one readily available to compare multiple projection systems. I got Shandler, James/Dewan, and Pecota into this by good old fashioned data entry.
   35. The Answer to the TWolves (GMoney) Posted: December 14, 2006 at 11:34 PM (#2261715)
Another intersting comparison would be R^2 of hitters OPS versus R^2 of pitchers OPS against.
   36. The Answer to the TWolves (GMoney) Posted: December 14, 2006 at 11:45 PM (#2261727)
You have Closser with the Rox still, pretty sure he's with the Brewers now.
   37. AROM Posted: December 15, 2006 at 12:47 AM (#2261805)
There's probably 100 guys like that I missed tansactions for. He'll probably be in AAA anyway, with Estrada and Miller ahead of him.
   38. Danny Posted: December 15, 2006 at 12:54 AM (#2261812)
AROM,

I'm not smart enough to figure it out myself, but does your "myth busting" post contradict the Davenport article? Supplement it?
   39. AROM Posted: December 15, 2006 at 01:32 AM (#2261837)
I'd have to see the Davenport article.
   40. philly Posted: December 15, 2006 at 01:38 AM (#2261842)
Here:

http://www.baseballprospectus.com/article.php?articleid=3946

Overall, the results are clear. The pitchers who made the major leagues are, not surprisingly, better than their counterparts who did not, by every measure of pitching you may desire--including giving up fewer hits per ball in play. Looking at the data for all 72 leagues, there were six leagues where the non-major pitchers had a better BABIP than the major league pitchers, just as there were six leagues where the non-majors allowed fewer home runs. Strikeouts broke "wrong" once; walks never did. The margins were not as large--the major league pitchers were typically 10-15% better in home runs, walks, and strikeouts, but only about 3% better in BABIP--but they were present and consistent. Just as Tom Tippett concluded, based on looking at pitchers by the length of their major league careers, one has to say that BABIP looks like just as much of a skill as home run, walk, or strikeout rates.

   41. Danny Posted: December 15, 2006 at 01:43 AM (#2261850)
Oh, I thought you were responding to it in that post. Here's Davenport's article, though it's probably subscriber only. I think his point was that players with poor BABIP in the minors generally won't make it to the majors--even once controlling for their other stats. In other words, there's a selection bias.

His methodology:
To do that, I looked at every minor league from 1996 to 2000, and I simply divided the pitchers into two groups: those who have played in the major leagues sometime through 2004, and those who haven’t. One would expect that the major league group would be better than the non-major group. Not 100% of the time; there’s always a Ryan Anderson type who, despite being one of the best pitchers in his league, blows out his arm before pitching in the majors, and there are pitchers like Jorge Julio who were lousy minor league starters and only made it to the majors after switching to the bullpen.

His conclusion:
Overall, the results are clear. The pitchers who made the major leagues are, not surprisingly, better than their counterparts who did not, by every measure of pitching you may desire--including giving up fewer hits per ball in play. Looking at the data for all 72 leagues, there were six leagues where the non-major pitchers had a better BABIP than the major league pitchers, just as there were six leagues where the non-majors allowed fewer home runs. Strikeouts broke "wrong" once; walks never did. The margins were not as large--the major league pitchers were typically 10-15% better in home runs, walks, and strikeouts, but only about 3% better in BABIP--but they were present and consistent. Just as Tom Tippett concluded, based on looking at pitchers by the length of their major league careers, one has to say that BABIP looks like just as much of a skill as home run, walk, or strikeout rates.
   42. AROM Posted: December 15, 2006 at 02:13 AM (#2261874)
Thanks guys. The answer is no, I'm not contradicting anything Clay did, since he's looking at pitchers who made the majors and those who didn't. I'm looking at only pitchers who made the majors.

I think his point was that players with poor BABIP in the minors generally won't make it to the majors--even once controlling for their other stats. In other words, there's a selection bias.

This may be true, the majority may not make it but a good number of them did beat the odds and make it to the majors, and their numbers translate at relatively the same ratios as the ones with normal BABIP.
   43. AROM Posted: December 15, 2006 at 02:15 AM (#2261877)
Just created a stat, runs + rbi per plate appearance. For CHONE in this sample, the r = .642, just a little lower than the r=.677 for OPS, but still way ahead of the pitching projections.
   44. Gaelan Posted: December 15, 2006 at 03:00 AM (#2261917)
I've heard some people claim that you shouldn't test a pitcher projection by looking at ERA anyway, you should look at component ERA or something. I don't buy that. I need ERA (or at least run average). Whether for fantasy baseball, where you need ERA to get results, or real baseball, where you need to keep real runs off the scoreboard, I don't give a damn if you can accurately predict component ERA. If you can't predict the real thing, then your system is useless.


The problem is that ERA is comprised of two things, the players performance and the timing of the performance. Projections make no attempt to predict the timing of that performance so if you are comparing projections to ERA you are introducing noise into the comparison.
   45. DCW3 Posted: December 15, 2006 at 08:03 AM (#2262107)
I've been doing a lot of work with Marcels using park adjustments, the lack of which might be the single biggest flaw with regular Marcels. I'm wondering whether this simple change would be enough to throw the system into the middle of the pack with the other systems.
   46. Dan The Mediocre Posted: December 15, 2006 at 08:16 AM (#2262114)
I've been doing a lot of work with Marcels using park adjustments, the lack of which might be the single biggest flaw with regular Marcels. I'm wondering whether this simple change would be enough to throw the system into the middle of the pack with the other systems.


If you feel like doing a lot of work to find out how much more accurate it would be, redo the 2006 projections with the park adjustments added in, and see how much better your correlation becomes.
   47. greenback calls it soccer Posted: December 15, 2006 at 09:12 AM (#2262127)
Besides PECOTA, is anybody else using hitter height and weight in their projections?
   48. Dan Szymborski Posted: December 15, 2006 at 03:36 PM (#2262210)
Besides PECOTA, is anybody else using hitter height and weight in their projections?

Not me - I personally feel there are too many problems with that. Height and weight are reported with various accuracy and weight fluctuates quite a bit over careers. Not to mention that height/weight don't necessarily mean equivalent physical attributes - anyone think John Kruk was LaDainian Tomlinson but a tad thinner?
   49. bibigon Posted: December 15, 2006 at 04:53 PM (#2262258)
Shouldn't the question be if height and weight add accuracy to the projections? I understand the unreliability is somewhat of an issue, but the test should really ultimately be if it makes the projection system more accurate, no? I'd be interested in seeing if Nate Silver could spit out some PECOTA projections with the height/weight variable excluded, so we could see if it adds anything. Given Gassko's research, I'd be surprised if it didn't. That and the gap between PECOTA and everyone else in the hitter projections makes me think there's something there.

But to reiterate, the test shouldn't be if it you can think of logical reasons why it might be unreliable, it should be whether it has practical significance. There seems to be a strong case to me that it does, in spite of the imperfections inherent. All statistics are imperfect - not everyone faces the same opposing pitchers, or the same weather conditions, but we generally assume these things balance out, when if fact, we know that not to always be the case. The same seems to be true of height/weight. There will be times when it misleads, but doesn't mean it's worthless either.
   50. pkb33 Posted: December 15, 2006 at 06:03 PM (#2262297)
I think the big question is whether if you took a two or three year sample of all these systems the results would look similar. That's really what we need to know; a single year probably doesn't truly tell us what we need to know about the relative merits of the systems.
   51. JPWF13 Posted: December 15, 2006 at 07:01 PM (#2262364)
I noticed how last year's FIPs is much better (much less worse) than using last year's ERA to predict furture performance.

I also noticed that many of the head scratching FA signings this year (ie: Marquis, Meche)
involve pitchers whose BBREF comps contain some suprisingly good pitchers- Meche for instance has Schmidt, Clement, Jennings, and Carpenter.

How did that happen?
1: Both pitchers, despite have ERA+ below 100 have winning records for their careers
2: Both pitchers, despite worse peripherals and FIPs have ERA reasonably close to their "comps"

I don't know if front offices look at similarity scores or not- but I wouldn't be suprised if KC was aware that Meche career to date LOOKS similar to the careers of Schmidt et al through the same date- the problem is I suspect that Meche's stat line looks similar as much through luck (good run support- ERA better than FIP/DIPs) as through similar talent.

I look at Marquis and see a pitcher I wouldn't want anywhere near my staff- I'm sure the Cubs see a pitcher with a career record of 56-52 who went 15-7 with a good ERA 3 years ago and has "won" 13 and 14 game since then.

I see a guy who with his runs allowed- should have gone 51-57 with league average run support-
I see a guy who "should" have allowed more runs than he actually did (and since there's only one Tom Glavine)- really has pitched like a 47-61 "talent".

Pitchers are unpredictable- Marquis could pitch much better the next 3-4 years than the last- but he's just a really bad bet simply because he has not pitched as well in the past as people like Hendry think he has.
   52. Fargo Posted: December 15, 2006 at 07:34 PM (#2262407)
To me the most interesting finding is the more accurate systems do so much better than Marcel in projecting Hitting performance: R-sq. (explained variance)=54.2% for PECOTA vs. 44.1% for Marcel. Remember that Marcel isn't just a dumb monkey. He uses a weighted average of performance from the previous three years, and he knows how to adjust for regression to the mean.

Giving him the intelligence to make a park factor adjustment (especially relevant for players changing home teams or stadiums) would probably finally require putting Marcel into the hominid line and his evolution would be worthy of study in its own right. As the systematic projection systems get better, moving the "baseline" by improving Marcel -- however attractive it might be -- would the purpose of having even the semi-sophisticated Marcel that people are using now.
   53. Fargo Posted: December 15, 2006 at 07:37 PM (#2262413)
The last line in #52 should have read: "As the systematic projection systems get better, moving the "baseline" by improving Marcel -- however attractive it might be -- would defeat the purpose of having even the semi-sophisticated Marcel that people are using now." (Sorry about that.)

You must be Registered and Logged In to post comments.

 

 

<< Back to main

BBTF Partner

Support BBTF

donate

Thanks to
phredbird
for his generous support.

Bookmarks

You must be logged in to view your Bookmarks.

Hot Topics

NewsblogOTP - July 2014: Republicans Lose To Democrats For Sixth Straight Year In Congressional Baseball Game
(2766 - 1:47pm, Jul 23)
Last: The Good Face

NewsblogFSAZ: D-backs cut off McCarthy’s cutter controversy
(27 - 1:47pm, Jul 23)
Last: What did Billy Ripken have against ElRoy Face?

NewsblogOT: Monthly NBA Thread- July 2014
(831 - 1:42pm, Jul 23)
Last: Jimmy P

NewsblogOT: The Soccer Thread July, 2014
(328 - 1:38pm, Jul 23)
Last: DA Baracus

NewsblogChase Headley traded to New York Yankees from San Diego Padres - ESPN New York
(114 - 1:38pm, Jul 23)
Last: SoSHially Unacceptable

NewsblogTony Oliva turns 76; Gardenhire: 'He should be in hall of fame'
(52 - 1:38pm, Jul 23)
Last: DanG

NewsblogSportsNet: Kawasaki: ‘A monkey never cramps’
(6 - 1:30pm, Jul 23)
Last: A big pile of nonsense (gef the talking mongoose)

NewsblogCowboy Monkey Rodeo taking the Minors by storm
(14 - 1:28pm, Jul 23)
Last: A big pile of nonsense (gef the talking mongoose)

NewsblogAs shifts suppress offense, time has come to consider a rule change
(62 - 1:27pm, Jul 23)
Last: Daniel in Toronto

NewsblogPadres’ Offense May Go Down as Worst Ever
(82 - 1:19pm, Jul 23)
Last: Ron J2

NewsblogRubin: deGrom for NL rookie of the year?
(21 - 1:10pm, Jul 23)
Last: SoSHially Unacceptable

NewsblogPrimer Dugout (and link of the day) 7-23-2014
(7 - 1:07pm, Jul 23)
Last: Batman

NewsblogGeorge "The Animal" Steele Mangles A Baseball
(126 - 12:35pm, Jul 23)
Last: Random Transaction Generator

SABR - BBTF ChapterWho's going to SABR??
(54 - 12:31pm, Jul 23)
Last: Mark Armour

NewsblogTrading for Price would be right move for Cubs | FOX Sports
(77 - 11:35am, Jul 23)
Last: SouthSideRyan

Page rendered in 0.2828 seconds
52 querie(s) executed