I agree with #200.

Dan: I think you raise a valid concern for dealing with a single year of data, though it concerns me less when thinking about careers, as with Tango's data. And the large majority of pitchers fall within a pretty narrow FB/GB range. Still, it's not perfect.

However, neither is your PBP approach. You say you found "less variation in BABIP" then you'd get flipping a coin, but that isn't true: you found less variation in an ESTIMATE of a what a pitcher's BABIP would have been in front of average fielders. So you've essentially removed some of the binomial variation. PZR may say a medium-hard GB to sector X, LHH-RHP, has a 65% out probability, but that's still a basket of BIP with varying out probabilities around a mean (hopefully!) of 65%. A pitcher might get a lot of 68%s or 62%s in a season.

Think about it this way: suppose a pitcher has a BIP distribution that sums to .298 according to UZR. Is it your position that this particular BIP distribution, given average defenders at every position, would yield exactly a .298 BABIP every time? Of course not. So you can't compare your UZR-based estimates to the binomial variance and say the "actual" variance is less. You aren't measuring actual variance.

Wow, Buck Ewing is an incredible match for Murph.

See John? I told you!

I don't see it. I don't look like I'm looking into a mirror when I see his photo.

GuyM, I understand what you're saying. What would be the right way to test my data then?

Dan:
Good question and I'm not sure about the answer. I remember we had a thread on The Book Blog where MGL figured out the random variance around team pythag record, which seems somewhat similar. Maybe those guys could help sort this out.

100 pitchers, with true BABIP talent distributed normally with a mean of .300 and a standard deviation of X. For each pitcher, BABIP in a particular season is distributed as normally with a mean of "true BABIP" and a standard deviation of .020

First problem: talent isn't distributed normally at the major league level. The distribution is truncated somewhere around .315. A pitcher with a .280 "true talent" will always pitch in the majors; a pitcher with a .310 "true talent" may, or may not; a pitcher with a .320 "true talent" will almost always be out of the majors fairly quickly.

Second problem: BABIP isn't distributed normally around the mean for an individual pitcher, either, again because of the truncation of the distribution at some level over .300. If a pitcher is giving up hits at a rate over .330 in an individual game, he's going to come out of the game early; if he's giving up hits at a rate under .270 in an individual game, he's going to stay in the game for a while. The net effect of this is that a pitcher's best outings have more effect on his BABIP than will his worst outings.

The effect that DanR sees - less variance in BABIP than one would figure from random variation - is EXACTLY the effect that you would predict if you were to assume that major league pitchers were pre-selected to pitch based on their ability to prevent hits. Almost all of the variability in the ability to prevent hits occurs outside the sample of major league pitchers, because pitchers without a minimum level of hit prevention ability don't pitch in the majors for very long. Every other factor within the game that affects the measurement affects it in ways that drive the variability between pitchers down.

The reason why hit prevention doesn't vary much among major league pitchers who pitch regularly isn't that it's unimportant; it's that it's so important that you simply can't pitch regularly without it.

-- MWE

Is there *anyone* who doesn't think that hit prevention in the minors is important? Again, my understanding of the current CW is that "the differences among major league pitchers in their ability to prevent hits on balls in play are extremely small, and the random variance in those outcomes is extremely large." The reason that there is only one Glendon Rusch is precisely what Mike says, no matter how good your DIPS numbers are, if you're too hittable you just won't last in MLB. I wasn't aware this was even still debated.

So essentially, the pre-selection issue is the "correlation between sequential time series point" issue? Same thing?

But the CW is likely wrong in saying the difference is "very small." The ratio of the true talent difference to the annual variance is completely irrelevant to the question of how important a skill it is. It matters only -- and this is a big 'only' -- for our ability to measure the skill. That's the mistake in some DIPS analysis: it confuses "hard to measure" with "not significant." What matters is how many runs allowed the difference creates. And a SD of .20 R/G for one specific skill is significant, though not huge. It's not as large as the difference in K and HR rates, but comparable to the difference in BB rates. Many great pitchers would have been merely good w/o this skill; many good pitchers merely average.

You want an unprejudiced demonstration that a guy can hit? You go to the statistics, not the video, not to what some old timer thinks. Johnny Damon has an uglier swing than 90% of the players in the NCAA, but who cares? Hitting stats provide wonderful detail based on hundreds if not thousands of opportunities (rookie ball, Class A, wherever) for success or failure. And people who do personnel evaluation for a living have a little saying, “Past performance is the best indicator of future success.”

Sure - for players for which you have a LOT of past performance data in the books, especially data against major league competition. But in the minors, where skill levels are unevenly distributed, the shape of a player's performance takes on increasing importance - you need to know something about how he achieved his numbers.

As for Johnny Damon's "ugly" swing - I know of no one who suggested that Johnny Damon wasn't going to hit major league pitching.

Remember in Gladwell’s “Blink,” where he describes the unusual and uncanny abilities of Vic Braden to instantaneously recognize when a 120 MPH tennis serve will be a fault? It is an automatic and almost unconscious reaction, based on the fact that Braden has watched several hours of tennis almost every day for 50 years. When a team sends a Birdie Tebbetts out to scout, they are relying on that kind of assimilated knowledge (unquantifiable, unconscious) from 10,000 hours spent watching the game.

Why, then, would you NOT believe that scouts apply the same sort of assimilated knowledge to evaluate hitters, apart from their numbers - and do a good enough job of it so that teams can rely on that knowledge?

-- MWE

Well, no, I think that argument is that a low year-to-year correlation doesn't show that an ability doesn't exist, just that there is a lot of noise muffling what may or may not be a signal. I imagine the same would be true of something like sac flies--some years you get a ton, some years only a few, very low year-to-year correlation, but obviously it's a real ability in the long run correlated to your flyball rate (and the # of guys ahead of you who get on 3rd with less than 2 outs).

GuyM, again, I think that .20 R/G stdev is too high, because it includes handedness, GB/FB tendency, and the "match" or "mismatch" between the quality of a team's particular fielders and a given pitcher's BIP distribution as well as the actual ability to prevent hits on balls in play. The real thing to look at, in my opinion, would be the stdev of true talent PZR (expected BABIP given a pitcher's BIP distribution and an average defense), after controlling for handedness and GB/FB tendency. I have the PZR data but not handedness and GB rate in the same spreadsheet to do it myself.

That's the mistake in some DIPS analysis: it confuses "hard to measure" with "not significant."

Exactly. It's hard to measure for a number of structural reasons that revolve primarily around the fact that the better you are at it, the more you pitch.

-- MWE

Dan: Why do you want to control for handedness and GB/FB tendency? Aren't those potentially part of what creates the skill difference? If I control for velocity, I can make true talent differences in K/PA look pretty small....

I guess what you'd really want is to be able to say A% intrinsic pitcher ability, B% handedness, C% GB/FB tendency, D% fielders, E% park, F% luck.

"Why did you do A?"

"I didn't do A?"

"You shouldn't have done A?"

"I just told you, I didn't do A?"

"Doing A was wrong, you shouldn't have done A?"

Billy Beane should not have written that book

I guess what you'd really want is to be able to say A% intrinsic pitcher ability, B% handedness, C% GB/FB tendency, D% fielders, E% park, F% luck.

This can all be modelled by an appropriate multiple regression/multi-level approach. But you still run up against the structural problem - the intrinsic pitcher ability determines, to a large extent, how much a pitcher actually gets to pitch, so unless your model is broad enough to encompass pitchers who don't get to pitch much, you're going to be filtering most of the variability in ability out of the sample.

-- MWE

Doesn't that argument prove way too much, though? Surely hitters who are good at hitting homeruns will tend to stick around and get more playing time than those who don't. Would you therefore contend that we can't measure the ability to hit for power? And if the answer to that is "well, that's a huge effect and the BABIP-hindering ability is much more easily masked", doesn't that amount to an argument that the BABIP-hindering ability isn't that important after all? (Again, among established MLB pitchers -- I get that.)

This thread has at least convinced me of one thing, though: Vörös is being arrogant again. (Well, I dunno about "again" since I dunno what he was like before, but at least once, anyway.)

Pitchers who strike out more batters get more opportunities to pitch. Even with that selective sampling we have no trouble estimating their ability to strike people out.

If we can't do the same for hit prevention, its because:

1) the spread in talent is not as big as for striking people out

and/or

2) Other factors are important to the end result - like fielders.

But you still run up against the structural problem - the intrinsic pitcher ability determines, to a large extent, how much a pitcher actually gets to pitch, so unless your model is broad enough to encompass pitchers who don't get to pitch much, you're going to be filtering most of the variability in ability out of the sample.

If you linked the amount that a pitcher *should* get to pitch to how good he is, you could try to address this problem. This sounds a lot like your basic approximately Missing At Random situation (you know that bad-performing -- not necessarily bad -- pitchers will have more "missing" innings than good-performing -- not necessarily good-- pitchers).

If I thought there was a chance of an interesting answer, I might actually try to model this using some freely available Bayseian modelling software (which handles this type of censoring reasonably well).

I like the DIPS research.

The growth of knowledge in social sciences often proceeds in the following way. A researcher does a study and finds a surprising conclusion and uses these to make a controversial claim. Other researchers do other studies that: (a) use other more appropriate methodologies to determine to what extent the surprising conclusion is true, (b) postulate and test for causes that explain the underlying phenomenon, and (c) see to what extent this phenomenon generalizes beyond the specific population being studied.

This is what happened when Voros posted the surprising findings of DIPS 1-- that there was a near zero YTY correlation on pitchers' BABIP-- and the controversial (and erroneous) claim that pitchers have almost no control of BABIP. This study prompted other research, by Voros and others, that have increased our understanding of pitching skill. As such, Voros' first study is, and should be regarded, as a substantial contribution to sabrmetric research. Below, I list the conclusions that I drew from Voros' studies and others' subsequent work. Let me know if you disagree with anything I wrote.

WHAT I THINK WE KNOW ABOUT DIPS AND BABIP

(1) Successful MLB pitchers' ability to prevent hits is much smaller than we commonly suppose. The SD of BABIP is only .009, which translates to 0.20 runs per game.

(2) MLB pitchers DO have the ability to limit BABIP. Although this ability is less important than strikeout rate or home run rate, it is not insignificant either, and it can account for a large portion of why some pitchers are successful.

(3) Pitchers' BABIP fluctuates wildly year to year, making BABIP skill very difficult to detect, especially by looking at raw statistics.

(4) As a consequence of (3), when pitchers have a BABIP that is far from the leagues' average, especially when it is less than the leagues' average, this is most likely due to luck.

(5) The pitchers who will and will not have a low BABIP is not entirely random (i.e., not distributed evenly throughout the MLB pitcher population). Knuckleballers, among others, systematically have lower BABIP's than normal.

(6) DIPS findings do not (necessarily) generalize to minor league or unsuccessful MLB pitchers. Using a DIPS MLEs to make major league projections is highly skeptical.

(7) Although BABIP is a pitcher skill, it also correlates well with K rate and HR rate, meaning DIPS ERA is still fairly accurate at predicting established MLB players' future performance, even though it doesn't count for BABIP directly.

(8) One practical consequence is that if a young pitcher is successful due to a low BABIP, this should signal a red flag to fans and GMs that he is especially likely to perform significantly worse the following year. It is not guaranteed, but examining why the pitcher's BABIP was so low (either through scouting or statistical analysis) would be a prudent thing to do in evaluating the future of the player.

(9) If a player does especially poorly in BABIP in one season, he would be a good bet to examine to see if an improvement would take place. Again, scouting should be done to see if he lost the ability to prevent hitters from teeing off on him, or if he was the victim of bad luck. It is probably not wise to conclude that the pitcher was just a victim of bad luck without a proper analysis.

(10) Because of the relatively low ability of MLB pitchers to prevent BABIP and the large amount of year-to-year variability, it is difficult to determine a pitchers' true BABIP, even with many years of data.

Surely hitters who are good at hitting homeruns will tend to stick around and get more playing time than those who don't.

The difference is this: the ability to hit home runs is not by itself sufficient to keep a player in the majors (otherwise, Rob Stratton would have been in the majors a long time ago), nor is the lack of ability to hit home runs sufficient by itself to keep a player out of the majors (otherwise, Luis Castillo would not be in the majors). For that reason, there is enough variability in the skill among major league players to allow it to be measured.

The ability to prevent hits on balls in play, on the other hand, IS used to put pitchers into, or to keep pitchers out of, the majors. Within a game, it is used to determine when to leave a pitcher in, or to take him out. Those are structural factors, intrinsic in the way that pitchers are selected and used, which reduce variability in the stats used to measure the skill. If hit prevention on BIP weren't essential to major league success, you'd see more variability in the skill - and you'd be able to assess it more easily, like you can HR power.

Actual BABIP can be misleading as a measure of a pitcher's true talent in hit prevention, not just because of noise but because the nature of the game makes it hard for a pitcher to stay on the mound long enough to post a high BABIP. I think that the actual spread of real ability, even among pitchers who stay in the majors for an extended period, is greater than the numbers indicate, because the structure of pitcher usage drives everyone toward the midpoint. That may not matter, in the broader scheme of things, as long as one is aware of it.

-- MWE

Pitchers who strike out more batters get more opportunities to pitch. Even with that selective sampling we have no trouble estimating their ability to strike people out.

Again, though, the ability to strike out batters is not by itself essential to staying in the majors. You can strike out 4 batters per game, or 10, and still succeed.

The less necessary an ability is, the more variability there is in the ability among players who succeed, and the easier it is to identify differences in the ability.

-- MWE

Again, though, the ability to strike out batters is not by itself essential to staying in the majors. You can strike out 4 batters per game, or 10, and still succeed.

I don't get this at all. Pitchers are not weeded because their BABIP is too high, they are weeded out because they give up too many runs (or they get hurt -regardless of what kind of stats they have). BABIP may be part of why they give up too many runs but its not the only reason, and certainly there are pitchers who fail despite good hit prevention because they stink at everything else.

You can strike out 4 men per game and be successful if you are far above average at other things. If there was a pitcher who was far above average at everything else but got hit at a .350 rate he could pitch in the majors. If he was slightly above average at preventing walks and homers AND struck out 10 batters per 9 innings, he could be lit up at .375 on balls in play and he'd still have a place in the majors.

Yeah, I think I agree. Mike E. kept emphasizing "by itself." Fine. I understand and accept that pitchers who are consistently allowing high BABIP are not making the majors to begin with. But how are you pushing it any further than that? Among major league pitchers, who is a good pitcher because of BABIP-hindering ability, by itself? Guys who are not particularly good at striking people out, or preventing walks, or preventing HR, and yet are good pitchers? Is it like a literal handful of players... Garland, Glavine, Wakefield, Pettitte... something like that? If so, that sure seems more like exceptions to a larger rule than a sweeping indictment of the general idea that we don't have to weigh the BABIP-hindering ability very much when evaluating veteran pitchers. I guess I don't really understand exactly how significant you think the BABIP-hindering ability is once we've limited the pool to those who have gotten over the hump.

The two guys who have made their careers on suppressing hits on balls in play are clearly Wakefield and Zito. Glavine's voodoo wizardry comes much more from his ability to strand runners by changing his approach than by stopping them from getting hits in the first place, plus he did a very good job of preventing HR's in the 90s. Pettitte has actually been below average at preventing hits on BIP.

Garland, Glavine, Wakefield, Pettitte...

Garland also doesn't walk anybody. Pettite doesn't walk many and before last year struck out a decent number.

Pedro Matinez last year allowed a .384 BABIP. (Disclaimer: Small sample size may not indicate true ability, yadda, yadda.) Fits in well with the theory of injured pitchers having higher hit rates, but with Pedro's control, strikeouts, and not allowing a homer I think the Mets might just give this kid a shot at the 2008 rotation.

I guess what you'd really want is to be able to say A% intrinsic pitcher ability, B% handedness, C% GB/FB tendency, D% fielders, E% park, F% luck.

I think what you really want is A) skill, B) fielders, C) park, and D) luck. Your B and C are just possible components of skill. If you wanted to know how much skill a pitcher had at preventing HRs, you wouldn't control for GB/FB tendency (though it might be interesting to know how much of the skill stemmed from that).

Although BABIP is a pitcher skill, it also correlates well with K rate and HR rate, meaning DIPS ERA is still fairly accurate at predicting established MLB players' future performance, even though it doesn't count for BABIP directly.

I think post 220 is a pretty good and balanced summary of what we know. The one thing I'm not sure about is point #7. The negative correlation with HR/9 seems to be a consistent finding, reflecting the fact that FB pitchers have lower BABIP, but it's pretty weak. As for Ks, some researchers have found a correlation, others haven't. At best, it's weak.

Much more importantly, though, is that some analysts seem to believe that the true hit-prevention talent is only what remains after you control for Ks and HRs. This is completely wrong. To the extent that the DIPS coefficients for HR and K pick up a little BABIP talent, that just means those coefficients are inaccurate in terms of measuring the true run value of Ks and HRs. Correlation with other skills doesn't diminish the value of hit prevention skill.

MWE,

I find the "BABIP too essential to find variance" theory pretty farfeteched but if true it should be testable.

1) First year pitchers should have larger variance in BABIP than expected b/c some of those without BABIP ability haven't been weeded out yet.

2) The best K/BB/HR pitchers would have greater variance in BABIP than expected since a pitcher who dominates in these categories can get by with a little less BABIP skill. There should exist some pitchers (or at least one pitcher?) who dominanted in these categories for their career and had a BABIP decided worse than average for his career. So much though that it wouldn't have been thought possible from the DIPS model. Are there such pitchers or even such a pitcher?

3) When hitters pitch in mop up duty they should have ridiculously high BABIPs. Is this the case?

J. Cross:
On point #1, look at this article http://www.diamond-mind.com/articles/ipavg2.htm, especially this table categorizing pitchers by batters faced. Those who faced fewer than 1,000 batters had a BABIP 15 points higher than that of their teams.

Career BF BF HBP BB K HR vsLg vsTm
1 - 999 401,138 .002 .027 -.017 .002 .017 .015
1000 - 1999 931,981 .001 .013 -.009 .001 .006 .004
2000 - 2999 1,105,712 .001 .007 -.005 .000 .002 .001
3000 - 3999 1,179,916 .000 .006 -.003 .000 .000 .000
4000 - 4999 906,271 .000 .002 -.002 .000 .000 .001
5000 - 5999 920,680 .000 .001 .000 .000 .000 .000
6000 - 6999 647,553 .000 -.004 -.002 .001 -.001 -.001
7000 - 7999 843,937 .000 -.003 .000 .000 -.002 -.001
8000 - 8999 716,200 -.001 -.005 .005 .000 -.002 -.002
9000 - 9999 788,532 .000 -.008 -.001 -.001 -.002 -.001
10000+ 2,589,409 -.001 -.010 .008 -.001 -.004 -.003

Let's say that a bunch of players on either side of that 1,000 batters faced had approximately the same talent level. Wouldn't you expect luck to be a factor in whether they crossed that (relatively small sample) threshold? I'm in way over my head here, I'm sure, but does that make any sense? How much is that really telling you about BABIP skill?

3) When hitters pitch in mop up duty they should have ridiculously high BABIPs. Is this the case?

I did look at that once. I'll see if I can find my notes. From memory it was something like .340-.350.

But there are extenuating circumstances. If Aaron Miles is on the mound the hitter might be too busy laughing to take an effective swing. Plus the score is lopsided so regulars might be pulled, and the hitters that stay might not be taking the AB 100% seriously. In theory the hitters might allow a .400 or higher if they had to face the middle of a good order in a close game.

Dan (if you're still out there): One way to get at the expected variance in your PBP data would be to examine hitters. Unlike pitchers, they face a pretty random assortment of pitchers and opposing fielders. So take a set of hitters and compare the SD for their actual BABIP and for their UZR/PZR hit rate. I think you'll find that the UZR SD is smaller (though still larger than the binomial variance, given the much larger spread in true talent for hitters). The difference in SDs should give you a sense of how the UZR rate relates to the true talent level.

Dan: Following up, I created a very simple version of projected BABIP for hitters, using the average hit rate for each BIP type (high for LDs, low for FBs). Looking at hitters with 200+ PAs this year, I get a SD for actual BABIP of .035, but only .016 for projected BABIP. Now, the gap won't be as dramatic with UZR/PZR data, because an Albert Pujols FB will be projected as a hit more often than a David Eckstein FB (I assume). But I do think you'll find the variance is much narrower than for actual BABIP. And if so, the same must be true for your PZR-BABIP: it strips out a lot of the binomial variance (I suppose because each "outcome" is not a binary hit/out outcome but rather a hit probability estimate).

Voros hasn't updated in close to two weeks. I hope it's just due to the holidays.

First year pitchers should have larger variance in BABIP than expected b/c some of those without BABIP ability haven't been weeded out yet.

They tend to, as the study done by Tippett and posted by Guy notes.

When hitters pitch in mop up duty they should have ridiculously high BABIPs. Is this the case?

As a group, they tend to. There's a wide range of variance there, though.

With the expansion of pitching staffs, by the way, hitters that pitch in mopup duty are become rarer, and pitchers that play the field (remember the old days where a team would move a pitcher to the outfield in order to keep him in the game?) are virtually extinct.

The best K/BB/HR pitchers would have greater variance in BABIP than expected since a pitcher who dominates in these categories can get by with a little less BABIP skill

The best K/BB/HR pitchers, in general, are also the pitchers who control hBIP the best. K rate, in particular, influences hBIP fairly strongly (difference of 7-10 points in hBIP between the best K pitchers and the worst). That tends to drive the variance down.

-- MWE