Phew...looking down that list makes me wonder if good defense isn't like a famous definition of pornography...you can't define it, but you know it when you see it.

If the Yankees aren't really a middle-to-lower-end defensive team and the Orioles aren't a lot closer to 20th than 10th, then someone is watching different clubs.

This just reminds me why I pay little attention to any defensive measurements; the concept obviously isn't as quantifiable yet as to be truly meaningful.

Phew...looking down that list makes me wonder if good defense isn't like a famous definition of pornography...you can't define it, but you know it when you see it.

That's a kick ass quote

This just reminds me why I pay little attention to any defensive measurements; the concept obviously isn't as quantifiable yet as to be truly meaningful.

I don't agree that PMR isn't meaningful. The methodology reveals that the Yankees surpassed what they should have achieved. But both their predicted and actual DERs are middling, with the actual one being a little bit better overall, but well behind the DER leaders - just as received opinion would suggest. The rest of the top 10 teams in the PMR rankings don't strike me as particularly surprising, except maybe the Phillies.

I don't see why someone shouldn't believe this. Shortstop is an important position, but it's not the only one. It's totally possible that Derek Jeter still sucks, but the Yankee team defense is really good.

The methodology is quite solid, the biggest issue is the modelling of the probabilities in that if the probabilities are modelled incorrectly then you can see potential problems in the modelling. If it's the pitching that causes balls to be fielded easily rather than the defense, then the defense will get credit here and that's obviously not what you want.

Also, our perceptions could be warped by looking at this data from a collapsed perspective, particularly since it's not weighted for run value. So the Yankees defense would look worse if they're making up a lot of ground on common plays with low run value and handling badly plays with high run value (which is probably what MGL's UZR would account for).


A quick suggestion for David: it should not be "too difficult" (says the man who's not going to do any of the work) to include an random effect for the pitcher in the modelling of the probabilities. This would correct for differences of the pitchers compared to the team that they play for. One problem that has probably been pointed out for the PMoR is that if you have two really good pitchers and they are affecting difficulty on BIP, then your team's defense will look better if those two pitchers have a disproportionate share of BIP. Adjusting for pitcher would allow you to get a better grasp on what actually belongs to the team. I'm guessing you use logistic regression to model the probability of an out... this would require something like a generalized linear mixed model or a generalized estimating equation.

Although this adjustment flies in the face of DIPS, this is still something you want to do because of the accumulation of lots of little effects.

But both their predicted and actual DERs are middling, with the actual one being a little bit better overall, but well behind the DER leaders - just as received opinion would suggest.

Which would explain the perception of the Yankees defense as "bad", because you're seeing them convert a small proportion of BIP into outs. But it's just that the Yankees defense shouldn't have been expected to convert a lot of those into outs based on the original probability model.

As a side note, these differences seem pretty small except at the extremes. How many games is converting 2% more BIP than expected into outs worth over the season? And how much better is that than converting 1% more BIP than expected into outs?

PMR definitely is not getting confused by crediting fielders for pitchers' hit prevention. For every batted ball, it estimates the likelihood that an average fielder would successfully make the play, and then compares that the the percentage of the time that the team actually did make the play. The gap between the two, summed over every batted ball on the season, is the team defensive rating. That leaves no room for misallocation of credit between pitchers and fielders.

One and two percent on BIP are definitely extremely important. If a team allows 4,500 BIP, then 1% is 45 hits and 2% is 90 hits. At .85 runs per marginal play, a 1% difference is equal to 38 hits/3.8 wins and and a 2% difference is equal to 77 hits/7.7 wins. At $2.5M per marginal win, a 2% difference on BABIP is worth nearly $20 million.

For every batted ball, it estimates the likelihood that an average fielder would successfully make the play, and then compares that the the percentage of the time that the team actually did make the play. The gap between the two, summed over every batted ball on the season, is the team defensive rating. That leaves no room for misallocation of credit between pitchers and fielders.

Of course it does. Let's say that I have a pitcher (let's call him Greg M) who is historically great at preventing hits on all possible BsIP. This affects the percentage of time that HIS team makes a play on the ball. Which makes the gap larger, which makes his team look better. This wouldn't matter if pitchers pitched equal amounts on the team (or, at least, contributed equally to total BsIP), because there would be no (true) variation in the contribution of pitchers to the gap. You can think that the "pitcher" effect would be balanced with respect to the "team defense" effect and so there would be no confounding. Of course, pitcher effects are nested within teams, but that's another issue entirely. The biggest issue for me is the lack of balance and it's "easy" to adjust for that.

One and two percent on BIP are definitely extremely important. If a team allows 4,500 BIP, then 1% is 45 hits and 2% is 90 hits. At .85 runs per marginal play, a 1% difference is equal to 38 hits/3.8 wins and and a 2% difference is equal to 77 hits/7.7 wins. At $2.5M per marginal win, a 2% difference on BABIP is worth nearly $20 million.

Thanks. I was feeling lazy this morning and couldn't find the number of runs per marginal play. 4 wins is a lot (the difference between being 1% better than expected and being 2% better than expected).

Huh?? That is EXACTLY what PBP metrics DO address and non-PBP metrics don't! If a team's pitching staff, on the whole, induces easy-to-field balls, that will be reflected in the team's "expected DER"--the percentage of balls in play that a league-average defense would have fielded, *given the batted balls generated by the pitching staff and the opposing hitters*. It is entirely possible for a team with low team BABIP to have negative PMR--if the pitchers induced a ton of easy-to-field balls (say 95% likelihood of an out), and the fielders converted those routine plays at a below-average rate (say 90% of them became outs), then the team would have a very high actual DER but an even higher expected DER, pegging it as a below-average defense.

I don't see why someone shouldn't believe this. Shortstop is an important position, but it's not the only one. It's totally possible that Derek Jeter still sucks, but the Yankee team defense is really good.

Yes, it's possible, but in the 25 or so Yankee games I watch a year, it's sure not what I see. (Matsui? Please!)

And I watch 130 or so O's games a year in person and on TV...NO WAY they are a middle-of-the-road defensive club. Period. End of story. Patterson was their only strong defender; Roberts and Markakis were plus defenders, Mora average. Everyone else was below average or bad. If that translates to being ranked somewhere between 11th and 13th, then there are either a lot of terrible defending clubs out there, or this is a toy with some serious flaws.

In my opinion, this is a case of those who are determined to measure defense nodding their heads sagely while the stats scream out 'we have no clothes!'

*given the batted balls generated by the pitching staff and the opposing hitters*

Conditional on the aspects of batted balls that are used by the model, i.e. (from the PMR website):



I agree that this is a fairly complete set of attributes. But some of the attributes (particularly 2 and 3) are fairly coarse measures (in fact, it means there are at most 12 types of hits and it's not clear from the description of whether the model allows for interaction between 2 and 3, i.e. does the effect of how "hard" a ball is hit change depending on whether it is a fly, line drive, etc.). There always exists the possibility that there are OTHER things that influence whether a ball turns into a hit and that these OTHER things are due to the pitcher. It's possible that there are no other things (unlikely) or the only things left are reasonably unimportant (more likely, and you would obviously argue in all probability) and this model complete captures what's going on.

I was merely stating that there was a way for allowing for pitchers effects and seeing if it changed what was in the model. If there are no other things due to the pitcher or the things that are left are unimportant, then the model will tell you that (and the model becomes much stronger).


I'm a big fan of the PBP defense methods (and PMR, in particular, is one of my favorites sabermetric tools), but it's critical in statistical analysis to clearly state and assess the limitations of the model.

Repoz, Rich McKinney is OURS...157 games in a White Sox uniform means you have to ask US before you can make fun of him.

I would've said "yes," but seriously, you have to ask first.

In my opinion, this is a case of those who are determined to measure defense nodding their heads sagely while the stats scream out 'we have no clothes!'

In my opinion, you have no evidence that this is true. As was said earlier, you should either criticize the model or provide some other evidence than "I know what my eyes saw". Because, as I said in a previous post, the results of PMR are EXACTLY consistent with what you saw (an observed low rate of defensive efficiency for the Yankees). If the PMR model is correct, then it is attributing that observed low rate of defensive efficiency to things OTHER THAN the defense.

It seems perfectly plausible that the Yankees have a good defense (even assuming Jeter is well below average). Cabrera and Cano are by most accounts top tier defensive players, Mient. and Phillips shared the duties at 1b, and ARod, Posada and Abreu appear to be solid defenders. So even if Jeter and Matsui stink, you're still looking at a decent fielding team overall.

Yes, it's possible, but in the 25 or so Yankee games I watch a year, it's sure not what I see. (Matsui? Please!) In my opinion, this is a case of those who are determined to measure defense nodding their heads sagely while the stats scream out 'we have no clothes!'

Cano and Melky were awesome. Abreu was ok. A-Rod seemed excellent this year though his ZR didn't back it up. Jeter was horrendous. The 1B Side Show was pretty good defensively. Damon in LF was terrific. Matsui had one of his better years, which isn't saying much, but still. Posada was ok. I don't know, it seems plausible to me.

Patterson was their only strong defender; Roberts and Markakis were plus defenders, Mora average. Everyone else was below average or bad.

Assuming the pitching staff is average in total and catchers rarely field a ball in fair territory, you've got seven positions. If three are above average and one is average, you've pretty much hit the definition of an average defensive team.

I'm going to pull some numbers out of thin air to quantify this, and not argue with your above descriptions:

Patterson +10
Markakis +5
Roberts +5
Mora +0
Tejada -5
Payton -5
Millar -5

That's a +5 team right there. Tejada might be worse, I don't think he is. I don't think Payton is a below average defender, and while the first basemen are bad, they aren't responsible for as many plays as the other positions. For the bench guys. Gibbons and Huff were bad playing the field, but Redman and Fahey were good.

i.e. does the effect of how "hard" a ball is hit change depending on whether it is a fly, line drive, etc.

If the instructions that BIS gives its major league scorers are the same as those given to its minor league scorers, it very likely does.

Balls in play are scored relative to all balls of that type. If you score a ball as a "line drive", the velocity is scored as soft/medium/hard based on line drives only - a medium line drive, for example, is almost always hit harder than a hard fly ball, but the scorer is instructed not to take that into consideration.

Fly balls are scored based on depth - a ball caught on the warning track will normally be scored as a hard fly ball, where one caught in shallow left will be scored as soft. In most cases, these rules make sense when you consider what BIS is trying to measure. But there are times where a line drive scored as "medium" will be harder to catch than a fly ball scored as "hard".

-- MWE

I don't think Payton is a below average defender

You'd be wrong. :)

Repoz, Rich McKinney is OURS...157 games in a White Sox uniform means you have to ask US before you can make fun of him.

Yea...but did Mckinney ever have a 4-error game for the White Sox...I believe this overules your permissionary position....:)

I don't think it's all that hard to believe that the Yankee pitchers are primarily responsible for the low DER, and that the defense actually bailed them out a lot.

Clemens had by far the lowest K rate of his career. Pettitte and Mussina had fairly low K rates as well - Pettitte had his lowest rate since 2000, Mussina since 1994. I don't think it's any great leap to conclude that hitters were not only swinging and missing less often, but making more solid contact when they were connecting against those three guys. In addition, the Yankees gave a lot of innings to marginal major league pitchers who normally DO get hit harder than does the typical major league pitchers.

I also don't think it's all that hard to believe that a lot of Baltimore's problems were more due to the pitchers than to the defense. The Orioles, too, gave a lot of innings to some marginal guys - Burres, Rob Bell, Birkins, Leicester, et al - and that makes it difficult for the defense.

If you accept Pinto's approach to determining underlying DER (and while I think that Russ makes valid points against it I don't know that there's enough information to do much better), then his conclusions make a lot of sense. I certainly think Pinto's comments on the Royals' D, for example, explain a lot about the seasons that Bannister and Meche had.

-- MWE

I posted this on David's site:

For the Yanks, here's how certain players did on the John Dewan plus/minus leaders and trailers:

Jeter -34
Melky -22
Abreu -12

Cano +17

If I assume the maximum possible rating for the others: A-Rod didn't make the leader or trailer list, so if the #10 3B is +7, then A-Rod could be at most a +6. So:

A-Rod +6
Damon +7
1B +1

That's -37 for the starters, so plus minus does not see the Yankees as the best defensive team, or anywhere near it. I'm pretty sure UZR did not either.

So my initial impression from watching 40-50 of their games is that the Yankees are not a good defensive team. I see two detailed defensive systems that agree with me, and one that says I was wrong, the Yankee defenders were actually the best in baseball. I hope you can understand if I don't buy that.

My response to #20, also on David's site, with some emphasis added here:

Well, for what it's worth I think PMR does a much better job at taking the actual distribution of balls in play allowed by a team, and modeling defensive expectations against that distribution instead of against a league-average distribution, than either plus/minus or UZR do.

What PMR doesn't do, and UZR does well, is to translate those missed plays into runs.

-- MWE

Thanks for all the comments! And thanks for the link!

Yea...but did Mckinney ever have a 4-error game for the White Sox...I believe this overules your permissionary position....:)

Sample size! 8-)

Dewan has Melky at -22 this year?? Wow, I was under the impression he could pick it.

Dewan has Melky at -22 this year?? Wow, I was under the impression he could pick it.

According to Szym in the Kazmir thread, Dial's method has Melky at +13. Pinto was a little surprised by the -22 as well - although he hasn't run the numbers yet for CF.

Dewan has Melky at -22 this year?? Wow, I was under the impression he could pick it.

Dewan's system, IMO, doesn't really do a good job of modeling the actual distribution of balls in play against the Yankees, and therefore many of the Yankee fielding penalties are actually a reflection of what Pinto's model picks up - the balls that Yankee fielders faced were, on balance, more difficult to field.

-- MWE

Dewan has Melky at -22 this year?? Wow, I was under the impression he could pick it.

I would imagine the -22 number is for Melky specifically, and not just "Yankees CF", which would include some gimpy Damon, right? If so, that is surprising.

I thought the systems made similar adjustments - speed and direction of batted ball, handedness of pitcher and batter, though they use different methods to do this. I may be comparing PMR to UZR, maybe plus/minus doesn't account for handedness or something.

By UZR, Melky didn't make the top or bottom 3, so he's somewhere between +12 and -7. I've got him at +7 by Stats ZR.

Gary Matthews Jr. takes a terrible beating in plus/minus. Seems strange because while he did poorly in Stats ZR (-11), he's +2 in RZR. While his zone rating is low, he did make a lot of out-of zone catches. This is a guy who I watched 90% of his games. He may be overrated if you only watch the web gems and think he's a gold glover. He dropped 3 flyballs on the year. But overall, its hard to reconcile what I saw was the worst CF in the game. He's better than any CF the Angels have played since Darin Erstad got hurt in 2003. I realize thats not a lofty standard.

(and while I think that Russ makes valid points against it I don't know that there's enough information to do much better)

I really want to make clear that I think the PMR rocks. My suggestions for improvement (which doesn't require any additional information, just some extra computing power and potential hair-pulling at getting the more sophisticated analysis to run "nicely") was really to give support to the PMR results. I can only see one or two ways to improve on it conditional on the data that David has AND I'm not convinced that you could actually see any improvement (so it may actually not be worth the work).

Basically, the PMR is utilizing some extremely good statistical ideas (controlling for confounding heterogeneity in pitchers, parks, etc.) and I think it's great from that aspect.

My suggestions for improvement (which doesn't require any additional information, just some extra computing power and potential hair-pulling at getting the more sophisticated analysis to run "nicely") was really to give support to the PMR results.

Well, yeah, it does - specifically, as you pointed out above:

some of the attributes (particularly 2 and 3) are fairly coarse measures

and what I was driving at is that you would want more finely-grained measures of batted ball type and velocity (as well as positioning info). Dewan's team is trying to get at some of that with the "fliner" category, however you might be getting to a point where you run into (a) diminishing returns by trying to get finer-grained data and (b) limitations of human scorers, who can't necesssarily MAKE consistent finer-grained distinctions.

-- MWE

and what I was driving at is that you would want more finely-grained measures of batted ball type and velocity (as well as positioning info).

Oh... gotchya. I guess that would be great too, but I was thinking more along the lines of just looking for the pitcher-level imbalancing effects and maybe putting in a fixed effect interaction between hardness of ball and type of ball hit. These effects might not do anything, but in that case it would eliminate two possibilities for the model error.

It's very clear that what the PMR is trying to do is the right thing. I was just trying to assess how well it did it and if small improvements could be made without too much additional effort (or data).

I'm not sure how much work it would be, but it would be neat to see this broken down by pitcher. I'd be interested to see how hittable different hurlers are.

certainly think Pinto's comments on the Royals' D, for example, explain a lot about the seasons that Bannister and Meche had.

Data for individual pitchers is up now. Bannister has a good ratio, Meche a not so good one.

I don't think it's all that hard to believe that the Yankee pitchers are primarily responsible for the low DER, and that the defense actually bailed them out a lot.

I think it's more likely that the Yankees' HITTERS are responsible for PMR telling us that Yankee pitchers gave up hard-to-hit balls. That's because the model relies on visiting fielders, which in turn means balls hit by Yankee hitters. A ball may look similar according to the six parameters, but when hit by better hitters it's more likely to become a hit. At a minimum, I'd like to see the GB/FB/LD distribution for each team before believing this. If Yankee pitchers gave up a lot of LDs, and few FBs, that makes this a lot more believable.

Here's what I wrote over at Pinto's site:

I think Mike Green and Rally have identified a serious problem with the visitor-dominated model: it effectively lowers the defensive bar for teams whose hitters have a high BABIP. I sorted the teams by predicted DER, and the 8 teams with the lowest predicted DER -- i.e. the teams whose pitchers ostensibly allowed harder to field balls -- had an above-average BABIP in every single case. The 8 teams with the highest predicted DER look just the opposite: 6 of the 8 have below-average BABIP, 1 was average (NYM), and just 1 was above average (CO). The low- predicted-DER teams had an average BABIP of .313, while the high-predicted-DER teams were just .295 (MLB average was .303). All figures from B-Ref.

This can't be a coincidence. What is likely happening is that the model, looking at the experience of visiting fielders in Anaheim trying to stop the rockets hit by Angels hitters (.315 BABIP), concludes "boy, it's hard to turn balls into outs in this park." But down the street in S.D. (.291), or up the road in SF (.281), it looks mighty easy to field those same balls.