Baseball for the Thinking Fan

Login | Register | Feedback

You are here > Home > Baseball Newsstand > Discussion
Baseball Primer Newsblog
— The Best News Links from the Baseball Newsstand

Friday, August 13, 2021

Would “Robot Umpires” Reduce Discrimination? Measuring Racial Bias in Major League Baseball Umpires

Utilizing thirteen years of Major League Baseball pitch-tracking and play-by-play data (all the data that is publicly available), this study investigates racial discrimination by umpires when making pitch calling decisions.

Bottom line?  White umpires show a statistically significant bias towards white hitters and pitchers.  Black umpires a smaller level of bias, Hispanic umpires a small reverse bias.

So yet another reason to switch to robo-umps for ball-strike calls.

John Northey Posted: August 13, 2021 at 09:41 AM | 22 comment(s) Login to Bookmark
  Tags: racism, umpires

Reader Comments and Retorts

Go to end of page

Statements posted here are those of our readers and do not represent the BaseballThinkFactory. Names are provided by the poster and are not verified. We ask that posters follow our submission policy. Please report any inappropriate comments.

   1. The Duke Posted: August 13, 2021 at 02:34 PM (#6034272)
Here is one of the conclusions

“Furthermore, baseball is a game that operates on increasingly thin margins as teams continue to apply a more empirical, Sabermetric approach to roster-construction — the effect that one missed call may have on a player’s career can be substantial.“

Feel free to read the rest but that seems quite hyperbolic

I’m not good at all the metrics presented but it would be great if someone could break this down into a metric that would be understood by the masses. Like “an average black batter will have 5 more strikes called against him than a white player over the course of the season”. Maybe it’s there but I couldn’t decipher it. If that number is 5, then ho-hum. If it’s 50 or 500, that’s interesting.

Finally, it doesn’t seem to account for the race of the catcher which one would think would be a huge variable. There are white, black and Hispanic catchers. It seems odd that a white ump would discriminate against a black pitcher/batter without regard to the race of the catcher. That’s a strange form of discrimination anyhow. You could even extend this to the manager as well.

   2. JJ1986 Posted: August 13, 2021 at 02:52 PM (#6034275)
There are white, black and Hispanic catchers.
Not for awhile.
   3. we all water; we all 57i66135 Posted: August 13, 2021 at 04:24 PM (#6034289)
Not for awhile.
whither charles johnson.
   4. winnipegwhip Posted: August 13, 2021 at 04:46 PM (#6034295)
Which way were the Derek Jeter at-bats measured?
   5. Bhaakon Posted: August 13, 2021 at 05:48 PM (#6034303)
The conclusions are pretty hyperbolic, because that's how scholars are often trained to write. You don't get grants for hedging. The actual effects appear real but minuscule and complicated in unexpected ways.
   6. gef the talking mongoose, peppery hostile Posted: August 13, 2021 at 06:47 PM (#6034309)
whither charles johnson.

Initially read that as "whiter Charles Johnson" ...
   7. kcgard2 Posted: August 13, 2021 at 06:48 PM (#6034310)
If that number is 5, then ho-hum. If it’s 50 or 500, that’s interesting.

Looking at balls that should have been strikes: 224K such pitches were thrown over 13 seasons when the umpire race matches batter race but not pitcher race (i.e., the times when the ump was "racially supporting a batter unfairly").
224K out of 3.4M pitches total means 6.6% of eligible pitches.
3.4M pitches in 12.5 seasons is 272K pitches per season.
Of which a single batter who gets 650 PAs in a season will see approximately 0.35%.
0.35% of 272K is 950 pitches.
6.6% of 910 is 63 pitches.

The typical batter, according to this study should see about 63 strikes called balls in his favor. However, he will also see about 87 balls called strikes in those same matchups, because umpires are much more likely to call a ball a strike than vice versa. So for the mythical "typical" batter playing a full season, he will have a net effect of 24 pitches to his disadvantage. However, please note this isn't the value due specifically to race, it's the value due to all factors combined.

The complicating factor is that most umpires are white, so white batters will benefit more than other batters. However, most pitchers are white also, leaving the real math of how this works out getting quite convoluted. For what it is worth, the typical full-time batter will see something like 2400 pitches in a season, so 1% of his (net) pitches will fall under the heading of unduly influenced, but note still, not by umpire race.

Another way to examine, using 2400 pitches as the full-time player amount. Only called strikes/balls are eligible for this bias, which puts the typical player around 945 eligible pitches. The effect of umpire race matching batter race is 0.0032: for every called pitch, 0.3% of them will result in an unfair ball in favor of the batter due to this race effect. 945 eligible pitches = 3 extra balls that should have been strikes due to race, however, matching the pitcher race as well removes 1.5 of those, so 1.5 net on the batter side. For the pitchers getting an unfair strike, the effect is almost identical: 0.0032 for 3 extra strikes, but matching the batter race takes away 2.5 of them, for a net loss here to batters of 0.5 pitches. Total net for batters, combining the two models = 1 pitch per season in their favor, due to race. Keep in mind this is for a mythical typical player, whose likelihood of facing a pitcher of the same race is equal to his likelihood of facing a pitcher of opposite race, and his likelihood of having an umpire of same race is equal to likelihood of umpire of opposite race. So for white batters, the effect is probably more like 3-4 pitches a year due to sharing race with umpire, and black batters the effect is probably like -4 pitches per year due to race, ignoring the contradictory effect of white umpires favoring non-white pitchers discussed below.

Whew, take a breath before we move on.

The strange thing is that they find no evidence that white umpires discriminate against non-white pitchers. So they discriminate for white batters at the expense of all pitchers, or only white pitchers? And they discriminate for white pitchers at the expense of...only non-white batters? That's the only way for the findings to be consistent. The following flow chart is what they suggest white umpires must be doing:
- Is the pitcher non-white? If so I have no bias.
- Otherwise is the batter white? If so, I favor him over white pitchers.
- Otherwise I favor the white pitcher over non-white batter.

As expected when you do multiple sub-analyses, you start to see results that get all over the place. White umpires significantly favor black pitchers on both pitches inside the zone and outside the zone. That doesn't quite fit a narrative of racism. They also significantly favor Hispanic pitchers on pitches outside the zone. Again, this starts to look like either noise, or unmeasured covariates, doesn't it? Also one of the very telling omissions to me are the omissions of same-race analyses in tables 8 and 9. I really consider this a major omission. I strongly suspect omission because of non-significant effects (ie, a lack of bias for same-race players in any of these sub-analyses, or particularly, at least a lack of bias for white-white data points). Black umpires are significantly biased against white and Hispanic pitchers. Which is consistent with being significantly biased in favor of white and Hispanic hitters. In other words no race effect at all, just a bias for hitters.

I have some nitpicks of this study. One, you need interaction effects for horizontal and vertical miss distance interacted with left- versus right-hand hitter (and pitcher probably). Two, you need miss distance interacted with pitch type. In fact, (three) you need pitch type main effects. Assuming that different races have equal distributions of their pitch mixes may be true or it may not. But pitch type interacted with location is either the 1st or 2nd most important factor in incorrect strike/ball calls, with the other being the count. Four, why did they include every inning except the 1st? Five, true strike zones were determined by Pitch f/x stringers, I guess that's the best we can honestly do, but it still represents a rather obvious rabbit hole of opportunity for its own biases to color this entire exercise. If we examine individual batters, how much variability is there in a single batter's strike zone via the stringers. This seems like highly pertinent information. Six, why didn't the authors themselves spell out the specific race effect on ball/strike net calls per season for batters and for pitchers? Is it because you get to a total of 4 calls a year for white batters? Is it because the sub-analyses don't even corroborate the main analysis and presenting the combined results would draw attention to it? Is it because that would be a classic case of mathematically significant doesn't mean real-life-impact significant? Seven, the throw-away comments about attendance being known as a significant confounder of ball/strike calls, but ignoring it for the entire sample of 13 years because it wasn't available for the 2020 season is suspicious.
   8. The Duke Posted: August 13, 2021 at 06:48 PM (#6034311)
I had no idea there were no black MLB catchers. I guess Russell Martin was the last one.

Lots of good Hispanic catchers: Molina, Contreras, Perez, Sanchez
   9. The Duke Posted: August 13, 2021 at 06:56 PM (#6034314)
6. Thank you for that - you should submit the analysis to their professor so she can make them go back and do more work :)

Your analysis makes more sense to me - that is, it’s a tiny but measurable impact. I’m guessing there’s plenty of other variables that could explain this away.
   10. gef the talking mongoose, peppery hostile Posted: August 13, 2021 at 07:06 PM (#6034315)
I had no idea there were no black MLB catchers. I guess Russell Martin was the last one.

I can think of only a handful over the decades -- Campanella, Earl Battey, Elston Howard, Hal King, John Roseboro ...

I hope I'm forgetting several others.
   11. Howie Menckel Posted: August 13, 2021 at 07:14 PM (#6034317)
for those mystified by the Martin reference, per wiki:

"Russell Martin Sr., is Black Canadian, while his mother, Suzanne Jeanson, is Franco-Manitoban. His full name is Russell Nathan Coltrane Jeanson Martin Jr. He was named Russell for his father, Nathan for a great-grandfather, Jeanson after his mother's family, and Coltrane to pay homage to musician John Coltrane, who his father admired."

Elrod Hendricks a more obvious example, also Earl Williams, Cliff Johnson, Terry McGriff (Charles Johnson's first cousin), Lenny Webster, Choo Choo Coleman, Bruce Maxwell - and of course Josh Gibson, Biz Mackey, Louis Santop, Quincy Trouppe, et al, as the Negro Leagues were major leagues.
   12. JJ1986 Posted: August 13, 2021 at 07:16 PM (#6034318)
"Whiter Charles Johnson" is the Giants owner.
   13. JJ1986 Posted: August 13, 2021 at 07:23 PM (#6034319)
I hope I'm forgetting several others.
I had hundreds of Lenny Webster baseball cards as a kid. And Floyd Rayford was also pretty common.
   14. Walt Davis Posted: August 13, 2021 at 07:43 PM (#6034323)
There was that Campanella fella.
   15. Walt Davis Posted: August 13, 2021 at 08:02 PM (#6034324)
Haven't read it. I am generally suspicious of studies like this, not because of the topic/conclusion but because anytime you see something with a huge sample size, even trivial differences will generally be "statistically significant" which just opens the field to wild speculation by authors. Now baseball is a game of small differences that can have surprisingly big effects** (or it's 99% random) so big samples are ncessary ... and big samples are really useful if your focus is on estimating individual-level random effects for pitcher/batter/catcher/ump.

In one of my jobs, I was regularly working with a dataset with about 250,000 people in it. My boss's key variables of interest kept coming in not significant. He'd keep asking for various refinements and I'd keep pointing out that, really, finding non-significance with 250,000 cases was the "significant" finding.

And sure, the bit about one pitch's effect on a career is well over the top. Maybe, maybe on rare occasion -- Buckner's error may not have had that big an effect but it's viewed as the defining moment of his career. Caught looking b9 in game 7 with the tying run on could have consequences.

** One of the framing nerds can correct me but my memory from early discussions is that stealing something like 2 strikes per game adds up to 2 wins over a season. Or more straightforwardly, we all know that 10 "points" of OBP is 1 extra time on base per 100 PA or about once per month for a player, about 4 times per 10 games for a team.
   16. gef the talking mongoose, peppery hostile Posted: August 13, 2021 at 08:04 PM (#6034325)
I have Elrod Hendricks' autograph (Elston Howard's too, for that matter) & thought of him, but since he was born in the U.S. Virgin Islands I've always assumed he's some variety of Latino. Earl Williams should've occurred to me, since he edged Willie Montanez, a favorite of mine, for Rookie of the Year when I was in junior high. Should've remembered Cliff Johnson as well. The other MLB guys I never knew anything about.

All-time bust Danny Goodwin was drafted as a catcher, but apparently he only ever appeared at 1B & as a DH in the big leagues.
   17. Pasta-diving Jeter (jmac66) Posted: August 14, 2021 at 02:29 PM (#6034384)

I can think of only a handful over the decades -- Campanella, Earl Battey, Elston Howard, Hal King, John Roseboro ...

I hope I'm forgetting several others.

Paul Casanova & the immortal Choo Choo Coleman
   18. Brian C Posted: August 14, 2021 at 05:49 PM (#6034406)
And sure, the bit about one pitch's effect on a career is well over the top. Maybe, maybe on rare occasion -- Buckner's error may not have had that big an effect but it's viewed as the defining moment of his career. Caught looking b9 in game 7 with the tying run on could have consequences.

When I first read that bit, my mind immediately went to cup-of-coffee type fringe major leaguers, who may only get a handful or so of PA, or for pitchers, face a handful of batters. Not established major leaguers.

Still hyperbole, obviously, but more defensible.
   19. gef the talking mongoose, peppery hostile Posted: August 14, 2021 at 06:25 PM (#6034410)
Paul Casanova

Also occurred to me, but he's from Cuba, so I think of him as Hispanic rather than Black for purposes of categorizing ballplayers.
   20. John Northey Posted: August 14, 2021 at 10:28 PM (#6034442)
The way I see it is the effect is small as the bias is probably small too - the type most don't notice. For example, when someone is hiring a person for a job they often hire people who make them think of themselves. Often not on purpose, but when the edge between 2 people is too small to decide you go with the familiar. I suspect the same applies here - if a pitch is 50-50 in the umps mind they go with unconscious biases - ie: this guy has a rep for good judgement so I give him the call, or that guy has a rep for swinging at bad pitches so I'll call it a strike. Mix that with racial stereotypes and you can easily get bad calls coming up racial, and also get those biases cooked in by the players who get the bad calls having more K's and fewer walks.

Robo-umps ASAP please. Make it so this is irrelevant in the majors and ideally wherever you can in the minors as well. I HATE seeing bad calls regardless of who it favors (the Jays had a painful game recently with the final out being on a pitch at least an inch or two outside the strike zone).
   21. we all water; we all 57i66135 Posted: August 14, 2021 at 10:42 PM (#6034444)
Which way were the Derek Jeter at-bats measured?
i think that answer depends on which grocery store you went to most recently, assuming they don't use plastic bags.
   22. Starring Bradley Scotchman as RMc Posted: August 15, 2021 at 03:59 PM (#6034504)
his mother, Suzanne Jeanson, is Franco-Manitoban.

I used to love her spaghetti as a kid!

You must be Registered and Logged In to post comments.



<< Back to main

BBTF Partner

Dynasty League Baseball

Support BBTF


Thanks to
Martin Hemner
for his generous support.

You must be logged in to view your Bookmarks.

Hot Topics

NewsblogNBA 2021-2022 Season Thread
(1302 - 11:24am, Dec 08)
Last: Der-K's tired of these fruits from poisoned trees

NewsblogESPN's Tim Kurkjian is 2022 winner of BBWAA Career Excellence Award
(7 - 11:23am, Dec 08)
Last: pikepredator

NewsblogMinnie, Gil, Buck among 6 elected to Hall
(133 - 11:19am, Dec 08)
Last: Moeball

NewsblogOn MLB-owned media, the players now barely exist. What’s behind that decision?
(18 - 11:15am, Dec 08)
Last: greenback used to say live and let live

NewsblogMcCaffery: Jimmy Rollins, Ryan Howard passed Hall of Fame eye test
(108 - 11:07am, Dec 08)
Last: SoSH U at work

NewsblogFlushing University: Reality Sets In
(8 - 10:39am, Dec 08)
Last: SoSH U at work

NewsblogMLB, union stopped blood testing for HGH due to pandemic
(41 - 10:39am, Dec 08)
Last: RJ in TO

NewsblogWhat Does Endeavor, Silver Lake’s Push Into Baseball Mean For the Minors?
(17 - 10:33am, Dec 08)
Last: RoyalsRetro (AG#1F)

NewsblogAthletics manager search: Mark Kotsay, Joe Espada, Will Venable among A's candidates, per report
(5 - 4:57am, Dec 08)
Last: Doug Jones threw harder than me

NewsblogOT Soccer Thread - Domestic Cups, Congested Fixture Lists and Winter Breaks
(65 - 11:08pm, Dec 07)
Last: AuntBea odeurs de parfum de distance sociale

NewsblogRed Sox, Astros Interested In Trevor Story
(7 - 8:39pm, Dec 07)
Last: Walt Davis

Hall of MeritGil Hodges
(77 - 8:36pm, Dec 07)
Last: Howie Menckel

NewsblogMajor League Baseball is headed for a lockout. Is Ted Cruz the only one who can stop it?
(12 - 5:10pm, Dec 07)
Last: Starring Bradley Scotchman as RMc

NewsblogJose Marmolejos Signs With Tohoku Rakuten Golden Eagles
(2 - 3:55pm, Dec 07)
Last: What did Billy Ripken have against ElRoy Face?

Sox TherapyLocked Out and Semi-Loaded
(20 - 3:19pm, Dec 07)
Last: Darren

Page rendered in 0.2645 seconds
48 querie(s) executed