Camden Depot: Measuring Bullpen Management: Bucking Up the Bullpen

Jon Shepherd suggests a way to evaluate a manager’s bullpen usage over time, drawing on Dan Lependorf’s article at Hardball Times:

A Hardball Times article suggested using WPA - WPA/LI as a measure of bullpen performance that would be useful in assessing bullpen management.  WPA is the acronym for Wins Probability Added.  This is calculated as the difference in win expectancy before and after an event. LI is the Leverage Index.  It is a measurement of how consequential a specific scenario is based on the inning, outs, score, baserunners, and baserunner position.  By using the two statistics in concert, you arguably have a measure that gives you a context neutral wins added metric.

To test whether this metric would give an indication that managers have skill associated with using a bullpen well, I decided to take the last five seasons of several recent managers.  I compare each manager’s mean for WPA, WPA/LI, and WPA-WPA/LI (ANOVA).  WPA will inform us simply if certain teams wind up with better outcomes for winning based on reliever usage.  WPA/LI will neutralize all situations and measure reliever ability in general (similar to wOBA).  WPA-WPA/LI is being used to inform us about whether relievers actually perform better in clutch situations.  If one of these statistics indicate an actual skill then the numbers associated with a manager should be (1) repeatable and (2) result in managers being different from each other.  Of course, this assumes that these things are, in fact, measurable.

The results are somewhat as you might expect: Showalter, Joe Girardi, and Joe Maddon do well, Eric Wedge, Ozzie Guillen and Mike Scioscia do relatively poorly. There are a couple of surprises mixed in - Ron Washington in the upper tier, Bruce Bochy and Jim Leyland near the bottom.

Not a perfect study, but an excellent start.