Clutch Runs, if you wanted to include LI, would be:

(WPA/10)-(LWTS*LI)

for Pujols

(591.7/10)-(43*1.08)=12.73


That logic's exactly right, but the runs-to-win multiplier that you use (10 here) makes a big difference. Use a value of 10.5 instead and Pujols' "clutch runs" drop to 9.91.

I didn't really understand Mike E's concern about the lack of a benchmark (#6). An average player will have a WPA of 0.0 (unless he performs better or worse in high-LI situations).

No, he won't; that's only true for an average player on an average team when the average LI of the situations in which he performs is 1.0.

-- MWE

Clutch Runs, if you wanted to include LI, would be:

(WPA/10)-(LWTS*LI)

for Pujols

(591.7/10)-(43*1.08)=12.73


That logic's exactly right, but the runs-to-win multiplier that you use (10 here) makes a big difference. Use a value of 10.5 instead and Pujols' "clutch runs" drop to 9.91.

Actually, looking at this, now I'm questioning the logic, which seems "exactly right" to me. But with a run-to-win converter of 10.0, you get 1.273 "clutch" wins for Pujols (12.73 / 10.0), but with a run-to-win converter of 10.5, you get 0.94 "clutch" wins for Pujols (9.91 / 10.5), which doesn't seem right to me.

If you use WSAA (an entirely different sort of win metric) Pujols accounts for about 70% of the Cardinals' wins over .500.

Actually, looking at this, now I'm questioning the logic, which seems "exactly right" to me. But with a run-to-win converter of 10.0, you get 1.273 "clutch" wins for Pujols (12.73 / 10.0), but with a run-to-win converter of 10.5, you get 0.94 "clutch" wins for Pujols (9.91 / 10.5), which doesn't seem right to me.

No, the logic is correct. When you have a higher R-W converter (10.5), that means that Pujols' total contribution (as measured in runs) is less, and so he gets less clutch credit.

I used 10 because that's generally used as a rule of thumb when converting runs to wins. If you wanted to get really complicated and somewhat more accurate you could come up with some kind of pythag conversion I suppose.

No, he won't; that's only true for an average player on an average team when the average LI of the situations in which he performs is 1.0.

If a player is average and plays average in every situation his WPA will be 0 regardless of his LI.

LI makes a difference when comparing 2 above average players, assuming they are also equally above average in their high LI opportunities. In that case the player with the higher LI will have a higher WPA.

No, he won't; that's only true for an average player on an average team when the average LI of the situations in which he performs is 1.0.

I don't think this is correct. Leverage increases the impact of performances above or below avg, but WPA for avg performance remains zero regardless of LI. Again, an average player could have a non-zero aggregate WPA, but only by performing better or worse in high-LI situations. And I think the team is irrelevant (except that average teams probably tend to have produce slightly higher LIs). But, I'm prepared to be corrected on any of above. Studes??

I can't imagine anyone really thinks that Marquis' 8 IP/2 ER performance was the 3rd most valuable here. If you were a GM and knew these players could repeat these performances once every 5 days, would you pay more to get Hancock or Wainwright than Marquis? Do you think a team with those pitchers would win more games than the team with the 2.25 ERA, 260 IP version of Marquis? And if not, how is their performance more "valuable?"

What you're missing is that the relievers' mediocre performances in close games will look a lot worse than the starters' mediocre performances in the same game. I don't think you can really extrapolate from single-game results to season-long numbers.

That said, I do agree with you on the primary point. I love the WP framework, but using WPA to assign value is pretty shaky; as great as it seems theoretically, the more you jump into actual results, the spottier it seems.

What you're missing is that the relievers' mediocre performances in close games will look a lot worse than the starters' mediocre performances in the same game.

To some extent, yes -- you live by high leverage, you die by high leverage. However, it seems to me that relievers, especially closers, have a special edge in terms of pitching the 9th. Pitchers can perform poorly in the 9th inning if team is ahead and yet still post a positive WPA (if team wins), but are always rewarded by WPA if they pitch well in the 9th. If you have a 3-run lead and give up 2 runs in the 9th, the pitcher is +5 (several AZ pitchers recently combined to give up 6 runs in the 9th, nearly blowing a 7-run lead, but got a WPA of +0.2 for the inning). (Hitters are the opposite: they are never rewarded for poor 9th inning performance -- it always reduces WE -- but can receive a negative WPA for good 9th inning hitting in a losing cause).

I can't imagine anyone really thinks that Marquis' 8 IP/2 ER performance was the 3rd most valuable here. If you were a GM and knew these players could repeat these performances once every 5 days, would you pay more to get Hancock or Wainwright than Marquis? Do you think a team with those pitchers would win more games than the team with the 2.25 ERA, 260 IP version of Marquis? And if not, how is their performance more "valuable?"

Assuming the league average is .5 runs per inning, Marquis' expected WPA would be:

((.5*8)-2)*10 = +20.0 WPA

Marquis' WPA is not much different than would be expected.

If one those relievers could average about +15.0 WPA per inning for a season, he would be better than Papelbon. There is no doubt he would be as valuable as the hypothetical Marquis.

Good lord. If you assume a replacement starter gives up around 6.0 R/G, then super-Marquis is 108 RAR. A replacement reliever is about 5.0 R/G (relievers have about a 1.00 R/G advantage over starters). Let's say this 0.00 R/G reliever delivers 1.5 IP to every 8 IP for Marquis, or 49 IP, so he's 27 RAR, exactly 1/4 Marquis. So he'd have to have a LI of 4.0 to match Marquis' value, which is impossible.

And even that overstates the reliever's value. Because all teams can use their better pitchers in high LI situations, top relievers really need to be compared to a much higher baseline than true replacement pitchers. That is, when WPA values the win probability gain of a reliever's performance, it assumes that pitcher's own team has the usual average chance of scoring in the remaining innings. In fact, the chance of scoring is often less because they are facing a good pitcher as well, so the pitcher's real WPA gain is less. To evaluate closers, we should probably be calculating Runs-Above-Looper, or something like that (then multiplied by LI).

In a neutral context those relievers expected WPA would be:

((.5*2)-0)*10 = +10.0 WPA

Pitching like that in highly leveraged situations would make them more valuable than that of course.

So how valuable is Papelbon?

To evaluate closers, we should probably be calculating Runs-Above-Looper, or something like that (then multiplied by LI).

If you want to evaluate closers more inaccurately, you could do it this way. If you have two relievers with the same RA and LI, they will come out the same this way. It won't capture whether one is a lot more effective in the save (high leverage) situations while the other is not.

See Huston Street versus Chris Ray or Bobby Jenks.

So how valuable is Papelbon?

I don't know, but a lot more than Looper!

I think it's one of the harder things to evaluate. He's given up 3 runs in 41 IP, so let's say that's something like 23 runs better than a replacement reliever. That's around 2 wins, and then his LI is 1.81 (not that high for a closer), so maybe around 3.6 wins? His WPA is 383, or about 3.8 wins.

But the question is, how much "credit" do you want to give Pabelbon, or any reliever, for the leverage. He didn't create it, his teammates did. On the other hand, it's there and he took advantage of it. At a minimum, I think you have to set a much higher replacement level for closers, since only good pitchers are allowed to pitch in high-LI situations (as a rule), so maybe the real value is more like 3 wins. Which is a hell of a lot in half a season -- I'm certainly not going to argue that a guy on track for 88 IP and a 0.57 ERA isn't valuable. But that's not exactly a typical closer performance.

Doing all these calculations with replacement level is confusing the issue, comparing apples and oranges, since WPA is supposed to be comparing players to average players.

In regards to LI and WPA, its illogical to believe in the concept of LI but not in WPA or vice versa. They are essentially the same concept.

Using LI and LWTS will give you a less accurate result than just using WPA.

Using LI and LWTS will give you a less accurate result than just using WPA.

It's true, to a certain extent, that LI and WPA are "essentially the same concept" or, at least, they derive from the same source. But, I don't know that I'd say LI and LWTS gives you "a less accurate result than ... WPA". It measures something different.

LI*LWTS measures the extent to which your performance was more valuable because of the average context of your performance, without considering the timing of your performance.

WPA - LI*LWTS measures the extent to which your performance was better in higher-leverage situations than in lower-leverage situations.

For batters and starting pitchers, by and large, LI isn't really something that they control, and even for relief pitchers, it's managers that control their LI, not necessarily them personally.

So if, for example, David Ortiz finds himself up to bat in "clutch" situations more often than Alex Rodriguez, his WPA would be better, even if both of them hit exactly the same in "clutch" and non-"clutch" situations. To judge whether Ortiz really did hit better in "clutch" situations you'd want to control for his higher leverage, so you'd want to look at WPA - LWTS*LI.

LI*LWTS measures the extent to which your performance was more valuable because of the average context of your performance, without considering the timing of your performance.

Is this still true if you figure it each PA?

It seems to me figuring LI*LWTS every plate appearance would weight performance according to the timing of it, but put it into a currency of runs (without having to choose a win/run conversion).

Is this still true if you figure it each PA?

I'm not sure, but I think, no. I think LI*LWTS figured by plate appearance would pretty much just be the runs equivalent of WPA, wouldn't it? I was thinking about doing it at the end of a season (or game, I suppose).

I guess if you did it by plate appearance, I'd still disagree with tigers77's statement that you'd get a "less accurate result." It seems to me, doing it that way, you'd get the same result, just in a different unit.

Having said that, though, I think "wins" makes more sense as a unit than "runs", so I don't personally see the point.

[Also, parenthetically, in #119, my formula WPA - LWTS*LI mixes units; obviously, you'd need to convert one of these into the other's currency, i.e., either (WPA/(Run-to-Win converter) - LWTS*LI or WPA - (LWTS*LI)*(Run-to-Win converter)]

I'm not sure, but I think, no. I think LI*LWTS figured by plate appearance would pretty much just be the runs equivalent of WPA, wouldn't it?

Not necessarily. With WPA you get credit for the actual events that happen - so if Jose Reyes scores from first on a David Wright single, Wright gets credit for changing the base/out state and the score. With LI*LWTS, he'd just get credit for a single in an important situation. To some extent I think it might filter out the effect of runners and fielders (beyond their role in a ball beooming a hit in the first place).

If you look at the Michael Young in the ASG example I posted above, his WPA was 59.1 or 5.9 runs. His Leveraged Linear Weights (to coin a phrase) was 3.75 runs.

I'm not sure which is more correct, but one of my issues with WPA is how much one play can be worth, so the smaller number feels better to me.

I think I'll run through an entire game and see how they compare. I'll post something this weekend with what I find.

Having said that, though, I think "wins" makes more sense as a unit than "runs", so I don't personally see the point.

It depends on what you want to do with it. I think runs makes sense in that's how we tend to measure offensive and defensive performance. If you can determine "clutch" performance, it makes for another contribution to determine a player's value, which you'll be able to express in a single runs above average rating.

To some extent I think it might filter out the effect of runners and fielders

I think this is exactly right. I think this points out a weakness of WPA in its current incarnation. If, in your example, you were to simply credit Wright with the change in win probability assuming an average baserunner and give the additional credit to Reyes for scoring on the play, then (a) you'll have a more accurate measure of the actual value provided by these two players on the play, and (b) get an answer that should be the win equivalent of LWTS*LI.

I think runs makes sense in that's how we tend to measure offensive and defensive performance. If you can determine "clutch" performance, it makes for another contribution to determine a player's value, which you'll be able to express in a single runs above average rating.

But if you're going to move toward a WPA framework anyway (and to get LI's you basically have to calculate win probabilities anyway), then you could just as easily express everything in terms of wins. It doesn't matter a great deal, I just think expressing things in wins makes numbers more readily comparable - no need to worry about run-to-win converters, it recognizes that certain events (bunts, stolen bases) are more valuable in terms of wins than runs because of when they tend to occur.

Ok, I charted yesterday's Yankees/White Sox game and calculated LI*LW for every plate appearance for the Yankees batters. Here's what I found along with the WPA from Fangraphs



I'll let you draw your own conclusions (since I'm not sure what mine are).

To get WPA Runs, you would probably want to divide by 10.

When you did your calculations, did you find the LI for each plate appearance, or did you use the players average LI for the game?

When you did your calculations, did you find the LI for each plate appearance, or did you use the players average LI for the game?

LI for each plate appearance.

Leverage increases the impact of performances above or below avg, but WPA for avg performance remains zero regardless of LI.

No, it doesn't. Quick-and-dirty example:

There were 9 times, in 2004, in which a bases loaded/two out situation occurred in the bottom of the ninth where the home team trailed by a run. The home team won one of those games, and lost the other eight, so the WP of the base situation would be 1/9 for the home team, assuming this was average performance.

Here's what actually happened:

7 times, the player made the third out to end the game, losing a total of 7/9 WPA
1 time, the player won the game with a two-run single, gaining 8/9 WPA
1 other time, the player drew a walk to tie the game. With bases loaded and two outs in the bottom of the ninth of a tie game, the home team's WP was 3/4 (15/20), so the player gained (3/4 - 1/9) WPA

Net WPA gained by players batting in those situations: 8/9 + 3/4 - 1/9 - 7/9 = 3/4 WPA - exactly the value of the walk that tied the game (which the home team ultimately lost).

WPA doesn't center around zero because win expectations are derived empirically, based on actual game results. For that reason, the calculated win expectation going forward from a given game state is dependent on the series of events that got you to that game state - but basic WPA assumes that WE going forward is in fact independent of the path you took to get there. In the example I gave above, the home team's WE when the game is tied in the bottom of the ninth, the bases are loaded, and there are two outs is .75 - but the path that I took to get to that specific game state includes none of the states in which the home team actually wins, and that difference is reflected in the baseline WE.

For early game performances, this effect doesn't have a huge impact. The further you get into a game, though, the more likely it is that certain paths will be foreclosed - and the more likely it is that those excluded paths will drive WE going forward average from the average, and WPA away from zero. And since we are *most* interested in the impact of late-game performance, it's absolutely essential, IMO, to recalibrate WPA.

-- MWE

I don't see why WPA needs to be based on empirical results, and it seems to me it shouldn't be. The Fan Graphs results that have been referenced in this thread are derived from a probability model, not real game data.

As for your example, the illusion of a positive avg WPA is created by your treating the walk PA in two different ways, once as leading to certain defeat but then -- correctly -- as leading to 3/4 of a win. Your initial WP is not 1/9, as the outcomes from that state are 7 immediate losses, 1 immediate loss, and a .75 chance of a win: 1.75/9 = .194. Then the players' totals are 7*(-.194) + .806 + (.75-.19) = 0. (Or, if you want to assume for the purpose of this exercise that the bases-loaded walk leads inevitably to defeat, then your 1/9 starting WP is correct, the value of the walk is now -1/9, and again everything nets to zero.)

I don't see why WPA needs to be based on empirical results, and it seems to me it shouldn't be. The Fan Graphs results that have been referenced in this thread are derived from a probability model, not real game data.

As for your example, the illusion of a positive avg WPA is created by your treating the walk PA in two different ways, once as leading to certain defeat but then -- correctly -- as leading to 3/4 of a win. Your initial WP is not 1/9, as the outcomes from that state are 7 immediate losses, 1 immediate loss, and a .75 chance of a win: 1.75/9 = .194. Then the players' totals are 7*(-.194) + .806 + (.75-.19) = 0. (Or, if you want to assume for the purpose of this exercise that the bases-loaded walk leads inevitably to defeat, then your 1/9 starting WP is correct, the value of the walk is now -1/9, and again everything nets to zero.)

Sorry for double post. And, should say "the outcomes from that state are 7 immediate losses, 1 immediate WIN, and a .75 chance of a win: 1.75/9 = .194.

The Fan Graphs results that have been referenced in this thread are derived from a probability model, not real game data.

This wasn't clear to me. I was looking at Studes's earlier article, in which he referenced Chris Shea's emprical WE finder, and assumed that the WPAs he was using (and provided to Appelman) were based on Shea's empirical approach. Tango, I know, used a mathematical model.

I think the problem that you run into, either way you calculate it, is that the probability of a win going forward from the current state is not independent of the path that you took to get to that state, and thus the WE for any particular state is not static.

I've done some preliminary work on changes in performance based on the LI of the state, using Tango's four-fold classification of LI situations (low, medium, high, and critical) and what's immediately apparent are three things:

1. Late-game performance (7th inning on) declines across every LI situation.
2. Hitter performance does decline in critical LI situations, although this effect is masked to some extent by the number of intentional walks given in such situations.
3. There is something of an interaction between 1 and 2. IOW, part of the reason why hitters don't perform well as a group in critical LI situations is that (a) they don't perform well in late-game situations in general and (b) most critical LI situations occur late in the game. But there still appears to a decline in performance in critical LI situations, even when the late-game effect is considered.

-- MWE

I think the problem that you run into, either way you calculate it, is that the probability of a win going forward from the current state is not independent of the path that you took to get to that state, and thus the WE for any particular state is not static.

Is this really true? WPA calculated using a probability model is based on a run expectancy model that can be calculated from a Markov chain. Previous states shouldn't have any impact (although because we are talking about an "average" context they might. For example, if you are at the top of the 7th with a two run lead how does the WPA depend on how you got there? Are you saying that the WPA context needs to be altered because it is based on average teams? If so this issue is discussed on Tango's blog at inside the book

1. Late-game performance (7th inning on) declines across every LI situation.
2. Hitter performance does decline in critical LI situations, although this effect is masked to some extent by the number of intentional walks given in such situations.

As you point out there is an interaction here. Isn't this a result of more critical LI situations occuring later in the game? Also we know in general the relievers have a lower ERA than starters -- for whatever reason. This could be creating the effect you note above.

How hitters / picthers change their approach to higher LI situations is interesting and one which is worthy of more study. I'd be keen to hear more about your findings ...

This wasn't clear to me. I was looking at Studes's earlier article, in which he referenced Chris Shea's emprical WE finder, and assumed that the WPAs he was using (and provided to Appelman) were based on Shea's empirical approach. Tango, I know, used a mathematical model.

Mike, David used the matrix math contained in my WPA spreadsheet, which is originally Tango's. Now, he uses a table derived by Tango for the current (estimated) run environment.

How hitters / picthers change their approach to higher LI situations is interesting and one which is worthy of more study. I'd be keen to hear more about your findings ...

Tango has actually posted some data on his site, which supports the view that any change in hitter performance is largely due to quality of pitcher (ie, better pitchers are used in higher LI situation.

Tango has actually posted some data on his site, which supports the view that any change in hitter performance is largely due to quality of pitcher

I control for this to some extent by using a two-way matrix, looking also at early/middle/late as well as the LI of the situation. The middle innings are especially interesting, because the number of plate appearances is almost uniformly distributed between situations and teams don't have a strong tendency to dip into their bullpens in the middle innings (and when they do, it's rarely one of their better-quality relievers).

-- MWE

To get WPA Runs, you would probably want to divide by 10.

I disagree. The whole point of WPA is that the win value of various events differ by situation. By extension, the win value of a run differs by extension. It's nonsensical to convert a wins-denominated value like this into runs using a constant multiplier.

If you want to compare WPA to a runs-demoninated measure, then convert the runs-denominated measure into wins.

If you want to compare WPA to a runs-demoninated measure, then convert the runs-denominated measure into wins.

And I agree that this is nit-picking. Tigers77 suggested dividing WPA by 10 to compare to runs, while I suggest multiplying runs by 10 to compare to WPA.

In regards to LI and WPA, its illogical to believe in the concept of LI but not in WPA or vice versa. They are essentially the same concept.

Others have already posted specific arguments against this idea, but my more general response is this:

They are not the same concept. They are different applications of the same general concept. I really like the WP framework. I think LI is very useful analytically (even if I am somewhat skeptical of LW*LI applications). I am far more skeptical about using WPA to evaluate the performances of individual players.

I can go into more detail about why, but I think much of that has already been covered by GuyM and others.

I think the problem that you run into, either way you calculate it, is that the probability of a win going forward from the current state is not independent of the path that you took to get to that state, and thus the WE for any particular state is not static.

MWE, what makes you think this?

(Hitters are the opposite: they are never rewarded for poor 9th inning performance -- it always reduces WE -- but can receive a negative WPA for good 9th inning hitting in a losing cause).

How do you generate a negative WPA without making an out? I'm stumped as to how a good hitting can result in negative WPA, even in that situation.

How do you generate a negative WPA without making an out? I'm stumped as to how a good hitting can result in negative WPA, even in that situation.

I guess a walk with first base open and the walkoff run in scoring position might do it, but I don't think that's normally considered "good 9th inning hitting".

How do you generate a negative WPA without making an out? I'm stumped as to how a good hitting can result in negative WPA, even in that situation.

I was talking about team totals in those situations. A pitcher (or staff) that gives up 6 runs in the 9th with a 7 run lead will get a positive total WPA. The hitters on the opposing team will, collectively, get a negative WPA. Of course, in the case of the hitters it's true that those who made the outs will get the negative ratings.

OK, thanks GuyM, that makes sense.