Baseball for the Thinking Fan

Login | Register | Feedback

btf_logo
You are here > Home > Primate Studies > Discussion
Primate Studies
— Where BTF's Members Investigate the Grand Old Game

Tuesday, October 01, 2002

Runs Allowed Average: A Better Way to Figure ERA

Eric breathes new life into the old standby.

Earned Run Average is regarded as the best mainstream stat for measuring a pitcher?s effectiveness. I generally agree, but as I see it, there are still two fundamental problems with ERA. One, it doesn?t account for unearned runs and two, it doesn?t account for inherited runners that score thanks to bad relief pitching. The statistic described in this article, Runs Allowed Average (RAA), is an attempt to correct those flaws.

 

Unearned runs are still runs. They hurt the team as much as earned ones, and pitchers should be held at least partially responsible for them. Very few runs are scored due to bad defense alone. If a run scores on a hit and an error, neither the hit nor the error alone caused the run, but the combination of the two. Besides, earned and unearned runs are based on errors, which are not a particularly good measure of defense anyway. With ERA, a run scored when Nomar Garciaparra bobbles a grounder is the fielder?s fault, but a run scored on a ball that rolls past the slothlike Mo Vaughn is the pitcher?s fault. In addition, when a pitcher gets blasted during an inning in which an error happens to be made, as Pedro Martinez was on May 7, ERA lets him off scot-free. In computing RAA, then, I decided to count an unearned run as 50 percent of an earned run. This is admittedly an arbitrary figure, but it is fair, I believe, to the pitchers who genuinely have poor defensive teams behind them, and is also fair to those pitchers who help their teams by giving up few unearned runs, and who are able to bear down when an error is made behind them.

 

The other major problem with ERA is that it?s polluted with runs that score when relievers allow inherited runners to cross the plate. A common argument regarding the MVP award is that players on bad teams shouldn?t be penalized for having bad teammates; likewise, neither should starting pitchers be penalized for having bad relievers. The pitcher who put the man on base and the pitcher who allowed him to score should each bear some responsibility for the run, but how much? Fortunately, we have play-by-play data for recent seasons, which tells us: a) how many of these runs a starting pitcher has been charged with, and b) which base the runner was on when the pitcher left the game. I decided to go with a simple formula, which is that each base counts for a quarter of a run. If a pitcher leaves a runner on first who later scores, 75 percent of that run gets charged to the reliever, 25 percent to the starter. If a runner scores from third, 25 percent of that is charged to the reliever, 75 percent to the starter. For a runner left on second base it?s 50-50. This system is admittedly flawed in that it doesn?t take into account the number of outs when the pitching change was made, but it is much better than simply charging the run to the pitcher who put the man on base.

 

Below are the 2002 RAAs for nine selected pitchers: those with the top three ERAs in the AL and the top five in the NL, plus Curt Schilling. I would have included more pitchers, but I?m not smart enough to figure out how to make my computer do all the work, so I figured these out by hand, game by game. I didn?t include relievers, although I do think RAA would be a far more accurate indicator of a relief pitcher?s value than ERA.

 

PITCHER

G

IP

ER

UER

adjER

adjUER

ERA

RAA

Pedro Martinez

30

199.1

50

12

47.75

12.00

2.26

2.43

Randy Johnson

35

260.0

67

11

66.25

11.00

2.32

2.48

Derek Lowe

32

219.2

63

2

60.25

1.75

2.58

2.50

Barry Zito

34

229.1

70

9

65.00

8.75

2.75

2.72

Greg Maddux

34

199.1

58

9

57.75

8.75

2.62

2.80

Odalis Pérez

32

223.1

74

2

73.25

2.00

3.00

2.99

Tom Glavine

36

224.2

74

11

70.75

8.00

3.01

3.07

Curt Schilling

35

259.1

93

2

89.25

1.25

3.23

3.12

                 

Totals

 

2048.0

627

66

605.75

64.50

2.76

2.80

As you can see, the end product of RAA is roughly equivalent with ERA. The nine pitchers collectively had an ERA of 2.76 and a RAA of 2.80. Most of the differences between the two stats are minor, but several are revealing. First, this year?s two ERA champions, Pedro Martínez and Randy Johnson, both have deceptively low ERAs because they allowed large amounts of unearned runs. Johnson, of course, is still easily the best pitcher in the National League. In the American League, however, it becomes apparent that the large ERA difference between Martinez and Derek Lowe is mostly an illusion. RAA is also useful in comparing two pitchers, such as Tom Glavine and Odalis Pérez, who have similar ERAs. Although Pérez had a slightly higher ERA than Glavine, he allowed far fewer unearned runs, and therefore was actually a somewhat better pitcher (at least before park factors are considered).

Another interesting tidbit: Greg Maddux?s ERA in 2002 clearly benefited from having arguably the best bullpen in baseball history to back him up. Braves relievers allowed only two of Maddux?s inherited runners to score all year, and both were already on third when Maddux left the game. Strangely, as can be seen on the chart, the Atlanta bullpen did not help Glavine nearly as much.

 

I don?t claim to be breaking any new ground with this stat. I?ve been contemplating variations on it since I was a teenager, and I?m sure others have had similar ideas before. I have no doubt that it can be improved upon, most obviously by incorporating the number of outs into the calculation for inherited runners. But this is what I came up with. It?s fairly easy to compute, and is clearly a better indicator of a pitcher?s performance than traditional ERA.

Eric Enders Posted: October 01, 2002 at 06:00 AM | 28 comment(s) Login to Bookmark
  Related News:

Reader Comments and Retorts

Go to end of page

Statements posted here are those of our readers and do not represent the BaseballThinkFactory. Names are provided by the poster and are not verified. We ask that posters follow our submission policy. Please report any inappropriate comments.

   1. John Posted: October 01, 2002 at 12:51 AM (#606504)
Interesting. And it goes a way towards giving us a simple number that gives us a snapshot of how well a pitcher has really done--not unlike OPS has for hitters. But I disagree with:

although I do think RAA would be a far more accurate indicator of a relief pitcher?s value than ERA.

How? In one sense, sure, it accounts for all the runners the set up guy leaves on for the closer to come clean up in the bottom of the 8th. OK, that's purely theoretical, since no closer comes in with runners on, but still... But RAA (like ERA) doesn't penalize guys who come in with runners on and cough up the runs. Seems like if you're going to "penalize" for runners left on, you've got to work those runners into the reliever's "RAA" before it's accurate across the board. I agree that the starter shouldn't get hit with the whole run when the reliever chokes--but don't we need to hit the reliever with the "rest" of that run? There's got to be a way to work that in, doesn't there? And conversely, don't we want to reward the reliever who comes in with the bases loaded (for which we've charged the SP with 1.5 RA) and wipes them all out--i.e., reduce his "RA" by (some portion of) those 1.5 runs?
   2. Dan Turkenkopf Posted: October 01, 2002 at 12:51 AM (#606505)
But RAA (like ERA) doesn't penalize guys who come in with runners on and cough up the runs. Seems like if you're going to "penalize" for runners left on, you've got to work those runners into the reliever's "RAA" before it's accurate across the board. I agree that the starter shouldn't get hit with the whole run when the reliever chokes--but don't we need to hit the reliever with the "rest" of that run? There's got to be a way to work that in, doesn't there? And conversely, don't we want to reward the reliever who comes in with the bases loaded (for which we've charged the SP with 1.5 RA) and wipes them all out--i.e., reduce his "RA" by (some portion of) those 1.5 runs?

The way I read it, the reliever was being charged for inherited runners that he allowed to score, in an opposite amount from the starter, i.e. runner on 3rd means starter gets .75 RA, releiver gets .25 RA. And as for the other situation, you don't charge the starter for the RA unless the runs score. In your hypothetical, no one receives any RAs.

But maybe I read it wrong.
   3. Mike Emeigh Posted: October 02, 2002 at 12:51 AM (#606506)
The way I read it, the reliever was being charged for inherited runners that he allowed to score, in an opposite amount from the starter, i.e. runner on 3rd means starter gets .75 RA, releiver gets .25 RA. And as for the other situation, you don't charge the starter for the RA unless the runs score. In your hypothetical, no one receives any RAs.

But maybe I read it wrong.


No, you read it exactly right.

A better version of this stat would be to charge the starter with the percentage of runs based on the percentage of the time that the runner scores in that situation; e.g. if the starter leaves with a runner on 3B and no outs, and that runner would score 90% of the time, the starter would get charged with .9 run if that runner scores, and the reliever would get charged with .1 run. But the quick-and-dirty estimator that Eric is using likely works just fine for most pitchers.

-- MWE
   4. MattB Posted: October 02, 2002 at 12:51 AM (#606508)
"A better version of this stat would be to charge the starter with the percentage of runs based on the percentage of the time that the runner scores in that situation; e.g. if the starter leaves with a runner on 3B and no outs, and that runner would score 90% of the time, the starter would get charged with .9 run if that runner scores, and the reliever would get charged with .1 run. But the quick-and-dirty estimator that Eric is using likely works just fine for most pitchers."

Except even this measure would undervalue the great reliever who comes in with no outs and a runner on third and strands him there. If you want to truly capture the value of that reliever, you may want to attribute 0.9 runs to the pitcher as soon as he comes out of the game (this reduces the riduculous, "We'll have to wait to see how Reliever does before we can close the book on Starter's game.")

Then, if Reliever lets the run score, he gets 0.1 runs added to his stat, but if he strands them, he gets a -0.9 runs added (or 0.9 runs taken away.) The value of this is that it reduces the perverse incentive to build up your stats through 3-out no-one on saves. The closer who starts out the ninth and makes three outs still gets his 0.00, but the reliever who comes in with a runner on third and doesn't even allow a sac fly for three outs gets a -0.90.

John Smoltz's ERA explodes with a horrible performance in April? He doesn't have to spend the whole year working that off 81.00. Bring him in in the bottom of the sixth with one out and the bases loaded. Not a save situation? No problem. If he induces a double play, he'll get over two runs taken away from his total. Two or three short, high leverage appearance like that his RAA is back down to zero, or lower, and more accurately reflects his value to the team. Meanwhile, the starter who loaded the bases doesn't get a "free pass" for Smoltz's inning ended DP, he gets the runs saved by Smoltz tacked on to his numbers, so that it evens out on a team level.

   5. tangotiger Posted: October 02, 2002 at 12:51 AM (#606510)
I think the last two posters make good sense, even though they are on the opposite side of the issue. To tally "official" scores, you should only count what actually scored, and therefore, the book stays open. I would use base and outs to split the runs, as Mike E suggested, but .25/.50/.75 is easy and clear enough for everyone.

For the "sabermetric" folks, what MattB is saying is spot on. As soon as the starter leaves, his book is closed, and he gets +.90 runs for leaving the runner at 3b with 0 outs. If the reliever gets him out of the jam, it's -.90, and if he doesn't, it's +.10. This is essentially what Wolverton, I believe, does.

The "% of inherited runners scored" and similar stats are good, because they give you a decent measure as to what's happened. But to get the "complete" stat, you should do as MattB says... but I would not put it as part of the "official" record. Rather, what Eric is doing here is better than the current system.
   6. Eric Enders Posted: October 02, 2002 at 12:51 AM (#606514)
Here are the numbers from 1974-1990, thanks to Tangotiger:

0 outs 1 out 2 outs
1B .38 .25 .12
2B .61 .41 .21
3B .86 .68 .29

Incorporating this would obviously make the formula slightly more accurate, but there are two problems. 1) The run expectancy chart changes over time. Do you change the RAA formula regularly as well, or do you leave it alone?

And 2) Using the run expectancy data instead of 0.25/0.5/0.75 would make RAA far less accessible to the average fan, which in my opinion should always be a consideration. (Though not necessarily the most important consideration.)
   7. Eric Enders Posted: October 02, 2002 at 12:52 AM (#606515)
Ugh... try again.
1B 0 outs .38/ 1 out .25/ 2 outs .12
2B 0 outs .61/ 1 out .41/ 2 outs .21
3B 0 outs .86/ 1 out .68/ 2 outs .29
   8. jeff angus Posted: October 02, 2002 at 12:52 AM (#606516)
1. A swell idea, well-delivered.

2. Perhaps there's a half-way between the full probability matrix and the 75-50-25% division marks that more accourate than the latter but far simpler than the former. For each base, what %age of the total runners who are on that base ultimately score (w/o regard to outs). Seems like it'd be easier to calculate (the inherent advantage of roughing it out to 25-50-75 in the first place), with fewer steps than the full monty.

3. I think pitcher's errors should be removed from UNearned runs. While there aren't very many of them, the resulting offense is the repsonsibility of the pitcher, and was 'earned' off of him.
   9. tangotiger Posted: October 02, 2002 at 12:52 AM (#606517)
The numbers change based on the run environment, but as a rough guide, you can use 26/43/60. I think that it's more easily understood though to say each base is worth 1/4 of a run, and for such a rough estimate, 25/50/75 seems more intuitive.
   10. Carl Goetz Posted: October 03, 2002 at 12:52 AM (#606528)
It seems to me that Eric is trying to create an alternative to ERA. In other words, a more accurate way of assigning blame for runs that actually score against a team to 1 pitcher or another. In this case, I don't think we should be assigning negative credit to a reliever who doesn't allow inherited runners to score. This would be more appropriate if we were creating a Runs Created type formula for pitchers. For this type of metric, I don't believe a starter should be charged with runs that don't score. I agree with the idea that a runner on third with no outs will score 90% of the time, so the guy who put him there should receive .9 of the blame and the guy who allowed him the score should receive the other .1. I just don't think we should cross our purposes. We're either assigning blame for runs that score, or we're taking the various factors the account for runs scored and determining how many runs the pitcher should have given up in a context-neutral environment. I don't think I wrote this very well, so I hope everyone understands what I'm trying to say.
   11. MGL Posted: October 03, 2002 at 12:52 AM (#606531)
Yes, perhaps you should not combine regular (as opposed to component) ERA with something that assigns theoretical runs to baserunners, especially if they get stranded. On the other hand...

Regular ERA is only concerned with how many baserunners a pitcher allows to score, as we all know, regardless of how many runners he puts on base or "how" he puts them on. However, since a pitcher does not get to determine whether any baserunners score or not after he leaves the game, it is fair to estimate how many "would have scored" by using the RE matrix, and then add this to his "previous ERA" (the part of the ERA that does not include the runners on base when the pitcher leaves).

In fact, the more that I think about it, the only way to do it is to assess some ERA runs to the pitcher who leaves, solely based on the run potential of the bases/outs (RE), regardless of whether any runs score or not.

As the author correctly states, if we simply "credit" the departing pitcher with baserunners that the releiver allows to score (the current scheme for ERA), we artificially and unfairly inflate his ERA if the reliever does a bad job and allows some runs to score. Assigning part of the actual runs that score to the reliever and part to the departing pitcher helps, but doesn't solve the whole problem. Even under this scheme, if the reliever does a good job, and doesn't allow any baserunners to score, we also artificially and "unfairly" deflate the departing pitcher's ERA.

The only solution is to assign ERA runs to the departing pitcher based upon the theoretical run value of the baserunners (bases/outs matrix), regardless of whether they score or not. Of course, the new pitcher only gets ERA runs if any runners score. However, he only gets that portion which the departing pitcher doesn't get, which would be "actual runs scored minus the RE when the releiver entered the game".

That is the ONLY way to do it if you want to accomplish what the author sets out to accomplish...
   12. MGL Posted: October 04, 2002 at 12:52 AM (#606535)
Jeff,

What you are describing has been called a "value added" approach to quantifying "theoretical performance". It has been discussed on other formus. The problem with this approach (at least for batters) is, while it seems rigorous and logical, it doesn't pass the "so-what" test.

What I mean by that is that it is a hybrid "stat", which attempts to combine apples and oranges. The apple is "performance", which is contributing to actual runs scored, such as RBI and Runs, and the orange is "theoretical value" or "context-neutral performance", such as linear weights or Runs Created. Unfortunately, there is really no such thing as "theoretical performance", as captured by a value-added stat such as the one you describe. Either you contribute to an actual run by getting on base and eventualy scoring or by driving in a runner, or you contribute a discrete event, such as a single, walk, out, or home run, which has a theoretical run value (linear weight).
The concept of applying different values to each discrete event, depending upon the number of outs, and the baserunners, is silly, particularily if you nake no distinction between a run actually scoring or not. While the result loks pretty, it has absolutely no practical significance. It doesn't tell you about the talent of the batter; it doesn't tell you how the batter contributed to his team's actual success or failure (how many runs they scored); it doesn't help you to predict his future performance. In fact, it doesn't tell you anything. You might as well assign differnent values to each event according to the day of the week!

It was a nice idea by James and others gone bad!

BTW, the fact that it doesn't "sum to zero" is trivial. You could easily make it do so by adding some constant per PA to each player's results like we essentially do for Runs Created or simply say that it represents "runs above or below league average" like we do for linear weights...
   13. Mike Emeigh Posted: October 04, 2002 at 12:52 AM (#606539)
The apple is "performance", which is contributing to actual runs scored, such as RBI and Runs, and the orange is "theoretical value" or "context-neutral performance", such as linear weights or Runs Created.

There is no such thing as "context-neutral performance". Performance is affected by context. For example, batters as a group tend to hit better with a runner on first base than they do with the bases empty - Bill James once noted this in one of the Abstracts, probably the 1987 Abstract when he did the rookie study (although I can't find the specific reference right now). The performance variation was virtually the same when groups of young players were compared to similar older players. There haven't been a lot of similar studies since, because people seem to have blandly accepted the idea that context doesn't matter, but when you think about it logically - think about some of the differences in the way that pitchers pitch when pitching with runners on base vs. the pitches that they throw when the bases are empty, for example - I don't see how one can come to any other conclusion.

What proper value-added methods attempt to do is to consider player performance in light of the actual contexts in which they are asked to perform. They account for performance variations across context far better than simple Linear Weights *average-context* value assignments do, and thus in my judgment measure the player's contributions to what his team actually does far better. Of course, the value-added approaches need to be team-context specific as well; the Dodgers have a far different team context, playing in Dodger Stadium, than do the Rockies in Coors.

-- MWE
   14. tangotiger Posted: October 04, 2002 at 12:52 AM (#606540)
Kent M, Jeff Smith: What you describe is LWTS by the 24 base-out states. You can check my site for a couple of articles on that.

The Mills Brothers have extended this to account for base/out/inning/score.

MGL and I agree on about 90% of whatever we discuss. But on this matter, he and I are polar opposites. There are plenty of threads at fanhome where MGL and I discuss this.

Jeff: You redistribute the ".454" based on the base/out state. It's a long explanation, and you can e-mail me if you want to discuss further.

Context-specific v context neutral: all events are based on the batter, pitcher, fielder, park, home/road, base/out, score/inning, handedness, style of hitting/pitching, speed of runners, etc, etc. Therefore, all events are context-specific. Larry Walker has a different "true talent performance level" for any combination of these. It is a b-tch to figure out. So, we start to mentally subtract those things that probably has little impact, or comes out in the wash after 600 PAs. You're essentially left with park, base/out, and handedness. I think most of us can feel comfortable to look at a player's "value-added" based on these criteria. Unfortunately, most run measures only take park into account. Base/out, especially when it comes to walk players, like Bonds, is important to know.

**************
Eric, didn't mean to hijack your thread here. I think the two essential points for the "adjusted ERA" is
1 - accounting for the run that did score, and how to split that off the best way
2 - accounting for runners left on base, and assigning theoretical run values

I believe that what Eric is doing is #1, and that's better than not doing anything. Doing #2 is more complicated, less elegant, more accurate, but less acceptable to the masses. I think we could easily sell a 25/50/75 system.
   15. MGL Posted: October 04, 2002 at 12:52 AM (#606549)
Jeff,

Yes, I understand exactly how this "stat" works. I know that it includes actual runs scored. I also know that it is diffcult to understand my criticism (it is difficult to articulate it). It is a little bit like the "tree falling in a forest." You have to think about it for a while to realize that while the value added stat seems to capture something realy "nice", it has no signficiance whatsoever (or you can simply disagree). Sure, it captures the "change in theoretical run value" after a particular event (as compared to before it), by definition. But "So what?" I challenge anyone to give me any use for this information.

What I'm trying to say is that unless we believe that a batter has significant control over the result of his PA vis-a-vis the "custom" RE value of that result, then the results of a value added stat are useless and meaningless - they are simply the result of chance! IF batter A hits more of his home runs with runners on 1st and second with 2 outs while batter B hits more of his home runs with a runner on third and 1 out, their "value-added lwts" will be quite different, everything else being equal. Who cares? Why would we even want to know this? Of what use is this stat? Do players even know what events are more valuable than others, given the outs/runners? Everyone would like to hit a home run with 2 outs or with runners on base. Every batter would like to get a hit and avoid a walk with a runner on second and 2 outs. Sure, a value-added stat would help us to evaluate the true "value" of a hitter like Bonds, where a lot of his non-intentional walks are probably less valuable than the standard .34 lwts runs, however, the "noise" in the value-added stat for any batter, including Bonds, would completely "drown out" any value it could provide...

As far as whoever posted that there is no such thing as "context-neutral performance", you apparently have no idea what I was talking about! Context-neutral performance, which lwts (and many other stats) reflects, is simply how many runs a batter would add or subtract from an average team, given his exact stat line! That's all! It is correct by defintion. Yes, we all know (yawn) that with a runner on first, the batter (especially a lefty) will hit "better" than with the bases empty, etc., etc. (in fact there is a different hitting "profile" for all bases/outs situations). It doesn't take a genius or even Bill James to figure that one out.

The way we arrive at a player's context-neutral stats is a different story and a little tricker, but it doesn't mean that it can't be done, to some reasonable degree. We can park adjust (whether you like park-adjustments or not), we can adjust for all the bases/outs situations, we can adjust for all the pitchers he faces, and we can even adjust for the hitters in front of and behind him (like Kent hitting in front of Bonds gets fewer walks and more good pitches to hit). Or we can not adjust at all and take a player's stats for what they are worth and still call them "context-neutral" if we want.
   16. MGL Posted: October 05, 2002 at 12:52 AM (#606552)
Speaking of "not knowing what I was talking about", Chuck, I have no idea what you are talking about and why "everyone is missing the point!"
   17. MGL Posted: October 05, 2002 at 12:52 AM (#606554)
Jeff,

I respect your "pause for thought". Most people on these and other sabermetrically oriented "boards", including myself, get their heels so dug in to their original positions that a team of wild horses "couldn't drag them away" (to quote the Stones, I think). Not that I expect or necessarily want you or anyone else to "come over to my side". That would be arrogant and unproductive on my part. Yours was just an expected and refreshing post.

For the record, I am not looking for a "deeper meaning". Actually, I'm looking for any meaning at all and can't find any. I haven't heard anyone posit any (meaning) either, other than simply parroting back the defintion of a "value-added stat", like if I proposed some "wonderful" new stat that told us how a batter hits on odd days only, and when someone questioned the efficacy or usefulness, I replied "It tells us exactly how a batter performs on odd days, you numskull - that's the value!"

When I fist heard of this kind of stat, I too thought it was the greatest, most rigorous thing since "blackjack basic strategy" (hey, I live in Vegas). After a while, I realized that it was completely transparent, and in fact, was utterly useless, other than as a mildly interesting sabermetric exercise...
   18. MGL Posted: October 05, 2002 at 12:52 AM (#606555)
Eric, didn't mean to hijack your thread here. I think the two essential points for the "adjusted ERA" is
1 - accounting for the run that did score, and how to split that off the best way 2 - accounting for runners left on base, and
assigning theoretical run values

I believe that what Eric is doing is #1, and that's better than not doing anything. Doing #2 is more complicated, less
elegant, more accurate, but less acceptable to the masses. I think we could easily sell a 25/50/75 system.


Tango, I can't believe the above comments!

First of all, re-arranging the ERA responsibility only for runs that score is terrible! Like the whack-a-mole game, you solve one problem (assigning too much responsibility to the original pitcher when a reliever allows runners to score), but another one immediately rears its ugly head - not assigning any responsibility to the original pitcher when the reliever does NOT allow any runners to score! In fact, I'm not even sure you've gained anything under this system! You HAVE TO assign theoretical runs to the starting pitcher when he leaves runners on base! There is no other way to do it properly if you want to have a better sample of a pitcher's ability to prevent runs from scoring than traditinal ERA, which I'm pretty sure is the goal of Eric's system! Sure, you end up with ERA's that don't match the actual earned runs scored, but there's nothing you can do about it, and it doesn't matter. Let's look at some extreme (actually not so extreme) examples:

A starting pitcher pitches 100 innings, in 10 of which he leaves lots of baserunners. If none of those baserunners score, by virtue of good relief pitching, good luck, or a combination, under the traditional ERA scheme, he gets credited no earned runs for those baserunners. Under Eric's new scheme (which Tango likes), he also gets no extra earned runs. We know for a fact that this will overrate his ability to prevent runs from scoring (understate his ERA), simply because he happened to leave the game several (10) times with runners on base mid-inning. Had he not left the game in mid-inning on these 10 occasions, there would be an excellent chance that some of these baserunners would have scored; in fact, our best estimate of how many baserunners would have scored would be the sum of the exact RE's of the bases/outs state when the pitcher left the game in each of those 10 innings.

Let's say that ALL the baserunners scored in those 10 innings, again, through no "fault" of the departed pitcher, but because of bad relief pitching, bad luck, or both. Under the old scheme, the starter would get credit for all of those runs, grossly overstating his ERA. Under the new scheme, the starting pitcher still gets credit for too many runs, even if we split them with the reliever in some fashion.

So when does the new scheme actually give the pitcher the correct credit for those baserunners? I really don't know! If we use the old (current) scheme, the same thing happens. If we give the starter all the "credit" for baseruners who score, only if a certain amount score (over and above what the RE's say) do we overstate his ERA. If only a few baserunners score (fewer than expected) under the old scheme (giving the starter the full credit) we might actually undersate his ERA or we might state it perfectly.

Sure, on the average the new scheme is better than the current one, since on the average the "correct" number of baserunners will score, in which case, under the old scheme the starter will get too much (all the) "credit". But there is so much variability in whether the baserunners score, that the new scheme is not much more accurate then the old scheme (notwithstanding what I said earlier in this post, I am conceding that the new scheme must be better) on a pitcher by pitcher basis, which is what we are looking for - a greater percentage of pitchers having their ERA's accurately reflect their ability to keep runners from scoring, at least as represented by a sample of that ability.

Yes, it may be slightly more complicated to do it the right way - assign theoretical run values to the baserunners (I'm not even sure that it IS more complicated - after all, we don't even have to know whether those runners actually scored, so we can do the whole thing with less information), however, the gain is enormous - we now have an exact sample of that pitcher's run preventing ability, as opposed to an estimate of a sample of that ability which, under the old scheme or Eric's proposed scheme, is unfairly influenced by whether the runners actually score, which is in turn influenced by the ability of the relievers and/or luck. We can live with the starter's own "luck" when he is on the mound (whether runners score or not always has a luck element), but to take the "luck" after a starter leaves the game and ascribe it to the starter seems a bit silly, doesn't it?

Getting back to how "complicated" it is to use an RE scheme - assigning runs to the starter based upon the RE when he leaves...

Eric's scheme is only simple and clean because we conveniently use a 20/50/75 system, or something like that, as a shortcut. Obviously, we would use a similar shortcut with an RE system. We could compute the average RE for 1,2, or 3 runners on base and round them off to a nice number, or we could use several numbers depending upon the outs, etc., etc. Any one of these shortcuts, including the simplest one (having 3 numbers), would be far superior to Eric's scheme (I'm not critcizing it BTW - I'm simply correcting or refining it - it is a great idea), and I don't see it as being that much more complicated than Eric's system, as Tango seems to think, if at all...

(BTW, you wouldn't use the RE when the starter leaves as the number of runs to add to his ERA. You would use the RE minus the starting RE of an inning...)
   19. Mike Emeigh Posted: October 06, 2002 at 12:52 AM (#606568)
What I'm trying to say is that unless we believe that a batter has significant control over the result of his PA vis-a-vis the "custom" RE value of that result, then the results of a value added stat are useless and meaningless - they are simply the result of chance!

And what makes you think a batter *doesn't* have significant control over the result of his PA based on the situation? Do you think players *don't* change their approach to hitting based on game situation? Do you think players don't try to give themselves up with a runner on second and no one out, ordon't try to get the ball in the air with a runner on third and fewer than two outs, or don't try to protect a runner trying to steal a base? Do you think that players have absolutely no control over where the ball is put into play, or how hard it's hit?

The only proper way that one can measure the true value of any offensive event is to evaluate it within the specific context in which the event occurred, because you cannot assume that the player would have generated the same event in a different context.

Do players even know what events are more valuable than others, given the outs/runners?

Of course they do. That's why they do things differently in different situations.

Context-neutral performance, which lwts (and many other stats) reflects, is simply how many runs a batter would add or subtract from an average team, given his exact stat line!

You can't know that, because you can't know how the batter would have performed in the context of an average team. You have to assume that the shape of his performance will not change, in spite of evidence that suggests that the shape of his performance does change, in order to justify assigning an average-context value to that player's offensive events. And that assumption, in and of itself, prevents you from really being "context-neutral".

-- MWE
   20. MGL Posted: October 07, 2002 at 12:52 AM (#606574)
Mike,

I probably used a poor choice of words by saying that "players do not exert a significant amount of control over...". Ditto (bad choice of words) for "players probably don't know..."

OTOH, I think that one could justify both statements as they stand, in part, depending upon you "definition" of the word "significant".

BTW, I think that this is the second time that you addressed some comments to me as if I haven't been doing (important) sabermetric research for almost 20 years now! IOW, you are preaching to the choir!

Yes, it is true that all (most) players (we hope) change their hitting approach, depending upon the "situation". Big deal! We either make the assumption that all players change their approach roughly equally, or if we think that they don't (that some players have a better "ability" to adapt to the situation - which is probably true), unfortunately it is almost impossible to distinguish such an ability among players. Value added-lwts might help us to do this if such an ability were significant and/or there were significant differences among players vis-a-vis that ability. Again, unfortunately, in reality value-added lwts does nothing mroe than provide tons of random noise. Sure, emebedded in that noise is a sample of a player's ability to adapt to his situation, relative to other players, but it can't be extracted! THAT was the point I was trying to make!

As far as the (paraphrased) statement that "Do players even KNOW what situations are what?", what I obviously meant (at least I thought it was obvious) was that do they know specifically at what bases/outs state a single is optimal versus a double or triple, etc.? As I said, all players know that a home run is better with runners on and/or with 2 outs, and that a walk is as good as a single with the bases empty, and that a hit is much better than a walk with a base open, etc. But they obviously don't know the differnce between the true value (the numbers that go into a value-added stat) of a single with a runner on 2nd and 1 out versus a single with a runner on first and 2 outs and that sort of thing.

To make distinctions in a metric among several different events, when the batter himself isn't aware of many of those distinctions, with the intention of the metric being to evaluate a player's ability to adapt his hitting "style" to a particular situation is illogical...

But this last argument is almost inconsequential to my principal argument that value added lwts does not distinguish one batter from another in terms of their ability to adapt - there is way, way, way too much noise, and the individual sample sizes are way, way, way too small. Therefore value-added lwts, as a stat, serves no useful purpose.

We are NOT in disagreement over the fact that yes, all batters adapt, some are better able or willing to adapt more than others, and that most, if not all, batters ARE aware (some are more aware than others) of the fact that they need to do things differently at the plate, depending upon the situation, AND that there are some "situations" that no batters (unless they carry the RE matrix in their pockets) are aware are different from other situations...
   21. tangotiger Posted: October 07, 2002 at 12:53 AM (#606585)
MGL, when I said "I believe that what Eric is doing is #1, and that's better than not doing anything. Doing #2 is more complicated, less elegant, more accurate, but less acceptable to the masses." :

There's no question at all that I prefer #2. And, I believe, this is exactly what Wolverton at Prospectus does for his reliever ratings. My point was the "crawl before you walk". That while it may seem obvious to you and me and most Primer readers what should be done, these theoretical RE charts haven't (and probably won't) take hold of the masses for many years to come. By at least putting out the idea that maybe we should be splitting off responsibility by bases, then it shouldn't be too long before someone says "hey, what about outs?" And then, someone will bring up "hey, what if the runner doesn't score, because Mo is on the mound?". And what about "WE"? The natural extension of RE is WE, and now you have to start walking to running. Now you need WE charts to know that leaving a runner on base with a 5-run lead is different than leaving him in with a tie game. I'll stop here, because we are only talking about "runs" and ERA.

I agree with your point that by doing as Eric is saying, you are solving one problem, but introducing another. I said as much, which is why I like RE-based solutions. But I still think that at least the concept of splitting up is a good one. I think we are in general agreement here.

**************************
Now, as for theoretical runs. First of all, a single is worth .46 (or whatever runs) because that is its weighted average value over the 24 base-out states. An out is worth -.27 runs (or whatever) because that too is its weighted average value over the 24 base-out states. The value of a single (and out) are of much much greater magnitude with the bases loaded than the bases empty.

You know all this of course, but I just want to make sure everyone is up to speed.

While events like singles, and HR, etc are more or less randomly distributed among the 24 base-out states, the non-intentional walk, and esp the IBB, are not at all distributed as such. This is why the walk is worth .33 runs instead of .35 if it were randomly distributed.

Howerver, MGL's point is that if a hitter is not able to perform, overall, at a better performance level, then why credit this hitter with the higher-leverage run values, just because, by sheer luck, he got more of those opportunities.

The answer is that if you are trying to account for past performance, past runs scored, then you can't do it any other way. You must know the context in which the event occurred. When Sammy Sosa hit 3 3-run HR in one game, the run values of THOSE PARTICULAR HR was more than 1.4 runs. However, if you were to use the weighted-average run value of 1.4 for HR and .46 for 1B, etc, etc, you would not be able to account for all of the Cubs runs in that game.

However, by using context-specific run values (by the 24 base-out states), then you can account for all runs precisely.

The extension of RE is WE. Continuing in this plane of thought, by now introducing WE charts that also include base-out, inning, and score, you now have a win-based value added methodology that will account for all wins. This is the approach of the Mills Brothers. In fact, you'd HAVE to do this to account for Eck's and Mo's value. Otherwise a context-neutral RUN-BASED value will completely miss out on their high-leverage performance.

The problem. The problem with this is that it doesn't help us forward-looking. That it is more likely that a run-based context-neutral approach would yield more accurate future-based win-based context-specific approach, than using the past-based win-based context-specific approach. This is the same problem with W/L of pitchers. You are more likely to predict a pitcher's W/L next year by looking at his past ERA, rather than his past W/L record.

So, when you want to account for all runs in the past, you need the 24 base-out states. When you want to account for all runs in the future (or to determine a player's "true talent level"), then you can stick with the context-neutral stats.
   22. MGL Posted: October 07, 2002 at 12:53 AM (#606590)
Tango, this idea of having to "account for actual runs scored" is silly (unless you are being audited by the SEC, I guess). You clearly don't want to do that sometimes, as in the "Better way to figure ERA" case, and frankly, who cares whether "runs add up" or not? Why do you use that as an argument for or against a particular methodology?
   23. tangotiger Posted: October 07, 2002 at 12:53 AM (#606591)
This containment system is a check to ensure that the methodology has some basis. Otherwise, we'll be adjusting at will, seemingly doing everything properly, and then you are left with results that don't make sense. It's a way for the reader to feel confident in the adjustments of any methodology without having access to the data or the inner workings of that methodology.

The SEC and most of Bill James' followers like the "accounting"/"balancing" system. So, to answer your question: LOTS of people care!
   24. MGL Posted: October 07, 2002 at 12:53 AM (#606595)
Tango, your arguments in favor of making sure that runs add up is disengenuous! Because people like it? Because it gives them confidence that your methodology is sound? So I should purposely bastardize my work so people will "like" my methodology? Please!
   25. Mike Green Posted: October 08, 2002 at 12:53 AM (#606606)
I like Eric's rough and ready approximation of accounting for runners left on base, who as between starter and reliever (or as between reliever and reliever).

As for the objection that it does not penalize the starter who leaves runners on, but is picked up by his bullpen, I have a suggestion. Don't change the stat and ask it to perform more than it can. My view is that there are 3 stats I want to know for every pitcher- Runs Allowed Average as Eric has it, Opposition OBP and Opposition Slug. If a starter is picked up more often than usual by his pen, it will show up because his Runs Allowed Average is lower than one would expect from his OOBP and OS. Now that OBP and Slug are widely reported and accepted stats, I'd like to see the same for OOBP and OS for pitchers.

In a sense, this would allow us to easily do the same mental calculation for pitchers as for hitters. We know when we see a hitter who scores and drives in 100 runs with an OBP of .340 and a slugging percentage of .460, that the hitter has had a lot of help to score and drive in those runs. Similarly, we will know if a pitcher has an RAA of 3.00 with an OOBP of .340 and an OS of .400 that he has had a lot of help from the pen.
   26. tangotiger Posted: October 08, 2002 at 12:53 AM (#606618)
disingenuous: Not straightforward or candid; insincere or calculating

MGL, let's be fair about what I said. You said, "who cares if runs add up or not", and my point is that a containment system is a check that whatever methodology you are creating adheres to some basic principles. You can't create a defensive system where every player is above average, and yet the team overall is below average fielding. You can't say that the Mariners "should" have won 105 games, when they actually won 116, etc, etc.

As well, any adjustments you make to an individual player's stat line will and should be scrutinized. Just because a guy hit 20 HR at the Astrodome doesn't mean he'll hit 40 at Coors, even if the league average players hits 10/20 respectively.

Earlier, you said "value-added stats serve no useful purpose". However, they serve to credit an estimate of value to a player's actual performance, while taking the context into account.

My position is that there are two questions to answer: past-looking / accountability / value and forward-looking / ability. You and Clay Davenport and others are clearly in the latter camp, while Bill James and others are clearly in the former camp. Me? I use both.

And for those in the "ability" camp, you will always "undervalue" Eck and Mo, since, by definition, you are assuming that those players receive "average" high-leverage situations.
   27. tangotiger Posted: October 08, 2002 at 12:53 AM (#606625)
Basil, excellent analysis!
   28. MGL Posted: October 08, 2002 at 12:53 AM (#606628)
I have a problem with assigning .75 runs to the pitcher (or .9, as other posters suggested) for runners left on third base. Suppose that relievers allow inherited runners to score from third base 75% of the time. Then the author's scheme of assigning the starter .75 runs if that runner scores only makes sense superficially. Suppose that the starter leaves a game with a runner on third 12 times over the course of a season. If all goes according to average, 9 of those runners will score. It would be perfectly appropriate to charge the starter with 9 runs, which is exactly what ERA will do. However, under the author's scheme, the pitcher leaving the runner would be charged with .75 * 9 runners, or 6.75 runs. The relievers, despite average performance, would be assessed 2.25 additional runs to their ERA. In other words, they are punished for entering the game with a runner on third.

Exactly why you HAVE TO use the number of baserunners left (and the outs and which bases, if you want), regardless of whether they (the baserunners) score or not! The "problems" associated with fact that the starters' RAA runs will not add up to actual runs scored pale in comparison to the problem you clearly point out above (understating the starters' responsibity and overstating (unfairly penalizing) the relievers' run responsibilty).

BTW, how do you format (like italics) these darn posts?

You must be Registered and Logged In to post comments.

 

 

<< Back to main

BBTF Partner

Support BBTF

donate

Thanks to
Phil Birnbaum
for his generous support.

Bookmarks

You must be logged in to view your Bookmarks.

Syndicate

Page rendered in 0.4250 seconds
63 querie(s) executed