great post. good clear explanation for why the numbers have to be wrong.

Very well done. Agree 100%

BP's WARP1 are a joke, but so are Win Shares. I find both publicly available uberstat systems tell you far less than you would know just by combining OPS+, PA, position, and say Zone Rating. As for where to stick replacement level, isn't that an empirical question that Nate Silver (of the very same BP) answered with his research on Freely Available Talent (FAT)? I don't know what win% that translates to, as I don't think he looked at pitchers, but it wouldn't be hard to calculate.

I'd be interested to hear thoughts on how I've calculated replacement level historically. I took Silver's FAT levels at each position, adjusted them for each league-season, and saw how many games an otherwise average team would win with 700 PA from a player with that production. Then I took an average of those theoretical team wins over the 42 league-seasons he considered (1985-2005). I then looked at the production of the worst three regulars at each position in each league-season and averaged those over the 42, and thus derived relationships between replacement level and worst-three-regulars average (eg, replacement 1B were 0.3 wins worse than the average of the worst 3 regular 1B per 700 PA). Then, to work backwards, I simply take a 7-year moving average of the worst three regulars at each league-position, and add on this gap between replacement and worst-regular average.

How does that sound to all you statheads wiser than I am?

FWIW, this reminded me of an exchange from a 2004 Prospectus Chat:

deadmonkeypaw (Chicago): I've read some criticisms of WARP3 that say that judging a player against both a defensive replacement level and an offensive replacement level is double counting. According to this claim, a player who is both a replacement level hitter and fielder would never get even close to the field. Do you have any sort of response to that?

Clay Davenport: It certainly is not double-counting; it just means that my overall replacement level is lower, on the order of a AA player rather than AAA.

And since there are numerous players who finish below even this extra-low replacement level, picking up a negative WARP-3 (the Orioles have 13 such players right now, although admittedly most have hardly played), the claim that they would never get close to the field is demonstrably false.

I asked Clay about this a while back, and he was kind enough to reply. I don't think he'll mind if I reproduce it here:

There are at least two reasons, Kostya.

One, the replacement level I'm using is lower than what you are citing, on the order of 25 wins for a team/season. Think Cleveland Spiders, 1899.

Two, and more importantly, WARP numbers for individuals don't add up to the same values that I would get if I evaluated them from their team numbers. Consider the 2004 Orioles. This team actually went 78-84. Since the replacement level win% is .153, their actual wins above replacement is 78-162*.153=53.2. Call it 53.

The sum of their individuals is 81, a big difference to be sure from the desired 53. But consider the team totals of batting rar (217), fielding rar (164) and pitching rar (381). These numbers have been adjusted to the standardized league of 9.0 runs per game, pythagorean exponent of 2. In the standard, a replacement level team scores 537 runs per 162 (replacement level EQA of .230, divided by league average of .260, raised to 2.5 power to convert it to runs, times 162 times 4.5 runs per team per game = 537) and allows 1262 (replacement pitching and fielding together yield a standard 7.79 replacement ERA, times 162 games = 1262).

A team that scores 537 and allows 1262 has a Pythagorean win pct of .153. The Orioles adjust to 537+217 brar=754 runs scored and 1262-164 frar-381 prar=717 runs allowed. That gives them Pyth% of .525 (85.1 wins), which makes them (.525-.153)*162=60 wins above replacement.

So we estimate the team as being 60 warp, and in real life they had 53 warp. That's not so bad, considering that they underperformed their real-world Pythagorean estimate by about 4 wins, and they allowed a few more runs than expected based on their pitching statistics.

The main point of that was to show that while the individual WARPs added together to 81, the team WARP only came to 60. There is a diminishing returns principle in the system, which follows from the mathematics of the Pythagorean model. A given number of extra runs will increase the win total of a bad team more than it will a good team. When Tejada or Mora or Lopez have their WARP calculated, they are adding their statistics to a replacement level team (simplifying; I actually calculate a team as average but for one replacement level player, compared to average but for this player) After you add Tejada, the team isn't replacement level anymore, and you don't get as many wins from from Mora as before. Now that you have Tejada and Mora, the team is even farther from replacement level, and Lopez' contributions don't count as much. That, in a nutshell, is the problem. The team environment does make a difference on how many wins a player's performance creates, but I've adjusted that context out.

Perhaps not a complete explanation, but it does shed some light as to the thinking.

I don't understand people who call stats "wrong". So are saves or wins or RBI "wrong" because they don't correlate perfectly with value? They might be "misleading" or "not perfectly useful for evaluating players", but calling a stat "wrong" is kind of going overboard. Just whenever we look at WARP1, we should just know that it rewards playing time a lot more than other metrics would.

And since there are numerous players who finish below even this extra-low replacement level, picking up a negative WARP-3 (the Orioles have 13 such players right now, although admittedly most have hardly played), the claim that they would never get close to the field is demonstrably false.

If Albert Pujols was injured 14 plate appearances into a season, there is a significant risk that he would rate as below replacement level during those 14 plate appearances. Take fringe players and the risk of them putting up below replacement level numbers in extremely limited playing time is obviously much higher. Clay Davenport cannot possibly be too dense to have realized this.

So in other words (and feel free to jump in if I'm wrong), when Tejada is listed at 8 WARP1 or whatever, that means that he, if placed onto a team of otherwise replacement-level players, would increase that team's win total by 8 wins. Is that right?

---

And since there are numerous players who finish below even this extra-low replacement level, picking up a negative WARP-3 (the Orioles have 13 such players right now, although admittedly most have hardly played), the claim that they would never get close to the field is demonstrably false.

that's ridiculous - players who put up negative warp3 are probably not worse than AA players (if that's what he thinks a true talent 0 WARP3 player is). he knows this. why bring up that example?

So in other words (and feel free to jump in if I'm wrong), when Tejada is listed at 8 WARP1 or whatever, that means that he, if placed onto a team of otherwise replacement-level players, would increase that team's win total by 8 wins. Is that right?

No, according to this parenthetical quoted in post 6:

(simplifying; I actually calculate a team as average but for one replacement level player, compared to average but for this player)

One thought that went through my mind reading this is that the Royals pitching staff this year was a pretty good definition of replaceable talent.

Working through the math, the Royals' ERA was 5.65 against a league ERA of 4.5. If the team's batters had scored 4.5*(4.5/5.65)=3.58 runs per 9 innings, the pythagorean winning percentage would be .286, or just about Tango's replacement level.

So if you ever need a mental picture of replacement level, think about the 2006 Royals pitching staff.

A team that scored 3.58 runs per game would score about 580 runs over the year. The worst team in the league (Devil Rays) beat that by 109 runs.

The most interesting thing to me there is the reminder that there's a diminishing returns issue with WARP. Amongst other things, that would mean that using a straight WARP1 in an MVP discussion (for example) divorces the player from their actual team context. Off the top of my head, this would seem to benefit players from good teams, or at least deeper teams, the most.

I thought this had been widely known. Roughly speaking, take about two wins off of WARP 3 to get something like what MGL would get "above replacement" ... but MGL usually reports above average, so take off about 4 wins.

Somewhere in the archives, there's an interesting discussion on these and related issues. I don't remember who, but someone made the pretty good argument that it's best to measure against average (since this is known) and if you want a "true zero" type measure then use true zero and just look at raw runs created (where the "true zero" is known). The latter is easier for hitters I suppose. True zero for pitchers would be???

that's ridiculous - players who put up negative warp3 are probably not worse than AA players (if that's what he thinks a true talent 0 WARP3 player is). he knows this. why bring up that example?

Well, they don't have to be worse (in true talent) than a AA player to make his point. Some people say that players as bad as BPro's "replacement level" would never make the majors. He is (or might be) pointing out that the fact that some players produce negative WARP suggests that players at or just above their "replacement level" do indeed make the majors.

Will Baseball Prospectus get their rabid lawyers going on this one?

I know all about the 1899 Spiders, and the recent Tigers.

Geez...lose one lousy playoff game, and suddenly you're the worst team of all time!

True zero for pitchers would be???

Pitching Runs Created

The Hardball Times tracks them in our stats section:

=1"]AL PRC leaders

Anybody want to compare VORP? How is it different than WARP? Is it better?

players at or just above their "replacement level" do indeed make the majors

All you have to do is look at the Rangers pitching staff to notice that.

players at or just above their "replacement level" do indeed make the majors

All you have to do is look at the Rangers pitching staff to notice that.

Or the Cubs Collage of AAA Pitchers.

Also, I have to add that the problem with WARP is not just a preference of baseline. Any study done with WARP is invalidated by the system, especially an economic one (i.e., Nate Silver's otherwise spectacular chapter on what A-Rod is worth in Baseball Between the Numbers) because the replacement baseline is set too low, which lowers the value of a marginal win.

Think about it this way, for example. Let's say we have two different baselines; one, in which a replacement level hitter contributes 3.3 "wins" (i.e. Win Shares) and another in which a replacement level player contributes 1.3 "wins." Now let's take an average team that spends $48 million on their hitters. If we value their hitters based on the higher (and essentially correct) replacement level, we'll find that a win above replacement is worth around $2.25 million (because the team is paying about $45 million marginal dollars and getting around 20 marginal wins). If we use the lower baseline, one win above replacement turns out to be worth $1.2 million. So how do you know what the correct amount to pay for Alex Rodriguez is? Well, if he's worth say 11 "wins," under the first model he ends up being worth (11 - 3.3)*2.25 + .3 = $17.6 million. Under the second, (11 - 1.3)*1.2 + .3 = $11.9 million.

If the first replacement level is correct, any model that uses the latter will be wrong, and if it sets its replacement level as low as WARP does, by quite a bit.

That's what happens in Nate Silver's chapter. And that's why it is imperative to use the correct replacement level.

Well, they don't have to be worse (in true talent) than a AA player to make his point. Some people say that players as bad as BPro's "replacement level" would never make the majors. He is (or might be) pointing out that the fact that some players produce negative WARP suggests that players at or just above their "replacement level" do indeed make the majors.

Walt, I still don't think that we can conclude that players as bad as BPro's replacement level make the majors just based on the fact that some players put up negative WARP seasons. Again, just because a player puts up negative WARP doesn't mean his true talent is negative WARP, any more than Adrian Beltre putting up 11.7 WARP in 2004 means he's an 11-WARP true talent player.

Try finding players who have more than a few hundred at bats or BFP and have negative WARP. I haven't found any, and I've plumbed the depths of the worst players in history (Ray Oyler, Mario Mendoza, Andres Thomas, Mike Ryan, Rich Morales, Jim Mason).

Furthermore, we know that players of 0-warp level do make the majors, so this is a silly argument anyway- teams frequently dig into AA pitching staffs for emergency starters or bullpen help, for example.

The Astros won 82 games, which is pretty much what their RS/RA numbers would have expected. 82 minus 58.5 is 23.5 wins. 23.5 / 162 = .145. Another perennial .500 team I like is the Seattle Mariners. Their team WARP-1 is 52.1, and they won 78. Their RS/RA would have expected around that as well. 78 minus 52.1 = 25.9 wins. 25.9/162=.160. The Yanks won 97, which is also around their pythag record. Team WARP-1 is 71.9. 97 minus 71.9 = 25.1 wins, or a .155 record.

I think there's a problem with methodology here.

By adding up a team's WARP, you're doublecouting the defensive runs above replacement level since run prevention is likely going to be credited to both pitchers and fielders.

By adding up a team's WARP, you're doublecouting the defensive runs above replacement level since run prevention is likely going to be credited to both pitchers and fielders.

If that's the case (and I don't know whether it is), it is a failing of WARP. They have to add up. If the reason they don't add up that way is due to marginal utility, that's reasonable. But double-counting defensive credit isn't.