Or maybe he'd be Goose Gossage, 1976.

Very good analysis, although the statement that chaining doesn't matter because you're comparing to the same baseline is not quite correct. As he points out, there are two deltas: from starter to first sub, then sub to replacement. It may be the case that, for some positions, the "first sub" tends to be closer to replacement level in production. (I'm thinking of the utility infielder replacing a starting 2nd baseman - e.g., Polanco to Infante/Perez - where the backup's value is in fielding/positional flexibility, and their bat may be at or near replacement level*. It's reasonable to expect that there's a fill-in for a 1B/DH that can hit above replacement level.)

I haven't looked at the data more generally, so YMMV. And Gassko does mention that the different is likely immaterial, not that it doesn't exist. I'm just splitting hairs.

* Disclaimer - "at or near replacement level" may not apply to Neifi.

Wow, this is excellent, primey material. I don't see how anyone could disagree with it, but I've never seen this type of analysis before (although I'm sure some of you will correct me on that.)

From the article:

So in other words, a great closer, one who posts an ERA that is half of the league average, is equivalent to a starter who is just 5% better than average!

Papelbon should be in the starting rotation. And in discussing replacement-level, we should pay more attention to chaining.

There are two problems with this. One, I'm not sure it isn't more likely that Papelbon can put up a 200 ERA+ in relief than a 105 ERA+ as a starter. Two, even if that is the case, considering the Sox' unsettled bullpen (their fault, giving away Bradford, Riske, and Meredith for nothing since last fall) they might this offseason be able to buy pitchers who comboed with Papelbon in the pen are better than pitchers comboed with Papelbon in the rotation.

Lastly, I'd like to see some sim studies before I buy the "value" = IP*leverage*ERA-above-replacement calculation for relievers. Has anyone seen/done this?

Of course, its a heck of a lot easier to find 105 ERA+ starting pitchers than 200 ERA+ plus relievers.

Lastly, I'd like to see some sim studies before I buy the "value" = IP*leverage*ERA-above-replacement calculation for relievers. Has anyone seen/done this?

***

How would a simulation help in any way, shape, or form here? This equation is a mathematical fact. 1+1 = 2, Runs over Replacement = (RA - ReplRA)*LI*IP/9.

Lastly, I'd like to see some sim studies before I buy the "value" = IP*leverage*ERA-above-replacement calculation for relievers. Has anyone seen/done this?

I don't know, but I admit I was thinking much the same thing when reading this. I thought this was great work, but to nitpick, I don't think it's correct to refer to a pitcher as being X runs above replacement after applying a leverage correction. A pitcher with 30 "RAR" isn't 30 runs above replacement; what he really means is the pitcher's win value was equivalent to a pitcher who is 30 RAR with average leverage.

Which got me thinking about Bill James' relief pitcher sims, where he calculated leverage but expressed it differently; he didn't center it around 1, but instead he expressed it in terms of runs/win.

I don't see how anyone could disagree with it

It's not a question of disagreeing with the model so much as applying the model to actual teams.

Across a lot of teams I suspect this model is pretty accurate in its conclusions, at least in general terms.

But on any given team, the dropoff between 'ace' and 'top setup' reliever can vary a lot and that changes the break-even point a good bit. On the 2006 Red Sox, it varies from Papelbon (an unrepeatably low 0.92 ERA) to Timlin (4.26 ERA). With that kind of spread, there's little chance Papelbon the starter will make up the difference...but if you assume a more normal dropoff, I think he very well could.

So the decision for an individual team depends a lot on their specific expectations for their bullpen, because the RA performance levels assumed in the model are, of course, not guaranteed for any particular team in any particular year.

Of course, its a heck of a lot easier to find 105 ERA+ starting pitchers than 200 ERA+ plus relievers.

It certainly is. Even the Red Sox have one.

DSG, this is a much better summary of my distrust of leverage and WXRL than I could have ever hoped to put together. I was trying to find the words for this in an AL MVP thread a few weeks ago. Thank you for making the argument so succinctly.

Which got me thinking about Bill James' relief pitcher sims, where he calculated leverage but expressed it differently; he didn't center it around 1, but instead he expressed it in terms of runs/win.

Same difference. I'm expressing it in terms of runs, he did it in terms of wins. It's a matter of preference, I guess.

But on any given team, the dropoff between 'ace' and 'top setup' reliever can vary a lot and that changes the break-even point a good bit. On the 2006 Red Sox, it varies from Papelbon (an unrepeatably low 0.92 ERA) to Timlin (4.26 ERA). With that kind of spread, there's little chance Papelbon the starter will make up the difference...but if you assume a more normal dropoff, I think he very well could.

Can't disagree with that. Of course, Timlin is probably better than 4.26 going forward and if he isn't, Hansen almost certainly is. Meanwhile, Papelbon isn't as good as 0.92. If the Red Sox closer next year will have a 4.00 RA, then Papelbon has to post a 4.40 RA (4.00 ERA) to be as valuable as a starter as he is as a closer.

I wish that Don Malcolm's terms wafering and rewafering caught on.

Or maybe he'd be Goose Gossage, 1976.

Well, he'll have Curt Schilling to advise him on how to make the transition; whether he wants his advice or not.

So far this article looks intriging, but I'll have to bookmark it for this evening.

DSG, this is a much better summary of my distrust of leverage and WXRL than I could have ever hoped to put together. I was trying to find the words for this in an AL MVP thread a few weeks ago. Thank you for making the argument so succinctly.

***

Just to be clear, I like both stats. LI tells us exactly how much more valuable a given reliever's innings were and WPA tells us just how valuable a player was. But note the past tense. Just because a player had a lot of value does not mean that the value is not replaceable. It just happens to be that it is easier to replace a closer's value than it is a starter's or a position player's.

This is good work, but I'd like to see the question posed differently. That is, rather than "what is Papelbon's value over a theorized replacement level given a theorized quality of other pitchers in the bullpen?" I would ask, what different arrangements of talent in the bullpen and what theorized replacement levels make it a good idea to used Papelbon in the bullpen or rotation?

I disagree strongly that "a great closer, one who posts an ERA that is half of the league average, is equivalent to a starter who is just 5% better than average." That is only true under the specific set of assumptions of "an average bullpen" and other values that DSG uses.

So, what if the team's bullpen features more 6.00 pitchers, or we drop or raise replacement level, or the arrangement of ERAs is staggered rather than ascending evenly? Or what if we use probabilities instead of set ERAs? Given different assumptions, what does the system say?

Given different assumptions, what does the system say?

Good question. David wrote really wrote an article about Silver's methodology, not the ultimate Sox's bullpen solution (though he gave his opinion at the end). An article like the one you're suggesting would probably posit a series of questions/assumptions, first of which would include something about the specific pitcher's strengths and weaknesses such as pitch selection, mental attitude, ability to warm up quickly, etc.

You'd probably also include questions about the managers' use of bullpen, the quality of the team's offense (impacts number of close games likely to be played, and ultimately LI) and some other things I'm not thinking of, and then ask who the specific alternatives are in the rotation and bullpen.

At least, that's how I'd approach it.

>hen a player goes down, his replacement in the lineup is generally a better than replacement-level player.

Maybe someone can explain this in words, too.

What the hell is the meaning of "replacement" level if replacement players are above it?

Given different assumptions, what does the system say?

Yes, you could do the exercise with an inferior bullpen, play around with LI assumptions, etc., but none of that would change the conclusion about Papelbon because it seems very likely he would be much better than a 105 ERA+ starter. The only reason to keep him in the pen is because you believe he would break down as a starter.

Of course, its a heck of a lot easier to find 105 ERA+ starting pitchers than 200 ERA+ plus relievers.

Not as much as you'd think. There are only 19 AL qualified starters at 105 ERA+ or better right now (21 if you include Weaver and Liriano), about 1.5 per team. It's a very scarce talent. And, of course, being more scarce does not necessarily mean more valuable.

Replacement level means "Freely available". AAAA guys. Often there is a pinch hitter or super sub on the team that is better than "freely available" that will take the place of the injured star. A replacement level player is brought up to cover for the super sub. This is how teams work for the most part, but not how replacement level value works... hence the need for this article and bringing up the concept of chaining. Doug of course brings up the point that this is not true for all positions, and in general is not true for all teams.

Last year Thome's actual VORP was probably negative as the guy who replaced him, Howard, was even better. For this reason, this is not a good way to evaluate players, but only a metric to evaluate a player on a defined team.

I'm sure all of us have thought about this in some way or another in our fantasy leagues when trying to value players.

Of course, its a heck of a lot easier to find 105 ERA+ starting pitchers than 200 ERA+ plus relievers.

Not as much as you'd think. There are only 19 AL qualified starters at 105 ERA+ or better right now

There are currently 20 "non-qualified" pitchers whose ERA+ is over 200; take out Weaver, Liriano, and 9 others who won't get near 50 IP, and that leaves 9.

This year, at least, it's twice as easy to find your 105 ERA+ starter.

I would ask, what different arrangements of talent in the bullpen and what theorized replacement levels make it a good idea to used Papelbon in the bullpen or rotation?

I have a spreadsheet that calculates this. If somebody has a place where it can be posted publicly, happy to send this over.

DSG, I think you make an excellent point with regards to chaining, but I think ultimately that in fixing one aspect of Nate's analysis, you have overlooked a number of even more important points. In other words, although this analysis might likely be more accurate than Nate's, I'm not sure it practically helps improve the decision the Red Sox have to make. Specifically, the Red Sox question entails a number of key issues, many of which other posters have already mentioned:
1) Given the specific talent on hand or likely to be available in the offseason, what are the possible arrangements of the rotation and bullpen?
2) What is the increase in workload expected to be borne by the remaining relivers if Papelbon moves to the rotation? In other words, if you remove option 1 from the bullpen, it does not necessarily mean that Option 2->Option 1, etc. It means that, if a talent of Papelbon's level is not found, then Option 2 moves to Papelbon's role, but with more innings, Option 3 moves to Option 2's role, but with more innings, etc.
3) What does the increased workload do to the expected performance and injury risk of remaining bullpen guys, and what effect would injury have on the remaining staff?
4) What effect will various roles have on Papelbon's expected performance?

Points 1 and 4 above seem to be outside the scope of the question you are asking (i.e. this is not a treatise on the Red Sox offseason moves, not Papelbon's ultimate projection in the rotation vs. bullpen), but questions 2 and 3 do need to be addressed within the analysis. Further, both of these questions seem to indicate a value of a good closer that is not currently accounted for in your model.

This year, at least, it's twice as easy to find your 105 ERA+ starter.

***

That's because the average starter is MUCH better than the average reliever, by almost half-a-run, if not more.

GuyM Posted: September 18, 2006 at 04:52 PM (#2180925)

There are only 19 AL qualified starters at 105 ERA+ or better right now (21 if you include Weaver and Liriano), about 1.5 per team. It's a very scarce talent. And, of course, being more scarce does not necessarily mean more valuable.

AND there is only 40 AL pitchers who have even pitched 149 innings.

looks like teams lucky to have the same TWO starters who even go at least 5 innings every 5 days
min - 3
tor - 2
cle - 4
det - 4
la - 3
ny - 3
bal - 3
chi - 5
bos - 2
tx - 2
sea - 5
KC - 1
tb - zero

There are only 19 AL qualified starters at 105 ERA+ or better right now (21 if you include Weaver and Liriano)

Kazmir, Wakefield, Burnett, Hendrickson, and Sowers all have at least 12 starts this year, but are "un-qualified". Plus, there's Rich Harden. All have ERA+ of over 105 this year (and you would expect all of them except Hendrickson to in any given year).

The question is

60*(Papelbon - MAX(Delcarmen, Hansen))

greater than 200* (Papelbon - MAX(Clement, Gabbard, CheapFreeAgent)).

It seems likely that it won't be (if we get a prime free agent, Paps replaces the projected #4 starter, not the #5). If our next best option for closer was say Timlin I'd suggest we had to keep Paps at closer, but it seems likely that either Delcarmen or Hansen will have improved enough by next year to be a pretty good closer, though probably not quite as good as Paps has been this year.

And some guys who currently are better than 105+ WON'T do it next year. The point is, league-average ERA starters who actually deliver the innings are not easy to come by.

Yes, 200+ relievers probably are more scarce. But that doesn't necessarily make them more valuable. David's analysis (and I agree) suggests they are not.

#21: DSG's analysis assumes that, when the #1 relief option is replaced, each other role moves up to take a spot of higher importance on the ladder.

So the #2 guy goes from 80 IP of a 1.2 leverage index (LI), to 80 higher-leverage innings. The #3 guy takes the 80 IP of 1.2 LI that the #2 guy used to have.

DSG has assumed in the article that the workload for each role does not change, just that the pitcher filling each role is somewhat worse. That is, he assumes that usage patterns do not change, just that different pitchers are filling each role. You may have other assumptions, and that would change the calculations/practical effects.

Per the general question of practical application to the Red Sox:

Let's say you assume the Sox could find an alternative starter with a RA of 5.50 (i.e., somewhat better than replacement-level). Let's further say that Paps will have a 2.50 RA, and the 5 other Sox bullpen types will have: 4.0, 4.0, 4.5, 4.5, 5.0. Dropping Paps from the 'pen for a replacement (5.0 RA) reliever costs 30 runs, using DSG's assumptions.

In this case, Papelbon would need a 4.14 RA (roughly 120 RA+) as a starter to merit the switch.

Feel free to make your own assumptions about what the alternatives look like. Basically, your #2 and #3 bullpen options have to be pretty darn bad to make the switch unappealing.

or what is harder to get

a starter with a 105 ERA+ who pitches at least 5 innings in at least 30 starts

OR a closer

I have a spreadsheet that calculates this. If somebody has a place where it can be posted publicly, happy to send this over.

***

I've made my own and it can accessed here: http://www.hardballtimes.com/images/uploads/Runs_Above_Replacement_for_a_Reliever.xls

You can mess around with the pink cells, but don't touch the yellow ones. At the bottom (E23, E24, and E25), the spreadsheet calculates the runs above replacement for the reliever(s) you have replaced, the wins above replacement, and the dollar value. So you can see that an average closer is worth about $5 million. A great closer is worth around $7 million. A great setup man is worth $4 million. And etc.

Sure, a starter pitching 200 innings at or above league average has a lot of value, even relative to the best relievers in the game, but that is because actually getting through 200 innings is a difficult chore. The lack of qualified pitchers only srengthens that argument. If Papelbon is having shoulder trouble now, then projecting 200 healthy innings next year seems like a stretch- heck, 80 relief innings would be tough enough. I don't feel like doing that math, but he would have to be substantially better in 160 innings, 120 innings, etc, than he is in 200 IP. I think a big part of the decision for the Sox will have to be whether they think the rotation or the bullpen would be more conducive to their young stud's health rather than his productivity, if the two can be separated at all. In a related issue, does anyone know how good of a comp John Smoltz would be in terms of shoulder stress and starter-reliever transitions? They aren't exactly similar pitchers, but there aren't too many data points from which we can choose here.

Another big question, as Andrew notes, is the assumption that Papelbon will be able to throw 200 IP at all. He's never topped about 160 and only done that once. I think 180 would be a better general assumption and maybe a bit generous in Pap's case.

DougG: Sorry, I probably didn't explain myself well. My point was that DSG does assume that each pitcher moves up one spot. However, this is not the practical reality of how the bullpen would actually work. If Papelbon is replaced with a replacement level pitcher, the manager will not keep the same 80 IP for each pitcher. The best pitchers will be asked to carry more load (as this would tend to reduce the impact of a replacement level pitcher in the bullpen). This increase in workload will have an impact both on the performance of the relievers and on the likelihood of injury.

I realize I am getting into a number of "soft" factors (i.e. difficult to quantify things), but my point is that I feel this analysis will systematically underrate the importance of good relievers. The reasons being (and adding some factors to my previous list):
1) Increased workload on remaining "trustworthy" pitchers
2) Increased risk of injury or underperformance due to increased workload
3) Valuable game time spent in "reoptimizing" reliver usage patterns (i.e. which relievers can be counted on in which situations, what bullpen configuration gives the team the best chance of winning)
4) "Psychic" benefit to relievers and team of established roles.

Another big question, as Andrew notes, is the assumption that Papelbon will be able to throw 200 IP at all. He's never topped about 160 and only done that once. I think 180 would be a better general assumption and maybe a bit generous in Pap's case.

***

ERAs he would need to reach to be as valuable as a starter as he is as a closer:

200 IP = 4.36
180 IP = 4.24
160 IP = 4.09

I've updated the spreadsheet to show you the RA and ERA a starter would need to put up to be equivalent to a given reliever, and you can change the IP levels to whatever you'd like: http://www.hardballtimes.com/images/uploads/Runs_Above_Replacement_for_a_Reliever2.xls

I can't conclusively tell you I've simulated the idea through, but it became obvvious to me while playing a sim game (Scoresheet) that this was true, and it has affected my draft strategy (particularly when playing with 'known' stats like a diamond mind or scoresheet players-of-the-past league). An elite closer (ERA+ maybe 200) is worth more than an average closer (ERA+ of 160?) by his runs saved times the closer leverage index (whatever that might be, say 1.7). But an average closer is only worth more than a decent set-up man by runs saved times a smaller index. So, it only pays to draft the few elites early, and everyone else can be picked up later. Later than most all of your starting pitchers.

AMBA: Agreed with that. To the extent that a manager (let's hypothetically call him Joe Torre) overworks certain trusted relievers to the detriment of their health and/or effectiveness, that's another consideration. You could play around with the IP assumptions, RA assumptions, etc. to model that, but it could potentially be a factor.

Not taking usage into account (since it is so uncertain), do certain types of pitchers profile for a greater probability of catastrophic injury in one role or the other?

Very good article and analysis by David. My guess is that the most important aspect in terms of deciding whether ANY given pitcher should close (or be used in any relief situation) or start is how the team thinks the pitcher will adapt to starting/closing and what the team thinks the performance differential will be, given the pitcher's style, pitches, health, etc. While the average difference between relieving (short, of course) and starting is around a run, that probably varies a lot depending on the pitcher. You have to give the Red Sox, or any other team, credit, in terms of being able to figure out what that differential might be.

The Red Sox want to protect Papelbon's arm but Papelbon's arsenal and makeup seem well-suited to high-pressure relief. How about making him a modified relief ace. He pitches 40-45 games, and goes ~one time through the order each time, resulting in about 110 IP. Sometimes he comes in the 7th and finishes the game, other times he comes into a jam a little earlier and turns it over to the traditional closer.

The paucity of appearances allows him to never pitch back to back, usually even getting 2 days of rest. The short stints still allow him to go all out and he rarely if ever faces a batter twice. If you can manage to get him an LI of 1.45 (halfway between closer and #1 setup), he's more valuable than a traditional closer (44 Runs to 38 Runs, per David's spreadsheet).

What MGL said. It's interesting to talk about the theoretical difference in value but by far the most important thing to consider is

1) the difference in performance between starting and reliever for any specific pitcher and
2) the difference in the likelihood of injury for any specific pitcher

It seems to me that this will vary a lot from pitcher to pitcher which precludes a standardized response to this kind of question.

It seems to me that this will vary a lot from pitcher to pitcher which precludes a standardized response to this kind of question.

***

I'm not sure it will vary "a lot," but certainly I'm just providing a baseline here. Beyond that, you have to apply any adjustments that are necessary.