This article would have been very much new and enlightening, in 2005.

I was hoping that this would be an article about actual, you know, analysis. The recent string of Mike Fast articles are amazing, I think we can all agree, and this is part of the beauty of pitch f/ and hit f/x. But scouting (as in the Morrow example) is just something which is much more difficult for an amateur to do, so it's no surprise that that sort of thing frequently gets brushed under the rug. Unmeasurable = doesn't matter seems like a pretty useful rule of thumb to me, unless the competitors in your fantasy baseball league are pro scouts.

I don't necessarily disagree with some of the points made in this article, but the tone and style is off-putting. I think he exaggerates the views of those who are serious about baseball statistical analysis. In a way, it sets up a straw man argument. Sure, you can find some fan bloggers or random commenters who ascribe every point of a pitcher's deviation from league average BABIP to "luck." But those kinds of opinions don't accurately describe the main stream views in sabermetrics.

Unmeasurable = doesn't matter seems like a pretty useful rule of thumb to me, unless the competitors in your fantasy baseball league are pro scouts.

If you're trying to get an edge over your opposition (whether that be in fantasy or real baseball), and you can figure out something that they can't because you have access to qualitative information that they don't, why would you toss that information away just because you can't quantify it? What Mike Fast's articles demonstrate is that given the right set of tools you *can* quantify some things that were previously thought to be unquantifiable.

And despite what RJ is suggesting above, there is still a fairly large body of people who continue to act as though "unmeasurable = doesn't matter".

-- MWE

Well, if we're going to be pedantic ...

"Unmeasurable" does mean "doesn't matter." "Unmeasurable" means that nobody can recognize it when they see it. A scout saying "this kid's got moxie" is a measurement. To say that moxie is unmeasurable is to say that anybody's guess is as good as mine which means "measures" of moxie are truly random and useless.

What MWE seems to want to say is that some people act as though "unquantifiable = doesn't matter." To the extent that's true, that's just laziness on the part of the analyst (at least those who work for teams). There's nothing to prevent an analyst from entering scouts' opinions into their model ... or their own opinions ... of BA's prospect rankings ... or Tango's "wisdom of the mobs" stuff. Those things might have to be crudely measured butj, at worst, it's a simple matter of entering some dummy variables into the equations. Seems a perfectly sensible piece of evidence by which to assess the quality of a team's scouts for that matter.

As to DIPS and BABIP, I can't imagine how "unmeasurable" or "unquantifiable" even enters into the discussion. I can see how measurement error enters into the discussion but that's a very different thing. And unless you can show that this measurement error is somehow introducing bias (i.e. some pitchers benefit from it consistently, others don't), you treat it as random which means that (depending on your purpose) it's mainly introducing uncertainty, not (well) bias.

So then what is really meant is "unmeasured" which is quite different than unmeasurable or unquantifiable. But this is a straw man and a copout. You can point to any statistical or experimental result in the world and claim that something relevant is unmeasured. This is meaningless unless you can offer a theory as to what is unmeasured. And over these last few years, it seems to take no more than a few months before person A suggests "we should look at X" before somebody has proposed a way to look at X.

Now it is at least a somewhat fair point to make that external analysts don't have access to as much of the qualitative or other proprietary data that a team analyst does so external analysts shouldn't get overly cocky in their claims as to what doesn't matter when, for them, those things are unmeasured.

This particular article also makes the fundamental flaw of thinking that the important DIPS claim was that BABIP is random (this is largely Voros' own fault IMHO). The important DIPS claim was that, after controlling for past K, BB and HR rates, past BABIP did not help in predicting future ERA. While this has been tweaked somewhat, it's my understanding that it's still largely true. This also seems true of the Fast article that is linked -- Fast lists 8 points about BABIP, all of which talk about variation in BABIP across pitchers, not at all about BABIP's usefulness for predicting future performance.

For those who aren't too familiar with statistical modelling, the basic notion is this -- BABIP (or any other variable) is only "relevant" in a particular equation* if it helps you better predict the outcome _after controlling for all the other stuff in the model_. To the extent that low "true" BABIP is correlated with good K-rates, low BB-rates and/or low HR-rates, BABIP might not help you predict future performance whatsoever. JC Bradbury (who isn't always the most careful analyst and there were limitations to the sample he chose) found that was the case. (I assume others have looked at this since and may have different findings.)

Did a quick read-through of the linked Fast articles. Too much info to take in without some real effort and I'm not gonna do that (feel free). But something missing from his analysis is the ball-strike count of that particular pitch. He brushes on it by looking at the correlation of speed off the bat and K-rates. The correlation is not huge but it is negative (-.44) -- though, in this sample, K-rate does not predict BABIP. The negative correlation is not surprising -- high K pitchers are more likely to pitch in favorable counts. Batters may swing more protectively early and are more likely to reach for tough-to-hit pitches. An interesting question is whether Livan Hernandez is hit harder when the count is 0-2 than Clayton Kershaw is.

Fast does show (graphically) a correlation between speed off the bat and HR-rate. The hardest-hit balls result in hits a LOT ... and they result in HRs a lot. HR rate will pick up a good chunk of the impact of speed off the bat.

Back to the original article, it's not clear to me that Fast has discovered much of anything that wasn't in some sense already known. Over the years, many adjustments have been made to the "claim" that BABIP is random. The spread of "true" BABIP for pitchers that he finds is about 285 to 315 which I think is about where it's generally estimated (maybe it's been more 290-310) and it's not clear (by my quick reading) what adjustment he's made for defense or park effects. It might well provide a better, certainly more detailed, explanation for why we find BABIP variation among pitchers but I'm not sure that it impacts on our ability to predict or our evaluation of true talent. I suppose it might help distinguish between the "lucky" BABIPs and the ones more likely to sustain by using speed off the bat.

* Not being relevant in a particular equation doesn't mean it's not important. A particular variable might play a key role earlier in the "causal" chain and/or the equation might be specified in a way that it's not testing what it would seem it was testing. See my standard rant against models of runs regressed on BA, OBP and SLG. Speed off the bat may prove to be similar -- i.e. HR rate may be dependent on speed off the bat similar to how OBP and SLG are dependent on BA. You would have to be quite careful in your interpretation if you put them all in one equation as is. For example, what is HR-rate after controlling for speed off the bat? That might be the impact of angle off the bat, not "HR rate" per se.

What Mike Fast's articles demonstrate is that given the right set of tools you *can* quantify some things that were previously thought to be unquantifiable.

I don't think anybody (or rather, most people) thought that things like catcher defense or quality of contact were unquantifiable in theory, just that they were unquantifiable with the tools that we had access to in 2005. At that time, it would have been useless and probably counter-productive for me (with no discernable scouting skill, and no access to pitch f/x, hit f/x, etc) to attempt to supplement my stat-based approach to a fantasy draft with my half-baked scouting skills.

I'm not claiming that MLB teams in 2005 shouldn't have had scouts -- obviously there is likely huge value to them in proprietary, difficult-to-replicate data like scouting reports. I'm saying that Baseball America and its equivalents were the only way that I could get access to non-quantifiable details like speed off the bat, etc, and so for me as a fantasy baseball player, I had to rely on stats, e.g. xFIP at Hardball Times and the like. And in analyzing and discussing stats like xFIP, what are analysts supposed to do? It seems like you would be satisfied as long as they made more obeisances to uncertainty -- which is all fine and good, and always deserves mentioning (Nate Silver's been on a real kick about this lately at fivethirtyeight), but useless to anybody trying to decide how good a player is likely to be in 2006.

I basically agree with Walt's post (and acknowledge my linguistic murkiness -- I did indeed mean "unmeasured", or alternatively, "unmeasurable" in a practical sense by me the couch-sitter), especially the following paragraphs:

So then what is really meant is "unmeasured" which is quite different than unmeasurable or unquantifiable. But this is a straw man and a copout. You can point to any statistical or experimental result in the world and claim that something relevant is unmeasured. This is meaningless unless you can offer a theory as to what is unmeasured. And over these last few years, it seems to take no more than a few months before person A suggests "we should look at X" before somebody has proposed a way to look at X.

Now it is at least a somewhat fair point to make that external analysts don't have access to as much of the qualitative or other proprietary data that a team analyst does so external analysts shouldn't get overly cocky in their claims as to what doesn't matter when, for them, those things are unmeasured.

I think this is more-or-less the right approach: if you want to claim a stat is ######## because it's missing a lot of things -- well, tell us what it's missing and how we can go get it. Otherwise, the critique boils down to "don't take xFIP too literally", which is a fair point but pretty obvious and uninteresting.