Baseball Prospectus 2005

In general terms, Baseball Prospectuses, published annually, are books of comments about major and minor league ballplayers that to use statistical analysis as the foundation for the comments. In the 2005 edition, player comments occupy about 420 of the book’s 570 or so pages. Another 80 or so pages comprise small essays on the major league teams; the player comments for a team’s players follow that team’s essay. The remaining 70 or so pages contain a very skimpy introduction, a few small statistical essays, a list of Top Prospects (according to the Prospectus writers’ collective), and an index of players with the page number containing each player’s comment. There is no general index, nor a glossary of terms.

Unfortunately, Prospectus has a fatal flaw, which it has had since its inception in the 1990s and which it has not corrected to this day. The flaw really is fatal; it renders Prospectus all but useless to serious baseball analysts. Worse, the flaw all but ensures that whatever analysis Prospectus does will be out of date; the book will never be on the cutting edge as long as it has this flaw.

So, what is this fatal flaw? Baseball Prospectus does not publish its methods, nor the standards that its analysts use to evaluate them. As a consequence, the reader cannot know what the statistical material in Prospectus actually means, nor how to compare it to anyone else’s results.

The flaw renders Prospectus absolutely useless to the casual baseball fan. I tried the book out on a test audience of a couple of such, and they found it to be incomprehensible. Prospectus does not publish actual raw baseball stats for the players; instead, it uses stats that have been massaged by the writer/analysts. It does not explain what methods the analysts use, and the methods are not obvious. In the comments and essays themselves, the writers continually refer to other unexplained stat massaging methods and use a lot of insider terminology for which there is no glossary. The house writing style - individual writers for the player and team comments are not credited - relies heavily on cheap sarcasm, insider terminology, and very obscure analogies from outside of baseball. (The Elementals comic book?)

You fantasy fans won’t find anything here, either. Prospectus claims to be analyzing real baseball, not any of the fantasy simulations, so you don’t get any numbers that are directly useful, like you do in John Benson’s books. There are projections for players for the 2005 season, but they are unreliable, especially in estimating playing time for minor leaguers about to make the big leagues. Nick Swisher, the A’s top pick in the Moneyball draft, is predicted to have 327 at-bats in 2005, as a center fielder. Last I heard, the A’s plan to start Swisher in right field, in which case, 327 AB is too low. On the other hand, they assign 176 AB to Jeremy Brown, presumably as the A’s backup catcher for Jason Kendall. Kendall has never left that many AB for his backup. On top of that, Brown, who was hurt last year, will very very likely be the A’s starting catcher in AAA ball, to get experience. He has no chance of getting 176 AB as a late-season callup behind Kendall.

As for the effect of unrevealed methods and standards, here’s what I’m talking about, using the old Bill James Abstracts for comparison. Bill started out with a method for determining how many batting runs a batter had contributed to the team effort, by massaging the batter’s component stats, such as singles, homers, strikeouts, etc. The earliest, simplest version of this method amounted to On Base Percentage times Slugging Percentage times At Bats; Bill called this “Runs Created.” I know what the formula is - OBP x SLG x AB - because Bill published it in his books (Bill broke it down into smaller components, but his RC is algebraically equivalent to my version).

Bill also told his readers what standard he used to decide that this was the best Runs Created method he had devised.  He would take an entire team’s component stats for a season, use the formula to estimate the team’s runs scored for the year, and then compare his formula’s results to the number of runs that the team actually did score. He did this for a lot of team seasons and then computed the standard deviation, which is the normal statistical measure for how close a formula gets to the actual numbers. The smaller this standard deviation was, the better the Runs Created formula. That is, the standard was “smallest standard deviation of estimated to actual team season runs scored.” This, too, he printed in his books.

The result of this was that a reader could evaluate Bill’s work for himself. If you wanted to try to dream up a better RC formula than Bill’s, you knew what Bill’s formula was, and you knew what standard he used to determine accuracy. If your formula had a smaller standard deviation than Bill’s, you had a better formula. And most important of all, if you sent Bill a letter with your formula, and he checked your math and your standard deviation was indeed smaller, he’d print YOUR formula in HIS book, and use it!

It was this access to Bill’s methods, standards, and forum that actually produced the revolution in baseball analysis now commonly associated with the book Moneyball. That’s because the access empowered Bill’s entire readership, turning them into a real community of scholars. With Bill, you were on firm ground, and never thought that his work was opaque or that yours was futile or redundant. You understood Bill’s work, you could avoid re-inventing his wheels, and you had standards to evaluate your efforts. Many people participated, and the field improved its results by leaps and bounds.

Contrast this to Baseball Prospectus 2005. The book has a morass of methods, none of which are presented in full formula fashion, much less with standards for judging accuracy. There is something called VORP, which supposedly evaluates both batting and pitching (though, oddly, not defense). It claims to adjust for everything from the strengths of individual minor leagues to, for all I know, the phases of the moon. But how? There’s no formula. How accurately? There’s no standard.

So, how can you decide whether you think VORP is worth using? You can’t. How can you determine whether you have a more accurate method? You can’t. How can you write a letter demonstrating that you have a more accurate method that Prospectus can use next year? You can’t. You can’t even do simple things like decide if you agree with Prospectus on what the replacement rate is for injured major league players. VORP is based on the concept of replacement rate (it stands for Value Over Replacement Player), but Prospectus never tells you what they think the replacement rate might be.

And all that means that VORP is essentially useless to you as a reader. So is PECOTA, the Prospectus method for projecting players into the future. The formula(s) aren’t there, nor are the standards. The back cover of the book claims that PECOTA is “exclusive (and deadly accurate),” but only one of those claims - “exclusive” - is verifiable. PECOTA is certainly exclusive; no one outside of the Prospectus crowd knows how to work it. But accurate? By what standard? Who knows?

But worse, the exclusivity means that no one except the Prospectus guys can even try to improve on either VORP or PECOTA, nor any other exclusive Prospectus methods. There’s no robust community of scholarly readers helping out. So what happens? Prospectus’ methods fall further and further behind the openly published methods. Further and further behind the cutting edge. Out of date, in addition to useless.

The nadir is the Top Prospects section. Each player gets a little essay and a few stat lines. But the stat lines make no sense. For example, Dallas McPherson, prospect #4, gets, among others, stats labeled “AVG” - presumably batting average - for the years 2003 and 2004. These are .253 and .256, respectively. Well, the lowest batting average that McPherson accumulated in 2003 and 2004, at any stop at any level, was .308. Obviously, the Prospectus AVG stats are not McPherson’s raw batting averages. What are they? Who knows? Major League Equivalencies of some kind? Maybe, but what kind? What’s the formula? There is no explanation of the stat lines anywhere in the book.

McPherson, like the other prospects, also gets a line labeled “5-Year WARP Trend,” which lists five numbers. Again, what does that mean? Who knows? Again, there’s no explanation. As a veteran sabermetrician, I might guess that WARP stands for Wins Above Replacement Player, but does that help? No. There’s no definition in the book of what Prospectus believes a replacement player to be worth, much less how Prospectus actually calculates player Wins. Without a baseline, a list of wins becomes just another set of unverifiable numbers. Worse, there’s no indication of which five years the trend covers. 2003 through 2007? 2005 through 2009? Who knows? Prospectus doesn’t tell us. And the casual fan has no chance of guessing what WARP stands for, even if my guess is correct.

I should mention one more feature of Baseball Prospectus 2005: the constant mention of the Prospectus web site. Frequently, the web site is cited as the place to go for further information on the methods in the book. I did not check the web site out; I was assigned to review a book. It apparently is at least partially a for-pay site, as the book mentions “profits” from it, but there is also mention of free stuff. I have no idea if methods and standards are actually fully presented on the web site, or if they are on the free or pay portion.

I will, however, say this: If you have to go to the web site to make use of the book, then the book is, essentially, an advertisement for the web site. I don’t know about you, but I am not accustomed to paying for advertisements. I expect advertising to pay for things like TV programming, not the other way around. The book costs $17.95 this year. The web site’s url is printed on the back cover. You can copy it down for free. Why buy the useless advertisement?

In actual fact, you cannot really call Baseball Prospectus 2005 an “analysis” book at all, because readers cannot approach it as analysis. All you can do is take the Prospectus authors’ word for their results. You have to take them on faith. You have to take them as - religion. Word from on high that you are not allowed to verify. The peer group of books for Baseball Prospectus 2005 is not Bill James’ Abstracts, nor his more recent Win Shares, which spends page upon page carefully detailing its methods and standards. Nor is the peer group any of the other fine baseball analysis publications that have appeared over the years. No, the peer group for Prospectus comprises works like the Bible, the Koran, and the Vedas. Books whose pronouncements you have to take on faith. Books that, however many truly wonderful characteristics they may have, are useless for analyzing baseball.