So basically.... Your calling people stupid if they don't read The Book for the reason that it's too complicated. Sounds reasonable.

You RTFA (or at least TFExcerpt) and came away with that?

Good read. I feel the same way about most of these things, though I do lean more heavily toward the type of analysis in "The Book." However, it is a lot more fun to read James, Poz, and others.

I will say that one of the things that makes this discussion frustrating is that unlike so many other research areas, every casual sports fan has an opinion about what's the right way and the wrong way to play baseball, and that's before you get into the rabid sports radio/sports blog arena. In a field like quantum physics or (my field) computer science, the public doesn't give a #### about the implications of research.

That's not to say that there aren't heated discussions, but Joe Schmoe on the street isn't going to start yelling at you about how garbage collected programming languages are superior to older alternatives if you posit that the benefits of deterministic resource management outweigh the hassle of manual resource allocation. You can't say the same thing about Derek Jeter's defense. Everyone who likes baseball, no matter how much they watch or how much they research, thinks they know something about that.

I think that's why people who want to research baseball find the situation so frustrating. Physicists and programmers don't have radio shows the regularly make sport of them and daily newspaper columns calling them eggheads and nerds. If this were any other field, these people (and others) would be acknowledged as experts; in sports, everyone's an expert.

So basically.... Your calling people stupid if they don't read The Book for the reason that it's too complicated. Sounds reasonable.

What? No, how is he saying anything like that?

In a field like quantum physics or (my field) computer science, the public doesn't give a #### about the implications of research.

That's not to say that there aren't heated discussions, but Joe Schmoe on the street isn't going to start yelling at you about how garbage collected programming languages are superior to older alternatives if you posit that the benefits of deterministic resource management outweigh the hassle of manual resource allocation.

Well, Joe Schmoe on the street has no idea what garbage collection is. Joe Schmoe on the street has idea what a constructor or a destructor is. He does not know what free() does.

And has no interest in any of that stuff. If Joe Schmoe ever gets interested in programming, because he wants to maybe dabble in writing games, who knows?

I think the articles excerpt is extremely unfair. To me they are basically saying that Mark Twain is a better expert of his times and culture then say some well regarded historical sociologist because Twain writes better stories. I think he completely misses the point that while Bill may be an enjoyable read the amount you will learn from him will be small compared to what you read from the experts. Bill was and is still always full of hunches and assumptions, his work is mostly opinions sprinkled with some data. Sure it reads better but it doesn't really help increase the level of what we know about baseball.

I think that's why people who want to research baseball find the situation so frustrating. Physicists and programmers don't have radio shows the regularly make sport of them and daily newspaper columns calling them eggheads and nerds. If this were any other field, these people (and others) would be acknowledged as experts; in sports, everyone's an expert.

I don't find it frustrating for that reason. I find it frustrating because the "experts" in this field are all too often self-anointed. Researchers in those other fields are subject to fairly rigorous peer-review processes and vetting before they become "experts" - something that we don't have in this field. Not that I'm saying we should have that level of discipline, mind you, but we certainly need more than we have now.

-- MWE

To me they are basically saying that Mark Twain is a better expert of his times and culture then say some well regarded historical sociologist because Twain writes better stories.

Well, Twain's going to do a much better at communicating with readers about his time and culture because he's a better writer -- that's what the MetsGeek post seems to be saying. Though why the writer neglected to make mention of what might be the best "middle man" in stat-centric circles -- that BP site some folks here have a beef with -- seems likes a very strange omission.

Bill was and is still always full of hunches and assumptions, his work is mostly opinions sprinkled with some data.

And how does that differ from The Book? Or Baseball Prospectus? Or anything I've ever written, for that matter?

Most researchers start with a hunch or an assumption - called a "hypothesis" or a "theory". The idea is to look at the data and see how well the data supports the theory. James does that with less formally and more intuitively than Tango/MGL/et al, or BPros, or someone like me, and his methods often aren't statistically sound, but he doesn't really care that much about how he gets to an answer, as long as the answer "sounds logical".

-- MWE

I think the articles excerpt is extremely unfair. To me they are basically saying that Mark Twain is a better expert of his times and culture then say some well regarded historical sociologist because Twain writes better stories. I think he completely misses the point that while Bill may be an enjoyable read the amount you will learn from him will be small compared to what you read from the experts.

I think what he's saying (or if he isn't, I'm saying it), is that while the historical sociologist is going to do a better job of researching and processing information on late 19th century culture, people being people are way more likely to read and be influenced by Twain because he is a good and entertaining writer.

Same with baseball. It's pretty clear that the stat-fu of James, Posnanski, FJM, whoever is relatively weaker than others (or non existent in some cases) but because they're entertaining and engaging writers, far more people are going to read and respond to them.

Ultimately this boils down to the question of whether it's better to have a lot more casual worshipers in the congregation or a smaller but purer and more devoted flock.

Edit: The Essex Snead beat me to it.

Rick Peterson should have never written that article.

why the writer neglected to make mention of what might be the best "middle man" in stat-centric circles -- that BP site some folks here have a beef with -- seems likes a very strange omission.

Most of BP's best writing - and writers - are behind the premium wall.

-- MWE

Don't know that I truly agree with you Mike. There's no formal peer review structure to be sure. But nobody becomes credible without showing their work. As such it gets meaningful review.

Yeah, of course 90%(+) of the the review process is at the "your rong" level. And there are a lot of dung throwing howler monkeys involved.

But anything important will get a fair review, and if you're interested you learn who does the meaningful work.

I think he completely misses the point that while Bill may be an enjoyable read the amount you will learn from him will be small compared to what you read from the experts. Bill was and is still always full of hunches and assumptions, his work is mostly opinions sprinkled with some data.

Different strokes for different strokes, I suppose, but I think this reduces James' contributions to an absurd level.

I don't find it frustrating for that reason. I find it frustrating because the "experts" in this field are all too often self-anointed. Researchers in those other fields are subject to fairly rigorous peer-review processes and vetting before they become "experts" - something that we don't have in this field. Not that I'm saying we should have that level of discipline, mind you, but we certainly need more than we have now.

This is certainly true. I guess that's sort of my point. Because there is no process that determines who is and is not a baseball "expert," anyone and everyone can claim to be one. Thus, those people, whoever they are, who really are trying a rigorous, intellectually honest approach to baseball analysis are easily drowned out by the scores of people who love baseball enough to be dangerous. In any other field, the people attempting the rigorous approach would have the processes you mention in place. Baseball is, in this respect, a victim of its own awesomeness.

But nobody becomes credible without showing their work. As such it gets meaningful review.

How did PECOTA become credible? What sort of "meaningful review" did it have? PECOTA's credible mostly because it comes from Baseball Prospectus; we accept it more or less uncritically because of the source.

How did Win Shares become credible? Who provided any sort of "meaningful review" on Win Shares before it was released? James showed it to his inner circle of friends, and did a presentation on it at a SABR convention right before the book was published, but all of the meaningful reviews on Win Shares (a) occurred AFTER the book was published, and (b) were buried in the rush to wide acceptance. Win Shares is credible mostly because it comes from James.

Mostly, you become credible in this environment not because of your work, but because you have a champion. James became credible largely because of Peter Gethers. BPros became credible largely because of Neyer (and to a lesser extent, ESPN as a whole). There is a lot of good work out there that has yet to surface in the mainstream because the researcher doesn't have a champion.

-- MWE

Marchman and Megdal would top my list of "middle ground" writers (plus they get props for being in my fantasy baseball league). I'm not sure where I would fit--my pieces can get technical at times and I do in fact develop my own statistics, but I write for a general audience. So rather than a middle ground, I suppose I'd be more of an ambassador for the stat geeks.

Mike, PECOTA became credible because it has the lowest RMSE of the various projection systems.

One thing I have tried to do in my column is promote the work of somewhat lesser-known sabermetricians, although I probably do rely excessively on BP/Nate Silver/PECOTA for forecasting purposes.

How did PECOTA become credible? What sort of "meaningful review" did it have? PECOTA's credible mostly because it comes from Baseball Prospectus; we accept it more or less uncritically because of the source.

I don't think that's entirely true. It certainly may have been in the beginning, but the reason people take it seriously now is that they do reasonably well in the projection business.

Most of BP's best writing - and writers - are behind the premium wall.

Well, yeah. To be honest, I was thinking of the days when BP still didn't have any walls (which might be farther back than I'd care to admit). That, combined w/ Rob Neyer's (also free) work on ESPN, was what got me interested in stat-centric baseball stuff. And as much as that interest was piqued by the theories and ideas bandied about, I can say for damn sure that (as a layman, and a noob) it was the actual prose -- the way these folks shared their findings, or others' findngs -- that drew me in and kept me around.

As for the PECOTA / BPro argument: I thought that PECOTA made Baseball Prospectus what it was, not the other way around. And I thought Win Shares was all but shown to be flawed?

As for the PECOTA / BPro argument: I thought that PECOTA made Baseball Prospectus what it was, not the other way around. And I thought Win Shares was all but shown to be flawed?

I found a Baseball Prospectus annual in the bookstore years before PECOTA. This was even before I was aware that there was a group of folks that talked baseball online. I thought that it was cool. Finally, here's a successor to the Abstracts that I read in high school. I wish that I came across the Big Bad Baseball Annual instead, but I think those weren't widely circulated.

I blogged about this article over at ShysterBall, and a commenter by the name of Roger Moore just made this comment:

The point that many hard-core stats people seem to miss is that you can get most of the benefits of serious analysis with a relatively small effort. You don't need to go to some convoluted formula like EqAvg or the technical form of Runs Created to get a pretty good understanding of an offense. 90% of the benefit comes from moving from BA/HR/RBI to BA/OBP/SLG and remembering to consider park effects.

The difference between BA/OBP/SLG and the technical formulas is at the level of noise. As far as I can tell, the huge number of hypercomplex formulas are mostly a result of pissing contests between their advocates. Everyone wants to prove that his formula is a tiny bit more accurate than the next guy's, or to mark his sabermetric territory by using a different formula. It winds up making things unnecessarily confusing and intimidating to the uniformed without adding any real benefit.

Thoughts?

Edit in order to expound a bit: I don't know if I can agree with him about the motives of people doing the analysis (I have never seen the kind of territorialism and pettiness he implies, even if it does exist), but I do agree that for the vast majority of people, a basic understanding of OBP/SLG and contexts (both park effects and era adjustments) is more than enough.

How did Win Shares become credible?

Personally, I don't see it as credible, but I cede the larger point, which is that it is more likely to be seen as credible simply because of its creator.

I do think that we should be careful not to overstate the differences between the rigorous academic research environment and the current state of baseball research. Certainly, there are more checks and balances on the academic side, but in any research situation, one's status in the community or the people and institutions "championing" particular research will influence the way the debate is framed and where attention is focused. No one will read my (imaginary) paper on quantum physics because I'm not a physicist and I don't have a university "championing" my cause. This is a good thing, because I know nothing about quantum physics. With respect the James and Baseball Prospectus, they receive more attention because they have already demonstrated baseball knowledge, which makes people more receptive to future ideas. I'm not saying this should spare them from criticism, but it isn't totally misguided to focus your attention on people and institutions that have already demonstrated the capacity for good research.

So basically.... Your calling people stupid if they don't read The Book for the reason that it's too complicated. Sounds reasonable.

Well, I call people stupid if they don't know the difference between "Your" and "You're."

90% of the benefit comes from moving from BA/HR/RBI to BA/OBP/SLG and remembering to consider park effects.

I think this strongly depends on what your goals are. At the risk of making far too glib an analogy, you can get 90% of the benefit of a physics education by studying only Newton. You'll have a really damn good idea of how just about everything works. That doesn't mean that there aren't better models out there. Sometimes you need to dig deeper.

you can get 90% of the benefit of a physics education by studying only Newton.

easier said than done

90% of the benefit comes from moving from BA/HR/RBI to BA/OBP/SLG and remembering to consider park effects.


I think this strongly depends on what your goals are. At the risk of making far too glib an analogy, you can get 90% of the benefit of a physics education by studying only Newton. You'll have a really damn good idea of how just about everything works. That doesn't mean that there aren't better models out there. Sometimes you need to dig deeper.

I would agree with this. The way I've thought of it is that you get pretty much all you need in terms of understanding what has happened by looking at BA/OBP/SLG and including park (and league) effects. However, if your goal is to project, to predict what a player will do next, there is a lot more data needed. In fact, to project really accurately, it may not be possible to get enough data. But to be sure, that first order evaluation isn't going to be enough. But I'm certainly no expert on baseball analysis, though I enjoy reading about it and thinking amateurishly about it a great deal.

I thought it was best said the other day: it's one thing to be ignorant (or glib) about hyper-precise statistical models if you're just a fan, but if you're a GM or baseball ops guy, that's pretty much your entire job.

EDIT: And if you are a professional baseball writer, you are probably somewhere closer to the GM than the fan, in terms of your obligations to be aware of, digest, and relate more exact information than a fan.

How did PECOTA become credible? What sort of "meaningful review" did it have? PECOTA's credible mostly because it comes from Baseball Prospectus; we accept it more or less uncritically because of the source.

For that matter, has any proprietary metric been meaningfully peer reviewed? I seem to remember MGL telling us that UZR had been peer reviewed by a few people (including Tango and Dolphin) but that he would not lay out the bare bones of the system to all of us. Neither has Dewan or James (regarding +/-) for that matter.

Peer review seems to become a lot less important very quickly when you're trying to sell something. The problem is that the end result is someone like Hughes2.50 waiving insanely on the street corner and making unsubstantiated claims between bothering passersby for change and asking for bites from their sandwich. That's not such a problem on the interweb, but it's also not terribly acceptable in most other endeavors.

For that matter, has any proprietary metric been meaningfully peer reviewed?

I'm not sure what makes a metric proprietary or not, but I think one can argue that DIPS was peer-reveiewed. Voros posted his work, others did independent studies of the data, and they critiqued his findings. Voros then adjusted his metric so as to account for lefties and knuckleballers. Does that fit?

Wow, it's great to see Roger Moore participating in this discussion, even if it's just tangentially.

One thing that James avoids that a lot of other researchers succumb to is falling in love with their own numbers. James almost always treats even his own proprietary stats skeptically.

you can get 90% of the benefit of a physics education by studying only Newton.

By study, do you mean read the Principia Mathematica? Because the book is only comprehensible if you work backwards, and already know what it is he wants to say. With James, by contrast, you don't have to read any academic baseball studies, you can just start with him.

A better analogy might be Darwin. The Origin of the Species is very well written, intelligible, and can give you about 90% of what you need to know about evolution.

I think a lot of this depends on the purpose of one's research. In fields like economics, there will always be a difference between academic research and applied research, for example. There is a difference between the economist who win a Nobel prize for abstract theory and the corporate employed economist who has to solve a specific problem with the information at hand. It doesn't mean one is better than other--they just have different purposes.

I have learned the hard way over the years that many times the least sophisticated model and the more basic descriptive statistics are the best in laying out a solution to a question or problem. The more sophisticated approach may be more self-satisfying, but it also may not add enough information to justify the complexity it adds.

As for Bill James' writing, what is wrong with making what you say readable and accessible?

nd how does that differ from The Book? Or Baseball Prospectus? Or anything I've ever written, for that matter?


I would say it differs greatly. Generally speaking Bill James is much more likely to present assumptions or opinion with with almost no data to support it or extremely simple and limited data to back it up. I didn't see that happening in the Book.

Different strokes for different strokes, I suppose, but I think this reduces James' contributions to an absurd level

Not really. James contribution to baseball analysis the last decade or so has been minimal. To me Bill James is the Cap Anson of the stathead world. A great player who played during a primitive era and whose greatness/popularity helped turn a primitive game into something more sophisticated.

I seem to remember MGL telling us that UZR had been peer reviewed by a few people (including Tango and Dolphin) but that he would not lay out the bare bones of the system to all of us.

That is false. UZR was explained in detail, right here on Primer. You can see the links on my blog.

If you had access to the STATS data, you could reproduce that version of UZR.

UZR has undergone several changes since then, but the basic form is the same. All the PBP systems follow the same form: try to figure out the probability of an out (or run value) given a particular ball that is hit, by establishing the characteristics of that hit ball (identity of the pitcher, batter, park, the count, weather, game state, hardness of hit, location, trajectory, fielder alignment, etc).

PECOTA on the other hand IS a black box. The characteristics that it uses have been described, but not enough that you could recreate it on your own (unlike the original version of UZR). In any case, since PECOTA is barely better than Marcel, the credibility of any forecasting system is tied in to its marketing.

I’ve never read a Baseball Abstract. I’ve never even seen one.

But I can tell you, anyhow: I'd rather see than be one.

I blogged about this article over at ShysterBall, and a commenter by the name of Roger Moore just made this comment:

Wow, that guy tears sabermetricians down like, umm, Roger Moore.

I agree that it's frustrating that James doesn't keep up with the research. It would be really neat if he were fully up to date on it; took a step back and diagnosed the state of the field; decided "ok, we understand X pretty well now, we need to work on Y"; and directed his research accordingly. But obviously his methodology is not so goal-directed, and he is the type of person that even if he just randomly explores whatever is interesting him at that moment, the end result is going to be something worthwhile. I'm signed up for Bill James Online (can we start calling him "William"? Yes, I know William James was an old writer, but "BJ Online" sounds incredibly wrong... although I understand Eliot Spitzer visited it a lot... I'll stop now), and it's silly to say that it's just stuff you can look up on Retrosheet. Forget about MGL's writing being inaccessible... huge spreadsheets of raw numbers are really inaccessible. There is a lot of value in pulling the data and getting it into a user-friendly format. And the articles aren't mostly the type of studies MGL and others do about run scoring, batting orders, converting defense into runs, etc.; a lot of them are pretty abstract meditations about the nature of sports (not just baseball), the relation of the fan to the game, obscure baseball history, etc. So they're not even "competing," really. The site is still obviously very much a work in progress, and has huge strides to take both in terms of the data being easily navigable, and in terms of just looking like it was designed within the last 10 years. But it certainly proves to me that James still has a lot to offer.

This was an eloquent, even-handed article, BTW.

Getting back to John Lynch's post in #3, I think the biggest reason the discussion is frustrating is not that sabermetric researchers lack formal credentials which the mainstream find credible, but that they're studying the work of human beings ... specifically, athletic human beings who in many ways are seen as symbols of masculinity. It's basically a jocks/nerds kind of thing.

And I agree with bunyon in my agreement with Craig: Once you understand the primacy of on-base/slugging, the differences in offensive expectation among the defensive positions, and that young players are more valuable (for a multitude of reasons) than older players, then going into the more advanced stuff carries increasingly diminishing returns.

Happy Base Ball

This thread makes me happy about my position in the internet sabermetric world - I'm just well known enough that I'm asked for an occasional quote or radio thing or people say nice stuff enough about me, but not well known enough to have a subset of people care enough to bother disliking me.

I'm not sure what makes a metric proprietary or not,

I just meant stuff that people are trying to sell/make $ on. UZR (when MGL was working with the Cardinals), +/-, and most of BP's stuff is what I had in mind. For the most part things that are designed for "us" seem to depend much more on the quality of the writing, whereas something aimed for sale to the teams themselves really needs to be much more rigorous.

That is false. UZR was explained in detail, right here on Primer. You can see the links on my blog.

Thanks Tango, I thought there was a good chance it might have come out and I missed it.

Happy Base Ball

The truest thing in this thread thus far!

but not well known enough to have a subset of people care enough to bother disliking me.

Wanna bet?

Naw, I'm just kidding. Everybody I know (among baseball geeks only) loves checking out Zips and Transaction Oracle. It gets me through the offseason. Thanks for all the effort!

Wanna bet?

Well, arkitekton and RETARDO dislike me, but that's mainly a personal thing rather than a "mean ol' proprietary stat guy" thing.

Well, arkitekton and RETARDO dislike me, but that's mainly a personal thing rather than a "mean ol' proprietary stat guy" thing.

That's got to be on a very short list of things you shouldn't worry about.

Honestly, the tattoo on my chest, when translated from the German, means "THE Szymborksi, THE!"

I prefer The Szymborski, The!

But you'll get yours from rakes.

Honestly, the tattoo on my chest, when translated from the German, means "THE Szymborksi, THE!"

Nobody who speaks German could be evil.

This thread makes me happy about my position in the internet sabermetric world - I'm just well known enough that I'm asked for an occasional quote or radio thing or people say nice stuff enough about me, but not well known enough to have a subset of people care enough to bother disliking me.

It's not that bad. The number of kind people far outweighs the haters.

My problem was I tended to let the relatively few people who were the latter get to me, while not fully appreciating the former. No longer.

Mickey and Tommy (I'm going to keep doing this until they tell me to stop) always struck me as two of the more intelligent and analytical people doing this sort of thing, and so it comes as no surprise to me that their book is considered to be outstanding work and also (to some anyway) a little dense to read.

We're pretty much all children of Bill James (whether we know it or not), but, like any child, it's silly for people to expect any of us to be close reproductions of our parents.

I will use the power of the Observer to turn Dan into Hillary Clinton.

Just wait until people find out he fired the entire ZIPS travel office.

I just want to point out that Tango took me to task here for calling his book "terribly written" when I had previously given a much more nuanced review.

Bill was and is still always full of hunches and assumptions, his work is mostly opinions sprinkled with some data.

This is absolute baloney. You ever hear of Runs Created? It still works just fine; ask Szym. Not to mention Offensive Winning Percentage, cERA, etc. etc. James is no math whiz, no, but he's an extremely competent numbers guy. No one can say for certain whether there would have been a Tango or MGL or BP without Bill James, but I'm comfortable saying it's unlikely any of these guys would have much of an audience if it hadn't been for James blazing the trail. What chaps these guys about Bill is that he's been reluctant to open a dialogue with them about all of the failings in his work that they have been so helpful to point out, and I can't say I really blame him for that.

Roger was one of the guys over at rsbb that I learned a lot from. With rsbb having pretty much both feet in the grave (unfortunately) it would be great to see him on some of the more populated internet places.

Maybe you need to start a website called NewRec.sports.baseball.com?

Then we could resurrect RLM and King Tut, and show people what a REAL troll looks like.

The Szymborski, The!

My tattoo artist couldn't spell very well, so mine came out "Teh Szymborski, Teh!

I popped into rsbb for old-time sakes a few months ago and Maynard was a-trolling as if there was no tomorrow. He really might live in his mom's basement.

One thing this article doesn't take into account enough: James works for the Red Sox now. I don't know what he is working on for them, but I would guess that it is pretty new and rigorous and takes advantage of the new technology that allows for better analyis of pitch-by-pitch/matchup data and fielding. As I noted when we discussed Gold Mine and Bill James Online, his stuff for sale to the public is by necessity more anecdotal. Anything he is doing that could create a real competitive advantage for a baseball organization--and that is what "cutting edge" ultimately means in sabermetrics--is property of the Boston Red Sox.

Another thing to remember: James did what he did in the 1970s and 1980s either with NO computers or primitive ones--a huge disadvantage, IMO. Yet, much of what he did is still accepted and is part of the standard ongoing discussion now. That, along with his ability to make it fun to read, which Tango, Dolphin and Lichtman absolutely do not have in spite of their obvious intellectual and research abilities--is why James, flaws and all, is such an important figure.

It is a much more enjoyable experience in reading what James writes at his worst, than what Tango (doing my Rickey) writes at his best. No one is arguing otherwise. I know who I'm reading first. I think we can concede that point and move on from there.

***

I've noted on my blog that James and Palmer are my inspirations, and they remain the two most important figures in sabermetrics.

While in some cases, you may think it's necessary to bring someone down to elevate someone else, in this case, it's not needed. Certainly none of us are going to be on 60 Minutes like he's about to be. Not unless one of us is about to be indicted on something.

***

James says he's excited about his online site because he can communicate with his readers in real-time. One would hope that he'd be as engaging as the rest of us bloggers, especially since he's limited his site to predominantly his fans.

but there’s a reason no one reads James Maxwell’s unedited manuscripts on electricity and magnetism

:

because Jeter is still active

because Jeter is still active

my brain keeps trying to forge a connection, but no luck yet

My tattoo artist couldn't spell very well, so mine came out "Teh Szymborski, Teh!

Spelling it wrong the second time is truly unforgivable.

I mean, doesn't twisting the left nipple do the same thing as F7 on a word processor?

Best Regards

John

However, if your goal is to project, to predict what a player will do next, there is a lot more data needed.

eh, not so much. Marcel works pretty darn well actually (weight the last 3 years 5/3/2, add a little age adjustment if you want to get fancy). So does a fancy internet poll of reasonable fans.

On offensive projections, the improvements have all been on the margin for sometime now. The biggest steps have been in terms of presentation -- adding confidence intervals, greater use of comps (though how much this actually helps is debatable I suspect), etc. I suspect the same is true of pitching projections -- look at Salfino's recent piece on the Mets staff, those numbers across systems are very close.

There are a lot of differences on the defensive side and those measures don't correlate particularly highly. This is the area where someone might make a real breakthrough.

But to me the biggest under-studied area is injuries and playing time (and to a lesser extent general decline), at least when it comes to longer-term projections. From a fan perspective, that's just kinda fun -- will David Ortiz make the HoF. But from a team perspective where they have to decide whether to offer a 34 year-old Ortiz a long-term contract in 2010 when his current one expires, it would be huge. PECOTA tries to get at this to an extent but it's not clear it's any good at it.

Mike, PECOTA became credible because it has the lowest RMSE of the various projection systems.

Over what time period? For what set of players? Based on what metric? Were the various systems normalized to the actual league context?

On the last point, comparing, for example, raw 2007 ZiPS projections with raw 2007 PECOTA projections in terms of how well they predicted actual 2007 performance is meaningless if the 2007 ZiPS projections assumed a different league context than the PECOTA projections did. How you would normalize is not clear but a simple way would be, if you were comparing based on OPS projections, to convert the projections to OPS+ in terms of their assumed league contexts, then convert each back to the actual league context. Of course these projections are also park-specific so for players who change teams anytime after the projection you're using, you need to adjust that as well.

There are lots of other questions about what populations you compare? All MLers over 200 PA? All MLers? All MLers weighted by PA? All players, minor and majors, using MLEs?

I'll be rather impressed if any comparison of the projection systems has done a good job of handling those issues.

Finally, as to marketing, champions, etc. the most interesting recent case was Dewan's Fielding Bible. Tons of mainstream sportswriters, guys who derided other sabermetric publications, took that thing on-board as if it were gospel. A lot of that was its clever title but I still don't really understand why it was so incredibly easily accepted relative to other sabermetric number-crunching.

Ack! Of course that simple method of normalizing won't work because OPS does not equal OPS+*lgOPS (park-adjusted). So you'd have to at least use OBP+ and SLG+. Alternatively, convert everything to context-neutral (park and league).

Another issue is around those park effects, to the extent that the different projection systems may calculate park effects slightly differently (in terms of projected park effects). Question is do you want to consider that part of the projection system or not?

I think PECOTA gets used a lot because it's accessible, not because it's shown itself to be so much superior than anyone else's projection system.

I'd be interested to see some post hoc evaluations of all these prediction systems. Just for grins, here's Diamond Mind's predictions for 2007. They got three of the four AL playoff teams right, which is good, but they had the World Champs in third place, missing the playoffs. And they went one-for-four on the NL playoff teams.

The Hardball Times was a little better, with three for four in the AL and two for four in the NL. That give them credit for the Cubs, whom the picked to win the Wild Card rather than the Central Division, but they get major props for picking the Diamondbacks in the West.

For someone who is poseur in these things, I find VORP a good enough tool to feel confident in any argument with my laymen baseball fan peers. It ranks players! Ranks them! There are people out there who think Ken Griffey Jr. was a better player in the 90's than Craig Biggio. I am superior knowing the VORPISH and win sharish truth. Of course, I don't think I know anything.

Walt, I'm analyzing all the forecasting systems, which you can see starting at post 30 on my blog

I like Bill James as a perceptive writer, and obviously as a pioneer. I don't like his extensive use of points systems. Too many of his systems seem to work because of his freakishly comprehensive knowledge of baseball history -- Win Shares has so many free parameters, I think of it more as a system perfectly designed to give you Bill James's view of a player's stat line, handily condensed to a single number. Anyone except Bill James should find some way of defining distances in units of runs, wins, pennants... hard numbers that can be put into other problems and tested for reasonability.

I'm very much of the school that says OPS+ tells you 75% of what you need to know. The difference in quality between offensive and defensive stats is so large that I don't trust hybrid systems like WARP3. In the future, I would like to see a "Win Shares Done Right" statistic that used play by play defensive data.

If you studied only Newton, and were as smart as Newton, you could do a lot. If you wanted to solve actual nontrivial problems, you would certainly want to use the discoveries and simplifications of the 19th century mathematical physicists... Lagrange, Hamilton, Gauss, Euler. Classical mechanics is a deep and interesting field that was more abandoned than completed when relativity and quantum came along. Quantum-classical correspondence is still an open field of study.

Ironically, much of quantum mechanics and relativity is easier in practice than classical mechanics. You can get a lot of mileage out of linear algebra and differential equations.

There are two kinds of peers: technical reviewers and subject matter experts (SME).

Sabermetric work requires both kinds of reviewers. Almost to a fault, a great majority of academic sabermetric papers are formally reviewed by fellow statisticians (with no SME), and a great majority of online sabermetric work are informally reviewed by SME (with some stats experts).

I was actually asked to do a peer review of an academic paper (as an SME), and the student practically dismissed all my criticisms. The online community is much more receptive of my criticisms. In general, the non-academic world puts out the best sabermetric work around.

As for MGL specifically, he does have some of the best stuff out there, and is an enormously helpful SME. He has enough statistical chops to get by, and there are enough statisticians who see his work to give him the necessary feedback.

In The Book, Andy Dolphin was our statistics expert (as well as being an SME). Certainly only an SME that could have done the Sac Bunt chapter that MGL did, one of the greatest pieces of sabermetric work that you will ever read. To analyze the sac bunt, the number 1 requirement, by far, is to be a baseball fan. You have to understand all the nuances of baseball.

I tend to side with Ron here, that the "Wisdom of the Crowd" essentially certifies the work to a good degree.

#60

But to me the biggest under-studied area is injuries and playing time (and to a lesser extent general decline), at least when it comes to longer-term projections. From a fan perspective, that's just kinda fun -- will David Ortiz make the HoF. But from a team perspective where they have to decide whether to offer a 34 year-old Ortiz a long-term contract in 2010 when his current one expires, it would be huge. PECOTA tries to get at this to an extent but it's not clear it's any good at it.

#68

There are two kinds of peers: technical reviewers and subject matter experts (SME).

That's the thing. Most of the statisticians and SME's in baseball, are not experts or even anywhere close to it on sports training, biomechanics, neuromechanics, sports science.

Also, until individual teams and individual players start revealing their training and practices regiments, or at least the general philosophies, their training and medical records, what if any restorative methodologies that they use, ie massage, sauna, cryotheraphy, hydrotheraphy, it really is pointless for an outsider to try to guess.

The quality of formal peer reviews is very hit and miss, and it's not clear that you would need such a system for as small a community as sabermetric research.

I think that there are lots of great results published in academic journals that are lost because they are poorly presented. James is one of those guys that, in academic research, people would say "oh, nothing he does is that advanced or original, he's only getting by on his name", when REALLY the reason that he "gets by" is because he's able to communicate those ideas to a wide audience. In research, you need both kinds of people: the technical wizards AND the prophets/salesmen/visionaries. There is some cross-over between the two groups (for example, James' quantitative limitations are only with respect to the very small percentage of people interested in sabermetrics who develop new tools -- even amongst more sabrmetric literate folks, he's in some small percentile as far as those skills go).

My old PhD supervisor once told me: "The most important thing in statistics is the idea. Not the software, not the proof, but the idea. If you can claim the idea, then it will be associated with you. Then you can let others refine it, substantiate it, and even implement it for a mass audience. But it's having the idea that is the tough part and the part that will make you famous. " And he definitely knew what he was talking about, because he's one of the most highly cited academic statisticians of the last 20 years.

Bill James is all about the big ideas. Now there are lots of people refining the ideas, proving that they work (and in some cases don't work) and are implementing them for mass audiences, but the important thing in methods research is still always the idea. That's what makes the widest impact. And my old supervisor definitely understood the importance of selling the idea. If you don't/can't sell the idea to others, you might as well have not even had it.

PInto doesn't. THe guy from Penn has a ways to go, and all of those don't even go back but about 7 years. SG has taken the ZR DRS back to 1987 - 20 years of pbp work that is really close to MGL. AROM's working on further back, but there's considerable peer-review to do on that.

That doesn't address Pinto nor SAFE.

SAFE is Shane Jensen. I said this about it

The fielding system approach I’ve been preaching

Whereby UZR and other systems use discrete zones, this approach uses a continuous function. It doesn’t look like they use as many parameters as MGL’s UZR (park, GB/FB tendency, base/out, etc), but nonetheless, they’ve got the basic framework down.

MGL's first words about the system:

The methodology is brilliant of course.

The author of the system stopped by and said:

Effect of Park:

As pointed out, this is one of the more obvious weaknesses of the current methodology, and the one that we are currently working on. There is a bit of tradeoff here, since fitting different bip densities for each park would make each bip density a lot more noisy. However, it is certainly worth it in at least a few cases, such as the effect of the green monster. I’m also excited to see if throwing in a turf vs. grass factor into our grounder model makes a difference.

We get into further discussions where we corrected him on stats of all things.

We've also had back-and-forth with David Pinto, who finally agreed with us to include distance in his model (he wasn't doing it the first couple of years).

UZR uses more parameters, so that's a plus for MGL. SAFE uses a continuous function, so that's a plus for Shane.

UZR, SAFE, PMR and Plus/Minus are all in the same ballpark. And I have no doubt that Pinto, Jensen, and MGL would agree that they are all neck-and-neck.

To think that UZR is way behind either of these models is to not make a fair opinion.

Yes, graphs are nice, but:
"So not only can you see that a given fielder has a flaw in his range, you can see why (he has trouble going to his left, for instance)."

Those things don't necessarily mean that. Which is part of the problem. People assign cause where it isn't proper to do so.

Sort of, kevin. People are a lot more sure of themselves with those graphs. And they are often wrong.

kevin, are you saying that the reason Pinto and Shane "crush" MGL is because of the visual representation, and not the underlying logic? I mean, MGL could if he wanted to give you the split data (and he has on occasion). I can't believe that's what makes it a "crush". In any case, UZR considers more parameters than the others.

But, Chris is right. The data itself already has an uncertainty level around the seasonal data for a player. If you break it down by left/at/right or other breakdowns, it'll make it that much more uncertain. A good reason not to show it is precisely for the reason that Chris is saying.

This is similar to the PECOTA percentiles. They've never been publicly proven, and yet, there they stand, looking nice and pretty.

As for the PECOTA / BPro argument: I thought that PECOTA made Baseball Prospectus what it was, not the other way around.

My recollection is that BP was around for a number of years before Silver joined them.

I subscribe to BP for two reasons:
1) to read Sheehan; and
2) to have access to their statistical data.

On Roger Moore's comments, I agree, and find OPS+ pretty good for most applications. The reason I sometimes use EqA is because it considers baserunning and (unless I'm mistaken) weights OBP and SLG more appropriately. So I think there is some real utility to using EqA above and beyond what OPS+ gives you. VORP also is great because it's a counting stat to supplement the rate stats, and I like that it considers defensive position -- but not quality of defense at that position.

I think the most interesting writing is by people (like Neyer) who choose interesting column ideas, have a fundamental grasp of the concepts, and sprinkle their assertions with statistics for support -- but don't flood the column with statistics (unless the column is more of a mini study).

As for James: I like his writing because it's a fun read and (usually) is logical. But I don't see how it would even possible for him to be on the cutting edge of serious baseball analysis unless he has kept up with the field.