Of-course, nate is assuming that clemens was 1.) Clean in 97, 2.) was on steroids in 1999.

Since the years mcnamee alleged clemens juiced are the only years everyone takes as gospel, why is nate ignoring the fact that in his 1999 season roger clemens was supposedly clean? That perhaps his drastic drop in performance in 99 was a result of not having mcnamee and winstrol around? I wonder what PECOTA would have predicted after 1999, a season where presumbly clemens wasn't juicing?

This analysis was skewed to favor clemens, what a shocker!

The only assumption is that he was clean in 1997. The article is a what if scenario, assuming he was clean the rest of his career, what would be the most probable scenario.

The hypothesis is that he is used steroids or something sometime after that, not the assumption. The hypothesis could be confirmed if his stats were significantly higher than what his projections would show over a non-trivial sample.

Now, his conclusion wasn't to prove or disprove something was fishy, he said so himself. The point is to show that the logical conclusion to a Hall of Fame career is pretty much a Hall of Fame decline.

"Now, his conclusion wasn't to prove or disprove something was fishy, he said so himself. The point is to show that the logical conclusion to a Hall of Fame career is pretty much a Hall of Fame decline."

A Hall of fame decline is something Hank Aaron had, which was aided by his homepark.

What happened with clemens starting in Toronto? Who the hell knows, but it doesn't seem "natural".

And, FTR, Assuming Clemens was taking steroids in 1999 is an assumption. We have no evidence that this is true, as we don't have evidence that he juiced in 97 (though I suspect he juiced both years).

Well, that's a false assumption.

I take it you have tangible evidence that he was juicing in 1997? I mean, it's certainly a questionable assumption, but who the hell are you to say categorically that it's a false one?

Kevin:

There is nothing objectionable to presume a "clean state" as a default without info otherwise.

Even though the person in question is objectionable.

I am less bothered by Clemens current discomforts than Bonds. BB never really pretended to be anything other than a guy who played baseball.

Anyway, that's my ha'penny on the issue.

Because there's no basis to assume he was clean in 1997.

Non-jackasses don't need a "basis" to assume someone is innocent. Everyone is, unless there's evidence of guilt.

#7, Nor is there basis to say he was already tainted. They could have tried projecting him from his age 25 year, but then, they wouldn't have a way to expect a HoF career from him.

However, in these exercises, everything after the cutoff point is split into a separate timeline. The point is to see if the actual events follow the logical conclusion if nothing changed. The PECOTAs are quite good at figuring out what kind of player someone is, much better than the Clemens Report and the "Anti-Clemens Report" did, to try to give as scientific an answer as projecting stats can give.

I'd be surprised if Clemens was clean in 1988, frankly.

Come on, people.

Clemens is a Texas school boy.

That means he has been roiding since junior high.

Well, since he did not get to Texas until he was 15, then high school.

What more do you need to know?

Oh right, people automatically believe someone's guilty when the media says so. No one believes to wait for the answer, they just take the first plausible answer they hear, regardless of accuracy.

I keep forgetting.

Oh right, people automatically believe someone's guilty when the media says so.

Where Clemens is concerned, who needs to wait for the media. I still don't understand why he isn't doing hard time for attempting to murder Piazza by shoving that bat through his torso.

This isn't really surprising at all.

I'm pretty sure that Clemens is lying. However, I don't think he got nearly the benefit from PEDs as Bonds or McGwire did.

As per BBref, there's a ridiculous jump in quality between Clemens' comparables by age after age 33 (career through 1996) and his age-comparables after age 34.

Comps through age 33 (6 retired)

W-L % Innings ERA+
185-113 .620 2693.1 121

Comps through age 34 (7 retired)

W-L % Innings ERA+
208-133 .610 3108 120

Comps through age 35 (7 retired)

W-L % Innings ERA+
227-143 .613 3405.4 122

After 1997, the average career IP of his comparables increased by 15%, with no decrease in career ERA+. Thus, I don't see 1997 as being a good year to run the projections for. Given the inherent time uncertainty of when (or if!) Clemens started using, the same procedure would give different answers after 1996, 1997, and 1998.

"I'm pretty sure that Clemens is lying. However, I don't think he got nearly the benefit from PEDs as Bonds or McGwire did."

Except that he did.

All that #16 proves is that Clemens outlasted most in his cohort. As they "die out" and he "continues on" he's going to be in an increasingly rare set of survivors in his initial cohort -- whether he juiced or not. As the PECOTA analysis points out, persons of his early caliber who "survive" to mid-late career in good shape are rare but Clemens is not unique among that elite set.

D'oh! Here's the career comparison I meant to make. (Statistics are now career totals for retired comparables)

After age 33:
W-L % IP ERA+
235-157 .599 3527.7 115

After age 34:
W-L % IP ERA+
259-180 .590 3978.6 116

After age 35:
W-L % IP ERA+
267-181 .597 4100.9 118

So after age 35, the quick & dirty career projection for career length increases by roughly 15% with no decrease in quality. I'd expect his Pecota comparables improved comparably.

But when you're comparing to a baseline established by a comparable cohort, it matters dramatically what that cohort is. If Silver had compared to PECOTA projections before Clemens went to Toronto, he would get a very different answer than he gives in this article. Since the time of alleged juicing is unknown, you should be very leery when the same method gives different results for different starting points.

I don't get the objections to this study. The accusation is by McNamee is that he began giving Clemens PEDs in 98. Nate's study shows that there is no statistical evidence that supports this claim.

If you think Clemens was juicing his whole career, then this is irrelevant, of course. But that's not what Nate is talking about.

One thing you (#20) should keep in mind is that Silver's "comparables" are not identified in the same way as B-Ref (and Bill James) chooses them. I haven't had time to think the implications through for your analysis, but keep in mind while B-Bef finds comparables by looking at the total career up to a given age, PECOTA finds comparables based on their performance only in the last three years, and further adjusts for a host of other factors (height, weight, speed, handedness, etc.). What this may mean is that your method is more susceptible to the attrition of a cohort over the career than Silver's method -- which also adjusts for body type.

Added edit: I also concur with Darren's point (#21).

"I don't get the objections to this study. The accusation is by McNamee is that he began giving Clemens PEDs in 98. Nate's study shows that there is no statistical evidence that supports this claim."

The accusations don't mention 1999, when clemens performance drastically dropped. Why doesnt nate take that into account?

FYI, Bonds tested positive for steroids. There is a baseball God.

The problem is, it's pretty much established that numerical analysis won't tell any useful information that will be statistically significant. We've seen two 'official' reports that contradict each other, and two unofficial reports here as well. Nate Silver's are probably as accurate as they're going to get.

The issue is that people are using PEDs as a witchhunt to chastise players who are 'too good', and this is just a means to it. People overperform and underperform all the time, and humans are quite good at looking for patterns where there are none, looking for a reason for each streak and slump.

So, whenever someone plays too good, is generally disliked for something, and tends to be paid a lot of money to act his own way, he attracts a lot of negative attention. So, people have all the reason in the world to accuse someone like Alex Rodriguez. Oh wait, they already have.

Regardless, performance enhancing drugs are such a vague topic that it's laughable. There's no issues for people having performance enhancing surgeries, such as Tommy John surgery - some are even having it done preemptively. People take cortizone shots, which slowly paralyzes the body in large quantities eventually, doesn't anyone think of the children about that?

And, if they suddenly decided to ban and test coffee, and other caffeinated products at workplaces, then, management suddenly decides to retroactively apply this policy and penalize people for having abused such a drug in the past, a lot of people would suddenly be up the creak without a paddle.

FYI, Bonds tested positive for steroids. There is a baseball God.

You do realize that this is the same claim made in the indictment, not some actual steroid test Bonds was given, right?

I don't get the objections to this study. The accusation is by McNamee is that he began giving Clemens PEDs in 98. Nate's study shows that there is no statistical evidence that supports this claim.

The study itself is fine as far as what it is - given Roger Clemens's career through 1997, his career through 2007 is not particularly surprising. I think the objections are to what this means. In order to be able to make any claims about "statistical evidence that supports [McNamee's] claim" you would need to know (1) the specific years when Roger Clemens used PEDs, (2) the specific years when Roger Clemens did not use PEDs, and (3) the statistical improvement one would expect for a pitcher who used PEDs. We don't actually know any of these things, so this study ends up not telling us anything at all.

I'm not at all sold on PECOTA as a projection system, to begin with. But any projection system that works by identifying comparable players is implicitly assuming that your target player will age in similar ways. Think of driftwood in a river. You want to project the point at which a particular stick will hit the shore by keeping careful track of where previous sticks that passed by the same place have come to shore in the past. But in order for that forecasting method to work, the population of comparable sticks must be constant as time goes by -- if the population of comparable previous sticks is changing with time, it means that your stick is caught in an eddy or the river's current has changed since you started keeping records. Since BPro has kindly decided to keep its entire hideous algorithm proprietary and hidden, I can only speculate about how it works in practice.

But any projection system that works by identifying comparable players is implicitly assuming that your target player will age in similar ways.

This actually raises another potential issue with using PECOTA to draw conclusions about whether a player used PEDs. If the comparable players in Clemens's projection used PEDs, then the fact that Clemens late career was similar to these guys could actually be evidence that he <u>did</u> use PEDs.

The point is, like the stock market, a player's results are essentially random and cannot be predicted in any way, shape or form. Statistically, if you accumulate enough of them, they will start to show useful data. Of course, you can never see the full population, and the population changes constantly.

This is why there is a technical maximum on the effectiveness of projection engines, because the number we're trying to project, and the information we can base it on is a sample, not the population. The effectiveness of projection engines fall under this maximum, but try to tend to it over time, as it's improved.

Granted, PECOTA isn't the perfect way to project. ZiPS isn't either, and it's just as proprietary. The only projection system that is truly open source is Marcels. There is no perfect projection engine, since we work with imperfect data. I doubt this is the issue. It's just the most sophisticated statistical study that has been performed on this data to date.

Some things just can't be statted, because we don't know them. We can't tell how exactly how good a fielder Ruth or Gehrig was because the only data is G, PO, A, E and DP. We can't tell exactly for some players when all data is available to us. We don't know who was affected, when they were affected, and how they were affected. We can at best use the information in front of us, and it's not sufficient, so we use what we have, and hope it fits.

If a system includes as much information as PECOTA claims but sees such a small improvement over Marcel, to my eyes the next step would be hard core triage to get all the information that wasn't improving the prediction out of the system. Every extraneous piece of information is a new opportunity for the system to screw up. It could be that Prospectus has done this triage and I haven't read about it, but the abundance of factors in PECOTA's system makes me like it less, not more.

But in order for that forecasting method to work, the population of comparable sticks must be constant as time goes by -- if the population of comparable previous sticks is changing with time, it means that your stick is caught in an eddy or the river's current has changed since you started keeping records.

The problem is that this line of reasoning invalidates neural networks, both the artificial kind and the kind that are in our brains and everything from weather forecasting to econometrics. Of course the model is going to change, the only question is how the model is going to change.

When you go to the grocery store, you're projecting what you want to eat over the course of the next week or two. The first time you go to a grocery store and shop for yourself, you might not do a good job of it, but after years, your mind has adjusted its model of proper grocery store shopping with knowledge of money needed, amount of food you can eat, practical limitations due to time and dietary needs.

It's no different here. The ability to infer a model from observations is and will be increasingly important in computers over the coming decades.

to my eyes the next step would be hard core triage to get all the information that wasn't improving the prediction out of the system. Every extraneous piece of information is a new opportunity for the system to screw up. It could be that Prospectus has done this triage and I haven't read about it,

That's the whole point of the Bayesian inference Nate's using.

Dan, are you referring to the reasoning I'm talking about or the reasoning I'm exhibiting?

Mental models of how things work are of course famous for failing badly and unpredictably, and being overly constrained by limited prior experience. If you suddenly had to shop for a diabetic or a platoon of hungry Marines, your mental model of shopping would fail miserably.

Regarding comparable players, the belief that similar players age similarly is why that information is included in the model, is it not? Hence the weighting of more comparable players higher? If you have a projection system that assumes Player A will age similarly to the ensemble of comparable players, then if things are working properly that ensemble should remain comparable to Player A as time goes by.

Mental models of how things work are of course famous for failing badly and unpredictably, and being overly constrained by limited prior experience. If you suddenly had to shop for a diabetic or a platoon of hungry Marines, your mental model of shopping would fail miserably.

True, but computers are far better at weighing long-term observations than people are. Baseball games are quite constrained in possible outcomes, which is a huge bonus here.


Regarding comparable players, the belief that similar players age similarly is why that information is included in the model, is it not? Hence the weighting of more comparable players higher? If you have a projection system that assumes Player A will age similarly to the ensemble of comparable players, then if things are working properly that ensemble should remain comparable to Player A as time goes by.

But again, it's a learning model - it's set to learn from past mistakes in a systematic manner.

That's the whole point of the Bayesian inference Nate's using.

Maybe this is as much a philosophical issue as a mathematical one, but why would you start with a universe of largely irrelevant data and then use Bayesian inference to carve down some of the weights rather than starting with a minimal model and adding data after it's known to be valuable?

Still going by intuitive reasoning, rather than actual math here.

If you have a model which is set up to map a given (incomplete) set of variables to a given (incomplete) set of outcomes while minimizing total error, then at any given time won't you believe your model is working better than if you could consult an Oracle which could tell you the true error on the universe of outcomes?

Huh? All the projection/prediction models can be tested by comparing the predictions to empirical observations from real life. You don't need to consult an oracle.

Not all possible player-seasons have been played yet.

Now that was truly oracular.

It should be added to the great declarations of the Oracle at Delphi.

"Nothing in excess."

"Know thyself."

"Not all player-seasons have been played yet."

"It ain't over 'til the fat lady sings."

They will play a lot of games this season, you know. Those games aren't included in the models yet. Suppose some phenom in deepest darkest Africa is busy developing the super-gyroball that you can keep throwing until you're 55 -- that's a possible career path which wouldn't be included in the model. Heck, there haven't been any left-handed catchers in a hundred years. The data which has already been collected is a tiny fraction of all the data which could ever be collected.