You've a list of people.
You've got your WARP ranking, you've got your BP ranking and you've got your BA ranking.

You wish to see whose ranking was closest to the WARP ranking...

As a practising professional statistician (meaning "someone who make a living doing statistics" as opposed to "sabremetrically inclined baseball fans") can I politely suggest you calculate the Spearman Rank Correlation Coefficient, rather than just pick over the results looking for obvious hits and misses.

If you're feeling really adventurous, you can do a significance test to determine the p-level of any difference in the r's

Very cool stuff, I think it's fascinating to look back and past prospect lists, and it's really interesting that even in their Jackie Rexrode days there is still lot of overlap between the BP and BA lists.

G.W.O.,

No professionals allowed, it would be a travesty to actually apply known and proven statistical concepts to any of these discussions.

BTW, I liked this article.

Rob

Or not

Very enjoyable. Thanks for compiling that, Philly.

I thought the point that BA and BP equally fawn over toolsy types to be a good one. If Jeff Francoeur fails to turn into much, it would be hypocritical for BP to sneer at him in retrospect, given the ranking he got this year.

PS, I liked this article too...

Chuck Spearman was a bludger.

Interesting that both BA and BP did better at projecting pitchers than hitters. We have been indocrinated to believe the opposite for quite a while.

Also, it seems pretty clear across the board that in 1999 BA's list was far superior. Is there a reason why we should think their list has gotten better over time, or should I go back to seeing BA as the "prospect people" until I someone actually puts together a list that nails more prospects.

Also, the Giambi quote will have me smiling all day!

A study I would love to see some club do is to study the correlation between the success of hitting prospects and the visual skills that are required for hitting. In the mid-90's an ophthalmologist who was a big baseball fan wrote an article for Baseball Weekly, if memory serves me, that talked about the several skills required to be a major-league level hitter. Besides the obvious ones, like superior eye-hand coordination and excellent depth perception, there were several others that are necessary. For example, saccadic eye movement refers to the way that the eyes constantly refocus on a moving object. There's a big variation in the speed with which people are able to do this refocussing. To be a major-league hitter, you have to be able to refocus much more quickly than average. There's also the ability to detect contrast. This ophthalmologist felt that this was the visual skill that separated the very best hitters from the rest. He said that when Ted Williams went duck hunting, he always saw the birds first, not because he had greater visual acuity than everyone else but because he had such outstanding ability to detect contrasts.

To my mind, such studies could prove invaluable. For example, there are always guys who can kill the fastball but have trouble with off-speed and/or breaking pitches. It would seem to me that it would be important to learn whether or not this was due to his lack of the proper visual skills. If it was, no amount of work will make the guy an effective hitter and it wouldn't be cost-effective to invest money in trying to develop him. On the other hand, if some guy had all of the requisite visual skills but was struggling, it might prove worthwhile to spend time developing his abilities.

If studies demonstrated that certain visual skills were essential to become a major-league-level hitter, then it would be in the interests of clubs to test all of their prospects, because there's no reason to waste time on one that doesn't have the requisite visual skills. These abilities are like foot speed. A guy who runs a 11.0 hundred meters dash might be able to improve upon that with training, but he's not going to become a world-class sprinter. Likewise, a guy might improve his visual skills, but unless they're at the appropriate levels in the first place, he's not going to be able to hit in the majors.

I recently let my BP subscription expire and signed up for BA. I'm happy with my decision, although I will say that both looked better from the outside, before I had the subscription.

The Giambi quote is funny, but I can't imagine there are ANY prospect lists that wouldn't have statements that sounded ridiculous 6 or 7 years later.

tikes, that is some bad english there in that last sentence. you know what i meant.

The Braves had Bruce Chen, Marcus Giles, and George Lombard on this list. I guess one out of three ain't bad...

Very cool stuff, I think it's fascinating to look back and past prospect lists

I agree. In college football and basketball, there are a lot of lists of who the best recruits are, but the "experts" are never accountable for their past projections. If the player doesn't pan out, it's always the player's fault, not the expert's.

Interestingly enough, Rany wrote an article reviewing his choices for the '99 list early in 2000. I came across it when looking for Jackie Rexrode's statistics.

Here it is.

BP was already acknowledging some of the things they get grief for here back in 2000. Good for them.

Nice article. As one of those who joked about the presence of Ruben Mateo on BA's lists over the years, I appreciate learning that he was also listed highly on BPro's lists as well.

Philly also touched on (and perhaps shares) one of the pet peeves I have regarding the listing of prospects:

It’s also one of my biases that I don’t think the specific sequential rankings are all that interesting. That one group may have a stud player at #4 and the other has him at #9 is completely meaningless to me. . . . Once you get past the very top premium prospects, I don’t think rankings that differ by 15 to 20 (or more as you get really low into the BA Top 100) are truly different.

Very well said.

From that BP article:
34. Daryle Ward, 1B/LF, Houston (BBA NR, Sickels #47)

What we said last year: "Our tough luck listing. Ward’s been ready to hit major league pitching for over a year, but the world is filled with first baseman who can hit, and Ward’s in the same organization as Jeff Bagwell. Trying to broaden their options, the Astros moved Ward to left at New Orleans in 1998. He didn’t embarrass himself, but if the way to a spot on the Astros’ major league roster is blocked for Lance Berkman, you can bet it’s even more blocked for Ward."

What he did in 1999: We finally got one right. He’s not playing first base for the Astros anytime this millennium, but the tornado that carried off half the Astros’ outfield gave him a much-deserved opportunity. He was having arguably the best season in the minor leagues (.772 slugging average and 28 home runs in 61 games) when he was called up in the second half. After slugging .473 for the Astros and hitting well in the playoffs, Ward figures to have the upper hand for a starting job for 2000.

Take-home lesson: It’s the offense, stupid. And good organizations find ways to give opportunities to good prospects rather than let them linger in the minors with nothing to prove until they stagnate.



Oops.

If Jeff Francoeur fails to turn into much, it would be hypocritical for BP to sneer at him in retrospect, given the ranking he got this year.

Should they continue to insist that he's a good player, even when the evidence mounts against it? They'd get killed for that. Sneering is rarely helpful in any context, but I don't see a problem with revising expectations.

Kurt: no, they should just admit they were wrong - or that they too drink the scouts' Kool-Aid from time to time. Nothing wrong with that. Rany does this all the time in the article I linked above, so I guess it's something I should expect from him. Really, the sneering comes from places like this one, so I think my comment above was unfair.

As a practising professional statistician (meaning "someone who make a living doing statistics" as opposed to "sabremetrically inclined baseball fans") can I politely suggest you calculate the Spearman Rank Correlation Coefficient, rather than just pick over the results looking for obvious hits and misses.

For god's sake man, get off your lazy duff and help Philly out.

The results are quite humorous.

First, all the NR guys for BA I coded in order of WARP 3 from 101 to 107 (got to give them some ranking). Ordering them by WARP3 will improve BA's correlation a smidgen. I also took out Ozuna since neither side should be penalized for him.

I'll also say I'm not sure what difference it makes that we've got the top 40 for BP compared to guys ranked as low as 100 (or not at all) by BA.

Anyhoo, BP and BA correlate pretty well at .62.

The hilarious part is that both of them have a negative correlation with WARP3. The correlation is -.37 for BP and -.33 for BA.

It's easy to see why this is for BA. Here's the total WARP3 breakdown for them by ranking (number of players in parentheses) ... again no Ozuna:

1-10 (8): 148
11-20 (5): 129
21-30 (5): 32
31-40 (5): 26
41-100 (9): 131
NR (7): 45

Who was the BA top 10 player not in BP's top 40 list?

It's not so easy to see why for BP:

1-10: 200
11-20: 131
21-30: 107
31-40: 74

For these players, in the aggregate, BPro did a good job, much better than BA did ... for these players.

Philly presents similar tables, using (I assume) BA's actual top 40 for their table, so let's reproduce that:

1-10: 145.2
11-20: 181.3
21-30: 130.4
31-40: 97.4

BA does much better in the aggregate here. A quibble though:

BA’s high success rate at the top of the list extends deeper all the way to #20

I think it's impossible to say this, with its implication (supported by an earlier sentence in the article) that this isn't true for BP. BA found 11 successes in the top 20 compared to 10 for BP ... trivial differences especially given that two of BP's "failures" (Davis and Giambi) have at least surpassed 10 WARP3. BA's top 20 compiled 326 WARP3 while BP's compiled 331. I'd say the two lists were equal for the top 20. BA's greater success from 21-40 is clear (but maybe not significant).

It's interesting that the 9 players on BP's top 40 that were 41-100 on BA's list did as well as BAs 21-30 and nearly as well as their 1-10. Despite Giles and Jimenez, BA's unrateds that made BP's top 40 weren't that impressive (average 6.5 WARP3 per)

Anyway, the upshot is that neither of these look any good by the spearman rank. I think this mainly reinforces Philly's argument that the difference between #20 and #40 is generally pretty trivial.

I'd really be surprised if PECOTA makes a difference. Or if it does, BP did a really bad job of "performance evaluation" in the pre-PECOTA days. Performance evaluation of prospects obviously requires "translating" minor-league performance to major-league performance. If players like Rexrode don't project well, there's no reason they shouldn't have known that before PECOTA.

Walt, don't you think it's unfair to judge BA just on the players that BPro ranked? Their entire top-100 is available here.

I think Ryan Anderson was the guy that they had in the top-10 that BPro didn't rank.

D'oh! The correlation should be negative -- high values on the prospect rank are bad, high values on WARP3 are good. So they're correlating about .35 give or take. Significant (barely) but not great (ranking explains about 10% of the variation in performance).

Walt, don't you think it's unfair to judge BA just on the players that BPro ranked? Their entire top-100 is available here.

I mentioned I don't know how this throws things off. And as soon as Philly takes the time to look up WARP3 for all 100 of their prospects and posts it in nice tabular form that I'm easily able to copy into a database, I'd be happy to run that correlation as well.

(Note, a rank correlation just takes into account the observation's rank on that variable. That is, if the 30th highest value on BA's ranking in that sample was 65, that player is treated the same as the guy ranked #30 on BP's list).

Also the last aggregate table is, I believe, BA's actual top 40 (it came from Philly's article too) and there's not a huge difference with BPro. We can't run correlations on that without the individual data, which I hope will be in Philly's eventual article on BA's ranking history. But their top 40 probably has a higher correlation to performance than their correlation on BPro's top 40 did, maybe enough that they inch ahead of Bpro, but there's not going to be a significant difference here.

Philly, if you have some spreadsheets with rankings and WARP3 (or WS or VORP or whatever), I'd be happy to run some correlations for you and maybe some fancier stuff. (Note, I'll be out of the country from 3/23 to 4/7 and have much to get done before then, so don't expect anything on short notice).

Thanks for the link Matt. Just got a chance to read through everything.

Walt

BA’s high success rate at the top of the list extends deeper all the way to #20

I think it's impossible to say this, with its implication (supported by an earlier sentence in the article) that this isn't true for BP.

You're right. I got ahead of myself there. I've crunched the data for 2000 and the 99/00 summary, I just haven't written it up yet. In 2000 BA is 2 or 3 players better in that interval which makes the difference look more significant. Obviously it's still just two years and a 2-3 player edge for BP in 2001 would even things out again.

Anyway, the upshot is that neither of these look any good by the spearman rank. I think this mainly reinforces Philly's argument that the difference between #20 and #40 is generally pretty trivial.

That and the high degree of overal we're reall the points I wanted to make. And a nice corollary to the idea the the difference between #20 and #40 is trivial, is that the difference between #75 and #105 is pretty trivial too.

Except that one MAKES the list and one doesn't. That leads to a lot of - I can't beleive prospect I hate made it at #75 and prospect I love (maybe) just missed at #105.

Thanks for the spreadsheet offer. I'm going to send them to the address in your profile.

Oh, and one thing I forget to add. The nice thing about BA going all the way to 100 is that you can see some good players that should have been in both Top 40s in retrospect.

Here are the top guys from BA #41-100 listed by BA rank with their WARP3.

51. Jeff Weaver 30.3
58. Mike Lowell 40.4
59. Kris Benson 16.6
60. Rafael Furcal 25.7
66. Ted Lilly 16.1
68. C Guzman 18.0
69. Vern Wells 20.9
76. A Kearns 9.8
80. Wade Miller 22.6
82. S Burroughs 9.5
86. Milt Bradley 14.9
87. Carlos Guillen 23.2 (BP honorable mention)
88. G Mota 15.8
90. Tony Armas 14.4
97. Randy Wolf 24.7
98. A Kennedy 25.2
99. Trot Nixon 29.6

A lot of those guys are only solid (at best) but there are a handful of very good players in there too. Lowell is jsut the 4th player to clear a 40 WARP and he's just barely behind Chavez/Beltran/Berkman. Weaver is one of roughly 10 players who cleared 30 WARP.

Personally, I think a key thing to remember here is that without Baseball America, the BP list-and, really, every other prospect list out there-would be completely useless. Baseball America's coverage of the players talent, rather than performance, allows guys at BP to have something of an accurate overview of the half of the evaluation process that they ignore.

If you take away BP, the BA list doesn't change at all. Their work stands alone. Take away BA, and I honestly don't think any of the heavily performance based minor league analysts could discern the difference between players like Brandon Moss, who didn't even earn an honerable mention on this years list, and Lastings Milledge, who BP ranked #19 overall.

Now, I think BP has the "right" ranking here, as I like Milledge more than Moss too. But what did BP base that ranking off of? Moss' Future Davenport Translation is higher, his PECOTA projection is better, he has better plate discipline, he's performed better at a higher level, and he's in a sabermetric organization. There's no performance analyst-based reason for Milledge to be ranked significantly higher by BP than Brandon Moss.

The work BA does is the backbone of the work BP does. Take it away, and BP's list falls apart. To me, that basically makes any comparison between the two rather pointless.

There are other differences between the list though. Or at least I think so.

BP's list is mainly Rany's baby. Other BP authors review and comment on the list, and apparently now they're making use of PECOTA, but it seems to me it's Rany who's making the call in terms of weighting "talent" and "performance" -- i.e. ignoring PECOTA projections and putting Milledge higher.

BA's list seems a full committee operation and that committee is getting input from a shitload of scouts, GMs, etc.

In a sense, it's the multi-rater approach that I mentioned in the Stark thread. Having 6 "experts" rate Milledge and Moss is almost certainly better than having 1 expert do so, often even if that 1 expert is the best of the bunch.

When you think of the time/money/effort/personnel/inputs/expert analysis that must go into BA's rankings vs. some stats and a dermatologist who probably doesn't even get to see much minor-league ball on BP's side, BA should pretty much kick BP's butt.

Maybe the difference in how much goes into them isn't as big as I think.

If "we" are going to have a serious discussion about how the prospects lists have faired we should do two things:

1) Consider John Sickel's lists as well. I don't have the 1999 Sickel's book, but maybe someone is willing to offer the list. (maybe John himself?) I would be curious to see how he does relative to BP and BA, given the fact that John mixes both scouting and performance analysis.

2) Consider how the lists have looked over time. The "ratings" we give the list today are different than they were in 2000 and they're different than they will be in 2010.

The other "issue" here is the age at which the players were rated. For example, take two players of equal ability/talent. If one of them is 21 and the other 24, the 21 year old will rank higher (I am assuming this would be the case). So, shouldn't there also be some sort of age adjustment to account for this in how well a particular list does?