Baseball for the Thinking Fan

Login | Register | Feedback

btf_logo
You are here > Home > Baseball Newsstand > Baseball Primer Newsblog > Discussion
Baseball Primer Newsblog
— The Best News Links from the Baseball Newsstand

Wednesday, January 15, 2014

Wired: Why Quants Don’t Know Everything

Michael Lewis’ Moneyball vividly recounts how the quants took over baseball, as statistical analy­sis trumped traditional scouting and propelled the underfunded Oakland A’s to a division-winning 2002 season.

. . . .

The reason the quants win is that they’re almost always right—at least at first. . . . The more a field is run by a system, the more that system creates incentives for everyone . . . to change their behavior in perverse ways—providing more of whatever the system is designed to measure and produce, whether that actually creates any value or not.

Give the system what it wants!

The Fallen Reputation of Billy Jo Robidoux Posted: January 15, 2014 at 08:53 PM | 22 comment(s) Login to Bookmark
  Tags: moneyball, sabermetrics

Reader Comments and Retorts

Go to end of page

Statements posted here are those of our readers and do not represent the BaseballThinkFactory. Names are provided by the poster and are not verified. We ask that posters follow our submission policy. Please report any inappropriate comments.

   1. madvillain Posted: January 15, 2014 at 09:10 PM (#4639969)
The most profound example of overshoot, of course, happened in finance, where the rise of quantification could concentrate decisionmaking—and moneymaking—within a relatively small group of people at a bank’s headquarters. Soon they were trying to optimize their algorithms to maximize profit, minimize risk, and make millions of dollars for themselves. Global regulators didn’t help: In 2004, in sympathy with the over-leveraged, hyper-quantified banking system, the Basel Committee—the Switzerland-based body that oversees world finance—put out the Basel II accord, more than 250 pages of regulations that effectively placed individual banks in the driver’s seat. The accord essentially embraced all of the quantitative techniques used by the wizards who would end up blowing up Wall Street, and it allowed banks to operate with astonishingly high levels of debt. As everybody knows, all of that ended in catastrophe in 2008. (You can read more about the particular math of that cataclysm in my March 2009 cover story for WIRED, “A Formula for Disaster."


Of course, Nassim Taleb has famously shot down the typical notion of "risk" as little more than a lie the vast majority of quants tell themselves in order to sleep well at night. I don't know enough about the math to really have a strong opinion on Taleb's thesis but it seems obvious that what he calls "black swans" are almost universally ignored by finance quants in their risk calcuations.
   2. greenback calls it soccer Posted: January 15, 2014 at 09:25 PM (#4639977)
it seems obvious that what he calls "black swans" are almost universally ignored by finance quants in their risk calcuations.

One definition of black swans is "the stuff that quants ignored that turned out to be important." It's not a useful concept, unless you like to scream "Look at those idiots!", as Taleb enjoys doing.

It's sad that I recognize a Felix Salmon column based on two excerpts.
   3. Shibal Posted: January 15, 2014 at 09:29 PM (#4639980)
unless you like to scream "Look at those idiots!", as Taleb enjoys doing.


Who doesn't like screaming "look at those idiots!"?

That's one of the great joys of life.
   4. madvillain Posted: January 15, 2014 at 09:36 PM (#4639983)
One definition of black swans is "the stuff that quants ignored that turned out to be important." It's not a useful concept, unless you like to scream "Look at those idiots!", as Taleb enjoys doing.


I think it's useful because it goes against the notion that the "Wizards of Wall Street" are infallible geniuses. It's helpful in the court of public opinion, and thus, is helpful to prod the political system into tighter regulation and monitoring of what exactly the Wizards are up to.
   5. Dale Sams Posted: January 15, 2014 at 09:38 PM (#4639986)
Quants?

I wager 50 quatloos on The Quants.
   6. You Know Nothing JT Snow (YR) Posted: January 15, 2014 at 09:43 PM (#4639991)
Who doesn't like screaming "look at those idiots!"?


Why it's my favorite song by Mr. Burns!
   7. Moe Greene Posted: January 15, 2014 at 09:45 PM (#4639994)
Taleb has some interesting points in The Black Swan, but man... what an arrogant jerk. He could've made the same point without grinding all of those axes.
   8. Walt Davis Posted: January 15, 2014 at 11:53 PM (#4640057)
Quants can't really get anything "wrong", at least not if they're genuine quants. Everything is/should be couched in terms of uncertainty and a corresponding likelihood of something outside that range occurring. It requires being wrong several times to provide sufficient information that the assumptions underlying the model were wrong and many, many more to help guide you as to what the better assumptions will be.

I've never read "black swans" but my understanding is that it's basically about outliers and mutants -- was it "The Mule" in Heinlein's Foundation? Easy to find after the fact and essentially impossible to include in the model in any useful way. But the key point is no other method or person can tell you the black swan is coming other than the completely useless "a black swan will pop up sooner or later." That leads to decision paralysis.

In short, it's true that a model can't distinguish Pujols from a similarly performing potential draftee on draft day -- but scouts haven't shown themselves to be any better at it in any consistent manner. Pujols was drafted 402nd -- everybody "missed" him including the Cardinals and the scout who found him.

Trivia question -- who was drafted one pick before Pujols? Amazingly he also made the majors, played parts of 9 seasons and has positive WAR. Hopefully useless hint: he was drafted by Albert's current team.

Perhaps even more surprising, that player is only the 2nd best 401st pick ever. The best ever honor belongs to Jim Eisenreich.

John Lowenstein with 10 WAR is the 2nd best 402nd pick of all time, not shabby.
   9. greenback calls it soccer Posted: January 16, 2014 at 12:40 AM (#4640073)
The guy the Cardinals drafted ahead of Pujols never made it past rookie ball. I've seen a few games in the Appy League, and it just doesn't seem quite right that scouts couldn't tell a below average player for that level from Albert Pujols only two years before his major league debut. Actually I'm guessing the scouting wasn't really the problem. Most models break when they get poopy inputs, and plenty of folks were convinced that Pujols was older than he was.
   10. villageidiom Posted: January 16, 2014 at 12:43 AM (#4640074)
Quants can't really get anything "wrong", at least not if they're genuine quants. Everything is/should be couched in terms of uncertainty and a corresponding likelihood of something outside that range occurring. It requires being wrong several times to provide sufficient information that the assumptions underlying the model were wrong and many, many more to help guide you as to what the better assumptions will be.
Ranges of reasonability, confidence intervals, failure probabilities, etc., are all heavily dependent on assumptions made. If a quant makes the wrong assumptions, they are wrong, and all their work is wrong. In the real world there is no partial credit for doing the right math with the wrong data.

I am agreeing with you, but emphasizing it differently. Many genuine quants get so far into the math and cool analyses that they neglect to check basic assumptions, the consequence of which is fatal to their work. They don't see it, and they get defensive when you point it out, which leads to brash overconfidence in the short run, and failure in the long run. (Wanna-be sabermetricians would be wise to remember this.)

Once I worked with someone who was asked to build a model so we could predict at the time an (auto) insurance policy was bought from us how likely it was to cancel before expiration. He built a very predictive model. The most predictive piece of info? How many times the customer called us after buying the policy. First, he assumed every field in our database was known info at the time the policy was sold. After getting his result he never went back to check his assumption, otherwise it would have been obvious that if we sell a policy we don't know at that moment it is sold how many times they will call us in the ensuing 12 months. Second, the field containing the number of times the customer calls us includes the call they make to cancel the policy. His assumption about the nature of the data led him to believe that that call wouldn't be in there. Consequently his model was telling us basically that we can be pretty sure someone is going to cancel their policy when they call to cancel the policy.

So, yeah, technically he was not wrong. But his model was useless.
   11. vortex of dissipation Posted: January 16, 2014 at 01:19 AM (#4640085)
Why Quants Don’t Know Everything


They know how to make a miniskirt...
   12. The Clarence Thomas of BBTF (scott) Posted: January 16, 2014 at 01:32 AM (#4640088)
Hey, who doesn't like a good Felix Salmon article?

Walt: I'd say black swans as described by Talib are more common, and not as out of the blue, than true aberrations without precedent. And then also second what VI wrote.
   13. PreservedFish Posted: January 16, 2014 at 01:45 AM (#4640089)
Great anecdote, #10.

I've never read "black swans" but my understanding is that it's basically about outliers and mutants -- was it "The Mule" in Heinlein's Foundation? Easy to find after the fact and essentially impossible to include in the model in any useful way. But the key point is no other method or person can tell you the black swan is coming other than the completely useless "a black swan will pop up sooner or later." That leads to decision paralysis.


When I read Black Swan a few years ago I was struck by how much of the book was stuff I had already learned from participating in the skeptical stathead community. There's a lot of stuff in there, for example, about mistrusting narratives (the daily market recaps: "investors were inspired by the morning's employment report").

Anyway, he's written another book on how people should plan for black swan events ... and he runs a hedge fund that is organized specifically to take advantage of such things. I'll shut up now because I don't know much about the subject.
   14. Shibal Posted: January 16, 2014 at 03:05 AM (#4640098)
Taleb's Antifragile book is outstanding. I didn't much care for the Black Swan though.

What are some other writers similar Taleb?
   15. madvillain Posted: January 16, 2014 at 03:46 AM (#4640103)
When I read Black Swan a few years ago I was struck by how much of the book was stuff I had already learned from participating in the skeptical stathead community. There's a lot of stuff in there, for example, about mistrusting narratives (the daily market recaps: "investors were inspired by the morning's employment report").


Although that stuff is important, it's also a what you learn (as you said) in Logic or Epistemology 101. It's (imo) the concepts of non-linear models in finance and predicting in general (ie, the sheer impossibility of it in most cases) that has greater implications. IMO, the thing that sticks with me most from Talib is the prudent position of : "we cannot predict, we should plan accordingly".

Glad this article was linked here, it's much better off season chatter than the endless articles on AROD/PED/HOF.

Oh wait, Taleb is wrong, we can predict -- there will be another AROD/PED/HOF article linked tmr.
   16. Monty Predicts a Padres-Mariners WS in 2016 Posted: January 16, 2014 at 03:51 AM (#4640104)
was it "The Mule" in Heinlein's Foundation?


Asimov.
   17. bjhanke Posted: January 16, 2014 at 06:11 AM (#4640114)
I remember Mary Quant. She was a 1960s mod model (mod was a British fad in the early 1960s - the movie Quadrophenia is about them), like Twiggy. Based on what she said at the time, I wasn't sure that Quants knew ANYTHING. (Coke to vortex) - Brock Hanke
   18. Dr. Vaux Posted: January 16, 2014 at 06:34 AM (#4640115)
I was thinking "I didn't realize Heinlein also wrote something involving a Foundation and a Mule! How odd!"
   19. Shooty Survived the Shutdown of '14! Posted: January 16, 2014 at 08:21 AM (#4640117)
Quants didn't "take over" the A's, they just supplemented the decision making apparatus. I know this is obvious to all of us here, but the popular perception of nerds with laptops I guess just fits too neatly a narrative niche for the general public to resist.
   20. Never Give an Inge (Dave) Posted: January 16, 2014 at 09:05 PM (#4640695)

Read most of Black Swan, haven't read Antifragile although it's been recommended to me.

First off, the criticisms of Taleb as a writer are all valid. He's insufferably arrogant, he makes up his evidence, he's repetitive, and he makes a point in 300 pages that could have been made in 30.

That said, I disagree with Walt (who admittedly hasn't read the book) in #8. Taleb's point is that outliers are more common than the models would predict, because many models assume the normal distribution for quantities that are not normally distributed (and our brains tend to assume the same thing qualitatively about the world even when we're not doing detailed statistical analysis). Rather than lead to decision paralysis, Taleb discusses ways to take advantage of this mispricing of risk.

I haven't RTFA but I do agree with one point that Salmon makes in the excerpt, although I would make it more generally - basically, once a model is developed that describes a system involving human behavior, the model will begin to influence the system itself.
   21. Pasta-diving Jeter (jmac66) Posted: January 16, 2014 at 10:38 PM (#4640742)
Wow-for someone my age , "Black Swan" meant Bob Lackey , power forward for Al McGuire's Marquette teams. One of the most vicious fights I've ever seen was between Marquette and Frank McGuire's (no relation) South Carolina team with Tom Riker in 1974--serious punches and several ejections. After the game Al McGuire sniffed "a bar bouncer wouldn't even have taken off his coat".

Andy and I (and Harvey) are prolly the onliest ones who remember Lackey
   22. greenback calls it soccer Posted: January 17, 2014 at 02:07 AM (#4640806)
The best example I've ever seen of bad stats was a pre-season projection of Darryl Kile for 2003. I'd like to think the author was driving a point home about the need to get out of the basement, but I've never been entirely sure.

John Hempton had a better column on the problem of quants. Excerpt:
Ambac and MBIA both failed because they forgot what they really underwrote which was fraud protection. Instead earnest people in their ivory towers (Armonk and the Southern Tip of Manhattan) did mathematical models of loss probability rather than combing though loan files and checking the individuals. The end game of mathematical modeling was CDO squared deals – where the individuals were lost two or even three deep in securitization structures and so there was no way that you were understanding just how corrupt the underlying foundations were.

You must be Registered and Logged In to post comments.

 

 

<< Back to main

BBTF Partner

Support BBTF

donate

Thanks to
tshipman
for his generous support.

Bookmarks

You must be logged in to view your Bookmarks.

Hot Topics

NewsblogWhich Players Will Be Most Affected by the Hall of Fame’s New Rules?
(7 - 11:03pm, Jul 29)
Last: pthomas

NewsblogOT: The Soccer Thread July, 2014
(508 - 11:02pm, Jul 29)
Last: frannyzoo

NewsblogOTP - July 2014: Republicans Lose To Democrats For Sixth Straight Year In Congressional Baseball Game
(3548 - 11:01pm, Jul 29)
Last: McCoy

NewsblogOMNICHATTER 7-29-2014
(33 - 11:00pm, Jul 29)
Last: cardsfanboy

Hall of MeritMost Meritorious Player: 1957 Discussion
(11 - 11:00pm, Jul 29)
Last: MrC

NewsblogFull Count » Tim Kurkjian on MFB: ‘I’m going to say that Jon Lester is not going to be traded’
(30 - 10:53pm, Jul 29)
Last: Nasty Nate

NewsblogMASN TV Contract Pits Selig vs Nationals vs Orioles
(20 - 10:51pm, Jul 29)
Last: Curse of the Andino

NewsblogSOE: Minor League Manhood - A first-hand account of masculine sports culture run amok.
(10 - 10:43pm, Jul 29)
Last: Der-K and the statistical werewolves.

NewsblogOT: Monthly NBA Thread- July 2014
(1014 - 10:07pm, Jul 29)
Last: robinred

NewsblogHoward: David Ortiz shaping up to become first steroid era Teflon slugger
(26 - 10:06pm, Jul 29)
Last: SoSHially Unacceptable

NewsblogTrader Jack? As Seattle's GM struggles to complete deals, some rival executives wonder | FOX Sports
(58 - 10:02pm, Jul 29)
Last: cardsfanboy

NewsblogValencia traded to Toronto
(12 - 9:52pm, Jul 29)
Last: Davo Dozier

NewsblogABC News: ‘Capital Games’: How Congress Saved the Baseball Hall of Fame
(46 - 9:38pm, Jul 29)
Last: Jose Can Still Seabiscuit

NewsblogBarney + cash to Dodgers for PTBNL
(28 - 8:56pm, Jul 29)
Last: Walt Davis

NewsblogMelky Cabrera smashed a windshield with a homer
(14 - 8:49pm, Jul 29)
Last: Walt Davis

Page rendered in 0.2291 seconds
52 querie(s) executed