Ass-backwards Analytics: From single case studies to general rules, from big data to little exceptions

Posted by in internet

People respond to data, or the lack thereof, in interesting ways.

Two tendencies that I have encountered as people respond to data (in excess or in shortage) would be that of (1) ascribing rules to the exceptions (i.e., inferring general rules from single case studies), or (2) asserting exceptions to the rules (i.e., highlighting slim exceptions when presented with big data patterns).

I think of these tendencies as biases, because I believe they may represent some underlying processing errors our brains fall into when dealing with data, particularly given the context within which the analyses take place.

Ideally, we should be responding quite differently, and oppositely, than these two tendencies when analyzing data.  Case studies should engage our understanding of exceptions, or the exceptional aspects of a specific situation.  Big data should trigger our pursuit of general patterns, or the unexceptional things that a wide range of cases might have in common.

Bias #1: Rules to the exceptions

How often have you heard, “What we learned from this example is that the best way to accomplish X is to do Y!”

Yet drawing broad conclusions from a single example is (most likely) not the way in which we should be responding to these sorts of data. In reality, we just learned about a single case of X and Y, and have no idea if across of range of attempts at X we would get a Y more often than not (or at an appropriate level of risk).

Case studies are compelling, however, because they are rich stories, capturing our interest in the specific and at times sordid details of some unique experience.  Oftentimes, the case studies we really love are those that recount truly exceptional experiences—breakaway successes or full on failures.

Bias #2: Exceptions to the rules

When presented with a large pile of data and the patterns therein, however, it seems that we can’t help but use each additional data point as a means to question, or counter, the general trends.  After (or while) presenting some large-scale analysis, how often do we hear, “Ok, but what about Z?” “Did you take Q into account?”

It is as if our brains just have to find something exceptional when presented with a pile of data through which we confront the unexceptional patterns therein. Or, problematically, we find it hard to believe patterns unless they apply to just about every instance we can imagine.

Yet general patterns are not supposed to be natural laws.  Patterns are simply links that exist “in general.”

The decision-maker’s paradox

So why do we respond to data scarcity and excess in these ways?  Are we fools?

I actually don’t think so.  And am trying to understand more.