Study on NBA refs and a possible small racial bias in foul calling

I didn't see a thread about this yet, but it seemed to be all over the sporting news today.

What I found more interesting than the story, though, was the high proclivity of people with obviously no statistical background making claims about the lack of validity of these statistics.

It also annoyed me that everyone kept calling it an "experiment" when it was just an analysis of pre-existing data. That isn't an experiment.

Ah well, just another piece of evidence for the "the less you know, the more you think you know" meta-analysis.

Yeah, that at least is a reasonable concern (or at least a good question to bring up).

So, there are 4 possible ref combos (based on race):

3 black
3 non-black
2 black, 1 non-black
1 black, 2 non-black

In the first two you can obviously tell which "color" ref made the call. In the that last two, you obviously can't.

However, if there was no bias, you shouldn't see a difference between any groups. While being able to tell exactly which ref made the call would have enhanced their statistical analysis, not having that information does not negate the potential validity of their results.

You can easily reword this to avoid any misinterpretation as well... "groups" or ref are more likely to call fouls against players who are differently colored than the "group" of refs officiating the game.

The really ignorant comments I keep seeing and hearing everwhere, though, have to do with their being more black players, less black refs, etc. Most people are immediately dismissing the data because of that... and the ones who get past that get hung up on not knowing the individual refs that made the call.

I'm not saying that this is groundbreaking research or anything, I'm just saying that people need to realize that when they don't understand statistics, they should ask questions, not spew false nonsense.

Think of it this way... I'll use a simpler example just to make it so I can type less... but the point will be the same.

Let's say we want to see if red cars are pulled over more than blue cars (by police).

In some area there are 10000 red cars and 1000 blue cars. Of course, you would expect more red cars to be pulled over based on overall numbers. When run statistics to look for a bias, though, what you are looking at is more like a %.

If 1000 red cars get pulled over, that 10% of the red car population. From that, if there were no bias, we would expect 100 blue cars (approximately) to have been pulled over.

Let's say that only 15 cars got pulled over. We would call that a potential bias in favor of red cars getting pulled over.

No, what if 500 blue cars got pulled over? That was less in overall number than the 1000 red cars, but because the starting sample was smaller, we would say that it is evidence that there is actually a bias to pull blue cars over more often.

One study can't prove anything... in fact 1 million studies can't prove anything... all data can do for us is provide evidence to support something. There are more years to look at, though, more sports that can be looked at, and so on. In addition, it might be interesting to look at changes in these biases over time.

No, you can never trust anecdotal evidence, that is why scientists don't say, well, it LOOKS like the lungs on these mice are more damaged that this other group... they have to run the statistics. It is even worse when human opinions come into play. The human brain is a big, lieing, shortcut machine that unfortunately can't be trusted. And until we find a way to destroy it and its evil lies, we'll just have to study it!

If the "margin of error" overlapped with 0, then it would not be a significant finding (and thus, no bias found).

I've seen it presented as 2.5-4.5% in the New York Times article. I don't know if this is 1 standard error of the mean, or possibly a 95% confidence interval (which is what I would hope). That would mean that the actual bias they found was 3.5% with a 1% margin of error (of some sort, like I said above). That would be a statistically significant finding of a bias.

Now, some people bring up a good point... how big of a deal is that? They tried to answer that question after looking at both increased foul calls and reduction in productivity based on the possible bias (i.e., scoring, rebounding, etc. less) and found that it could cost about 2 games a season for every extra black player you have (I would assume that this number uses random ref groups drawn from the actual ref population... i.e., something representative of the actual ref sample... and also that it assume significant playing time, so a starter).

Seeing as most teams probably have similar racial makeups, the effect is probably small... but a 2-4 game swing can still matter come play-off time.

Also, there methodology to determine that 2 game swing per player is certainly not perfect, there is a margin of error around it as well, but it is still interesting to consider.

I also haven't read the actual paper, so I can't comment directly on it, but I hope that clears some things up.

No, you are exactly right. Like I said, I was making an oversimplified example to get the main point across. You would actually have that other variable to account for as well (ref racial makeup). The same ideas apply, though, I just didn't want to type out a 2 page example to explain that, but that would also be accounted for.

No, they didn't care about individuals. They had a 4 levels variable which I'll call Ref Team Racial Makeup with the 4 levels being:
3 black
2 black, 1 non-black
1 black, 2 non-black
3 white

They are only showing a bias in reffing "teams" not individuals. However, if there was no difference in individuals, with such a large sample, all the groups should end up being even in their foul call rates.

The other possibility is that whenever a mixed group is reffing, the minority ref makes almost all of the calls. That would throw this data off if it happened consistently... so it is something to consider. If it does happen, though, why would the minority ref make more of the calls? It also wouldn't explain the bias in the non-mized reffing teams.

In other words, it is a good question to bring up, but it would seem unlikely to matter (which means it should be looked into more).

I do agree with this. If this happened in a more scientific field, the professor would very likely get at least minorly ostricized and no reputable journal would publish his paper. That actually hapenned recently... I think it was a biologist of some sort, although I don't recall the details. Ecomonics professors and journals might work differently, though.

But, yes, it should have been peer reviewed and published in a reputable journal before it made it to press. I do question that.

Yep, that would be accounted for in the statistical analysis.

They are looking at changes in foul rates due to changes in ref team racial structure, and not overall foul numbers or rates in general.

You are talking about the difference between statistical and practical significance. If they "did their math right," this appears to definitely be statistically significant.

You wouldn't be able to deny that, but you could then go on to argue that it is of low practical significance (which is what you are doing), but that is a different argument entirely.

See my post above about individual refs. This study makes no claims about individual refs, just about trios of refs and their racial makeup as a whole.

You still wouldn't expect to see any differences if there wasn't a bias somewhere, however.

Sure, I agree that the bias is probably primarily "under the surface" and not outright bigotry.

Sure, this study of the data shows the number to be relatively small. That is why I said it would be interesting to do a timeline of the change in this bias. This could turn into something positive for the NBA. In other words, in the 80s, the refs were less than bias than they were in 70s, and in the 90s they were less bias than they were in the 80s, etc.

I wasn't even arguing about the importance of the study, however, just the misunderstanding of the statistical analyses that get used with data like these.

If the range they did give was the 95% confidence interval, you'd have to more than triple the variability in the data to make it not significant (at an alpha level of 0.05). If it was a single standard error measurement they gave, you'd still have to almost double the variability to "get rid of" the effect.

What factors would have that much of an effect on the variability?

I agree that that would be an interesting question. You do realize the near impossibility of that study, though, right? Unless the NBA has reliable data on that already collected (and who is deciding accuracy?), that could take years and years to analyze and double check everything even with a small team of people watching tape in all of their free time.

No, it wouldn't be surprising... but that shouldn't account for changes in the rates of fouls called depending on race. That should just result in a general increase (or decrease) in falls called by various individuals... which would have no effect on the results of the study (that gets factored out).

I'm not sure I follow your point.

If black refs had the "same sort of racial bias" (to me that reads that they are also prejudice against blacks and call falls against them at a higher rate), then all refs should be pretty much the same, and the data would show no significant bias for any particular triad of ref color.

If you mean the "same sort of racial bias" as is bias against races that aren't them (i.e., whites), well that is indeed what the study found as well.