Polling: Accuracy and Sample Size

Practical Issues Concerning Accuracy

Suppose the problem is to determine which singer bridge players prefer. If polling 10 people yields an answer that 80% prefer Presley, you can be reasonably confident that Presley is the answer. If 60% prefer Presley, Presley is your best guess, but there is a reasonable chance of that conclusion being wrong. And if 50% prefer Presley and 50% prefer Sinatra, you have no idea who is actually preferred most.

Suppose instead that you are trying to decide which music to play. If 50% of the people prefer Sinatra and 50% prefer Presley, then it doesn't matter which one you play. If 60% prefer Sinatra, it is a small mistake to play Presley. But only a small mistake. The big mistake occurs if 80% like Sinatra and you play Presley.

For the second issue, much smaller sample sizes are needed. Which is to say, you can poll fewer people and be fairly confident of making a reasonable decision. If 3 people are polled and they all say they like Presley, you can stop there. It is unlikely that Sinatra is preferred by the majority, but even if that is true, then most likely Sinatra doesn't have a big edge over Presley and it is only a small mistake to play Presley music. If 10 people are polled and half like Sinatra and half like Presley, then you aren't complely lost. To the contrary, it probably doesn't matter (much) who you put on.

The bridge issue is probably caught between the two. Suppose "significant proportion" is 30%. Then it makes a big difference to the ruling whether the true value is 29% versus 31%. (This assumes you can't give a weighted score.) However, in terms of overall justice, it in a sense doesn't make a difference what the ruling is. Someone who bids when pass is an obvious LA has committed an obvious infraction and deserves to have their score taken away. Someone who bids when pass is obviously not an LA deserves to keep their bid even if UI suggested pass. In the middle, there is no clear right or wrong.

And anyway, if weighting isn't possible, you don't have much choice to make a decision in the middle area that might be right or wrong.

I will suggest the following. First, when the adjusted score can be weighted, poll enough people to get a reasonable answer. My suggestion concerns when weighting isn't possible. Poll a few people, such as 3 or 4. If they results of the poll are reasonably clearcut, stop polling. As a practical matter, three things can happen when you poll more people, and none of them are good. The first is that the results become even more clear cut. Then you end up making the same decision, using more time and effort (including the time of the players being polled). Or the results might stay just as obvious/fuzzy as they are now. Then you are faced with the same decision. Do you still poll more players or make a decision that might be wrong? Or, the new results might come in for the opposite direction. If they do, then the results will now be balanced and you won't know what to do.

If the results of the poll don't seem at all clear, then poll more people until they do seem clear. If further polling isn't clearing things up, then decide on a stopping point (for example, "I am going to poll eight people and then stop") then make the judgment based on that.

NEXT: Practical Issue of Committment to Avoid LA Problem