Why Even the Best Exit Polls in India Are Useless

Conclusions about caste and community vote patterns in Indian elections are all based on erroneous data.

From the last day of polling to the day results are announced, analysis in this country becomes a spectacle subject to the absurdities of exit polls which are almost never close to being correct. This is understandable. The diversity of India extends to every village, so it is almost impossible to ensure that any sample size is completely representative. And even when through some measure of luck a pollster arrives at a reasonable estimate of vote share, converting this into actual seats is perilous. Thus, the only time exit polls get it right is when they reflect an obvious change in popular mood, or when they are right only because they are wrong, obtaining the right ballpark figure for seats won even when their vote share prediction is incorrect.

For some reason, this rather straightforward predicament does not seem to be understood by political analysts; perhaps their lack of grounding in basic mathematics is a problem. But as a consequence, analysis based on exit polls continues to be the basis of our understanding of elections even when the actual polling percentages are available and we can easily ascertain how badly off the exit polls were. This becomes especially problematic when the analysis uses exit poll data on voting patterns among different castes and religious communities, something that actual vote share figures cannot reveal. We are forced to suffer the consequence of the belief that wrong data is better than incomplete data.

Consider the conclusions drawn in the case of the UP results. Writing in Outlook after the verdict, Christophe Jaffrelot states, ‘Mayawati’s defeat will be explained—again—by her megalomania, corruption, bad governance and so on. But the pundits who kept telling us that she was bound to lose because she was only paying attention to her core constituency, the Dalits, will have to think of some other grouse. The party has lost 27 percentage points among the Jatavs, according to a CSDS (Centre for the Study of Developing Societies) post-poll survey (which excludes the last phase of voting). The Jatavs and other SCs (down 15 percentage points) deserted the party, while the Brahmins have remained with it.’

In an India Today issue after the verdict, Sanjay Kumar uses CSDS data to tell us ‘How Uttar Pradesh Voted’. There is no suggestion in the piece that this is exit poll data, but it does reproduce what the CSDS exit poll had claimed. Through the piece there is no suggestion that the conclusion the introduction line offers so boldly—‘Muslims moved en masse towards SP, while Dalit Jatavs turned away from Mayawati’—is tentative at best.

Clearly, as the two pieces suggest, much of the analysis is based on the CSDS exit poll data since it is seen as the most accurate. The channel which ran with the poll, CNN-IBN, has gone to town claiming how right they were. But look at actual vote shares and what was predicted by the poll (see table below).

This clearly shows that the only reason any poll prediction based on CSDS data—which grossly overestimated SP performance and significantly underestimated BSP vote share—came close to the result is that there was one significant error in sampling (which is why vote shares are so off) and a second significant error in converting vote shares to seat percentages (which is why such high vote shares gave seat counts close to the actual). The two large errors more or less cancelled each other out, but two wrongs do not make a right. Clearly, any analysis of the poll results based on CSDS data is not just erroneous, it is laughable.

Consider the main claim made by both Christophe Jaffrelot and Sanjay Kumar: Jatav votes moved away from the BSP. The 1.9 per cent difference between the BSP’s overall vote share and the CSDS estimates is large enough to take care of almost all the 25 or 27 per cent of Jatav votes that the analysis claims may have moved away. The only conclusion, and even that is very tentative, we can draw from the actual results is that the move away from the BSP was far less than the 25 per cent being claimed, and could be as low as 5 per cent for all we know. That is not much of a conclusion, but it is the only one we are entitled to, despite the hyperbole. The same problem comes up with claims about the return of Muslim votes to the SP after the general elections in 2009. This is bound to have happened to some extent, but the data available in light of the 5 per cent error in the SP vote share does not allow us to say anything substantive.

At the end of all this number crunching, all we are left with is the conclusion that the 3 per cent difference between the SP and the BSP is the result of some shifts in votes, but we are unable to make any sense of these shifts from the CSDS data in light of the actual results. Simply put, we can say as much with the CSDS data as we can without it, which actually amounts to saying that the exit poll is useless.

While this conclusion is based on an analysis of this specific CSDS poll, it holds even truer of all the other exit polls that were used in this election. This reveals problems with much of the caste- and community-based analysis of Indian elections. The actual vote percentages do not provide us figures for estimating caste or community votes. The exit poll data in almost every election has enough error to render all quantitative assessments wrong. In effect, we are trying to draw conclusions about numbers that are smaller than the margin of error in the estimates. Consequently, much of the nonsense regurgitated by our political ‘scientists’ is not worth the paper it is written on, and as far as qualitative assessments are concerned, we are better off reading journalistic analysis that deals with these issues in broad terms.

This does mean that all the shows you watch on TV, or have been watching for over two decades—where ‘learned’ analysts pore over opinion or exit poll figures to give you insights on how caste and community influenced the actual result—are largely an exercise in fiction, the pretence of expertise an attempt to paper over the subjectivity of the analysis. In all honesty, if psephology has to have an honest future in India, it needs to go back to the drawing board, relook at sampling techniques derived from Western models that cannot factor in Indian diversity, and come up with some reasonable mathematical and statistical models that can deal with the problem—if that is at all possible. That is unlikely to happen. So in the meantime we can continue to watch such shows, provided we remember they are not meant to be taken seriously.

About The Author

Hartosh Singh Bal

Hartosh Singh Bal turned from the difficulty of doing mathematics to the ease of writing on politics. Unlike mathematics all this requires is being less wrong than most others who dwell on the subject.