NEET 2018: The solution is normalization

If there is one exam worse than JEE, it is NEET. And it is compulsory for all medical admissions. Earlier, we have seen different question papers in different languages, but no attempt to see if the two papers have similar difficult level, or normalize the marks if they aren't. That is not a surprise to those who have followed CBSE's conduct of JEE Mains in the past. There is no normalization in the multiple papers they offer (online and offline).

This year, a whopping 49 questions in the Tamil version of NEET had wrong translation. Well, at least they translated so that everyone gets the same set of questions. Better than completely different papers of the past. But 49 wrong questions out of 180 total questions. Obviously students would complain. CBSE refused to do anything about it. The matter goes to high court. CBSE argues that students should have read the English version since they had already issued the instructions that in case of any problem in translation, English version would be considered authentic. But they couldn't answer a simple question. How would the students know which questions are wrongly translated, and hence they should look at English version. And if it is to be assumed that everyone understands English question paper, why give options of local language to begin with, or was it the case that students from vernacular background are required to read both papers and spend extra time (without any compensation, of course).

The court was aghast at such arguments. CBSE should have suggested how one could fairly treat the 24,000 students who had taken the test in Tamil language. And sure enough, the court ordered that all 24,000 students should get full marks in all 49 questions.

Obviously, it is unfair to others. It is no one's argument that all these 24000 students would have solved correctly all 49 questions, if there was no error in the translation. And by awarding them full marks, they have been given vastly higher marks than what they would have scored in an error free exam. And thus, they have been placed higher than those students who took exam in English.

How do we know how many marks these 24000 students would have scored in an error free exam. We really can't say about an individual. And then there are second order effects like wasting time on a question which is now ambiguous, say. But statistics does allow an approximate answer. We can normalize marks based on 131 questions that everyone has been able to attempt.

The problem is that CBSE has never learnt how to normalize marks. They haven't done it in JEE before. They do a completely horrendous "normalization" of 12th class marks, which is nothing but grade inflation. So they don't know how to provide at least a statistically fair solution to these 24,000 students.

Is there a way to study correlation between 131 questions and 49 questions. Can we do this analysis for different categories of students - all India, students from Tamilnadu, students from TN Board, etc., and figure out which class is most similar to the group of 24,000 students who chose Tamil as the medium of question paper. Is there strong correlation of some questions out of those 131 with each of these 49 questions. How difficult is each question for that cohort. Based on these types of questions, one can start looking at what would have been the likely score of the student if there was no error in translation and if they were like an average student of the cohort that we are comparing them with.

But, I suspect I know what will happen eventually. CBSE will look at the marks of 131 questions for each of those 24000 students. We will then see how many marks did other students getting identical marks in 131 questions got in 49 questions. We will assign that many marks in 49 questions to those students. This is a very simplified version of normalization but something is better than nothing and courts might agree with this, and frankly, CBSE will only care what will go through the courts.

Any solution based on statistics will cause heartburns. But it is the best you can do, short of conducting the exam again.