The mess called JEE Advanced

Supreme Court, yesterday, stayed the process of admission to IITs, NITs, and several other institutes. A student has filed a petition a couple of weeks ago, in which she has claimed that awarding marks to all students for wrong questions was fundamentally flawed and has resulted in significant change of merit, and that there were alternative ways in which the problem of wrong questions could have been sorted out. The writ petition can be read on the same page as this news report.

I read the petition. I don't know if everything being claimed there is correct or not. It is for IITs to decide on how to respond to those points. However, if even ONE of those claims is correct, I think it is a matter of shame for the IIT system. The news paper reports suggest that IITs responded yesterday to the petition by arguing that the process of admission has already started, that about 30,000 students have already accepted admission, and that it will take a long time to recheck everything, etc. I hope this is incomplete response and that they have said more things. I hope IITs are able to argue that the suggestions made by these students are somehow wrong or unimplementable. Because if they have not argued on those lines and cannot argue on those lines, then we can have no trust in JEE any more. And I believe that a fair JEE was always one of the most important reason behind brand IIT. If JEE is no more fair, then it won't take long for brand IIT to go down as well.

But I have my doubts. People who have followed this blog would know that my confidence in fairness of JEE and in the competence of IIT system to conduct a large public exam has always been very low. Every year, we see wrong questions. Every year, we see that IITs don't respond to alternate answers by external experts (or even internal experts).

Let us look at the main issue. Apparently there are questions which got misprinted in Hindi medium. The claim in the petition is that because the question was misprinted, IITs decided to award marks to every single student. Now, what is the percent of students who sought Hindi question paper. I believe that the number is 3-4%. Only a couple of variants out of 10 variants of the question paper had a misprint. Which means that about 0.5% of the students received a different question. Now the petitioner is claiming that it was possible to identify these 0.5% students who may have given a different answer, and for these 0.5% students, one could do different grading. There is no reason to give full marks to all students. Makes eminent sense. If you incorrectly identify (why would you) 5-10 students, say, and 1-2 of them could have been in the top 10,000 ranks, then you are unfair to 1-2 students out of 1.6 lakhs. But by giving marks to all 1.6 lakhs, you have been unfair to may be 10,000 students who could solve that question. Now, if you are faced with a difficult situation, and the only two options are that you be unfair to 10 students or you be unfair to 10,000 students, what would you choose. I am sure the answer is obvious. You would rather be unfair to 10 students than 10,000 students. But IITs don't think like that. (And I have been inside the room with JEE Implementation Committee in the past, so I know the culture there.) The IITs would argue that let us give bonus marks to everyone. It does not matter how many students you are unfair to, since no one would do such back of the envelope calculations. If you give marks to everyone, there is a greater chance of happiness all over, and less chance of someone going to court. And the only thing that matters is a court case, and not fairness in the system.

At another place, the petitioner has pointed out that while the IITs have given out one single answer to a question, a large number of coaching places have given alternate solutions which result in different answers out of the four choices. Now, let us remember that some of these coaching classes have some of the finest Physics/Chemistry/Maths faculty, some times better than the paper setters. If these people are saying that there is an alternate solution, and showing that alternate solution on their websites, IITs must respond to them. They must point out why those solutions are not acceptable. I know what IITs will argue. How can they respond to each and every answer given by any tom, dick and harry. But sorry, they are not tom, dick or harry. There are typically, only a handful of answers which are different from IIT answers, and there is no reason why IITs can not respond to those handful of answers, and that too, as I said above, by some of the finest brains of the country. Again, I have had experience of being in the room. Typically, it happens because the question is ill framed. The language is ambiguous. And no paper setter would ever admit that the language is ambiguous. And if they were to respond they will have to admit in writing that a particular statement only means something and that in their opinion alternate meanings are wrong. Now, this interpretation can be easily challenged in a court of law. So, as I said above, the fairness does not matter. Only court cases matter. So we won't agree to those alternative solutions, since our ego is hurt, and we won't explicitly respond saying that, since fairness does not matter. Only court cases do. Now, in this particular case, either IITs must respond as to why multiple answers are incorrect or agree to regrade as per multiple answers.

I also want to know if there is any accountability of paper setters. Does their substantial remuneration depend on their performance. Are they told that they will get x amount of money, if there is no dispute about their answers, but will progressively get lesser and lesser money, if it turns out that they made a mess. Is there a process of black listing professors who set erroneous questions. Why does it take weeks to publish the answer key, when the coaching classes normally publish them within an hour. Wasn't a key prepared several months in advance. Why is that key not announced within an hour of the exam.

IITs have argued (as per media reports) that it is too late, that about 30,000 students have taken admission, that regrading will take too much time. First of all, regrading will take a few minutes. Yes, a few minutes. If IITs are claiming that it will take a long time, then they are lying, and someone should be persecuted for lying to the court in an affidavit. All the answers have been scanned and stored. Now, only a key needs to be decided and that needs to be applied to the stored answers. We don't need to scan the answer sheets again. And applying the new key will take a few minutes.

If the scale of the problem is really as serious as this petitioner is claiming than it does not matter if 30,000 students have taken admission. We must restart the process, and if that means that the semester will be delayed by a week or two, so be it. Let us not forget that in 1997 when the JEE question paper was leaked, the semester started 5 weeks late and the heavens did not fall.

My only concern is that the petition seeks as the last resort (page 20, prayer 'e' of the petition) a direction to the IITs that all those who took JEE Advanced this year be allowed to take JEE Advanced next year. I suspect that IITs will latch on to this demand and agree to it, if they are allowed to continue with the admission with the current merit list. This particular petitioner may be ok with such a decision, but it would be grossly unfair to thousands of hard working students (again assuming that what is written in petition is correct).

It is becoming increasingly clear that conducting large public exams is not a core competence of IIT system. I am sure some of you would disagree and point out that in an earlier era JEE was conducted wonderfully well. Well, what was done in the pre-RTI era can not be termed as wonderful. It is just that we don't know what happened then. A few years ago, when the previous government wanted "One Nation, One Test" and we successfully resisted that, one of the commitments that IITs had made was that besides the JEE Implementation Committee (consisting of all JEE chairpersons and vice chairs), there will also be a new standing committee which will look at the longer term horizon and advice on how JEE could be made better. That committee does not exist today and all the issues of JEE are left to be resolved by the current chairmen and vice chairs who are under pressure to somehow manage this year's JEE with as few court cases as possible. I think it is high time IIT system has a standing committee to look into the longer term.

Added on 8th July:

A few questions have been asked offline, and I thought I should address those questions.

Q1: IITs appear to have claimed (see the ToI news where this is mentioned) that it is not possible to identify the students who solved the specific Hindi paper with that misprint. Is it possible.

Ans: JEE knows exactly who applied for Hindi question paper and who applied for English question paper. JEE also knows which code paper was given to whom. Now, the only issue is that a student could possibly have both Hindi and English version of the question paper and looked at the English paper and answered it accordingly, even though in JEE data base s/he is shown as a student who received the Hindi question paper. There may be 5-10 such students in the country, but theoretically, they could be 100 or 200 students. So technically, IITs are right in saying that they do not know who have looked at only the Hindi paper before answering.

However, this is just an argument to not do anything to solve the problem. Remember the principle at this stage should be that out of all bad options, we should select the least bad option. So what if JEE were to say that we will grade you according to the language you were supposed to be looking at. This would perhaps be unfair to 5-10 students. The current solution is unfair to 10,000 students. There are other options. What if we consider both 5 and 6 as the right answer for those few students and consider only 6 as the right answer for everyone else. Or even, consider 5 or 6 as the right answer for all students. So let us understand that the only options with IITs are not binary (as Attorney General, the highest law officer of the country is claiming in the highest court of the land). They have multiple options with varying impact on fairness.

Q2: Isn't there a Supreme Court judgment in 2005 that bonus marks in case of wrong questions can only be given to those candidates who attempt the question. Why did IITs award marks to all.

Ans: I have not read that judgment. But I doubt if that would be relevant in the current case. Earlier, the exams required long answers to be written. So if someone has written something in the answerbook which has anything to do with the question, it could be taken as a proof that the answer has been attempted. But now with fully objective type question, students try to solve the question in rough sheets and then fill in a bubble. If they couldn't reach any answer, or if they have a doubt about their answer, they will not fill in the bubble. Since those rough sheets are not kept, and sometimes rough work is done on the question paper itself which is allowed to be taken away by the student, there is no identification possible of students who have "attempted" the answer. Filling in the bubble is certainly a proof of attempt, but not filling in a bubble is not a proof of lack of attempt. But note that this petition is really not dependent on 2005 ruling. This petition is arguing that a better solution key would be more fair than giving bonus marks to all.

Q3: What happens to a student who has rejected BITS/IIIT-Delhi/IIIT-Hyderabad, etc., and is now told that his JEE Advanced rank is much poorer. Wouldn't revising the grading scheme be detrimental to his/her interests.

Ans: Yes, it will be unfair to such a student. However, remember the philosophy that I am suggesting. If you have to be unfair to some, prefer a system with least unfairness. And would you rather be unfair to someone who did not deserve an IIT seat or would you rather be unfair to someone who deserved an IIT seat. Also, it is not as bad. If there is a new ranking and new counseling, the student may not get IIT but will get NIT/IIIT etc. So unfairness to an undeserving student is going to be rather small.

Also, even this unfairness can be handled. The court can order a 10% increase in all IITs in all programs as a one time measure to reduce this kind of unfairness. A single batch being 10% larger will not be a disaster for any IIT. And it will give relief to large number of students whose ranks change. Note that if someone's rank changes drastically downward, there may still be cases where they will not be fully compensated, but heck, these guys didn't deserve to be in IITs by a wide margin.

Added on 9th July:

As a comment on the next blog, Mr. Kandasamy Subramani has sent the link to a petition on change.org. There the issue of bonus marks for supposed misprinting has explained very nicely. Please read that.