JEE 2013: More on Normalization

That some students and parents are upset will be an under-statement of the year. They are really angry. And the least I can do is to write my side of the story.

First of all, let me explain my role and JIG's role in deciding the formula.

The process to find a normalization process started soon after the August 14th notification of the MHRD regarding the admission process to be followed by IITs, NITs and other CFTIs (Centrally Funded Technical Institutes) in the year 2013, which talked about 60:40 ratio of JEE Mains marks and Board marks (normalized on the basis of percentiles) for admission to NITs and other CFTIs. A committee was formed under the chairmanship of Dr. S K Joshi. The committee asked and received several suggestions from people working in the areas of testing, measurement and statistics. The committee first shortlisted two methods as the only reasonable methods to do normalization and then suggested one of them. Their report was discussed in several forums and there was a huge amount of debate on which of the two methods should be chosen for normalization.

In April, 2013, the JEE Apex Board decided to set up a JEE Interface Group, and ask this group to look at S K Joshi committee report and give its recommendation. I was a member of JIG. The JIG decided that we should recommend the average of the two methods. The JIG recommendation was then discussed in JEE Apex Board and they accepted the recommendation of JIG. Soon after that, CBSE published the formula on its website. So note that JIG's role was fairly limited in deciding the normalization formula, and even if I wanted to play a role, it wasn't possible.

As soon as CBSE made the formula public, I decided to write about the formula in this blog. Immediately a lot of concerns were raised about it. I then asked in one of my comments for people to suggest an alternative. The comment page was kept open for about 5 weeks, and there was no alternative which I could be convinced about as being better than the formula arrived at by the S K Joshi Committee.

Indeed, almost all the suggestions were a variant of just taking the percentile score of the student in the board. I had written in one of the comments, why this was a bad idea. This idea essentially said that all boards are equal, which I could not agree with. Also, this idea meant that everyone will have very similar marks at the upper end, and at the upper end, the ranking essentially becomes same as JEE Mains ranking, with minor perturbations being introduced by board marks.

Since, there was no proposal better than the one which had been proposed by Joshi Committee, despite such a large number of people reading the blogs, and in my final comment on that blog, I clearly announced my support for this formula, and said that it was now too late to change, and any newer suggestion, I will forward to who ever are in the committee for 2014. It was clear to me that one cannot hold admissions in the entire country in order to continue a debate in which for over 5 weeks, no better idea has been generated.

I find it extremely strange that people are talking about my appearance in Delhi High Court as going back on my words. I had announced my public support to the formula (which, as I have repeatedly said, is not my formula or even JIG's formula) about one month before the Delhi High Court appearance, and I have not said anything in the court which I had not said on the blog. So why are people surprised, shocked, angry, I do not understand. I have only repeated which I have publicly said earlier, nothing else.

Let me come back to the Delhi High Court case a little later. Continuing with the formula, I did take all those initial complaints and examples seriously, and requested CBSE that I want to see the data of all those students before the result is announced. I went to Delhi on 1st July, despite the visit of President of India to IIT Kanpur campus on 5th July, where I was the Chairman of the Organizing Committee. CBSE had told me that they were trying to announce the result on 1st evening, and latest by 2nd.

And let me tell you what I noticed in all those complaints. In most cases, the student had a decent JEE Mains score, but was not in the top 2-3 percentile in the board marks, and hence lost the ranking. I even exchanged a couple of emails with a couple of such students to understand what their complaint really was, and I quickly realized what the problem was.

When MHRD announced the new admission policy which said that the board performance will have 40% weight, it had two kinds of impact on the students and parents.

In one group (let us call them Group 1), there were students who were convinced that somehow a formula will be used for normalization that will render the difference in board marks meaningless. They were convinced since many of the influential persons were going around in media giving some formula and claiming that this formula will only cause minor perturbations in the ranks of JEE Mains. (This was wrong as one could easily see that even if we were to use that formula for normalization, it will lead to top ranks being decided on the basis of JEE scores and board marks causing minor perturbations as they called it. But the latter ranks being decided by the board performance with JEE marks causing minor perturbations. So the best institutes will continue to admit students on the basis of JEE marks, while the weaker institutes will have to admit students based on board marks. And all this is besides the fact that such a process would have been completely unfair to students from good boards.)

There is another set of students (let us call them Group 2) who did not believe this theory that 40% weight should lead to only a minor perturbation of the ranks. These students realized that a 40% weight means that the board exams would be almost as important as JEE Mains and should cause serious impact on ranking. They did not know what that formula was going to be, but they were convinced that there can be a mechanism where 40% weight would lead to and should lead to a serious impact on ranking.

The Group 1 students continued to focus on JEE Mains, while the Group 2 students gave some additional time to Boards as well.

Now, it was unfortunate that the students had to decide their strategy for the year (to be in group 1 or group 2) without knowing what would be in their own interest. And this is the reason why I said on my blog that the 60:40 rule should be challenged in a court. A few groups approached me in August/September last year. They wanted to file cases in High Courts, but only on issues related to IIT admission, whether a limit of 1.5 lakh is reasonable, whether an 80 percentile is a reasonable restriction, etc. I advised them that these things, while we may like or dislike, are on strong legal footing. I have read in the last decade more than 100 Court decisions on education related cases to understand what courts sympathize with, and what they don't. I advised them that if they have to go to court, they should challenge 60:40 rule and ask the courts to postpone its implementation till we have clarity on implementation details, and some other things.

Surprisingly, no one challenged 60:40.

Anyway, I digress. Back to normalization and Group 1 and Group 2. Which assumption about normalization is more sound. The assumption of Group 1 is based on statements made by some influential persons. The assumption of Group 2 is based on a more common sense approach - 40% weight should have significant impact on the rank. The final normalization process is essentially saying that Group 2 was right in its assumptions, and not Group 1.

Group 1 took a gamble because they believed some statements, they lost the gamble, and now they don't want to admit that they took the gamble and lost. They need to put blame on someone else, and I become the villain.

Let me give you an example of the complaint. There is a student who says that he had a little more than 98 percentile in JEE Mains, giving him a rank of about 21,000. He has about 88 percentile in boards, and he is shocked that his rank has gone down to 39,000. If his rank in one exam is 21,000 and in the other exam about 1 lakh, and we are going to have 40% weight of the other exam, should he not expect that his rank will go down to 39,000. The formula has worked perfectly for him. But he wants his rank to remain 21,000 and that is why he is upset and angry.

So the anger in Group 1 is really about the 60:40 rule. They always believed that 40% weight would have close to 0 impact on their rank, and this formula is causing 40% impact, and they are not willing to accept that 40% weight should have 40% impact. But I am surprised that your anger is not directed towards people who told you that 40% weight will have 0% impact. (Don't direct your anger at them. They are still very powerful people. They may even now go to Ministers and ask them to change the normalization to make a 0% impact. So work with them, and keep your anger directed at me, who kept warning you through out.)

If there are only 15,000 general seats in all CFTIs (including NITs), to claim those seats you need to be in the top 15,000 ranks. If there are two exams which, for the sake of simplicity let us assume will be combined in 50:50 fashion, it is obvious that you need to be in about 20-30K ranks in both the exams. If you are 10K rank in one exam and 100K rank in the other exam, you will not make it. If you don't understand this, I can't help it. And most of the complaints are from such students, those who have a significantly better rank in JEE than in the boards.

So, as I was saying, I sat down in CBSE office on 1st July, checked out each complaint and figured that in most of the cases, the issue was his/her expectation that the board marks will have 0 impact. Of course, in some cases, the issue of "long tail on the right" distribution of JEE Mains versus the "long tail on the left" distribution of Board marks was causing unexpected gains or losses. But that would happen in any normalization of 12 lakh students.

After the result was announced on 2nd, another round of criticism started. It was mostly, again, based on Group 1 expectation that board marks will have no impact on the ranks. Many people in the JIG were concerned, and we requested CBSE to organize a meeting between JIG and the Professor in Oxford who had written the original paper on this sort of normalization, and another colleague of his. We had this meeting on 9th July. They went through the entire data carefully, showed us some graphs, and explained to the group that a few unexpected outcomes are because of reasons that we had suspected - the long tail stuff that I said in the previous paragraph. But these are few and mathematically acceptable. In any case, these are not the complaints.

Unlike the original decision of the government to do 60:40 without any data, analysis, etc., S K Joshi committee did extremely exhaustive analysis, how the scores in different boards are distributed, what would be the rankings (working on 2012 data) with different tweaks of normalization process, how much is the impact on each board, and so on and so forth. And they have communicated with so many experts in ISI, IITs, NITs, and so on. A rare situation where a decision has been reached at after talking to so many people outside the committee. But those who feel that 40% weight should have 0% impact will find no solace in all this.

For a long time I had no idea what is happening about normalization. The first time I heard of S K Joshi committee was sometime in January. I was given a copy of the interim report. I did not know at that time that they had shared it with several people. I felt that I have been given a confidential copy, and hence I did not share it. But when I read it, I realized that the so-called Group 1 (as above) would be upset with this. People who have got good ranks in JEE Mains and had a weaker performance in Boards would have to have access to some good institutions, if they will not find access to NITs and CFTIs. So, in January, I sent email to a few institutes, including IIIT Hyderabad, asking them to do admission on the basis of JEE marks alone for this year. I mention IIIT Hyderabad since they had a detailed discussion with me. Others either did not respond, or told me that they will just use the final rank on the JEE score card. The confusion that happened in May/June was because they put on their admission website that admission would be based on JEE rank. They were assuming that there will be something called "JEE Rank" which will be based on JEE marks alone, and there will be some other name for overall ranking after 60:40. Anyone who approached them for clarification on "JEE rank" between February and June, they explained to them that it was only JEE marks. But in June, when it created confusion, since CBSE started using "JEE Rank" for the final overall rank, I (and lots of others) asked them to clarify and they did.

Now coming to the case in the Honorable High Court in Delhi. As I said in the beginning, there is absolutely nothing that I have said in the Court that I had not already said on my blog. In fact, I did not have to say much in the court anyway. The judges were brilliant. They had read the formula, understood it, and understood it so minutely that I was absolutely amazed. Their main area of expertise is law, but they showed better understanding of nuances of the normalization than what I have seen in most academicians. I mostly had to only confirm whether their understanding of the formula is correct. They only asked me about the outliers, and I told them that it was due to long tail distribution of JEE Mains marks. They intuitively understood what a long tail distribution is. And I did not have to say anything else. I may have spoken for
about 2 minutes only.

Now the learned judges started asking questions, very similar to the questions that anyone should ask of normalization, and what I have asked on my own blogs. And throughout this discussion, CBSE lawyer was silent and I was silent.

The first contention was that there is a change in the process. Judges asked how the process has changed. They could say that the August 14th MHRD notification was different from the formula. The judges said that the formula is not different but just implementation details. Nothing in the formula is inconsistent with the August 14th notification. If it was obvious to a judge, shouldn't this be obvious to students, parents and academicians.

Regarding the delay in announcing the formula, the Court was clear that if the formula was announced a few days in advance or even a few weeks in advance, it would not have had any impact on the study pattern of the student, and hence the delay is irrelevant in this case. (Of course, this does not answer the question why the formula should not have been announced six months in advance of the board exams, when the students could indeed take advantage of that knowledge and adjusted his/her study pattern. But CBSE lawyer, I am sure, would have said that the implementation of a policy requires at least some time, and hence the formula could not be announced in September. So if delay in knowing the exact formula was affecting you, you should have gone to court in September 2012, and not in June/July 2013.)

The judges even asked them to give an alternate normalization formula, if they were not happy with the current formula. They could only come up with the "0.4 * percentile in board" formula, which many of the students and parents are asking for, and about which I have already explained why it is not a good formula. And something amazing happened. The judges took just five seconds to understand the implications of this formula, and they said, "but this formula means that the topper of one board has identical academic credentials as topper of another board." When the other party agreed, they said that they cannot agree with this, and asked a counter question, do they believe that Delhi university topper in a particular subject has same academic credentials as topper of another university. Thankfully, the other party agreed that that is not the case, but I could see where the learned judge was going. If they had said that indeed the two toppers are same, the judges would have perhaps suggested that in that case it does not matter where the appellant studies. All universities are identical. Let him study where ever his current rank is allowing him to study in. So something that most academicians find difficult to understand (that percentiles of two boards are not comparable), the judges understood it intuitively without either me or the CBSE lawyer having to explain.

I was thoroughly mesmerized by their brilliance, not just in this case, but in all cases that preceded this one since morning.

The other party told the judges that I have written on the blog that the policy of 60:40 was absurd. (I chose to not react to it, since what they had said was correct. I still believe that 60:40 policy is absurd. The current issue is not about 60:40, but if 60:40 is forced upon you, then what is the best way to implement it. And I am only supporting one implementation of it. And it does not mean that I like the 60:40 policy.) But the judges, brilliant as they were, were quick to respond. "We are not asking him to explain his views on the policy. He is here only to explain the process." The judges could immediately see the difference between the policy and the implementation, which unfortunately most people fail to see.


So the summary of all this is:

1. I continue to oppose the inclusion of board marks in ranking.
2. But I do not want the admission in the entire country to be jeopardized, and hence have worked towards the best possible implementation of that policy.
3. The formula being used was neither devised by me nor by JIG, but it was given to us by S K Joshi Committee, who have done extensive consultation, and data analysis, and I have had no impact in acceptance of this formula. The charge of JIG was fairly limited.
4. However, as an individual, I am yet to see another normalization process which is better than one being used.
5. The real problem with the normalization process is that students expected 40% weight to have 0% impact, while this process is based on the premise that 40% weight should have 40% impact.
6. I have been publicly saying that this formula is the best (read the comment #201 on my blog, where I have summarized the other suggestions and why this one is better), and that is exactly what I said in the Delhi High Court. There is not an iota of difference between what I said in the Court and what I have written on the blog. So I don't know why some students are upset and feel that I betrayed.


And, of course, I won't allow any abusive comments on this. They all go to spam folder.