Search This Blog

Friday, June 15, 2018

JEE Advanced 2018 Results: IITs Blunder

To understand what went wrong with the ranking list of JEE Advanced 2018, we need to understand the difference between eligibility and ranking cutoff (and admission cutoff) and various legal and other requirements governing admissions. In this article, I first explain the process of admission, which could be quite boring to most readers of this blog, but I do this for completeness sake. Sorry for a very long article. Those of you, who understand the admission process, please go directly to the next section "What happened to JEE Advanced this year."

Eligibility requirement implies that those who do not meet this requirement are not likely to perform satisfactorily in the program under consideration.
If eligibility means the minimum standard to follow a program, why is eligibility different for different categories. One reason is that people from disadvantaged backgrounds often work harder and are able to make up for some deficiencies that may be there at the time of admission. And hence a small difference may be allowed in the eligibility requirements of General candidates and others. Indeed, the courts have decided that the eligibility requirements between General and OBC-NCL (Non Creamy Layer) can be at most 10 percent. However, when it comes to admission of SC/ST students, courts have pronounced that the educational institutions must provide additional support to them to succeed, and hence in their case the difference in the eligibility requirements can be larger. (I am not aware of any legal case regarding Persons with Disability or PwD, but IITs have decided that their eligibility will be similar to those of SC/ST students, essentially admitting that they will be provided additional support to succeed.)
The admission of a student in a particular category can be allowed only if they meet eligibility requirement of his/her category.
IITs have a rather complex set of eligibility conditions (which wasn't the case when I sought admission to IITs a few decades back). These include performance in 12th class (75 percent marks or 80 percentile performance, which ever is lower), when one has passed 12th class (in the year of admission, or previous year, and in case of an improvement exam, the year of first passing the 12th class is considered), minimum number of marks in JEE Advanced in each subject (10 percent for General, 9 percent for OBC-NCL, and 5 percent for SC/ST/PwD), and minimum total marks in JEE Advanced (35 percent for General, 31.5 percent for OBC-NCL, and 17.5 percent for SC/ST/PwD). In addition, one needs to be in top 1 lakh approximately in JEE Mains for General and similarly in top few for other categories.
Finally, IITs look at all those candidates who meet all eligible requirements and rank them. The ranking order is the total marks in JEE Advanced with some tie breaking rules. Everyone who meets all eligibility requirements of General class gets a Common Rank. All reserved class students who meet eligibility requirements of their respective categories get a Category Rank. It means that students belonging to reserved classes may have a General rank too, in addition to their Category ranks.
The Ranking Cutoff is the performance of the student who has the last announced rank. Since IITs announce a rank for all those students who are eligible for admission, the ranking cutoff is same as eligibility requirement in their case. Please note that it is not necessary for the two to be same. If the number of eligible candidates is very large, IITs could decide to rank only as many as they think will be able to fill all seats. But it is convenient to announce all ranks and then ask only some of them to participate in the counseling and in case seats do remain vacant, ask more students to fill up their choices.
The Admission Cutoff is the performance of the student who is the last person to get admission in a particular program in a particular category. Note that admission of reserved class students depend solely on their rank among eligible students and does not depend on the admission cutoff of general students.
To consider the implications of all that is written above, let us take a hypothetical university which has only one program, and it has announced that the eligibility is 60% marks in 12th class for General candidates, 55% marks for OBC-NCL, and 50% marks for SC/ST/PwD. A large number of applications are received, much more than the number of seats, and they announce a ranking for all eligible applications. Then they announce admission cutoffs for different categories. For example, they may say that it is 98% for General, 85% for OBC-NCL, and 60% for SC/ST/PwD. This means that all students with 98% marks or more will get admission to the program. Among OBC-NCL students, all those with 85% marks or more will get admission, and so on.
Note that the difference in the cutoff for OBC-NCL and General can be more than 10 percent. Similarly, the difference between SC/ST and General can be very large as well. One cannot have a rule which says that OBC-NCL students' admission cutoff can be at most 10 percent less than General, or that SC/ST admissions cutoff can be at most 20 percent less than General, etc.
What happens if the number of eligible candidates in a particular category or in all categories is less than the number of seats (or some people who had earlier shown interest by applying now do not want to seek admission). 
There are only two options in this case. One, you have a smaller class than what you planned for. That is, leave the seats vacant. Two, you change the eligibility conditions. Changing eligibility conditions means that we are either willing to admit academically unprepared students, or our earlier eligibility conditions were arbitrary and ad-hoc and therefore, there is no harm in changing them.

What happened to JEE Advanced this year?

When the result was prepared, it was found that less number of students were eligible for admission to IITs than any time in the past, compared to the number of seats on offer. The Joint Admission Board (JAB) which consists of all Directors and all JEE Chairpersons decided that they will announce the results by sticking to the eligibility conditions that had been announced long ago. In public statements, they said that the number of successful candidates were 1.6 times the number of seats, and they were confident that all seats will be filled by these eligible students.

This only means that none of the leadership in our greatest institutes had done their homework. Let us look at the claim of 1.6 times, for example. They said that there are 11279 seats and 18138 eligible candidates. This was patently false, since there are about 800 extra seats to accommodate women reservation as well. And therefore, the ratio of eligible candidates to seats is only 1.5 and not 1.6. That JEE Chairperson keeps repeating 1.6 as mantra only shows his ignorance and perhaps a secret desire to not have 14% reservation for women.

There were other issues with this. As we have explained above, admissions are done in different categories. So while overall, the number of eligible candidates may be 1.5 times the number of seats, one had to look up whether this is true for each category of students. If they only had looked at numbers category wise, they would have noticed that this time the number of eligible candidates in OBC-NCL category were actually less than the number of seats reserved for them. On the other hand, for SC category the number was 2.3 times the number of seats. How could this be missed by everyone in JAB. It shows how completely unprepared they come for the meetings where such important decisions are to be taken. That says a lot about the leadership of our IITs.

Media reported some extremely stupid statements. I find it difficult to believe that even this clueless leadership could have made those statements, and perhaps there is a communication gap between IITs and Media. For example, it was said that in case the number of seats remaining vacant is large, we can always announce a second list then. That a special admission round could be held at that time. This was stupid because you would have converted OBC-NCL seats to General in the usual rounds. Then when you announce the second list in which there would be more OBC-NCL students, would you tell these students that they can't get any seat since all vacant OBC seats have been converted to General (as per the law), or would you tell those General candidates, sorry, we are canceling your admission now that we have more OBC-NCL candidates. Obviously, the solution of extended list had to be implemented now, and not later.

Also, the last General candidate to get any seat in IIT system last year had a rank of about 15,000 (Architecture courses at Kharagpur and Roorkee). This time, the total number of students who had a Common Rank was less than 15,000. While the number of seats had gone up by 1000. How were they sure that this year, the number of candidates were enough to fill all seats.

Added on 18th June:

It has been pointed out to me that last year's closing rank for General seats was around 11,000, if we ignore Architecture courses. The closing rank of Architecture depends a lot on who all give AAT, and there are only a couple of seats in each program that were filled by students beyond 11,000 rank. This year, the last Common Rank announced was around 12,000, and hence there were enough candidates.

My response to this argument is that last year, no OBC-NCL seat was converted to General. This year with so few OBC-NCL candidates, about 1000 seats would have been converted to General, if the new list had not been announced. So, 12,000 would have been insufficient to fill all seats. In addition, there are more than 1000 extra seats this year, including the women reservation quota seats.

It would have been at least courageous to announce that IITs recognized that the number of eligible students is less this year and that they will leave seats vacant, since they didn't find enough good students. Not admitting this, and yet announcing only a small list of successful students was stupid to say the least.

Of course, if they had indeed admitted that they don't intend to fill large number of seats in IIT BHU and IIT ISM and may be some other IITs as well, it would have been based on a completely arbitrary eligibility condition.

In general, the eligibility condition should be based on some research. Is there any evidence that statistically speaking students getting 40% marks in JEE perform significantly better than those getting 35% marks. Also, have IITs not compromised with their pre-announced eligibility conditions in the past. What does giving 18 bonus marks do other than compromising eligibility conditions. Indeed, every single year ever since the system of JEE Mains and JEE Advanced has been created, bonus marks have been given to students on some pretext or the other. Most boards in the country give bonus marks. So why suddenly we are saying that this year despite the less number of eligible students we will not give any bonus marks to anyone and artificially increase the pool of eligible students.

I am glad that finally, JAB has decreased the eligibility condition. But I am sad that ministry had to intervene to get it done. The ministry intervention is nothing but an interference with IIT autonomy, but if the entire leadership of all 23 IITs combined will act in the way they did, the political leadership is not going to remain silent. I wish IIT leadership does not give occasions for political leadership to intervene in future.

Why IITs have 35% as eligibility requirement?

Let us go back in history and see how IITs admitted students then. When I was seeking admission a few decades back, the eligibility condition was simple. Pass 12th class with 33% marks with any subjects (not even PCM was required). Same eligibility for all classes.

But there was something hidden, and these were pre-RTI days and hence remained hidden. After JEE results were tabulated, they will decide individual subject eligibility. Those who did not get that many marks in that subject were not eligible for admission. This eligibility condition was never announced but was implemented, and was different for General and SC/ST. (There was no OBC or PwD reservation at that time.) Every year, the eligibility could be different, but it would usually be a number which would make about 15-25% students to be eligible. If we take the set of students who were eligible in all three subjects, it would typically be 15-20% of those who have taken the exam. Out of these about top 3% would be assigned a rank, and top 2% would actually be offered admission. The rank was purely based on total score.

The idea at that time was that IITs wanted to admit only those students who were good in all three subjects, and did not want to admit students who were excellent in two subjects and poor in third, and hence the subject wise eligibility, which was never publicly announced.

For SC/ST, there was another hidden condition. They had to score a total of at least 75% of the total scored by the last general student on the rank list. The idea was that if the gap between the General and SC/ST students in the same class was very large in terms of their academic preparation, that would lead to very poor academic performance by SC/ST students and may lead to termination of their programs and worse. However, the seats reserved for SC/ST were filled up by admitting students against the vacant seats for a one-year preparatory program. This was an excellent program, and most SC/ST students who finally took admission to IITs after one year were able to compete well with the rest of the class at times even better than the students who had directly got admission that year.

The secrecy and the absence of RTI helped IITs continue with this system till 2005. In 2006, it was alleged that the subject wise eligibility conditions were decided in a malafide fashion to keep a specific JEE candidate out of the final list even though his performance was better than any cutoff decided in the past. RTI, which had just become the law, helped the cause. The newspaper headlines forced IITs to stop these hidden conditions and announce all eligibility requirements in advance.

Bitten by what happened in 2006, IITs pretty much decided to not have subject wise eligibility conditions. The subject wise eligibility conditions were kept so low that there was no danger of disqualification of any student in future. It was 20 percentile score in each subject in each category. Now, 20% of the students giving JEE get close to 0 marks in a subject. So, the marks based eligibility was so low that it did not bar anyone from selection on this count. However, when media asked for the exact cutoffs under RTI and put that out as a news item, it was embarrassing to see single digit marks being touted as eligibility condition. The right thing to do at that time would have been to completely do away with subject wise eligibility (or enforce a decent number as the eligibility), but IITs decided to have a 2-digit number as eligibility. They announced that the minimum eligibility in each subject would be 10 percent for General and 5 percent for SC/ST.

What was remarkable about reducing subject-wise eligibility was that it was taken without any application of mind as to what will be the implication of this on the quality of admissions. There was a long standing principle in the admission process which I just stated a little while ago. That IITs will admit only those students who were good in all three subjects and not admit students who were excellent in two and poor in one. This principle was being thrown out of the window without any discussion in Senate of any IIT. It was said that in the era of RTIs, we had to be transparent and hence this. This was a wrong argument, since transparency did not mean giving away your principle of admission that had stood the test of time for 45 years. For example, if they had decided the eligibility to be 20 percent and 15 percent for General and SC/ST respectively, that too would have been transparent. But, IIT leadership wasn't much better at that time.

What is interesting is that in 2007, the batch that was admitted based on the new eligibility had lots of students who were exactly the type we were avoiding for the past 45 years. While JEE and IIT system would refuse to do any research on the implications of their decision, I decided to inquire from my contacts in different IITs if they felt that the student cohort was any different that year, and in multiple IITs, the impression was that a much larger number of students were weak in Maths in that batch. At least two IITs reported that a significantly larger number of students had received F grades in Maths courses in the first year. Of course, not a scientific study, and impressions should not be used to change policies, but impressions such as these should have encouraged IITs to do a proper scientific investigation. But that was never done. And we continue to offer admission to students who are excellent in two subjects and poor in the third.

May be I should clarify that I am not against the new admission philosophy. What I am against is that rules for admission have been changed without even understanding that the admission philosophy has been changed as a side effect. If someone were to propose with good arguments that we change our admission philosophy, it was discussed in faculty at large, and then changed, it would be fine with me. But the changes in IIT JEE are either forced by MHRD or by fear of an adverse ruling by the courts. In my 25 years at IIT Kanpur, I rarely have had a discussion on how to change JEE so that we get better students.

Anyway, coming back to subject wise eligibility conditions, removing subject wise eligibility (or requiring such low marks) would have had a serious repercussion on how we admitted SC/ST students. You could not deny admission to any SC/ST student who had received such low marks in JEE. The previous policy of only admitting SC/ST students who had received 75% marks of the last ranked general candidate would have been struck down by the courts, if people knew about it. And in the RTI era, IITs could not continue doing so. But the system of admitting only good students directly and rest through a one-year preparatory program had worked extremely well and really trained these students to compete with everyone else on an equal footing. IITs did not want to give up on that. IITs knew that if we were to fill up all SC/ST seats directly, they will not be able to compete with the other students and lots of them will face their programs being terminated, or will take more than 5 years to graduate (which is happening now).

Indeed, in 2008, the landmark judgment regarding OBC reservation (and a later case seeking clarification) resulted in a very clear principle. You cannot restrict admission in reserved classes based on the difference between the marks obtained by general category and reserved category. So what was felt as illegal was actually declared as illegal, though in the context of OBC reservation.

So another eligibility condition was created, which was to simulate the condition of only good SC/ST students getting direct admission and remaining through one-year preparatory course. This was the total marks in JEE.

Initially, based on the previous distribution of marks, it was felt that 35 percent as eligibility requirement was a safe requirement that would always result in sufficient number of general candidates to fill up all seats. And SC/ST eligibility was proposed as 75% of that.

But that is when the issue became political. The two tier admission policy of IITs, which I believe has helped SC/ST students tremendously, was under attack by activists, and it was said that IITs are deliberating causing most SC/ST students to graduate in 5 years. The reality was that many of these students would not graduate in even 5 years if they were given direct admission without the solid support that the one year preparatory course provided. But IITs succumbed under pressure and decided to have SC/ST eligibility as 50% of General students. So it was announced as 17.5 percent, which means that most seats are filled now directly, and there is hardly any student in the preparatory program. The number of students in the preparatory program is so few that only a few IITs run that program on behalf of all other IITs.

What should be done going forward?

If we are only looking at how to avoid the embarrassment of this year's JEE fiasco, then the solution is simple. Reduce the General cutoff to 30 percent, and in future, if there is still an issue of less number of eligible students, just add bonus marks to all participants (like what CBSE does). That will take care of it.

But in my opinion, each such fiasco is really an opportunity for the IIT system to think of admission process as a whole. Who is the right student for us. How do we test for that. How do we decide our eligibility requirements to ensure that we attract such students.
In 2011, when the then Minister wanted to make significant changes to IITs' admission process, the IIT system had promised that we would have a standing committee that would look at the admission process on an ongoing basis and keep making suggestions to JAB. A committee was indeed made, but with a very limited mandate and a limited timeframe. It is time that we make a standing committee to look into these issues from a long term perspective. The JEE system currently is geared to somehow finish the task each year. It is not geared to think of any innovation. And IITs must encourage research on different aspects of admission.

Added on 15th June:

A few friends have told me that I am being harsh towards JEE, and if there were a handful of seats (they put the number in single digits) vacant, it wouldn't make a difference, and the pre-announced eligibility conditions are legally binding. So let me make a prediction here. At the end of counselling session in July, there would not be more than 10,000 students from the first set of 18138 selected candidates who will accept admission in IITs. So we are talking about 2,000 seats being filled in by new students (or remaining vacant). I think a few women seats would still remain vacant. So I am criticizing the result for having a few vacant seats. It is enormous number of seats and many of the less popular programs would have got 0 students.
I agree that there is a legal issue in changing eligibility conditions. But as I mentioned above, IITs have compromised eligibility conditions every single year either by directly reducing the eligibility or by adding bonus marks. And they have not been challenged on this count. To challenge this would require the existence of a person who stands to lose by this. I don't think there is any such person in this universe.


sunita sachdev said...

Well said!

But I think you missed on bonus marks! The qualified number of candidates would have been much much lower if the authority wouldn't have changed the rules for the new pattern questions from decimal places to integers and introducing range for answers. It was obvious that being the paper online, the result could be seen instantly by the authority and to make the qualified list at least 1.6 times, the new set of rules were introduced. It was as good as giving bonus marks but only to the selective candidates.

gautam barua said...

It is indeed time to ponder: why have cut-offs at all? Having cut-offs implies that the standard of the test papers are uniform over the years (yes, cut-offs were relative to performance earlier, but was deemed illegal by courts). But this is clearly not so. Then there are aberrations like wrong questions resulting in grace marks which raises the average marks. If students are getting admission after getting 0 marks in JEE Advance, does this not indicate that the tests were too tough compared to what students have learned / know? So if students qualify with 0 marks, it will act as a regulatory mechanism, forcing test setters to adjust the level of questions the following year. Can Govt. funded institutes insist that we will admit students only if they meet our standards? Must we not instead meet the standards set by society in the form of what is taught in schools and coaching institutes etc?
Coming to coaching institutes, by now it should be clear to all that if you want to do well in JEE(A), you must attend a coaching institute. They thrive because the standard of the JEE(A) tests are so high (not, as IITs have been taught to think: the standard of schools are very poor - see my argument above). Because of the coaching institutes, the number of students getting marks above the cut-off are still more than the number of seats. So the presence of coaching institutions adds comfort to the view point of IITs that standards are global (or as decided by IITs). But we know what coaching institutes do to students. We have seen how the quality of students have gone down over the years. How students learn to match patterns (rote learning - even for problem solving, by remembering solutions to different types of problems), but forget how to analyse, how to innovate, how to apply first principles, etc.
So, why can we not ask the Govt. to ban coaching institutes? You know the type that thrive in Kota, Hyderabad, and now in most big cities? It can be done, and I am confident it will pass legal muster.
What will happen then? Firstly IITs will be under direct pressure to make question papers as per students' abilities. It will make parents put pressure on school authorities to introduce extra classes to "coach" final year students for entrance exams. Schools will have to improve their teaching then. Students entering IITs will not be like zombies, burnt out from excessive "coaching". Who knows? The standard of Board exams may also improve as pressure for better teaching increases all over the country?

iitmsriram said...

Just to add some data to your comments on the preparatory courses; IITM has been running preparatory courses for many years and I have looked closely at the students who went through this. A significantly larger fraction of them were completing their degree program in the assigned time than reserved category students who were admitted directly. Also, as a group, the graduating CGPA of the preparatory course students was about 10% higher than the group of directly admitted students. Clearly, the preparatory course was working as the students who went through this were doing better than the very students who had scored more than them at the JEE stage (or, perhaps, by some strange twist, those who were scoring a bit lower in JEE were actually the "better" students). As a minor aside, I believe the "old" cutoff for SC / ST students was two third of the last general candidate, not 75% as mentioned in your post.

Sunil Joshi said...

excellant... thanks for enlightening.

Dheeraj Sanghi said...

Sorry for late response. It was due to blogger not informing me of any new comments for the last one month.

Dheeraj Sanghi said...

@Sunita, I would disagree with you. I think it is perfectly fine to consider 7 as 7.00. In fact, if the answer is an integer, 7 is in some sense more correct answer than 7.00. And this hasn't changed too many marks anyway.

Dheeraj Sanghi said...

@Gautam, I agree with you first part, but do not agree with banning of coaching. It will only make things underground, more expensive, and accessible to only people with resources and connections. IITs should learn to work with the boundary conditions they find in society and not try to insist that society changes itself. If we can lead that change somehow, well and good, but don't depend on courts/legislature, etc. There is really no reason why we can't initiate a research cell to study admission tests and the rest of admission processes.

Dheeraj Sanghi said...

@Sriram, thanks for your comments. A couple of other emails have also said that it was 2/3rd.

sunita sachdev said...

@Dheeraj I know logically 7 makes more sense then 7.00 but while writing instructions in the paper, the paper setter also must be knowing the same but the instructions stated to write two decimal places even if it's 7. That's one part, the second part which I mentioned range of answers considered correct, for eg. paper 2 question 13, the correct answer is 8500 but the answer considered correct in range [8430, 8470]. If it remained unnoticed earlier (before 20 May) then how come suddenly after the exam was conducted, next day morning the revised instruction came that range of answers will be considered correct for the questions. In my opinion, one can't and shouldn't change the rule of game once the game is over but it was done as there were very less students were qualifying the exam.

Dheeraj Sanghi said...

@Sunita, IIT JEE has a system of checking answers by all faculty members and not just the paper setters. Paper setters may have one possible answer in mind, but later on, the faculty members point out alternate ways in which students can do, and what reasonable approximations they can use. Instructions in the exam are given on the basis of what examiners have said. But the grading is based on what a reasonable student may have done making sure that someone who followed the instructions exactly does not lose marks. And I think this is going to be the way in any exam anywhere in the world, except when you have standardized tests where each question has been given out as test (ungraded) question to a large number of students in the past.

Unknown said...

I agree. If answering 7 made as much sense as 7.00 then why did the jee conducting authority create an hoax by not mentioning it clearly in the instructions provided earlier?

Dheeraj Sanghi said...

@Unknown, we are further discussing this on another blog. Please visit there.