Search This Blog

Monday, July 16, 2018

Mountain out of molehill: Is 2 = 2.0

After a few days of suspense, the Madras High Court has stayed the single judge order asking IITs to reorder their ranks. (That the order really wouldn't have changed anyone's rank is not important.) And after a few days' gap, the counseling has restarted.

Let me recap the issue. The following was the instructions on the exam paper.

  • For each question, enter the correct numerical value (in decimal notation, truncated/rounded off to the second decimal place; e.g. 6.25, 7.00, -0.33, -.30, 30.27, -127.30) using the mouse and the on-screen virtual numeric keypad in the place designated to enter the answer.

Now, there were two interpretations of this line.

One, this instruction is meant only for those answers where the answer can be truncated/rounded off, and more specifically, was not meant for integers.

Two, presence of "7.00" and absence of any integer in the possible solutions implies that even integers had to be written up to two decimal places.

In my opinion, the first one is the right interpretation. However, the second one is a "reasonable" interpretation and since IITs are more interested in finding out whether you know the answer or not, we may award full marks for those who used second interpretation.

Second thing, I said was that instructions only suggest what is the minimum you need to do to get full marks and doing more should not be penalized. So even if writing two zeroes was suggested and was the correct interpretation, writing an integer as an integer is certainly a better answer and hence should be given full marks.

The opposite view point that was presented to the court and was written by lots of people as comments on various social media, including this blog was that the second interpretation is the correct interpretation and the first interpretation is so wrong that those who followed that should be penalized. And, following instructions is of supreme value in exams like JEE.

Frankly, I am bothered by the opposite view point. I am unable to look at it as a difference of opinion. There is something else at play here. What they are saying is that lack of an integer in the list and presence of 7.00 implies that integers had to be written in a particular way. Do they have any idea of what is implication, what is proof, etc. And then, saying that it is not enough to show that you know the answer, but following instructions is an integral part of an exam. Do they really believe that IITs are looking for students who can follow the orders and not think independently. If yes, IITs need to do a lot more in terms of improving its image.

In an exam, if I ask them to write names of any 4 prime ministers of India, and someone writes five names (all correct), should I penalize this guy for not following the instructions. Obviously, additional information, if wrong, can lead to penalty, but should there be penalty if the additional information is correct.

And as, someone commented, should IIT penalize someone who comes only one hour before the exam, when the instructions were to reach three hours before the exam. (I may be off in terms of exact time.) Obviously not, since following instructions are useful only to the extent of peaceful conduct of the exam and not beyond that. Similarly, following instructions in the exam are useful only to the extent of displaying your ability to solve problems in the exam and not beyond that.

To me, this alternate opinion is borne out of inability to take responsibility for one's actions. Almost every student tells his/her parents/friends that they deserved better in any exam than what the result indicates. I talk to so many JEE passed students every year for counseling, and the conversation always start with, "I was expecting a better rank, but...." And they are always looking for excuses to convince themselves and others that they are indeed better than what that rank indicates. (And they very well may be, the rank indicates nothing about them as a human being.) And as soon as they find out any excuse that anyone has used, they are very happy and start using that excuse themselves. Sometimes, even small coaching places, will tell their students that they deserved better rank, and it is all because of someone else (IITs mostly) that their ranks have suffered.

Because of this reason, one can easily predict that the kind of problem we had this year, will actually happen every year. Such issues have arisen every single year in the last few years, and will continue to happen in future too irrespective of how good the language is of all instructions and all questions.

IITs are normally very reluctant to change since they think that following what was not questioned in previous years would ensure less court cases this year. But I think they should learn from JEE 2018 where there were major changes (women reservation - I am really surprised there is no court case on this, fully computerized exam - there was a case, but dismissed early, numerical questions with answers not between 0 and 9). Irrespective of how many changes you bring about, all those who want to blame IIT JEE for their performance will latch on to one thing and the other changes will go unquestioned, and they will latch on to that one thing, even when there is no change.

The only way to reduce court cases is to reduce the value of the exam. Give students option. If not doing too well in one exam leaves good options open for students, they will feel more confident of owning up their performance and not look for excuses.

Till that happens, court cases will happen irrespective of what IITs do. So they shouldn't be afraid of changes. Changes do not increase the number of court cases.




NEET 2018: The solution is normalization

If there is one exam worse than JEE, it is NEET. And it is compulsory for all medical admissions. Earlier, we have seen different question papers in different languages, but no attempt to see if the two papers have similar difficult level, or normalize the marks if they aren't. That is not a surprise to those who have followed CBSE's conduct of JEE Mains in the past. There is no normalization in the multiple papers they offer (online and offline).

This year, a whopping 49 questions in the Tamil version of NEET had wrong translation. Well, at least they translated so that everyone gets the same set of questions. Better than completely different papers of the past. But 49 wrong questions out of 180 total questions. Obviously students would complain. CBSE refused to do anything about it. The matter goes to high court. CBSE argues that students should have read the English version since they had already issued the instructions that in case of any problem in translation, English version would be considered authentic. But they couldn't answer a simple question. How would the students know which questions are wrongly translated, and hence they should look at English version. And if it is to be assumed that everyone understands English question paper, why give options of local language to begin with, or was it the case that students from vernacular background are required to read both papers and spend extra time (without any compensation, of course).

The court was aghast at such arguments. CBSE should have suggested how one could fairly treat the 24,000 students who had taken the test in Tamil language. And sure enough, the court ordered that all 24,000 students should get full marks in all 49 questions.

Obviously, it is unfair to others. It is no one's argument that all these 24000 students would have solved correctly all 49 questions, if there was no error in the translation. And by awarding them full marks, they have been given vastly higher marks than what they would have scored in an error free exam. And thus, they have been placed higher than those students who took exam in English.

How do we know how many marks these 24000 students would have scored in an error free exam. We really can't say about an individual. And then there are second order effects like wasting time on a question which is now ambiguous, say. But statistics does allow an approximate answer. We can normalize marks based on 131 questions that everyone has been able to attempt.

The problem is that CBSE has never learnt how to normalize marks. They haven't done it in JEE before. They do a completely horrendous "normalization" of 12th class marks, which is nothing but grade inflation. So they don't know how to provide at least a statistically fair solution to these 24,000 students.

Is there a way to study correlation between 131 questions and 49 questions. Can we do this analysis for different categories of students - all India, students from Tamilnadu, students from TN Board, etc., and figure out which class is most similar to the group of 24,000 students who chose Tamil as the medium of question paper. Is there strong correlation of some questions out of those 131 with each of these 49 questions. How difficult is each question for that cohort. Based on these types of questions, one can start looking at what would have been the likely score of the student if there was no error in translation and if they were like an average student of the cohort that we are comparing them with.

But, I suspect I know what will happen eventually. CBSE will look at the marks of 131 questions for each of those 24000 students. We will then see how many marks did other students getting identical marks in 131 questions got in 49 questions. We will assign that many marks in 49 questions to those students. This is a very simplified version of normalization but something is better than nothing and courts might agree with this, and frankly, CBSE will only care what will go through the courts.

Any solution based on statistics will cause heartburns. But it is the best you can do, short of conducting the exam again.

Sunday, July 15, 2018

Institute of Eminence Report

In the last one week, a lot has been said about the Institute of Eminence process and report. There are far too many questions about the way this policy has been handled. And my take on this is, that it may have been strategic. Make so many errors, violate your own policy so many times, that different people will comment on different issues and there would not be one single issue on which all the people are united in criticism.

The report that has been made public on UGC website says nothing about how specifically these institutes were decided. There are a lot of general things that they say (but do not seem to follow themselves). When I was reading the report, I was repeatedly asking myself, if this is a competence issue, or is it that they were not willing to spend enough time on such an important assignment (after all, other than the chairperson, members are extremely busy people, but then they should have refused to be a member rather than sign such a poor report).

The report clearly says that the primary goal of the committee was to look for universities which would be in top 500 of any popular world ranking within 10 years. One would have assumed that those universities which are already in top 500 would be automatically selected, unless this committee is convinced that those universities have seen their zenith, and are now on the declining path, and in the next 10 years, they will be out of top 500 list. And if committee is convinced of this, it would actually give reason for the same. So what is the basis of keeping IIT Kanpur out is not clear.

The committee talks about private institutions not doing enough research and hence the possibility of them getting into top 500 is less. And then, it ignores the private university which arguably has the best research output (Amritha University) and because of this research output they already have comparable or better ranking in various lists compared with the two private universities that have indeed been selected. There was certainly a possibility of having the 4th private university in the list.

In the report, they mention that some of the newer universities are so small (less than 3000 students) that they couldn't really become world class, but at the same time recommend a university with 0 students to be in the list. Also, when the goal is to have a university in top 500 in 10 years, how come a university which is claiming to be in top 500 in 13 years has been selected. Jio University seems to have suggested that it will start in 2021 and will be in top 500 by 2031. Based on their only higher education venture (DAIICT) which is nowhere in the world rankings even after 20 years, it seems difficult to believe that they will be in top 500 within 10 years of starting, but in any case, the goal was 10 years from the selection, and they themselves are saying that they will not be in top 500 within 10 years from selection. For a good analysis of this decision, read the following report by Prof. Sandeep Shukla.
Money cannot buy excellence in education, but Jio's 'Eminence' tag is worth crores.

This is not to say that Mr. Ambani cannot be the promoter of a great university. I hope Jio will be one. But as of today, there is no reason to believe that they will be in top 500 within 10 years. Indeed they themselves are saying that they won't be. So the tag is clearly misplaced.

(People are too sensitive. When I say DAIICT is not in world ranking, I am saying only that much. I am not saying that it is a bad institute. It is a good institute, but not in top 500 ranks in the world.)

The committee says that many universities applied only to get autonomy from UGC and they can be dealt with within the graded autonomy regulations. But why should that be an issue. My motivation may be to gain autonomy, but you please look at me from the perspective whether I can get into top 500 ranks in the next 10 years or not.

The committee has recommended 8 public universities. When they had space for 2 more, they have still decided to ignore those universities which are already in top 500 list, the primary goal as stated by the committee. One would have expected an explanation in the report (and not now to the media, even that is not forthcoming on this issue though).

On what basis has government chosen 3 out of 8. There appears to be some feedback from the committee, since the Chairman of the committee has said that IIT Madras was not selected because Chennai has bad weather because of which international students and faculty don't come to Chennai and hence their international ranking is poor.. Gimme a break. Weather in Chennai much different from Mumbai? And other IITs have a large number of international students and faculty members?

IIT Kanpur is a curious case. Not only it is not in the list of 8 universities, it is not even in the list of specialized universities focusing on science and technology. A university which has been consistently in the top 5 of the country in all rankings over the last so many years being ignored without giving any reason as to why the committee believes we will not be in top 500 in the next 10 years. Do they know something that we don't know. Was our presentation so poor. Was our proposal so poor. (I was hoping to work for an Institute of Eminence. I am disappointed though I guess I have the option to work for Jio :-)

Combine all this with what all has happened in the last one year, and the optics is really poor. Reopening of the application submission even though 100 universities had already applied. Changing the shortlisting after the committee had decided to shortlist only 40 institutes and asked them to give presentations.

What is certain is that the four members of the committee have not displayed any eminence in signing this report.

Other articles on IoE saga:

UGC's laziness has led to needless 'Institutions of Eminence' controversies by Prof. Pushkar
Imminent Eminence: Ambani's Egg by Prof. Mukul Kesavan
Institutes of Eminence status given without field visits, rankings by Anubhuti Vishnoi
Jio Institute: Why the Modi government is misguided in giving the eminence tag to a select few by Arihant Pawaria