Designing an Objective Type Test (like JEE/GATE)

This week, I had an interesting discussion with a few colleagues. This was about the effectiveness of interview as opposed to the marks in an objective type test. The context was MTech admission. The CSE department at IIT Kanpur admits a few top GATE rankers directly to the MTech program, and invites several more for another test/interview. It was felt that students whom we admit through interview process generally perform better in the program compared to those who are admitted directly on the basis of GATE score, even though the former have lower GATE scores.

My explanation for this apparent anomaly is this. In an interview, we value approximate answers. In GATE, we strongly discourage approximate answers. Let me give a hypothetical example. Suppose the question is: What is the minimum size of Internet Protocol header. The right answer is 20 bytes. If this question is ever asked in GATE, the likely choices are 16, 20, 24, and 28. The GATE wants to make sure that guesswork is strongly punished, and hence all the answers are such that they are close enough to the right answer. Either you know the answer, or you don't. But if the same question is asked in an interview, and the candidate does not know the exact answer, and says that it is one of 16, 20, and 24, we will give him a chance to explain. If he tells us that the size of the header is a multiple of 32 bits so that the header can be processed fast by CPUs having 32-bit architecture, and therefore the header size is a multiple of 4 bytes. Then he explains that the IP address is 32 bits, and the header has both source and destination IP addresses, and hence 8 bytes of addresses. There are other fields that he does not remember, but certainly all the remaining fields cannot fit in 4 bytes, so the minimum size cannot be 12. If someone knows so much about IP, we will say that we don't care whether he knows the exact number or not. So, in our interview process, we are trying to find such students who know a lot, can reason about it, but don't remember specific details to be the topper in GATE.

So, if we value guesswork in an interview, what can be done in an objective test to allow similar guesswork. Going back to the same question about IP header size, if I were to set this question in GATE, the choices I would give are: 12, 20, 30, and 40. And now, one can argue that 30 is not a multiple of 4, that 12 is too small, that 40 is too large, and hence "guesses" it to be 20. If someone can reasonably remove other options, it is a sign that the student knows something about the topic. But we frown upon guesswork in objective tests.

If you look at JEE, it is argued that the only thing coaching classes do is to give training about how to guess or how to eliminate. And that is why the paper setter keeps coming up with questions and potential answers where without solving the question fully, it is very difficult to mark the right answer. Now, the JEE papers are so difficult that it is not possible by a single Mathematics faculty member of an IIT to completely solve the paper in stipulated time. But should we get into this race at all. My own view is that ability to guess is often based on a decent understanding of the subject, and therefore, it is alright to prepare an objective type test which allows guesswork.

An extension of guesswork is approximate answers. The subjective tests were great because one could assign partial credits to someone who had partial knowledge, who could do some steps, but not solve it fully. Can we incorporate that property in an objective type test. Prof. Rajeev Kumar of IIT Kharagpur sent me a writeup explaining how this could be done. I am paraphrasing what I could understand from his scheme. The method is actually quite simple. When we have 4 choices in an objective test, we should select these choices in a way that one is absolutely correct, another one is wrong but a plausible answer, and the remaining two are completely wrong. Now, instead of assigning +1 for right answer, and -0.33 for the wrong answers, we should assign +1 for the right answer, 0.5 for the plausible answer, and 0 for the remaining two answers. There is no need to penalize guess work through negative marks.

This will encourage people to intelligently remove options, think of ways to quickly get approximate answers, and even do the guesswork amongst the remaining options.

It is rather interesting that while most faculty members I talk to admit that interviews are great because we encourage approximate answers and guesswork. They admit that subjective tests are great because the student gets to write whatever he feels like and if the examiner feels that the student is partially correct, he may give partial credit (and of course, there is no negative marks in a subjective test). But when it comes to objective tests, all our energies are spent in finding out ways to stop guesswork and partial/approximate answers.

What is wrong with our Higher Technical Education

Everybody agrees that the higher technical education in our country is of very poor quality. The top 10 institutes provide very good quality education. The top 50 institutes provide good education, and everyone beyond that is bad. Talking specifically of education in IT related fields, different people talk about anything up to 80% graduates being unemployable (that is, they can't even be trained).

People from industry would love to argue that this situation has come about because academia does not listen to them. They do not have curriculum in line with industry needs. They don't teach them the latest technologies. They don't expose them to live projects. They don't put enough focus on non-technical skills like communication skills, teamwork and what not.

I have a very different take on this. It is not that these institutes (beyond top 50) are not teaching their students English, or Dot-NET, or any other latest technology. The problem is actually worse. They are not teaching them anything.

Recently, I had a chance to look at some statistics from the CS paper of GATE 2010. The average marks were 12 out of 100. This is after considering the lowest marks to be 0. (In reality, the lowest marks were -21.33.) So, if we consider real marks achieved, the average would be around 9. And the median would be even lower,
around 7 marks. More than 10% of the students had negative marks overall.

It set me thinking. What would happen if we were to ask 1 lakh 12th class students to give GATE. I would guess that unlike BTech 4th year students, the 12th class students would leave most of the answers blank (and hence avoid negative marks), and only answer those 5-7 odd questions which s/he is confident about. (There were a couple of easy questions on aptitude. There were a couple of questions on programming that 12th class students have been exposed to. And there were a couple of questions which had so much information given that one did not need to known any computer science to get those right.) My gut feeling is that if we were to give the same GATE paper to one lakh 12th class students, the median marks may be only a couple of marks less than what has been the case with these 7th semester students.

It means that 7 semesters of technical education has enabled our graduates to get 2-3 extra marks (out of 100) compared to what 12th class students can get. Remember, GATE paper is about basic computer science only. It is not about the latest technologies. It is not about industry trends.

So the problem is very simple. There is no education going on in thousands of colleges around the country. These students are not being taught even basic programming, or data structures, simple algorithms, basic computer organization concepts, etc. And hence any attempt to improve the employability of these graduates by training them in communication skills, dot-net, java, software engineering, and so on, is futile. One has to first see how we can ensure that they learn computer science basics. Unfortunately, I have no solution to offer.

Another interesting statistics from GATE 2010 paper was that in several questions, the average marks received by the students was negative. As people who have given GATE would know, you get 1 mark for the question if you answer it correctly, and -0.33 if you answer it wrongly. The scheme has been designed so that, if people were to guess randomly, then the average marks obtained would be zero. (Assuming, 1/4th of the students will answer each of the four options.) If some people know the answer and mark it correctly, some have left it blank, and others have given a completely random answer, then the average score should be positive. If we take out those who have not attempted, and those who genuinely knew the answer, and consider the rest, 75% of those should (statistically speaking) give a wrong answer. And if you add those who genuinely knew the answer, the percentage of wrong answers should be less than 75%.

But, in some questions, more than 85% people (out of those who have attempted the answer) have done it wrongly. This is too high a number (compared to 75%, in a sample of 1 lakh) to be considered a statistical anomaly. I discussed those specific questions with a few colleagues, and it occurred to us that the only reason why this can happen is if the students are not answering it randomly, but are confident of the wrong answer being right. Which means that they have been taught the subject matter of that question, but have been given wrong concept or information (which is worse than not teaching at all).

Recently, there was a proposal to have an exit test for MBBS to ensure that the degrees given by all universities in the country are adhering to a minimum quality for MBBS. I think there is a need to have an exit test for all BTech in this country. That will be a sure shot way to weed out poor quality institutions.

Should IIT Directors Retire at 70

The IIT Council apparently has decided that one can stay as Director of an IIT till s/he reaches the age of 70 years. Two reasons have appeared in the media for this decision. One is that the sixth pay commission had already raised the retirement age of vice chancellors of central universities to 70, so why discriminate against IIT Directors. The second reason is the shortage of younger people (read 60 years old, inclusive) who have the right experience, energy, dynamism, vision and whatever else the current IIT Directors have.

The first reason is very interesting. We never stop shouting about the superiority of IITs over everything else in India. But when it comes to seeking benefits, we have absolutely no problem in comparing ourselves to the same institutions whom we besmirch so much.

Coming to the other reason. It is difficult to fathom. We have been able to find IIT Directors below the age of 60 for all IITs in the last 60 years. Each IIT has now grown much bigger than what it was a few decades ago. Which means that we are providing leadership experience to a much larger number of faculty members in each IIT. It should be easier to find younger Directors now than at any other time. It is very strange that the youngest nation on earth has the older leaders on earth. And the leaders are telling the younger ones - we are the leaders because all of you are quite incapable of leading, so even though we would rather enjoy our retirement and time with our great grand children, we are forced to do this duty of leading for your sake and the sake of the nation.

Don't get me wrong. I am not proposing age discrimination. I am not suggesting that everyone above a certain age loses ability to lead. I am only finding the arguments given by our leaders amusing. If there is lack of leadership talent, what is the government doing to tackle it. Or would there be another decision 5 years later that we still don't have sufficient leadership talent in the country, and therefore, the retirement age is increased further to 75 years.

Even in the current setup, the lack of leadership talent was being handled properly. While the preferred age was mentioned in the requirements for a post, the selection committee was empowered to recommend someone who did not meet those requirements, and in some cases, a genuine deserving older person has been given leadership roles.


Also, why is it that a 65-70 year old can only do academic administration or politics in this country. Why can't we have bureacrats in that age group. Why not judges, and we do have serious shortage of judges.

In the last 15 years, we have increased the retirement age of faculty members in IITs from 60 to 62 and then to 65. Has this helped. Hardly. We have not postponed retirement by doing this. Even earlier, good faculty members were able to get contractual positions in IITs or find a job outside the IIT system. So they were active earlier, and they continue to be active now. The only difference we have made is that now even non-performing faculty stays till 65.