Search This Blog

Thursday, September 30, 2010

Designing an Objective Type Test (like JEE/GATE)

This week, I had an interesting discussion with a few colleagues. This was about the effectiveness of interview as opposed to the marks in an objective type test. The context was MTech admission. The CSE department at IIT Kanpur admits a few top GATE rankers directly to the MTech program, and invites several more for another test/interview. It was felt that students whom we admit through interview process generally perform better in the program compared to those who are admitted directly on the basis of GATE score, even though the former have lower GATE scores.

My explanation for this apparent anomaly is this. In an interview, we value approximate answers. In GATE, we strongly discourage approximate answers. Let me give a hypothetical example. Suppose the question is: What is the minimum size of Internet Protocol header. The right answer is 20 bytes. If this question is ever asked in GATE, the likely choices are 16, 20, 24, and 28. The GATE wants to make sure that guesswork is strongly punished, and hence all the answers are such that they are close enough to the right answer. Either you know the answer, or you don't. But if the same question is asked in an interview, and the candidate does not know the exact answer, and says that it is one of 16, 20, and 24, we will give him a chance to explain. If he tells us that the size of the header is a multiple of 32 bits so that the header can be processed fast by CPUs having 32-bit architecture, and therefore the header size is a multiple of 4 bytes. Then he explains that the IP address is 32 bits, and the header has both source and destination IP addresses, and hence 8 bytes of addresses. There are other fields that he does not remember, but certainly all the remaining fields cannot fit in 4 bytes, so the minimum size cannot be 12. If someone knows so much about IP, we will say that we don't care whether he knows the exact number or not. So, in our interview process, we are trying to find such students who know a lot, can reason about it, but don't remember specific details to be the topper in GATE.

So, if we value guesswork in an interview, what can be done in an objective test to allow similar guesswork. Going back to the same question about IP header size, if I were to set this question in GATE, the choices I would give are: 12, 20, 30, and 40. And now, one can argue that 30 is not a multiple of 4, that 12 is too small, that 40 is too large, and hence "guesses" it to be 20. If someone can reasonably remove other options, it is a sign that the student knows something about the topic. But we frown upon guesswork in objective tests.

If you look at JEE, it is argued that the only thing coaching classes do is to give training about how to guess or how to eliminate. And that is why the paper setter keeps coming up with questions and potential answers where without solving the question fully, it is very difficult to mark the right answer. Now, the JEE papers are so difficult that it is not possible by a single Mathematics faculty member of an IIT to completely solve the paper in stipulated time. But should we get into this race at all. My own view is that ability to guess is often based on a decent understanding of the subject, and therefore, it is alright to prepare an objective type test which allows guesswork.

An extension of guesswork is approximate answers. The subjective tests were great because one could assign partial credits to someone who had partial knowledge, who could do some steps, but not solve it fully. Can we incorporate that property in an objective type test. Prof. Rajeev Kumar of IIT Kharagpur sent me a writeup explaining how this could be done. I am paraphrasing what I could understand from his scheme. The method is actually quite simple. When we have 4 choices in an objective test, we should select these choices in a way that one is absolutely correct, another one is wrong but a plausible answer, and the remaining two are completely wrong. Now, instead of assigning +1 for right answer, and -0.33 for the wrong answers, we should assign +1 for the right answer, 0.5 for the plausible answer, and 0 for the remaining two answers. There is no need to penalize guess work through negative marks.

This will encourage people to intelligently remove options, think of ways to quickly get approximate answers, and even do the guesswork amongst the remaining options.

It is rather interesting that while most faculty members I talk to admit that interviews are great because we encourage approximate answers and guesswork. They admit that subjective tests are great because the student gets to write whatever he feels like and if the examiner feels that the student is partially correct, he may give partial credit (and of course, there is no negative marks in a subjective test). But when it comes to objective tests, all our energies are spent in finding out ways to stop guesswork and partial/approximate answers.


Ishan said...

I fully agree with you sir.We should encourage guesses, and point of giving .5 for plausible wrong is quite nice.

Anonymous said...

That's a very good post.

Also, I wish to add: Apart from what you call "approximate" answers (illustrated so well by you with that IP address example), what happens in an interview is that you also have an opportunity to see in action the candidate's ability undertake those "back of the envelope" calculations. Such exercises are important because they reveal the ability of the candidate to undertake those crucial wider-scale or base-level(conceptual) integrations. Unless you have an open-ended examination (like take-home or open-book, etc.), there is little opportunity to test this particular skill. Interviews are close enough to such open-ended tests, and also practical enough that they can be conducted within the time and other constraints.

One downside of interviews is that "politics" (i.e. favoritism, cronism, biases against a particular viewpoint such as against a candidate who supports capitalism or Ayn Rand's philosophy by a socialism-committed professor say at a Russia- or UGC-aided Institute of National Importance, etc.) can easily creep in. Here, I am sure I need not elaborate further; the last two pages of my detailed resume (available at my Web site: bear adequate testimony to it.



KUNAL said...

I will say no and yes ...both for ur suggestion ....if u see this GATE result...
GATE 2011
Electronics topper score 91/100
CS 94.33/100
Mechanical 87/100

and around 60 marks ..rank around 600-700 ...that's insanity..... people crossing limits no one can reach IITs without guess ..Good knowledge is required room for flukes... for these marks

if u want good students....follow pattern of UPSC Engg services Exam pattern.....which have objective subjective everything which IITs Profs want ....

selected candidate also like
IIts prof.............

everybody will be happy .......

Rishab Mahajan said...

Hello Dheeraj
Myself Rishab Mahajan faculty of electronics at BVP Delhi.Your article regarding JEE has shortcomings
1}What if Student marks both correct?

2}As all know in JEE least 50K aspirants are serious others all guess in paper if one knows such scheme he would mark Two options out Four in Hope least one would be correct?

3}As per my knowledge your IIT is hosting this year JEE lets see how would the pattern be anyways person/student who have research on art of guess is quite strong probable to make least to cut=off

Dheeraj Sanghi said...

@Rishab, Marking both is invalid in my scheme. Since it is invalid (and not wrong) the candidate will get 0 marks and not negative marks. Just because "my" IIT is the organizing IIT, it is not bound to listen to my ideas. I am not part of the team that makes policies, or even implements them.

Rishab Mahajan said...

That would be pretty fine! if it is clearly mentioned in instructions and I hope one day such nobel ideas would be implemented.

Satish Kumar said...

This is a great article.This will encourage conceptual learning and quality of student certainly increase.