Search This Blog

Thursday, April 20, 2017

Merit-based admissions

People who do well in JEE Advanced will mostly tell you that merit is a one-dimensional beast, never mind that it consists of performance in three heterogeneous disciplines, mapped into a single number by the simple addition of three numbers without checking if these numbers are comparable or represent something which can be simply added. It is further believed that admission to higher education is a reward for performing well in an exam under the conditions of extreme pressures. And hence the only justified admission policy is to admit students in the decreasing order of merit or performance in that exam (modulo any laws passed by the parliament, but of course, not before they have pointed out that merit has been violated by those laws).

Should a university consider admission as a reward for performance. Or should it be something else.

Well, it has to be something else. In fact, no university will couch its admission policy in terms of rewarding past performance. University goals are varied: Who is likely to do well in our programs? or Who is likely to do well even beyond our program and be successful and enhance our reputation?

If university goals are somewhat in line with what I have mentioned above, then it is obvious that the admission through a single test is not ideal. No one can perhaps argue that performing in a particular test can both be an indicator of future performance in Computer Science, as well as an indicator of future performance in Economics. No one can argue that India has the best admission process since this is the only major country in the world that does not take language abilities into account for admission. No one can argue that the ability to answer MCQs is a strong indicator of the ability to write long notes necessary in any studies.

The reason for following a single exam is that it provides an objective, transparent mechanism while simultaneously measuring one variant of irrelevant merit. In other words, the most important outcome of JEE based admission has been that we have kept out pressure from the rich and influential as well as won most court cases. And by no means, this is a small achievement. In India, one may genuinely consider keeping pressure out and winning court cases as more important property of admission process than getting students aligned with the university goals. However, over a period of time, people doing well in JEE have started believing that the prime purpose of JEE is to evaluate merit, and that there can only be one definition of merit, which is the JEE rank.

Besides the obvious inadequacy of a single number predicting success of all programs in all disciplines in all institutes, there are other problems as well.

If a university asks the question who is likely to do well in our programs, and let us assume (despite absence of any data) that performance in PCM has a strong positive correlation with performance in Mechanical Engineering. What if we were compare the performance of people who have 80% in JEE after intense coaching, whose parents were rich and provided a good study environment whether at home or in a hostel, with the performance of people who have 75% in JEE without any coaching, and who come from modest backgrounds, and are the first generation learners. When both these groups compete, who is likely to do better. Most universities would have data to show that latter group performs better, and hence they would actually create admission policies to offer more admissions to the latter group.

When we point this out, an immediate reaction is, but how do you evaluate deprivation, and what has been the impact of that deprivation, and how much extra "credit" should be awarded to compensate for that deprivation so that we can compare the merit of the two. Now, in this question, there is an inherent belief that JEE score is a strong indicator of future performance of every sort. Gold standard of transparency and honesty (?) has been converted to Gold standard of merit. Do you ever question why people with same total in JEE score are ranked differently based on marks in some subject or the other. Does that imply different merit. Isn't it true that luck plays a part. If you had a headache that day, your merit may be way down. That any exam performance is only an indicator within a significant band is conveniently forgotten. But when it comes to any affirmative action, we want exact data, why 5% and not 4%. Why not ask data about JEE also.

Let us do another thought experiment. Again, let us assume that PCM performance is an indicator of future performance in all programs. Consider two possible admission decisions. One, take the top 100. Two, take the top 90. Study whether there is sufficient diversity in the class. If certain backgrounds are missing, let us fill the last 10 seats through them. Now, there is a belief (and educationists may even have data, I am not one) that diversity is good for education. That diverse inputs will cause more innovative projects. That having diversity in your peer group in college will prepare you for a much more diverse workplace that you are likely to face. So both performance within the university and success beyond the university is likely to be enhanced if the class is more diverse. Because of these reasons, universities encourage student exchange programs besides having diversified student admissions to begin with.

Let us for the sake of argument assume that there is data to show that diversity helps. What should a university do? Should you admit 100 students whose average performance will be 60, or should you admit 90 students whose average performance will be 70 (helped by diversity) and 10 students whose performance would perhaps be only 50. There is certainly a plausible argument for not admitting strictly through merit (if at all merit can be determined by a single number). But ask someone who was 91st in our imaginary list. Will he agree. If you assume that admission is a reward for performance in a particular test, you will not like what the university is doing.

Universities may have other goals. For example, should we admit those who are the smartest and will benefit very little from university education, or should we admit those who are likely to gain a lot more from university education. So on a scale from 1 to 10, should we admit students at 9 who can be helped to reach 9.5, or should we admit students at 7 and take them to 8.5. In other words, should "merit" be the only criteria for admission or should we also look at what will benefit the society most. This question is particularly important when the society is funding the cost of that education. Again, I know what the "merit" crowd will say, but I will only suggest that most universities around the world will admit both kinds of students in some proportion. (We also are mandated by law to do affirmative action, which means admitting students at lower "merit" but they are likely to benefit the most from our education.)

A lot of universities have a stated goal of helping the society that nurtures them. One way to do this is to prepare entrepreneurs who will create jobs in that society. Depending on the geographical location (and particularly so, if the location does not attract a lot of investments), universities may prefer to have some weight to the nearness criteria, since students belonging to that society are more likely to stay there beyond graduation. A lot of universities will, therefore, admit more students from within the state even when that magic number representing merit is lower for these students.

In summary, the insistence on admitting based on a single number is misplaced. As long as the single number is considered a proxy for transparent/honest process, the insistence has a logic, but when this number starts getting treated as a proxy for merit, there is a problem. On the other hand, it is necessary for public funded universities to not only admit the best, but also give an appearance of admitting the best. As long as public universities like IITs have a transparent/honest process, they should have flexibility in both defining the merit (as combination of multiple parameters, perhaps), as well as bypassing merit to achieve the multi-objectives of  the university.

4 comments:

Sanket said...

Sir,

As IITs are educational achievements, success in IIT-JEE should be correlated with success during IITs. In operational terms, JEE ranks/scores should be positively and significantly correlated with CGPI. How strong is the evidence for that? Also the magnitude of correlation, in absolute terms, should be higher than any other option that is cheaper to implement. If there is a cheaper option with the similar magnitude of correlation as JEE exam, it should be implemented. For example, does JEE rank have a stronger correlation with IIT-JEE CGPIthan board marks? If it doesn't one can even consider the blasphemous proposal of selecting a percentage of students purely through board marks. That would automatically take care of the concerns regarding the lack of gender diversity.

Dheeraj Sanghi said...

@Sanket, we have seen limited studies. I have for example looked at CGPA of students in different programs. Assuming that CSE closes before EE, and so on, one would have expected that the mean CGPA or median CGPA would be highest for CSE, then EE, then ME, and so on. But what we notice is that the number is highest for CSE (around 8), marginally higher for EE (around 7.5), and then everyone else is between 7.3 and 7.4. There is a perception on campus that top 200 rankers are definitely statistically better performer, and then the difference is marginal, if any. And remember that the gap between a 1000 rank student and a 5000 rank student is significant. These studies are not scientific since we are not able to look at the performance of people who didn't make it. Also, even if we find some correlation, is there causation. Prior to 2012 debate on including 12th class marks in the admission process, there were small studies done which showed that the correlation with 12th class performance was much higher than with JEE rank, and it would suggest that may be we should admit students based on 12th class marks. However, we still opposed it, partly because of practical reasons (how will we normalize, will results come in time, etc.), and partly because of the fear that if we started taking students from 12th class, the corruption and arbitrariness in grading will only increase and this correlation will cease to exist (and we did see that happening even though they were being used only for NIT admissions and not IIT admissions).

The primary purpose of this blog is not to give out a solution or even hope that the discussion here will give hints towards a solution, but to only point out that one shouldn't look at a single number as the sole metric of merit, and that every time an IIT or a university comes up with a plan to modify admissions, don't jump up and see this as erosion of merit. You may critique any new plan on a variety of reasons - not practical, prone to pressure, and what have you. But criticizing something as erosion of merit argument assumes that the single number identifies merit.

iitmsriram said...

@sanket states "one can even consider the blasphemous proposal of selecting a percentage of students purely through board marks". Actually, this was done in the good old days when I got my admission (late 70's). I had a handful of classmates who did not qualify through JEE, but were admitted on the basis of being within the top N ranks in their respective school board exams. Reserved category students were also admitted without writing JEE, based on school marks alone. This was discontinued in the 80's I believe. We could go back to doing this ...

Unknown said...

As many negative points related to many possible solution to this(e.g. political interference, court cases) already discussed in this blog. One way to extend whole process is adding some component where candidate judged by machine(learning) using old data points.
Say, introduce some JEE Main paper(computer based) for essay writing or speaking skill and evaluate it against the success of candidate after admission(irrespective of institute, map with Aadhar for keeping track). Keep it first few years as optional(trial), then make it part of selection using trial data points. As years progress, system will have enough learning to make it as good as human without any bias.
For more data points, 10-12th exam papers can be added, keep it as something national database, make it available to all institutes. Lets institutes decide what are the criteria for selection. Make whole process as transparent as possible.