Search This Blog

Monday, May 13, 2013

JEE 2013: Normalizing Board Marks

Last year, MHRD had decided that normalized board marks (12th class) shall be taken into account for admissions to engineering programs in CFTIs (Centrally Funded Technical Institutes) except IITs. The marks in AIEEE (now, JEE Main) and the normalized board marks shall be added in the 60:40 ratio to come up with the final ranking.

Of course, I continue to believe that considering 12th class marks in ranking is not a good idea for reasons we had explained in the last year's debates, but something had to be done for 2013. The main issue was how do we normalize marks across diverse boards. Unfortunately, there is no simple answer.

Again, if we go back to last year's debates, there were essentially two sides - one side suggesting that all boards are equal. That in any large board, the distribution of native intelligence had to be similar, and hence X percentile of one board had to be of similar intelligence as X percentile of another board. The opponents, on the other hand, suggested that board exams did not test intelligence but academic preparation, and hence whether the native intelligence is equally distributed or not is an irrelevant point. Also, there are several types of intelligence, and we don't need to ensure intelligence of all sorts for engineering admissions. Engineering admissions on the basis of demonstrated performance is a fairer mechanism rather than some unverifiable claims of someone having raw intelligence.

The opposite side, therefore, suggested that there has to be some mechanism of comparing boards. A student with same percentile in a "better" board has to be given higher marks out of 40 than another student with same percentile in a worse board. And this side had suggested that we find a way to compare the boards and come up with a method where appropriate adjustment can be made to someone's percentile performance based on whether the board was better or worse.

In the last 8-9 months, there have been numerous committees, numerous experts from not just within India, but outside India as well, who have looked at this problem, and there have been several reports, minutes of the meetings, comments, and so on.

Out of all the suggestions for comparing the boards, the one which has caught fancy of many experts is where students of all the boards are tested in one common exam (called an auxiliary exam or an anchor exam), and how students of a board perform in that common exam will be taken as a measure of performance of that board per se. And one notices that we do have one common exam across all the boards to whom these students belong, and that exam is JEE (Main). So we could look at the performance of students of different boards in JEE Main and based on their relative performance, we could adjust or normalize their board marks. If from a particular board, most students perform very well, then the same percentile score of that board should correspond to higher normalized score.

There are some issues though. This method of having a common auxiliary exam is a good method to normalize performances if some conditions hold. These conditions include: students appearing in JEE Main from a particular board constitute a representative sample of that board, and students of a particular board do not enjoy any particular advantage or disadvantage over students of another board, in respect of the JEE Main examination.

It has been argued that the stronger boards indeed have an advantage over weaker boards in respect to their performance in JEE Main. This advantage comes from a greater degree of alignment of the syllabus of 12th class with the syllabus of JEE Main. Second, the stronger boards happen to be those who offer education in Hindi and English, and since JEE Main was in three languages - English, Hindi, and Gujarati this year, this would have given some advantage to the students of stronger boards.

It has also been argued that the students from a board giving JEE Main do not constitute (based on 2012 AIEEE data) a representative sample. In fact, we see that even a 50 percentile student of a strong board feels confident of giving JEE Main while even a 80 percentile student of a weak board does not give JEE Main.

Another argument has been that the auxiliary exam should have a similar mix of subjects as those exams which we are trying to compare, which means that at least a language paper should be there too in the auxiliary exam.

However, I consider them as weak arguments. If there is greater alignment of syllabus in case of stronger boards, it is not because other boards are teaching different topics, but it is because other boards are teaching less. Exams like JEE have been around for 50 years. Why haven't those boards aligned their syllabi to such exams in 50 years. It is primarily because they can't. They are the weaker boards.

The language could be of some help, no doubt, but a large number of weak boards have a large fraction of their students learning in Hindi. Second, this option of regional language paper was given to all states, if they agree to use JEE Main for engineering admission. Third, the impact of language is reduced in an objective type test. It would be much higher in long answer type exam. But let us agree that there could be some impact of this.

The students not forming a representative sample is true but its impact on the final normalized marks is not likely to be significant. For example, if there is a weaker board where only top few are giving the exam (this is what the data from 2012 shows), then the performance of that board will "artificially" improve. So this is actually helping weaker boards.

As far as JEE Main not being a good enough common test, even if we agree for the sake of argument, then which is a better test. Unfortunately, there is no other common test. Can we argue that the JEE Main is such a bad common test that it is throwing up all random results and therefore just considering all boards as equal is a better assumption. If this is the case, then perhaps we need to junk JEE Main itself.

So the debate on normalization has been intense. All boards being equal versus JEE Mains being a good enough auxiliary exam to compare boards. Notice that for 2014 onwards, if the government persists with this idea of using the board marks, perhaps someone can design a better auxiliary exam as well (what will then happen to One Nation, One Test). But we need to do something for 2013.

This has been debated in so many committees that any resolution is impossible. The data of 2012 cannot conclusively tell us whether the advantage that a student from a good board enjoys is exclusively because s/he is a better student. I think that those who argue for all boards being equal have no data at all, and hence they arguing that the data on the other side is imperfect and therefore their side wins is quite funny. But then one does realize the limitation of considering JEE Main as the auxiliary exam to compare all boards.

And that is how a compromise has been evolved for 2013. The compromise essentially accepts that students from stronger boards will show a strong performance in JEE Main compared to other boards. But it argues that it is not clear how much of that "stronger" performance is due to their being better academically prepared versus other factors mentioned above. And hence we don't give them the full benefit of stronger performance.

In other words, we consider what would be normalized marks if we were to treat all boards being equal. And also, what would be normalized marks if we were to consider JEE Mains as an ideal auxiliary exam and treat performance of students of different boards in JEE Mains as reflecting on the quality of academic preparation of all students of those boards. And then take the average of the two normalized scores.

I am sure this will annoy everyone. Those who strongly believe in equality of boards will see this as an attempt to favor certain boards. And those who equally strongly believe that some boards have a much greater fraction of academically better prepared students will see this as a political compromise under pressure from weaker boards. But I see this as just a way to ensure that we can do admissions in 2013 without much litigation, and a way to make some headway in a bad situation that we have all been forced into. I also think that we have been able to establish the principle that there are stronger and weaker boards, which was extremely important to me. In absence of relevant data, we may or may not have been fair to stronger boards, but I do hope that someone will start thinking of 2014 right away, and come up with a better process to compare boards (or as one lives on hope, we will stop comparing board marks for admission purposes).

Of course, if you read the exact method on JEE Main website, it won't read like what I have written above. The reason is as follows.

When you are adding two quantities, they must have same units. For example, if I say that I walked for 100 meters and then I walked another 50 meters, how much did I walk, one can add the two quantities and say that I walked 150 meters. But if I say that I walked 100 meters and then I walked for another 30 seconds, now it is incorrect to add the two quantities and give a total distance.

Similarly, it would be improper to add JEE Main marks with either percentile or board marks, etc. All the committees before us had therefore suggested that board performance has to be converted into an "equivalent" JEE Main performance. So both JEE marks and normalized board marks have to be from the same total and should have similar distribution. So if you consider board performance, one looks at it in percentile terms, and then see what would that percentile in JEE Main be and give that many marks to the student. Notice this would be the case when we consider all boards to be equal, and whether a student has received X percentile in board 'A' or board 'B', s/he will be mapped to the same marks which correspond to X percentile in JEE Main result.

On the other hand, when we consider relative performances of the board, then X percentile in board 'A' would map to X percentile in JEE Main amongst those students who gave 12th class exam from that same board. These marks would be higher for a board whose relative performance in JEE Main is better.

And yes, I was a member of the JEE Interface Group whose task it was to look at everything that various committees had done in the last 8 months, and recommend a "final" solution for 2013.


«Oldest   ‹Older   201 – 205 of 205
Dheeraj Sanghi said...

I have gone over all the 200 comments.

@Vinod Kumar Sharma proposed that anyone getting greater than 99.75 percentile in boards should be mapped to 99.75 percentile in JEE. The problem with this is twofold. One, the number 99.75 needs a better mathematical explanation. Two, the issue of long tail distribution affects many more than 0.25 percent.

Several persons have alluded to some decision last year according to which the formula was to be 0.6*percentage marks in JEE Mains + 0.4 * Percentile in Board.

I have never seen this decision. Even now, if someone can find out such a decision, I am sure it can be discussed, or it can be taken to a court. But, no one on this blog has been able to point out the exact document which said this. The formula was really something that Prof. Barua was trying to push. Of course, he being a Director, and a member of IIT Council, a large number of students and parents may have believed that that was accepted, but it was not.

There are two problems with this formula. One, as Prof. Barua himself says that this will give such a small weight to the board marks that its impact will be just the choice of branch within an institute. Would that be equivalent to giving 40% weight. Many persons have criticized the JIG formula as giving huge weight to board marks. Is the solution to deny any weight to board marks. Also, the debate last year was that the weight should be such that students have an incentive to study for boards. Such a formula gives zero incentive to study for board, since studying for JEE and getting a few more marks as a result would be so much more efficient thing to do than to invest that much time in board exams. Prof. Barua will have to decide whether he wants a system where there is an incentive to study for the board, or he wants a system where there is no incentive.

The second problem with this formula is that it is based on the assumption that all boards are equal. That is an assumption which is so absurd that I really do not even wish to comment upon it. I had written enough about it last year.

@Sanjeev has suggested a mapping of marks from other boards to CBSE, but it is effectively same as taking percentile. Hence the discussion above applies to his proposal as well. @Ak_amazing has also suggested a scheme which is just a rehash of percentiles. @Seshagiri has suggested a formula which is also equating all boards.

The process suggested by @Sunil Kumar Chillkuru will aggravate the effect of long tail, not reduce it.

There are some suggestions on dividing seats into 2012 and 2013 students, and also having some seats (40%) filled only on the basis of normalized board marks, and other seats (60%) filled in only on the basis of JEE marks. I have no issues if NITs and other universities (including state and private) deciding how they will do their admissions.

Considering only 3 or 4 papers in the board as some including @Yashpal has suggested does not make any impact on the long tail distribution of JEE Mains.

So in over 5 weeks and 200 comments, there is no suggestion which I would consider as better than the one which JIG has come up with. This statement does not mean that there can not be a better proposal. This does not mean that the proposed model has no flaws.

The primary problem with the model that has been articulated by giving so many examples is that a mapping is being made from one distribution (board marks) to another (JEE marks) when these two distributions are so completely different. As I said in my blog, this is something that a fairly large number of statisticians came up with, and I personally just accepted their recommendation.

I suspect that it is too late to do anything for 2013. But if someone came up with a better system, I will circulate it so that we can have a better system in 2014.

రౌతు విజయకృష్ణ said...

These new changes to JEE exam of 50 years history are completely political. I see a clear analogy between percentage allocations for backward and rich Communities And reservations across the weak and stronger boards. Any how , one more action by the parliament to degrade the IIt system also. Jai ho Bharat

Pradyumna said...

Normalisation process is fine but
the weightage numbers 0.6 and 0.4 should be decided based on degree of difficulty.
E.g. Out of 14 Lacs students appearing for JEE Main
How many students score more than 90% in board--> 40,000

How many students score more than 90% in JEE Main--> 400 (may be less)

This proves JEE Main is 100 times difficult exam and weightage should be 0.99 for JEE
And 0.01 for Board

Deepak Bhootra said...

Very interesting. I am the father of two sons who have no interest in IIT or engineering. However, they want to study economics at Delhi University. The converse situation exists there. Cut-offs are used irrespective of the board that one comes from. This entire blog clearly accepts one reality -- that boards are different in the scope of difficulty. Then the question is how many thousands of deserving students where left behind by DU and other colleges that refuse to believe that boards differ in difficulty? Has anyone ever raised a legal challenge on this matter or are these debates and discussions limited to IITs where a very small % of India's talent will ever go to or aspire to?

Govin said...

Since the topic is under discussion by various panels now and the NITs and IITs would have witnessed the experience, its time that this topic is reopened to guide the law makers on exactly what would be an optimum solution to keep up the morale of students on admission to these institutions.

I am also not against consideration of board marks for engineering admissions. But the method they are adopting for ranking is too punishing for students with higher order thinking skills, i.e., who score well in JEE.

For example say the top 10000 students in any board score between 95%(10000th rank) to 98%(1st rank). Among these 10000 students I dont think there is any major difference in knowledge level.

But the same top 10000 students in JEE mains score between 210(10000th rank) to 355(1st rank) out of 360. There is a huge difference between the one who scored 210 and the one who scored 355. This difference is in the application of knowledge, which is very important for his success as an engineer through premier institutions.

Let us consider two students A & B who perfom just opposite in board and jee.

As per existing nomalisation procedure, student A who scored 98% in board is given 355 marks for the 40% weightage (because he is number 1 in board). If he had scored, say 210 marks in JEE, he is ranked with (0.40*355)+(0.60*210) = 268 marks, rank ~2000.

student B who scored 95% in board is given only 210 marks for the 40% weightage (equivalent jee score adopted in normalisation procedure). If he had scored 355 marks in JEE, he is ranked with (0.40*210)+(0.60*355) = 297 marks ~500 rank.

That is, based on JEE their rank difference was 10000, but based on normalisation their rank difference is reduced to just 1500. That means a small difference (3%) in board marks %age can cause 85% effect on overall ranking.

So it is NOT just 40% effect.

«Oldest ‹Older   201 – 205 of 205   Newer› Newest»