If newspaper reports (one here) are to be believed, the IIT Council has decided to dilute the requirement of 12th class performance to get admission in IITs. Instead of 80 percentile as of now, it will be lower of 80 percentile and 75 percentage from the next year.
Apparently, the Directors and Ministry are very concerned that about 2 percent students are denied admission to IITs even after passing the JEE Advanced.
This is very interesting, to say the least. When this system of 80 percentile was introduced in 2012 (from JEE 2013 onwards), it was stated that the criteria for school marks should be such that a couple of percent of students are denied admission to IITs. This was supposed to put pressure on all JEE candidates to take 12th class seriously. And the policy achieved exactly what was predicted - about 2 percent of students were denied admission to IITs. So when the policy achieves what was predicted to be achieved, shouldn't we call this a successful policy and the Directors and MHRD should pat themselves on their respective backs. So what went wrong?
There was another problem that people like me had pointed out. It was pointed out that in many boards, the grading is totally arbitrary, and there can be wide variation in marks depending on who grades the paper, and this wide variation in marks can result in significant variation in percentiles, and hence a high percentile in boards can be a matter of lottery and not a matter of academic performance. This was particularly true of boards which have extremely liberal marking.
If the marking is consistent, and the luck can only cause a difference of couple of percentiles, then a steep cutoff to encourage focus on school education can be justified. Someone with 79 percentile cannot claim that in a fairer grading s/he could have got 80 percentile. Well, you should have focused more on 12th class exams and tried for 85 percentile. But if someone with 79 percentile can claim that in a fairer grading system, s/he could have got 90 percentile, then the argument is that much stronger for not having the 80 percentile cutoff. Unfortunately, some students could actually point to such arbitrary grading in some of the boards.
Just two years ago, MHRD and Directors had argued that once we start focusing on the 12th class performance, the boards will be under pressure to reform. They will start having better question papers, more consistent grading, and so on. So it was just a matter of time, when everything in this country will improve and we can all live happily ever after.
Yesterday's decision of the Council is essentially admitting defeat. It is an acceptance of reality that MHRD and Directors have no control over the boards. That the boards in the last two years, instead of improving the exams and grading, have actually made it more random, more liberal. The 80 percentile cutoff in 2014 was higher than 80 percentile cutoff in 2013 in many large boards. And hence there is no evidence that boards will improve in future. I stand vindicated.
But there is an interesting side effect of this. If MHRD and IIT Directors have started believing that the boards will not improve and that the grading is quite random to the extent that different people grading the same exam copy can result in wide variation in percentiles, should the use of 12th class marks be not stopped even for NITs and other engineering colleges.
You can't argue that boards have arbitrary grading and hence we need to dilute the 12th class marks requirement to an extent that it becomes a mere formality for IIT admission, but the same arbitrary grading can be considered for admission to all other engineering colleges.
But then consistency has never been the strength of Indian academic leadership, regulators and administrators.
Posts
Mr. M. K. Gandhi and Nathuram Godse
1 month ago
43 comments:
Indian education system is getting weirder and weirder with time...!!!
.
.
God bless the future generation... :-/
In case people are wondering about "arbitrary grading" here are mine and Debarghya Das's blogs about what the arbitrary marking is:
Ten Year CBSE Score Analysis
Suspicious CBSE and ICSE Scoring Patterns
ICSE/ISC Marking Snafus
Honestly though, school level assessment is a serious issue and the govt. should have cracked the whip on the boards a long time ago. 95 and 99 in English Language - what does it mean?
I think instead of their wanting students to take xii board seriously , they should attempt to make xii board a serious affair so that what us asked in competetive exam is in unison with board...this draw, derive , define and gift of 30 marks of practicals have made a mockery of whole examination...that really needs to be addressed rather than making the policies to bunch the top to make evrryone happy
Dheeraj Sanghi has raised the right questions. We must appreciate that all State Education Boards function under the control of the State Governments. I have not seen much improvement in functioning of school education and Education Boards for several reasons, e.g: shoddy appointments of teachers; mismanagement of Boards, large scale copying and wide variation in quality of examination and evaluation. Any dilution in the cut-off of percentile would raise the same issues which forced the dilution of 80. The political pressure would continue to mount and gradually we would keep reducing the cut offs. It is most unfortunate, the IIT system that worked notwithsatnding shortcomings, is being tampered with. The UGC's inteference with the IITs and IISc is yet another example of how the good system built with hard work spanning more than five decades could be destroyed. In any case, the track record of the UGC has hardly been a matter of pride. It should be the last agency to tinker with the IITs.
Oh, come on Dheeraj; don't act as if you are not aware of what went on at JAB and who are all the people pushing for this "admissions also for those who score 75 marks". Was the debate along any of the logical lines you point out?
Sriram, I know what you are saying. The real issue is political. The political masters could not take the heat from 200 odd students and a lot more who felt that they could similarly be denied admission. But the point is why is it that the political masters could not take heat. It is because they realize the arbitrariness of the grading by boards. I think that there is abroad agreement in the Ministry that boards are doing a very poor job, but it is not possible to say this publicly. They also do not want to appear as if they are undoing everything that the previous government did, and hence the NIT admission will continue to be based on 12th class marks. However, if NITs and other stakeholders raise as much noise as has been raised on 80 percentile issue, it is clear that the new dispensation is not wedded to the old decisions. Unfortunately, the politics is more important than academics.
By the way, the gist of what I have written in the blog was indeed said in the meeting. It was specifically pointed out that dilution of 80 percentile amounts to admitting that 12th class marks should not have any significant role in IIT admissions.
Interesting piece! Prof. Sanghi, you may find this post on board-exams interesting. It supports your argument: http://www.globiansperspective.com/2012/04/unscientific-practices-of-state.html
Dr. Sanghi, the fundamental problem here is the reliance on a 3 hour or a series of 3 hour exams to decide entry into higher education. One can try whichever permutation they like, and it will be impossible to be convince everyone that the system is fair. It is structurally incapable of being so.
Limited time tests can only be used for assessing general intellectual ability and some threshold level of preparation. They cannot tell you who is better prepared and will contribute the most to the community at your particular college.
We are trying to overcome our fear and mistrust of schools and teachers by simply bludgeoning the problem with this or that hammer (read entrance/board/etc exams). Colleges and universities need to do the hard work and select candidates by a thorough appraisal of their applications. Professors need to be reviewing applications and reading student essays rather than be preparing and grading 12th class Math/Physics/Chem exams.
About 15% of the population goes to university in this country, and at most 1% of the population has this obsession with getting into 'the best' (of course our media relentlessly focuses on these folks, and they are also the most vocal). But how long are we going to punish the remaining population, schools, teachers and professors for this 1% ?
I went through the links provided by Prashant. There seems to be a lot of confusion about what the Central Limit Theorem is and what it can be used for.
The CLT states that given a sample of N independent and identically distributed random variables, the difference of the sample mean and the expected value of the underlying distribution asymptotically approaches a normal distribution.
This does not imply that the distribution of scores for a class of students in any given year for any given subject has to look normal. Here's what the CLT can do for us in this setup:
Suppose we found a way to make all the distributions of scores for a subject across different years identical, and say we call this distribution P.
Then the mean of all the scores for a given year would approach the expected value of the underlying distribution P. It is this mean of scores that will have the normal distribution (asymptotically), not the individual scores themselves.
The individual scores are random variables, but their distribution is P itself, whose shape is dictated by CBSE's grading policies. And while you may not agree with these policies, the data show that it is being applied consistently across student populations and in time. The data in the 'Ten Year CBSE Score analysis' clearly shows that this is indeed the case, and the grading scheme is consistent across the years.
In fact, I will argue that the CBSE percentiles are a more reliable metric to 'rank' students. Given a student and a rank, it is very likely that if the same student took the exam next year, the rank range will remain similar. The CBSE exam is stable and consistent as a ranking procedure. It may not be desirable for other reasons, but thats a different matter. For the JEE exams, this is not at all the case, as will become apparent on seeing the extreme skew in the distribution. It becomes even more drastic if one factors in the individual scores in Maths, Physics and Chemistry. It is the JEE that is a lottery not the CBSE.
Vikram, if the grading scheme was consistent across the years, surely the shape of the curves would be similar? And they clearly aren't. It indicates low repeat-ability of the examination. FYI I have got this analysis read by academics working in the assessment space in the UK, US etc. and they all agreed that the curves are bizarre. They don't even coincide year to year which makes the examination unreliable. Plus, if you club certain subjects (where you stick to the raw distribution) with other subjects (where you flatten out the curve to an approximately uniform distribution) the aggregate ranking will not correlate fairly with different subjects. As I have mentioned no one is expecting some perfect bell curve - there will always be some kind of skew (and spikes at grading boundaries) but it doesn't take any deep analysis to discover that the scoring curves are changing drastically on a year to year basis. The varying shapes of the curve on a year to year basis indicate very low repeat-ability of the exam. A similar case in the United Kingdom led to the cancellation of an exam for something relatively minor GCSE Case
I think everyone agrees that the JEE alone is a far from perfect metric, the problem is that no one knows how the boards function. There are even rules such as - what happens if your answer script gets lost (what score you should get) - and the candidate has no way of finding out how the score is computed in such cases?
@Vikram:
If by "Professors need to be reviewing applications and reading student essays rather than be preparing and grading 12th class Math/Physics/Chem exams." you mean something analogous to the system most US universities follow, then I would invite you to read two recent essays: the first is by Steven Pinker (a Harvard psychologist), and the second is a supportive piece by Scott Aaronson (an MIT computer scientist). Both argue that the admission systems at their universities undermine students' academic achievements, and call for (surprise!) standardized tests to have a much larger role to play in the admission process. I guess the grass is always greener on the other side. I would especially like to draw your attention to the "prep schools" described by Pinker which get their students to do sham community service in order to bolster their admission essays: somehow I find this rather more undesirable than even the dreaded "coaching institutes".
However, in so far as this system can be applied in India, I think it is already being applied: IISc admits some of its students through KVPY, which, at least when I was a student, used to have an open-ended interview as a major step. But if the idea is to adopt the US model wholesale, then the evidence in favour of such a drastic change seems much too little.
While I agree with you that there is no a priori reason to expect a Gaussian distribution in CBSE scores, I don't know what is your evidence for the claim that "The CBSE exam is stable and consistent as a ranking procedure" and that "Given a student and a rank, it is very likely that if the same student took the exam next year, the rank range will remain similar". The fact that the graphs for different years look same does not imply that the criterion used across the population in a given year was uniform. Indeed, some people would argue that some state boards (perhaps not CBSE, but I doubt that) are rather inconsistent even in the same year---in the sense that different examiners can award significantly different scores to identical answer books. In other words, the same student, hypothetically taking the exam in two different places in the same year and writing exactly the same answers could end up with a very different rank depending upon where her answer books are sent for evaluation.
I would also like to see the "skew" you point out in the JEE data. The JEE appears to be a lottery simply because it is a competition for a very few places (unlike a board exam). Even so, the weirdness of the board scores (especially at the higher end of scores) means that JEE is much less of a lottery that the eminent farce of 100% cutoffs that plays out at DU colleges every year.
@Vikram, no doubt JEE is a lottery, but it is a very different kind of lottery than boards. In an exam, variation can happen either because the student performs differently on different days, or the grading can happen differently on different days. In JEE, the student performance can vary from a day to day, which means that a slight headache can cause a few less questions to be answered, and those few questions can have a HUGE impact on the rank, which of course has a huge impact on the admission to a program.
The boards on the other hand have a different problem. Different graders can result in significantly different marks, which of course matter a lot for not IIT admission, but for all other admission.
The luck or lottery aspect of JEE is something that people are able to live with. They understand that if they solved 2-3 problems less on a given day, they have performed poorly on that day. While other systems like US can be thought of, but I can tell you that the population in India would have much more trust in an objective exam than subjective reading of CV or essays or reco letters.
On the other hand, variation in grading is something that is very difficult to accept by anyone. If two persons performed similarly on the day of the exam, they should get similar results. This has to be ensured. I don't think most academicians and pedagogy experts outside India would even consider that such a thing is possible what is routinely happening in Indian boards.
And this consistency of JEE grading is not a recent phenomenon. Of course, with MCQ and machine grading, it is to be expected that all machines will grade the answer in same way, but even when JEE had long answers, the system would take extreme steps to ensure consistent grading.
Dr. Sanghi, that the JEE is a lottery (no matter what kind) is clear from the distribution of scores. Beyond a certain rank (around 800), and for the bulk of the population, the distribution is essentially uniform. A uniform distribution is meaningless for ranking a population.
From the data I am looking at, it is not clear that the accusation of systemic fraud and inconsistency in the board exam evaluation can be verified. There is a curve which is consistent across different years. It has a non-zero slope across the entire population. That curve looks much more suitable for ranking a population than the JEE curve.
One may say that the board curve would look similar to the JEE curve for the highest 20000 students. That would only show that the whole idea of ranking students beyond a certain proficiency level is extremely problematic and there is no option but to rely on subjective mechanisms to select students.
"Vikram, if the grading scheme was consistent across the years, surely the shape of the curves would be similar?"
Prashant, I am not sure which curves you are looking at. But the ones on this page (http://www.thelearningpoint.net/home/examination-results-2013/cbse-2004-to-2014-bulls-in-china-shops, that you yourself posted), look very similar across all years for all subjects (Hindi, Physics, Biology) for all years. Can you point me to the specific data you are alluding to ?
"but it doesn't take any deep analysis to discover that the scoring curves are changing drastically on a year to year basis."
I am sorry, but I just dont see this in the link you posted. In fact, they seem to show the very opposite.
"the problem is that no one knows how the boards function."
I fully concur and would support wholeheartedly an initiative to make the boards much more transparent.
@ IITK student,
"I would also like to see the "skew" you point out in the JEE data."
Please see the curve here, note that the y axis is on a non-constant scale. http://4.bp.blogspot.com/-1BDsdkCwJ38/U61nDmdfTZI/AAAAAAAAA0Y/yuBEs9P78Iw/s1600/Graph1.tif
Clearly, the bulk of the curve from ranks 800 to 16000 is essentially uniform. If you want a mathematical argument, we are on working on bounded domain [800, 16000], so a uniform distribution has maximal entropy and contains virtually no information. And so the JEE is useless for ranking the bulk of the candidates it is ranking. How is this not a lottery ?
If you would break the scores subjectwise the entropy would increase even more.
The JEE clearly does a superb job of identifying the top 500 from the rest, as one would expect from a maximally difficult exam. The same could be achieved via the Olympiads. The JEE has also had a huge positive impact in that it compels around a 100000 Indian kids to have world class quantitative and basic physics skills on graduating high school.
However, it is worthless for ranking the bulk of the students it ranks.
Vikram - I think you are looking at Hindi and CS curves for CBSE which are among the only ones consistent across the years. Please look at Physics, English, Mathematics, Chemistry and Economics and you will see how have changed year to year.
Prashant here are the summary statistics for Physics from all the years presented in that page:
Year Mean Std
2004 59.52 15.93
2005 60.63 14.41
2006 63.34 15.61
2007 63.09 16.54
2008 61.8 15.56
2009 62.94 16.66
2010 61.93 16.8
2011 62.41 16.42
2012 61.6 16.93
The variation in the means over these years is 3.82, and the std is 2.52, which are small fractions of the overall mean and std, indicating that the distribution is stable. Both location and scale (?) parameters are not changing, although I am not sure what precise scale parameter is used for bimodal distributions.
Why do you feel these distributions are not stable ?
I'd expect a slightly better sense of observation, I hope this presentation of my data makes things clearer.
So there are two things
(a)a drastic change of distributions within a decade and
(b)the non standard distributions themselves. Which isn't just my analysis - as you can see an Australian Professor who was appointed by CBSE itself, has expressed his surprise at this.
So in Physics we have the curve shape drastically different for 2004 and 2013.
Differences year on year may not be so evident but when you compare the curves with a 5+ year difference the changes become a lot more noticeable.
In English:
2004: Mean 59.7, Std. 17.7
2013: Mean 65.2, Std. 21
In Mathematics:
2004: Mean 55.2, Std. 23.4
2013: Mean 51.58, Std. 26.85
In Economics:
2004: Mean 51, Std. 20.87
2014: Mean 49.73, Std. 25.7
Changes of magnitude >3% in Standard Deviation are pretty considerable. The increase in STD is basically a consequence of pushing more people into the 90% band in particular (for whatever reason).
Also, I haven't put up this table, but there's a significant change in the % of people in the 90+ band as well per subject. Though that is intuitively evident from the curves as well.
One more example:
The CBSE 20 percentile mark was 78% in 2012, and 83.2% this year.
ISC 20 percentile mark was 81.2% in 2012 and 85% this year.
Most of those who get into IIT are within the top 10% of CBSE/ISC anyway. Adding a 20 percentile cut-off just ends up increasing tension by making people worry about the consequences of an exam which they have relatively little control over.
Prashant, you are making two claims,
(a)a drastic change of distributions within a decade
I see no change in the nature of distributions (in the years 2004-2012) given the summary statistics you have provided. I wrote them down, but I can see no trend. Now you are bringing in the year 2013 to justify your claim. I am not sure if this is admissible, since the examination regime changed in that year. The scores started being used for selection to highly sought after and selective institutions leading to evaluators taking a minimum risk option and pushing everyone in the high range up. This makes comparing 2004 and 2013 curves meaningless.
Now this might be a policy making error on CBSE's part, but still does not justify your claim of tampering. Exam evaluation is not done in a vacuum, there is a context to it and it has to be taken into account.
(b)the non standard distributions themselves
I have already addressed this point, there is no reason to expect any 'standard' distribution. It depends on grading policies and the nature of questions and what they are designed to test among other things.
Vikram states "Clearly, the bulk of the curve from ranks 800 to 16000 is essentially uniform". This statement is not correct. The data source cited is a plot of marks vs rank and contains no information of how many candidates scored particular mark / rank; I assume this is what is meant by distribution. If so, the cited curve simply does not have the information required to comment about the distribution. The JEE advanced 550+ page report is available at all IIT JEE sites and contains the marks distribution in histograms (well, in curve form), appears to have been plotted in 4 mark blocks for maths / physics / chemistry and 10 mark blocks for the total score. The curves looks reasonably bell shaped with longer tail on the high marks side. The chemistry score is close to having symmetric tails: peaking at around 30 marks with about 11000 candidates, and approaching zero near -10 on the low side and 90 on the high side. The total score distribution has peak 11000 candidates in [70 80] marks range. The claimed "essentially uniform" distribution of ranks 800 - 16000 corresponds to score range [134 234]. If the distribution were uniform we would expect about 1500 candidates in every 10 mark block, but the data is far from this. [130 140] marks has about 4000 candidates while about 2000 candidates are in [170 180] range and around 500 candidates in [210 220]. This does not look anything like a uniform distribution to me. Beyond score 230 or so is extreme end of the tail and it is hard to tell if the distribution is near uniform at that end. Data from JEE 2013 is presented in a different form in the JEE 2013 report and that shows that the distribution is not uniform near that end either. Here is the data on marks drop per 100 ranks that shows that the distribution is not uniform: rank 1-101 drops 56 marks, 101 - 201 15 marks, next 100 rank blocks have marks drop of 8, 7, 6 and 4. As we keep going down, we see a drop of 1 mark covering 100 ranks, then 200 ranks and so on till at around rank 10000, a drop of 1 mark fills 400 ranks. All this suggests that we are working in the tail range of a distribution that is certainly not uniform.
Thanks iitmsriram. I was working on a comment to retract my earlier claim. Indeed, there is not enough on the ranks vs marks plot to say anything about the distribution. And it is certainly not uniform. Apologies to all for my earlier haste.
My position is still that the JEE score cannot be used for ranking, although, I obviously need to develop a better argument.
Even if a comparison of 2011/2012 curves was made (wrt 2004) the priceonomics article would be almost entirely the same.
If there are separate explanations for 2013 then here's another one for a quick visual inspection.
Physics-2004 vs Physics-2011
Physics-2004
Physics-2011
Physics-2004 vs Physics-2011
Also, for Biology, the article I linked above - even if you replace 2013 with 2012 or 2011 the Priceonomics article would have been the same. (all three graphs have a very different pattern from the exams around 2005). Look at the shapes of the curves apart from the mean & std. marked. Unfortunately I now don't get a sense that you looked carefully at the graphs at all. The year on year increase in the 90 percenters in each subject even between say 2004 and 2012 doesn't speak much of the stability of the CBSE exam. It is visible in the graphs though I may add a tabulation as well.
Anyway I am not commenting further on this, since most people I know could quickly spot the changes based on a quick visual inspection of the graphs (similar to the Priceonomics article). I hope this is not a case of sour grapes.
Vikram, why do you say JEE score cannot be used for ranking? JEE used to be "insanely" difficult in an attempt to ensure that the distribution at the high scoring end spreads out, so we don't get the CBSEesque bunching. Nowadays, there is a designed attempt to increase the mean score while retaining a good fraction of tough questions so that both a respectable mean and high end spread are obtained. In some previous years, the mean and median scores have been below zero (so turning in an empty answer sheet would actually have resulted in better rank, though this rank would be too far away from the qualifying for seat mark). If your basic argument is that JEE can be used to set a qualifying cut off (all those scoring above xxx qualify for a seat) but not for ranking, that would be difficult to counter, we may not have sufficient data to accept or reject.
But, we seem to have digressed. Dheeraj's original point was about using XII marks. And, as Dheeraj can certainly be described as academic leader and administrator, we should ask for his opinion on whether we should use XII marks at all - for ranking or for cut off, whatever. Yes, Dheeraj has posted in the past about this, but now that we have more data available, what would be the sane thing to do?
Vikram, I would not say that both JEE and Board exams are lotteries and hence in some sense equally bad. (In fact, your contention is that at least CBSE Board is better than JEE for ranking.) I see that an exam being a lottery because of extreme competition and students losing half a percentile due to a small error and that causing serious impact on admission is one thing. At least the student understands that there is competition and they just did not do well on that day. Unfortunate, and we need a better system for admission, but it is not as bad as a student doing well, and loses several percentiles just because the grading is arbitrary. (Unfortunately, the statistical data will not capture erroneous grading, and this is a clam based on what you hear and see in your neighbourhood schools.)
Since my "belief" is that grading is perfect in JEE and bad in boards. CBSE is the best board in the country. Even that is not good enough is another matter. But our admission process can not be depend on CBSE being good enough or not being good enough. Some of the other large boards are just horrible, and unless most of the large boards become good enough, we should not have admission based on board performance.
There is no doubt that we need to reduce the impact of a single chance poor performance on one's career. But I am not sure if the US style of applications is possible in India. Any subjectivity is not likely to be tolerated by our potential students and parents, not to say the possibility of influence. But there is a way out. We can have the exam conducted multiple times and the best performance of everyone is considered. (Of course, will lead to issue of comparison across papers, but we have a handle on that.) We could then have multiple rounds. In my opinion, some public exam which includes very basic PCM, any one language, some analytical stuff, etc., should be there one year in advance, taken a few times, and select 100,000 odd for the next level. By doing this, we are ensuring that a single bad performance does not affect you and a slightly poor performance is tolerated in the sense it is unlikely that someone not in the top 100,000 in the first exam will be in top 10,000 after all steps are over.
These 100,000 could be given another exam, which is the advanced subjects that we want to test on (say, PCM, as we do now), and we further shortlist may be around 30,000 or so. Again the exam could be conducted on a couple of occasions to ensure that poor health, etc., type of issues do not affect the result and since the number of candidates is less, we could design the question paper in a way that a couple of questions being wrong would not thrown your rank out from 10,000 to beyond 30,000, so as to give another chance to people with slightly poorer performance on a given day.
These 30,000 could be asked to go for another step (I don't know what, but equivalent to whatever else is checked by top universities in their admission processes) and we get 10,000. Again, remove the lottery aspects by having a performance metric where a small error can cause a dip in the rank by only a small amount. And it is possible to do so if the number of candidates is small and a significant fraction of that will go to the next stage.
And finally, these 10,000 should choose only an IIT and not the branch. So small variation in ranks should not decide all important department. And the department should be decided after looking at the performance of entire year.
But this is the wish list. If there has to be a choice between ranking based on board marks and ranking based on JEE (Advanced), I will any day vote for JEE (Advanced).
@Prashant,
I hold no brief for the CBSE or any other board, nor am I contesting your hypothesis based on a personal vendetta. One of your claims is that the CBSE graders are deliberately inflating people's grades. You base this claim on the spike in the 90s that seems to appear from the year 2008 onwards. Could you tell me what the proportion of people scoring over 90 has been since 2004 ? (You can also point me to the data and I would be happy to perform the calculation myself)
@iitmsriram, I do believe that a system where the JEE acts as a benchmark for high achievement (which colleges use in their admisssion process in ways they find suitable) might be the way to go. Earlier I felt that the JEE could be conducted for subjects like CS, Math, Physics and proficiency levels like A+, A- etc be earned which colleges can use as minimum requirements and/or weights for admissions/ranking.
I now feel that even a general proficiency test which hands out just achievement levels will do. The the intersection of the top 10000 and those who score above 95 be given the highest proficiency level. Then the IITs can admit those with the highest proficiency, have them go through one year of college and then start the process of branch selection based on GPA + proficiency + counselling.
The system we have right now punishes IIT students no end. Especially, those that are not able to do CS and EE.
Dr. Sanghi, maybe JEE is better than the boards right now. But that is not stopping it from doing damage to MechE, Civil, Aero and other disciplines, the IITs and the whole Indian education system itself.
"And the department should be decided after looking at the performance of entire year."
This point cannot be emphasized enough. It is unbelievable what we have done to Civil, Mechanical and Chemical Engineering in the IITs. I have observed no such hierarchies on US university campuses. Yes, its slightly harder to transfer from Aero to Mech than vice versa, but this does not permeate into a broader feeling that some branch is 'superior' to another one on campus. The ranking system of the JEE is among the principle drivers of this hierarchy on Indian campuses. And the arbitariness in ranks beyond a certain level, makes this even more painful.
The ideas you have outlined are not only good from the point of view getting rid of the tyranny of JEE ranks, but also distributing the admissions process over a longer period and thus reducing the stress on students at the end of the 12th standard year.
This is the basis on which I'm making the claim about inflation.
Grade A1 is assigned to candidates ranking in the top 1/8-th of the passed candidates in a subject in the CBSE exam. The minimum score required for the A1 grade in 4 common subjects (EPCM) in CBSE is given below at 7 checkpoint years in this period (2004-2014)
As you may see the lower cutoffs for A1 grade changed drastically between 2004 and 2012 for English and Math (+11 and +6 percent) and
medium (+5 and +3 for Physics and Chemistry).
Variations are even more pronounced between 2012-2014.
If we look at 2004-2014
The upward climb in the A1 grade cut-off for the subjects has been-
English: +15%
Mathematics: +9%
Chemistry: +13%
Physics: 4%
Even if you compare 2004 with 2012 instead of 2014 (before making board scores a requirement for the JEE) the differences are noticeable.
2004
MATHEMATICS==086=A1
PHYSICS==080=A1
CHEMISTRY==080=A1
ENGLISH CORE==078=A1
2007
MATHEMATICS==091=A1
PHYSICS==086=A1
CHEMISTRY==081=A1
ENGLISH CORE==082=A1
2010
CHEMISTRY==081=A1
MATHEMATICS==086=A1
PHYSICS==085=A1
ENGLISH CORE==084=A1
2011
CHEMISTRY==082=A1
MATHEMATICS==093=A1
PHYSICS==084=A1
ENGLISH CORE==085=A1
2012
CHEMISTRY==083=A1
MATHEMATICS==092=A1
PHYSICS==085=A1
ENGLISH CORE==089=A1
2013
CHEMISTRY==092==A1
PHYSICS==086==A1
MATHEMATICS==091==A1
ENGLISH CORE==090==A1
2014
CHEMISTRY==093=A1
PHYSICS==084==A1
MATHEMATICS==095==A1
ENGLISH CORE==093==A1
@Vikram:
I do not quite understand your claims about the "uniform distribution" being worthless for ranking. In fact, the same argument can be made against any distribution that is smooth near the cut-off point (for JEE, that would be at roughly the 98th percentile), since that means that there will be a non-negligible fraction of people who would miss selection just because of individual random variation (the kind Prof. SAnghi talks about above, not the kind induced by variation in how the test is graded). Any distribution that is very smooth close to the cut off will have this problem.
Also, you are probably looking at the two graphs at very different scales: for JEE, you have only the first 16000 points out of a sample space that is some 600 times larger, while for the board exams you have the whole space. A look at the distributions of some of the boards around the higher percentiles might be instructive to see how useful they are for ranking (anecdotes do not count as evidence, but you would be surprised to know that one of my friends had a score in excess of 96% in the Andhra board, while having a rank outside the top 100). The fact that so many DU Colleges have 100% cutoff should immediately show why board exams are far more "worthless" at ranking.
I am also sorry to say that your argument from entropy is irrelevant. Firstly, the claim that "so a uniform distribution has maximal entropy and contains virtually no information." is exactly the opposite of what entropy means: the uniform distribution having high entropy means that it carries the "maximum" amount of information: in the sense that a sample from it gives you more information (at least, as quantified by entropy) than a sample from another discrete distribution would. But any of those interpretations are completely irrelevant to the issue. The crucial point for ranking is how non-smooth the distribution is near the cut-off points.
@iitmsriram:
I think that Vikram is correct in pointing out that what matters for JEE to be a good ranking test is for the curve he showed to have non-smooth behavior around the cut-off points. The ideal ranking test would be for which everyone who is selected has a score that is much higher than the score of everyone who is not (or, in other words, that the curve should have a discontinuity at around rank 10000, which I believe is the number IITs select). OF course, such a test is hard if not impossible to design. Why Vikram tried to pose this in terms of distributions and entropies, I do not know.
But I think the major flaw in his comparison that is looking the plot of the 98th percentile of the JEE, and comparing it with the plot of the whole population of the JEE. Theoretically speaking, that plot does have enough information to construct a distribution. In effect, it is a plot of percentiles vs observations, or marks, so that if you rotate the graph by 90% clockwise, it would exactly be a (modified, since the author is using log-log scales) graph of the cumulative probability distribution. But the crucial point is that it would be a graph of the cumulative probability distribution only beyond the 98th percentile, and would be perfectly consistent with your claim that the graph on the whole population looks approximately normal.
@IITK student, you are correct in pointing out the flaws in the entropy argument. However (if this digression can be indulged) I am not sure I agree with your interpretation of entropy in this context. Entropy is a measure of disorder/randomness/uncertainty. So when we say the uniform distribution has maximum entropy among all continuous (and discrete) distributions on a bounded domain, we mean that it captures the situation where we know the minimum possible information about the random variable.
In contrast, if we say the random variable has a truncated normal distribution in the same bounded domain, its entropy will be lower, because now we have more precise information about the variable.
My thought while making the entropy argument was that if one conducts an examination in an unbiased population with the hopes of ranking them, then a test whose scores follow a uniform distribution would not be useful. Because then all the variation in the scores would purely be due to randomness and not the discriminating abilities of my test. I felt that there is a similar situation going on in the case of the JEE marks and the people in the 800-16000 range, there were certain easier questions which all of them knew how to do and certain hard ones which none of them could answer, so all the variation in their scores arises purely due to randomness, eg: being ill on the day, misreading a question, machine error in grading etc.
@Prashant, thank you for presenting that statistic. It made things much clearer for me. The trend is most stark in the English language marks, while its fairly limited in Physics. Are you sure this is down to biased grading, and not a change in CBSE guidelines in grading/setting English language papers ?
Although, that probably wont explain the trend in the Chemistry statistics.
@Vikram:
The interpretation of entropy in my post is the contrapositive of (and hence equivalent to) the interpretation you present in you new post.
I agree with you that a distribution whose cdf looks very vertical for a range of marks would have rank highly susceptible to small changes in marks in that range of marks (as such, there are distributions that would be much worse offenders than the uniform distribution, by virtue of having even larger slopes in the target range). However, having at least one such region of large slope is a mathematical necessity, I think the best place for this would be around the cut-off point. However, designing a test to have exactly one clear cut-off is perhaps easier said than done. Nonetheless, as both you and Prof. Sanghi point out, perhaps the biggest change needed is greater flexibility in choice of departments, which is almost entirely based on ranks in most IITs (I believe IITK provides, by comparison with others, rather flexible rules for change, but these are only allowed to very few students).
Also, following the principle of truth in advertising, I should mention that I am not a student at IITK and my screen-name is purely a product of laziness.
An IITK Student feels that "The ideal ranking test would be for which everyone who is selected has a score that is much higher than the score of everyone who is not". This essentially presupposes that the population of test takers can be divided into these two distinct groups, the chosen ones (shall we say) and the also rans - all the test has to do is to be able to sieve them apart. I seriously doubt if this assumption is correct. Dheeraj is suggesting that we should have multiple test administrations so that the randomness of a single test can be overcome. I suspect that if we do multiple tests etc, the curve would actually become smoother and less double humped. On the other hand, if we look at the recent trends in the CBSE scores, we do see some evidence of two distinct groups and then the question becomes whether that is an artefact of scoring (or manipulation of scoring) or the underlying truth.
@iitmsriram:
"This essentially presupposes that the population of test takers can be divided into these two distinct groups, the chosen ones (shall we say) and the also rans - all the test has to do is to be able to sieve them apart."
I agree with you that designing such a test would be near impossible. But purely as a mathematical curiosity, it does not necessarily require such a division in the actual abilities of the population to exist.
Suppose for example that there is some notion of intrinsic ability that varies smoothly with percentile across the whole population. Let us call this measure X(p), where p is a function of the percentile, and assume that it varies from 0 to 1. (As a n aside, I seriously doubt such a measure exists, but test-setters seem to assume this anyway). Now X(p) must be a increasing function of p to numbers between 0 and 1, but it can otherwise be as as smooth as you want.
Our "test" is now an increasing function f over [0, 1] mapping it again into [0,1]. Suppose that the test is to select people with X > Z := X(0.98) [that is, those above the 98th percentile in terms of "intrinsic ability"]. Then all I am asking is that f should have a large derivative only around Z, and not elsewhere. Note that no assumptions on X are made, only f needs to be non-smooth.
I know that when I was a college student, instructors already tried to do something of the sort by introducing, say, a somewhat harder problem on the test that you are only supposed to be able to solve if you had worked hard enough to get "beyond" the 90th (or 95th or whatever) percentile of the population. I do not know how successful it was in practice at producing jumps in grade graphs, but certainly they had some version of the above argument in their minds while doing this.
w.r.t. the posting on dilution, I have difficulty in agreeing with the various stated interpretations on why and how the new criterion was added, but I agree with the overall conclusion that this change of heart is illogical.
However, the main point I have to make here is that we are digressing.
I view the top 20 percentile as a good thing. Not because it is fair for the IITs. Or it is logical for ranking. Etc. Etc. As Dheeraj argues it is not.
But it is "fair" to the overwhelming majority of the school children. A overwhelming majority of students and teacher are at the raw end of the spectrum because the small "smart" minority bullies the rest of the class with impossible problems. They tend to view "proofs by induction" as irrelevant, and language as a monstrosity. The treatment of girls (who happen to be a minority in coaching classes) in the school is undesirable. The will not copy in the TopicWiseTest conducted in the coaching class but will not blink to copy in the school unit test bring dishonesty to a different level.
You may call it perverse, but it is a desperate attempt to bring some sanity in classes XI and XII in a system crazily manhandled by making promotions compulsory till class X, CCE notwithstanding.
So yes, the 75% fix is confusing to the say the least. Yes there should be a better solution than the arbitrary 80 percentile. But till then, can you leave us people who want to learn (grow up) in Class XI/XII alone and not bombard us with obscure massless pulleys and the wavy curve theorem?
Do the NIT Directors aka Leadership have the courage to confront the Govt on this? They meekly submitted to the 2012 diktat while the IIT leadership at least vehemently opposed & thus brushed aside the inclusion of XII percentiles for ranking.
askingForSanity: I am having some trouble understanding your argument here. Do you think that the 80 percentile criterion is somehow more effective at making people "learn" than the 75% criterion? If so, what does it have to do with massless pulleys and the "wavy curve theorem"? I must, however, confess I have no idea what the "wavy curve theorem" might be.
It is true that there is an undue emphasis on "tricks" in JEE coachings. On the other hand, it is not as if the current school curricula actually emphasize anything better. When I was a high school student, my experience was that I learnt much more mathematics, chemistry and physics out of the textbooks I used for "preparing" for the JEE than out of my school books. But then I never really went to a coaching class (I depended mostly on self-study and sometime on my teachers at school, who were all very knowledgeable), so I am not sure what goes on there.
askingforsanity, you are quite correct about the negative impact an evaluation regime based on solely three hour tests and other limited time entrance exams has on teaching in schools.
One of the main outcomes is that school instructors have the power to evaluate and assess (at least assessments that matter) taken away from them. I have seen first hand the demoralizing impact this has on schools in Mumbai.
AnIITKstudent is right about the curriculum for the boards not being necessarily better than the JEE, but the problem here is fundamentally with the testing regime, not the curriculum.
As it is teacher quality and motivation are big problems in India, http://ajayshahblog.blogspot.com/2012/01/first-pisa-results-for-india-end-of.html
@iitkstudent I was not arguing for 80 percentile versus 75 percent or vice versa. I was arguing for why the basic premise of increased (compared to the earlier 60 percent) school weightage, while being potentially unfair to the IIT system, is beneficial for the overwhelming larger majority.
If you agree to this premise, then
the percentile model is definitely better. The percentile model by definition does not let you pick definite target. It depends on how others perform. So if you think others will perform better, you push yourself. This same logic works for the others so the whole education system has the potential to benefit.
The percentage model favours Andhra board (for example) to the detriment for eastern states board, and appears to me as quite unfair to these boards. It also creates and encourages an inflation of marks which is absurd.
The percentile model (and percentage model) is flawed because of a different reason -- double counting principle. The JEE advanced tests PCM, which is precisely what constitutes the board exam for a vast, indeed 100% majority. However, that is not a problem for me here because if an IIT aspirant is going to study in the school, and (more important) let other non-IIT aspirants study, then everyone wins. Further, if the IIT aspirant wants to go to the Coaching Class and join an integrated programme, all power to him (use of male gender deliberate). Let him quit the school and let others who want to learn in school be in school. This might jack up the profit margin of the coaching class but I am not concerned with that.
As to whether the school curriculum is better or not, that is a different (and complex) issue. Many points there but any curriculum that emphasizes learning with respect for all is beneficial.
@askingForSanity: I am still having trouble understanding the reasoning behind your claim that preparing for the IIT-JEE somehow raises obstacles in actual "learning" which is supposed to take place in schools. As I said in my post, the challenging and open-ended format of the JEE often provides the "push" (your word) for actual learning to several students in a way that the bland school curriculum (especially in the sciences) does not. You seem to be of the opinion that people who prepare for IIT-JEE are not interested in "learning". My experience has been quite the opposite, so I would be interested in seeing what data you have in support of your claim.
Also, I should point out that the debate here is that your basic premise -- that giving a large weightage to the board percentage is "beneficial" -- is the one that is being challenged here. The inconsistency and arbitrariness of grading standards in board exams means that putting a very high threshold on board marks would actually throw out people for completely random reasons, having nothing to do with how much they "learnt" at school. I fail to see how putting a high premium on an inconsistent metric is going to be beneficial for anyone.
@IITKstudent
The blog format does not allow us to converge easily. I agree I could be more precise but I have been writing more casually than I should have.
First, I agree with you. Preparing for IIT JEE (advanced) is a good idea and a good learning experience. I would love to prepare for IIT JEE if I were a 11th/12th standard student.
However.
IIT JEE is not for all. It is only for a small minority.
The issue here is that there are different things to learn. What you call as bland curriculum might be a personal preference (valid for you in that case). Many IIT JEE aspirants scoff at "proof" techniques, scoff at the importance of drawing beautiful pictures (e.g. in physics), or scoff at the ability to write.
Or it could be that it becomes bland because the lack of importance given to the system.
And different people learn at a different pace. In fact most people cannot learn at the pace that the IIT JEE Advanced wants you, given that promotions upto Class X is "free".
You need to give them space to learn at their pace.
Increased board weightage does not help IIT JEE aspirants (as Dheeraj articulates, and as agreed by me).
Post a Comment