The mess called JEE Advanced

Supreme Court, yesterday, stayed the process of admission to IITs, NITs, and several other institutes. A student has filed a petition a couple of weeks ago, in which she has claimed that awarding marks to all students for wrong questions was fundamentally flawed and has resulted in significant change of merit, and that there were alternative ways in which the problem of wrong questions could have been sorted out. The writ petition can be read on the same page as this news report.

I read the petition. I don't know if everything being claimed there is correct or not. It is for IITs to decide on how to respond to those points. However, if even ONE of those claims is correct, I think it is a matter of shame for the IIT system. The news paper reports suggest that IITs responded yesterday to the petition by arguing that the process of admission has already started, that about 30,000 students have already accepted admission, and that it will take a long time to recheck everything, etc. I hope this is incomplete response and that they have said more things. I hope IITs are able to argue that the suggestions made by these students are somehow wrong or unimplementable. Because if they have not argued on those lines and cannot argue on those lines, then we can have no trust in JEE any more. And I believe that a fair JEE was always one of the most important reason behind brand IIT. If JEE is no more fair, then it won't take long for brand IIT to go down as well.

But I have my doubts. People who have followed this blog would know that my confidence in fairness of JEE and in the competence of IIT system to conduct a large public exam has always been very low. Every year, we see wrong questions. Every year, we see that IITs don't respond to alternate answers by external experts (or even internal experts).

Let us look at the main issue. Apparently there are questions which got misprinted in Hindi medium. The claim in the petition is that because the question was misprinted, IITs decided to award marks to every single student. Now, what is the percent of students who sought Hindi question paper. I believe that the number is 3-4%. Only a couple of variants out of 10 variants of the question paper had a misprint. Which means that about 0.5% of the students received a different question. Now the petitioner is claiming that it was possible to identify these 0.5% students who may have given a different answer, and for these 0.5% students, one could do different grading. There is no reason to give full marks to all students. Makes eminent sense. If you incorrectly identify (why would you) 5-10 students, say, and 1-2 of them could have been in the top 10,000 ranks, then you are unfair to 1-2 students out of 1.6 lakhs. But by giving marks to all 1.6 lakhs, you have been unfair to may be 10,000 students who could solve that question. Now, if you are faced with a difficult situation, and the only two options are that you be unfair to 10 students or you be unfair to 10,000 students, what would you choose. I am sure the answer is obvious. You would rather be unfair to 10 students than 10,000 students. But IITs don't think like that. (And I have been inside the room with JEE Implementation Committee in the past, so I know the culture there.) The IITs would argue that let us give bonus marks to everyone. It does not matter how many students you are unfair to, since no one would do such back of the envelope calculations. If you give marks to everyone, there is a greater chance of happiness all over, and less chance of someone going to court. And the only thing that matters is a court case, and not fairness in the system.

At another place, the petitioner has pointed out that while the IITs have given out one single answer to a question, a large number of coaching places have given alternate solutions which result in different answers out of the four choices. Now, let us remember that some of these coaching classes have some of the finest Physics/Chemistry/Maths faculty, some times better than the paper setters. If these people are saying that there is an alternate solution, and showing that alternate solution on their websites, IITs must respond to them. They must point out why those solutions are not acceptable. I know what IITs will argue. How can they respond to each and every answer given by any tom, dick and harry. But sorry, they are not tom, dick or harry. There are typically, only a handful of answers which are different from IIT answers, and there is no reason why IITs can not respond to those handful of answers, and that too, as I said above, by some of the finest brains of the country. Again, I have had experience of being in the room. Typically, it happens because the question is ill framed. The language is ambiguous. And no paper setter would ever admit that the language is ambiguous. And if they were to respond they will have to admit in writing that a particular statement only means something and that in their opinion alternate meanings are wrong. Now, this interpretation can be easily challenged in a court of law. So, as I said above, the fairness does not matter. Only court cases matter. So we won't agree to those alternative solutions, since our ego is hurt, and we won't explicitly respond saying that, since fairness does not matter. Only court cases do. Now, in this particular case, either IITs must respond as to why multiple answers are incorrect or agree to regrade as per multiple answers.

I also want to know if there is any accountability of paper setters. Does their substantial remuneration depend on their performance. Are they told that they will get x amount of money, if there is no dispute about their answers, but will progressively get lesser and lesser money, if it turns out that they made a mess. Is there a process of black listing professors who set erroneous questions. Why does it take weeks to publish the answer key, when the coaching classes normally publish them within an hour. Wasn't a key prepared several months in advance. Why is that key not announced within an hour of the exam.

IITs have argued (as per media reports) that it is too late, that about 30,000 students have taken admission, that regrading will take too much time. First of all, regrading will take a few minutes. Yes, a few minutes. If IITs are claiming that it will take a long time, then they are lying, and someone should be persecuted for lying to the court in an affidavit. All the answers have been scanned and stored. Now, only a key needs to be decided and that needs to be applied to the stored answers. We don't need to scan the answer sheets again. And applying the new key will take a few minutes.

If the scale of the problem is really as serious as this petitioner is claiming than it does not matter if 30,000 students have taken admission. We must restart the process, and if that means that the semester will be delayed by a week or two, so be it. Let us not forget that in 1997 when the JEE question paper was leaked, the semester started 5 weeks late and the heavens did not fall.

My only concern is that the petition seeks as the last resort (page 20, prayer 'e' of the petition) a direction to the IITs that all those who took JEE Advanced this year be allowed to take JEE Advanced next year. I suspect that IITs will latch on to this demand and agree to it, if they are allowed to continue with the admission with the current merit list. This particular petitioner may be ok with such a decision, but it would be grossly unfair to thousands of hard working students (again assuming that what is written in petition is correct).

It is becoming increasingly clear that conducting large public exams is not a core competence of IIT system. I am sure some of you would disagree and point out that in an earlier era JEE was conducted wonderfully well. Well, what was done in the pre-RTI era can not be termed as wonderful. It is just that we don't know what happened then. A few years ago, when the previous government wanted "One Nation, One Test" and we successfully resisted that, one of the commitments that IITs had made was that besides the JEE Implementation Committee (consisting of all JEE chairpersons and vice chairs), there will also be a new standing committee which will look at the longer term horizon and advice on how JEE could be made better. That committee does not exist today and all the issues of JEE are left to be resolved by the current chairmen and vice chairs who are under pressure to somehow manage this year's JEE with as few court cases as possible. I think it is high time IIT system has a standing committee to look into the longer term.

Added on 8th July:

A few questions have been asked offline, and I thought I should address those questions.

Q1: IITs appear to have claimed (see the ToI news where this is mentioned) that it is not possible to identify the students who solved the specific Hindi paper with that misprint. Is it possible.

Ans: JEE knows exactly who applied for Hindi question paper and who applied for English question paper. JEE also knows which code paper was given to whom. Now, the only issue is that a student could possibly have both Hindi and English version of the question paper and looked at the English paper and answered it accordingly, even though in JEE data base s/he is shown as a student who received the Hindi question paper. There may be 5-10 such students in the country, but theoretically, they could be 100 or 200 students. So technically, IITs are right in saying that they do not know who have looked at only the Hindi paper before answering.

However, this is just an argument to not do anything to solve the problem. Remember the principle at this stage should be that out of all bad options, we should select the least bad option. So what if JEE were to say that we will grade you according to the language you were supposed to be looking at. This would perhaps be unfair to 5-10 students. The current solution is unfair to 10,000 students. There are other options. What if we consider both 5 and 6 as the right answer for those few students and consider only 6 as the right answer for everyone else. Or even, consider 5 or 6 as the right answer for all students. So let us understand that the only options with IITs are not binary (as Attorney General, the highest law officer of the country is claiming in the highest court of the land). They have multiple options with varying impact on fairness.

Q2: Isn't there a Supreme Court judgment in 2005 that bonus marks in case of wrong questions can only be given to those candidates who attempt the question. Why did IITs award marks to all.

Ans: I have not read that judgment. But I doubt if that would be relevant in the current case. Earlier, the exams required long answers to be written. So if someone has written something in the answerbook which has anything to do with the question, it could be taken as a proof that the answer has been attempted. But now with fully objective type question, students try to solve the question in rough sheets and then fill in a bubble. If they couldn't reach any answer, or if they have a doubt about their answer, they will not fill in the bubble. Since those rough sheets are not kept, and sometimes rough work is done on the question paper itself which is allowed to be taken away by the student, there is no identification possible of students who have "attempted" the answer. Filling in the bubble is certainly a proof of attempt, but not filling in a bubble is not a proof of lack of attempt. But note that this petition is really not dependent on 2005 ruling. This petition is arguing that a better solution key would be more fair than giving bonus marks to all.

Q3: What happens to a student who has rejected BITS/IIIT-Delhi/IIIT-Hyderabad, etc., and is now told that his JEE Advanced rank is much poorer. Wouldn't revising the grading scheme be detrimental to his/her interests.

Ans: Yes, it will be unfair to such a student. However, remember the philosophy that I am suggesting. If you have to be unfair to some, prefer a system with least unfairness. And would you rather be unfair to someone who did not deserve an IIT seat or would you rather be unfair to someone who deserved an IIT seat. Also, it is not as bad. If there is a new ranking and new counseling, the student may not get IIT but will get NIT/IIIT etc. So unfairness to an undeserving student is going to be rather small.

Also, even this unfairness can be handled. The court can order a 10% increase in all IITs in all programs as a one time measure to reduce this kind of unfairness. A single batch being 10% larger will not be a disaster for any IIT. And it will give relief to large number of students whose ranks change. Note that if someone's rank changes drastically downward, there may still be cases where they will not be fully compensated, but heck, these guys didn't deserve to be in IITs by a wide margin.

Added on 9th July:

As a comment on the next blog, Mr. Kandasamy Subramani has sent the link to a petition on change.org. There the issue of bonus marks for supposed misprinting has explained very nicely. Please read that.

IIITD: Information Technology and Social Sciences

This article is about a very exciting program being started by IIIT Delhi this year. If you have done 12th class from a school in Delhi and you were NOT a science student but did study Mathematics, and your aggregate marks (in 5 subjects) is at least 80%, you have a chance of getting admission into this really fantastic combination. But hurry, deadline is tomorrow (24th June), 3PM. (Deadline extended till midnight.) Here is the link to apply online. If you are a Science student and has taken JEE, you can apply for this program through Joint Admissions Counselling website.

This program is a unique combination of Computer Science and Social Science. I am not aware of any other under-graduate program in India with this combination. Of course in US universities (and other places around the world), one could do a double major but such flexibility is missing in Indian universities. And even where such options are available, most students consider studying anything other than Computer Science as waste of time. So, IIIT-Delhi has decided that instead of only offering a regular BTech (CSE) program with options of doing a second major in different disciplines, they will offer programs where the student commits to the second major right at the time of admission. This allows tailoring of program right from the 2nd semester. (There are other programs of this nature: Computer Science and Applied Maths, Computer Science and Design, and there are plans to offer more such programs in future.)

The program requires you to study all the core Computer Science courses, and a few electives in CS, but it also exposes you to social sciences. There are three Social Science streams offered: Economics, Psychology, and Sociology/Anthropology. The student has to choose two streams to do at least four courses in each of them, and do at least one course in the third stream. Besides, there are some foundation courses, and a few electives that have to be done in any social science discipline. (Besides these three disciplines that we will be offering several courses in, we do offer a few courses in Philosophy, Literature, History, and so on.)

Since one of the goals of this program is to produce social scientists who can understand and use computational technologies, the social science content has been chosen with a view that a graduate can seek admission in any reputed Master's level program. So suppose you want to do Masters in Economics, not only you will do minimum 4 courses in Economics compulsorily, but can also choose a few more electives, and with 6-7 courses in Economics, will be ready for admission to the relevant Masters programs.

The society and the industry today desperately need people with such combinations. If one wants to be an entrepreneur, understanding technology is not enough, the exposure to social science is extremely important. The same is true for those who want to be managers and administrators. Even product development will be significantly enhanced if one really understand the society for which that product is being developed. On the other hand, there is a serious need for social scientists who can use Computational technologies to improve our understanding of society. Data analysis has become an important tool for social science research. And of course, one can do inter-disciplinary work that overlaps with both CS and Social Sciences. Since there is a huge gap, it translates to a huge opportunity for the graduates of this program.


More details of the program can be read from this link on IIITD website.

We had an open house yesterday, which had a focus on the two new programs, including this one on ITSS. I am mentioning here a few typical questions or concerns that were expressed by students or parents in the open house.

The foremost was this: I am only interested in Computer Science, and I am considering ITSS only because I am unlikely to get admission to CS. Is the social science component a waste of time for me, or is it useful.

There is one question that many students ask us all the time. Why should a CS student study Chemistry or Thermodynamics or Sociology. Vikram has recently explained the need for diversity of courses in an educational program really well. To add to what he has already written, broad based education will really help in future because today, we have no idea what kind of jobs and careers would exist just 15-20 years from now. And remember, the person entering higher education today will probably be an active worker 50-55 years from now. How do educational institutions ensure that today's education remains relevant in an uncertain future world. Really speaking, the only thing we can do is to impart some knowledge and skills which are likely to remain relevant in the next 10 years, and impart the most important skill of learning new things on one's own. And learning becomes easier if you can make connections of new knowledge with the knowledge you already have. And therefore, having a broad based education ensures that you can learn more easily throughout your career, since it would increase the chances of making connections with past knowledge. This is one reason why liberal arts education is getting so popular lately. The hope is that a broad based education will enable you to do anything that you may want to or have to do in future.

So, coming back to the original question, my take is that even if you are not deeply interested in social science, having a diverse set of courses would be immensely useful to anyone. Of course, we would love to have students who are equally interested in CS and SS. That would make for a more interesting class and everyone learns better as a result.

Also, anecdotally, when ever we have asked senior alumni of IIT Kanpur regarding what courses have really benefited them in life, surprisingly, a large number of them mention social science courses. My personal belief is that studying a large number of social science courses is very positive even if someone was only interested in CS related careers.

The second question was: Is this program better or worse than Computer Science program.

It is difficult to compare two programs. However, one should note that whenever any university will think of starting a new program, it will certainly think of whether the program fills a need of the society and whether it can be offered with at least similar quality as its existing programs, if not better. It would be trivial for IIIT-Delhi to expand by simply increasing the number of seats in its existing very successful and popular programs. There was no need for a massive one year effort in planning a new course, which included taking feedback from 100s of people in India and abroad, personally visiting several universities and checking out websites of a lot more, having multiple workshops of subject experts as well as people from industry and academic thinkers of the country. So it is a program that we have invested in heavily because we believe that this combination is a great need of the society and the graduates will contribute in leading India to greatness.

Question 3: Would it be difficult for non-science students to compete with science students in the same class.

The IIIT-Delhi curriculum does not have any compulsory physics/chemistry/biology courses. And Mathematics at the 12th class level is required for admission. There may be 1-2 courses (like a course on circuits) where some exposure to Physics could possibly help, but in everything else Maths is sufficient background to perform well. We believe that a mixed class would lead to better learning and we are really hoping that many non-science students (from Delhi) will seek admission to the program.

Question 4: The all important question of placements/internships.

Howsoever I may like to convince students and parents that if you get good quality education, you will not have to worry about jobs and careers, this question keeps coming up. Well, obviously, we don't have data and won't have data for 4 more years. The only thing we can say is that even the first batch of IIIT-Delhi that graduated in 2012 had great placement, and they are currently either studying in great universities around the world, or working in top companies of the world, or are doing other interesting things like their own companies. And we have not looked back since then. The placement (if we consider the median salary offers) is one of the best in the country. And this is purely based on the quality of education that IIITD provides. If we could have a great placement for our first two programs when no one knew us, it should give confidence to prospective students and parents that it can only get better after we have made a mark in the education sector. So look at our faculty, talk to our students, convince yourself that we continue to provide high quality education and trust us that quality leads to great jobs and careers.

In terms of sectors and kinds of jobs/careers that could find this education very useful (though I must warn again that good quality education allows you to learn many new things quickly and hence get into other sectors/careers quickly), I think the Data-centric approach of some ITSS courses (Econometrics, Networks (Social Science Research Methods), Machine Learning (Pattern recognition, etc.)) are contemporary areas with a lot of job opportunities that will only grow in the shorter-run. Specific sectors include e-commerce/retail, Banking/Insurance sector, credit card companies with risk-management roles, consulting sector, data-driven journalism, new media/social media roles, and many more. Of course, any IT product company like all well known MNCs, would find this education highly desirable.

To end this article, let me quote from an article in Quartz:

As the vociferous Shark Tank host and entrepreneur Marc Cuban has recently observed about business careers: “I personally think there’s going to be a greater demand in 10 years for liberal arts majors than there were for programming majors and maybe even engineering, because when the data is all being spit out for you, options are being spit out for you, you need a different perspective in order to have a different view of the data. [You need] someone who is more of a freer thinker.”

While we are not in the business of liberal arts education, we certainly believe that a more liberal education would be hugely valuable in future.

Why are Parents Confused about College Choices

In July, 2015, I was at the Pan IIT Conference in Santa Clara, US. In one of the chat session, the legendary Vinod Khosla said something close to my heart. Let me paraphrase him and change the context from Silicon Valley to India:

"It is understood that you need to live comfortably, that you would want to rent a decent house, and eventually buy an apartment, that you would want to buy a car, have all the modern amenities in your home, take care of all your responsibilities like childrens' education in good places, supporting parents in their old age (including their medical bills), save some for a rainy day, have a vacation every year, and you can add a few more things that you really want to enjoy. This will translate to securing a decent job after your education. But once you have achieved all this, what next. Would you want the second house, a third car, a fourth iphone, more jewellery, and so on, or would you then think of jobs that you enjoy doing. May be the job that you like is also the job that gives you lot more money. But if there is a choice between doing something not so enjoyable but getting lots of money, and doing something that you enjoy and still getting money greater than your threshold for comfortable life, what would you prefer."

I am sure, if the question is asked in this fashion, most people would say that they would follow their interest once they are reasonably secure financially. But do they mean it.

In our country, someone starting with a salary of 4-5 lakhs and a decent growth (say, 5% more than inflation) would mean that everything that we have written above could be done very easily. In fact, one could start at even lower figures and grow from there. And pretty much every discipline in any good institute would enable a good student to get these levels of jobs. So the placement question that every parent is so interested in should really boil down to whether the median salary of the graduates is more than 4 lakhs or not. And beyond that, the student should be free to decide the program and college. But we all know that is not the case. Parents and students are easily excited about large salaries and do not even attempt to figure out what they might like.

Often, the goal is to earn the maximum. The optimization function is the money to be earned over the next 50 years. Unfortunately, this function is very difficult to compute. We don't even know what jobs and careers would be there 10 years from now, what to talk about 50 years. An average person will not remain in the same career for so long, and indeed people change careers every 10-15 years. So what is needed for success is not a particular discipline, but the ability to learn new things. Even if you happen to be in a low paying career in the beginning, when the time comes to change your career, you could then pick up a better paying career, if you had the ability to learn. On the other hand, if you do not pick up the ability to learn, being in the highest paying career is of no use if you are going to lose that job in 10 years.

So from monetary point of view, one should look for education which gives a decent start and prepare you for change of careers (or teaches you the most important skill of self-learning). Again, it would seem that pretty much any discipline in any good institute would meet this requirements.

Since it is pretty obvious to anyone that earnings over the next 50 years are impossible to compute, parents use a proxy. The first month's salary. The assumption is that if the first month salary is high, it would be a strong indicator of higher lifelong earning. That is a huge leap of faith, since as we just said we don't even know what kind of jobs will be there after 10 years.

Let me give you my example. When I was in my final year, and was applying for higher education, the hottest areas were theoretical computer science and Artificial Intelligence (which at that time was really about search and expert systems). I had thoroughly enjoyed the course on Networking and wanted to pursue that. Each of my well wisher advised me that I should write TCS/AI in my statement of purpose, but I was adamant. It had to be Networking. I was lucky to get admission in a good school (UMCP). And I did my PhD in Networking. By the time I finished, all the smart folks were competing for a diminishing number of jobs in TCS/AI and there were extremely few networking PhDs. I got a job in IIT Kanpur, which was considered the best CS department in India at that time, where around the same time, we rejected the applications of PhDs from top departments of the world in TCS/AI, since there were just so many of them. If you were in networking in 90s, the industry lab to work in was Bell Labs, and I could get a visiting position there for a semester. Not having to compete with an army of smart people has helped me a lot in my career. And those who only looked at what is the current favorite actually were so disappointed with the job market, many of them had to change their careers right away. And, of course, today, the networking is no longer hot. One should have morphed into a security expert, or a cloud expert or an IoT expert, or something else.

So, if we don't know what jobs we will be doing 10 years from now, we can not take it for granted that high first month salary will necessarily mean high income over 50 years

But, the first month's salary is important to parents. The next issue is that even if you know exactly what salaries have been offered to everyone in the graduating batch, how do you know what will your ward be offered 4 years from now. Let us consider two programs, both having 10 graduates. In one program, each of those 10 graduates has a job with a salary of Rs. 10 lakhs per annum. In the other program, there is one graduate with a salary of Rs. 1 crore, another one with a salary of Rs. 10 lakhs, and the remaining 8 with salaries of Rs. 2.5 lakhs each. So the average of the first program is Rs. 10 lakhs, and the average of the second program is Rs. 13 lakhs. On the other hand, median of the first program is Rs. 10 lakhs, while that of the second program is Rs. 2.5 lakhs. I can bet that most people will select the second program. My son is the best in the world, and will certainly grab that Rs. 1 crore job. Never mind that since most parents think like that, the competition would be much tougher in that program. So to guess their first month salary, people often look at the highest salary, and if they are a bit more grounded in reality, then average. Hardly anyone ever asks median and hardly any college will give out median.

In summary, the way things get decided are as follows:

1. The optimization function is maximum earning over the lifetime.
2. Since that is not easy to compute, assume that the first month salary would be strongly correlated with the lifetime earnings.
3. Since first month salary is also difficult to guess, assume that the highest salary offered last year would be offered to your ward (suitably enhanced, of course) 4 years hence.
4. And hence the program which had the highest salary package offered last year is the one we want to join.

If the algorithm is so simple, why are parents confused. Well, there are multiple reasons:
1. Do we really know the placement data. (Usually, No.)
2. What happens if the data for the last two years is contradictory (which is often the case).
3. Continuing with 2, will the relative ordering of programs change over the next four years. (That is, if Program 'A' had one student who was offered 100L, and program 'B' had one student who was offered 99L, is it possible that 4 years from now, there would be one program 'B' student who would have a 100L offer, while the highest offer in Program 'A' would be 99L.) Of course, it will happen.
4. The ordering based on highest salary is different from the ordering based on averages, which is different from the ordering based on median. Which one to adopt, particularly when most of the data in public domain is false anyway.
5. Other unknowns.

There is a simple solution that will take care of all your stress. Read the second paragraph again, and convince yourself that beyond a reasonable salary, you wouldn't worry about placement. Stop being a rat. Become a human. And you would be astonished as to how you will actually be able to think of your interests. You will actually be able to think of what you want out of college. And then choosing a college will become that much easier.

I was talking to a philosophy professor a few days ago and discussing this issue of stress during admission season. He told me that the problem of decision making is that a lot of people want to solve an unsolvable problem, and since the problem is unsolvable, they are stressed because they are investing time and effort without any result. If people start focusing on what is possible they will be able to take better and quicker decisions.

Also, as this excellent article by Hunter Rawlings in Washington Post points out, "The value of a degree depends more on the student’s input than on the college’s curriculum." Hard work is a more important key than the college itself. So if you are confused, just toss a coin.