Search This Blog

Loading...

Friday, August 26, 2016

Normalization of Admission Tests

So NEET is back in Supreme Court, this time a candidate asking for normalization between NEET I and NEET II. Here is a news item:

NEET I and II combined results challenged in Supreme Court

First of all, why is it that no one challenged JEE. They too have multiple papers, and by no means the papers are of equal difficulty levels. The JEE Mains has not been challenged, because someone who gave a slightly tougher exam but still still got shortlisted for Advanced JEE, does not care. And someone who didn't get shortlisted for the Advanced JEE, knows that even if s/he was shortlisted, the chances of getting through to IITs was remote. And frankly, people don't quite care for admission to NITs as much as they care for admission to IITs and for admission to MBBS programs.

I recall a meeting in CBSE regarding JEE. Before the meeting started, out of curiosity, I asked the officer handling JEE Mains whether the averages of the paper version and the computer version were same. Without blinking an eyelid, he said, yes they are identical. I kept quiet.

At the end of the meeting, the CBSE Chairperson was wondering why so few candidates give online JEE, even though there is significant benefit in giving online JEE. The paper exam is just after your board exams are over, while the online exam gives you additional time to revise everything. And the conversation went something like this:

Me: Do you recall the question I asked before the meeting about the averages of the two papers.
CBSE Chairperson: Yes.
Me: I can guarantee that the officer was lying. He has never seen the two averages.
CBSE: How can you be so sure.
Me: He has a laptop in front of him. Ask him to check the information now, and let us know the exact numbers.
JEE Officer: Sorry, Sir. I don't know the averages.
CBSE: (Now very curious), But how could you guess?
Me: Averages of lakhs of students can be close but not identical. If he had seen the numbers and they were indeed very close, he would have said that they are within 1% of each other, or something like that.
CBSE: But what does this have to do with students not taking up online exam.
Me: The perception is that the online exam is much tougher. Your processes are very opaque and you don't give out any statistics. So you do nothing to tell people that either the exam is not much tougher, or yes, it is tougher, but then we normalize.

At this point, the officer who handled JEE Mains exam informed us that the way they handle the differences between the two exam is that they ask the group who prepares the question papers to ensure that the two papers are of equal difficulty levels, and we get a certificate signed by them that it is indeed so.

This was, frankly, very shocking. Think about it. There are persons out there who are willing to vouch in writing that the two question papers are of exact same difficulty level without any data, and just on the basis of their experience. I don't know who are these persons with no understanding of testing, but I can only say that these people don't deserve to be anywhere close to that question paper. And for CBSE to involve such persons in preparing the most important question paper for over 10 lakh students shows how CBSE lacks competence about testing. And if, doing this was expedient in the beginning, wouldn't you at least look at data later on. And if the data miraculously pointed to similar distribution of marks, make that information public so that the public perception about your processes become more positive. But once the exam is over, everything about it is best forgotten.

I am extremely happy that someone is challenging the lack of normalization in NEET. I only hope that there is no miracle here and NEET I and NEET II indeed have different distributions of marks. Otherwise, we will be strengthening the argument that it is possible for a man to look at two question papers and guarantee same distribution of marks.


Sunday, August 21, 2016

IITs as accreditators?

We have problems in the country, lots of them. But we also have solutions in the country. And thanks to successive governments' indulgence, lots of them too, 23 of them at the last count. And there is a promise that if these 23 can not solve all problems of India, we will create more of them.

One of the problems is the poor quality of higher education. It was hoped not too long ago that India will reap the immense benefits of a young population, the demographic dividend, as they said. But without a good quality education, that dividend seems to be becoming a liability. One of the ways of improving quality is to insist on accreditation by a bunch of supposedly quality conscious individuals following some guidelines and checking things in each college. NAAC was born out of this requirement. But NAAC has not been able to grow fast enough. It is said that at the current rate of accreditation, they will take 38 years to accredit only the existing institutions, and in these 38 years there will be many new ones. So in the foreseeable future there is no hope of making accreditation compulsory for all.

The new HRD Minister is quick to figure out that we need multiple accreditation bodies. This is, not a new revelation, of course. A lot of people have been saying that we need accreditation agencies in the private sector. But when it comes to education, private sector is bad, not to be touched with a 10 feet pole, never mind that most of our students are studying in private institutions and that many of these are superior to most government institutions. So the minister has been advised that perhaps IITs could be the accreditation agencies.

Here are the links to some of the news reports regarding this:

Centre plans 10-20 more accreditation agencies besides NAAC, says HRD Minister

Prakash Javadekar for giving IITs and IIMs accreditation body status

Do IITs have the bandwidth. Will this not be a distraction. Remember, compulsory accreditation once in 5 years for 38,000 institutions means 7600 accreditations per year, and if 20 organizations are involved in accreditation, then each of these organizations on an average will have to accredit 380 institutions in a year. That is a huge workload in addition to teaching and research.
 
I am sure some IITs will see a huge business opportunity in this. We can hire an agency to do this job under some bit of supervision from our side. Charge heavily so that a decent profit is made which can improve the quality of education at IITs further. Basically, a transfer of wealth from poor institutes to richer institutes.

But there is a more fundamental question. Should someone who has scant regard for the accreditation process be doing it. Today, accreditation is no longer based on inputs. You don't just look at the infrastructure, number of faculty, etc. You look at what are the vision, mission statements of the institute. What are the goals of each academic program. What is the outcome of each course in that program, and how it contributes towards the overall goal of that program. Having looked at all of it, one then asks questions regarding, how do we know that these outcomes are being attained. We also look at the whether there is a process for continuous improvement, looking at those outcomes and goals and discussing whether they remain relevant, etc.

I can't say about all IITs, but I can confidently say that IIT Kanpur does not do anything of this sort. We do not have a vision statement even 56 years after we were set up. We don't have goals for our programs, we don't have outcomes for our courses, etc. We are perhaps a unique institution which says that while there will be a feedback of all courses every semester, that feedback can not be used for any administrative decision. If you look at the way proposals are drafted, they invariably would not talk about options and what are the pluses and minuses of those options. The way minutes of the meetings are written, one could be forgiven to think that our language of instruction is not English. Overall, they have complete contempt for this process, which they believe is too bureaucratic, and is designed to generate employment. And that is fine. They still have done wonderfully well in academics. They still teach well, and they still do good research. They may fail the accreditation test, but if they don't want to go through accreditation (as long as it is optional), that is their call. And the market has already decided whether they need accreditation or not.

But while, it is alright if IIT Kanpur does not follow what Washington Accord suggests as best practices, or what NAAC will come and check, but it is not alright for such an institution to inspect other institutions and demand that they follow the same best practices which we have contempt for. And I suspect that it is not just about IIT Kanpur, but a lot of IITs are in the same boat.

So while I appreciate that the Minister has identified a very serious shortcoming of our educational system, but asking IITs and IIMs to solve that problem is not correct. Particularly when it is obvious that IITs can not afford to do this with their own faculty, and can at best provide a very slight supervisory role with an outsourced agency. The better thing to do will be to seek private participation in this.

Saturday, July 23, 2016

Why do Seats remain vacant?

Every year, a few weeks after the end of the admission season, there will be a report on vacant seats in IITs, NITs, etc. A fairly large number will be mentioned in the report. And we will be told that this happens because certain disciplines are not at all popular, and should perhaps be closed down, and may be those many seats should be increased in the more popular disciplines.

Indeed, there is no doubt that some disciplines are fancied by the herd of sheeps, assumed to bring in the riches. But is that the only reason. Are there really no students interested in studying those disciplines. In particular, given that a significant number of graduates don't go for core jobs, but instead go for MBA, finance jobs, IT jobs, and other such careers, how come such students are not taking admission in these "branded" colleges.

The reason for vacant seats, unfortunately, is not the lack of demand, but the stupidity of the admission process.

It is a common belief among the public as well as academic administrators that if we do a large number of rounds for admission, all or most seats will be filled. So we have seen our admission process go from 1 round to 2 to 3 and this year to 6 rounds for the admission to NITs and IITs. It is also believed that if we do joint counseling of larger and larger number of institutions, the number of vacant seats will go down substantially. Of course, both joint counseling and larger number of rounds would help a bit, but they can not solve the problem on their own.

If we look at the statistics of the 6 rounds, we notice the following: After the first round, there were 6490 vacancies. That means that so many people did not accept the admission offers. This is pretty reasonable given that there were almost 35,000 seats and many among those 35,000 would prefer places like BITS, IIIT-Delhi, and so on. The next round filled up these 6490 seats (well, barring a few that remained vacant. I am ignoring them since they are too small and don't change the main argument of this post). However, at the end of the second round, there were 2716 vacancies. Again, pretty reasonable given that they must have been offered unpopular programs and many would have preferred colleges outside JOSAA and not accepted admission. At the end of the 3rd round, the number of vacant seats come down to 1837, which is consistent with the argument made above, and is telling us that if we just keep doing a few more rounds, we would be able to find students for most vacant seats.

However, the story changes after this. At the end of the 4th round, the number of vacancies actually increase to 2021. Why did this happen. Well, some (or many) of those 1837 did not accept the admissions offered to them. But now, even those who had accepted admissions earlier, have started withdrawing. And this withdrawal becomes a serious business during the 5th round. At the end of 5th round, the number of vacancies is 4094.

A large number of students have withdrawn during the 5th round. Why did they not withdraw earlier if they were unhappy with their admission offer. Well, there was no incentive for them to withdraw, so might as well hope against hope and see what they can get in the 5th round (and what they can get in other colleges outside JOSAA). They withdrew during the 5th round since they were told that if they did not withdraw, their entire payment may be forfeited. (Some dd not withdraw even then, since they know that in the future MHRD will come to their aid and ask the institutes to return the money).

The government insists that there can be no financial penalty for withdrawal till the beginning of the semester. So we need to have all but the last round before the deadline of withdrawal, and only one round after the last date for withdrawal.

Unless people, who are not going to join, withdraw, we don't admit more students.
There is no incentive for people to take early decisions and withdraw as soon as they have multiple options.
But most importantly, and this is something that is often ignored by our academic administrators, even when someone withdraws, and there is a vacancy, we are filling that vacancy by someone who showed willingness to accept that admission more than a month ago, and who, in the last one month, has probably got many other admission offers. But since the counseling group does not know who is still interested one month later, they end up making offers to next in the queue who are not interested.

So, if consider the JOSAA admission process, the 6th round has filled up about 4000 vacancies, but if we take the survey of all institutes who have got admission through JOSAA, some time in August, I would not be surprised if there are still 5000 vacancies, which will not be filled.

Is there any way that we could have offered admission to more students. Of course, yes. But not by more rounds. We will have to solve problem by looking at the genesis of the problem.

So the problems and potential solutions are:

1. We admit students only after some people withdraw. Why can't we admit more students then the so-called number of seats. We have data for many years and we know roughly how many people will not accept offers. Based on historical data, we can always admit more students. We can be a bit conservative not to get into a situation where we have more students than what we can handle. But let us face it. If in a particular year, we do get a few extra students, heavens are not going to fall.

2. There is no incentive for people to withdraw early. This is a huge problem and a lesson. When you try to be populist and do things which common people will consider as friendly to them, you will actually end up doing things which hurt common people. Allowing students withdrawal till the last minute without any penalty will result in unfilled seats. And thousands of students not getting admission hurts more than a few thousand rupees loss to a few people. The right thing to do will be to declare that free withdrawal can only be done till X days before the semester and after that deadline, every day, there will be an additional deduction of the money if a student withdraws. This will ensure that people withdraw early and we are able to conduct not just one round but multiple rounds after people have started withdrawing.

3. We don't know who all are still interested in the programs one month after they filled in the choices. This is really the biggest problem. Currently, the way we fill up seats is by way of a "Spot" round (which is not happening in JOSAA this year). In this round, people have to apply afresh. So those who are no longer interested are out of the system. And invariably they have to be at a location physically and give a significant amount of money within minutes of getting admission offer and since it is being done through physical attendance, if someone does not want admission, the next person is offered the same.

Spot round has its own problems, of course. Traveling on short notice is not easy and airlines make a lot of money in this season. (Here you go, the money that you saved through full refund in one college, you paid to the airline. So you really did not save much.) Everyone has spot round in the last week of July or 1st few days of August. So there isn't much option regarding traveling. Invariably, spot round happens after the semester has started. So the students are joining late, have missed out on the orientation program, have missed out on the first assignments, first lab, introductory lectures, etc.

Another way to solve this problem will be to have an incentive for students to withdraw from the counseling process. Again, you don't have any other handle on the students except a bit of financial handle. So if JOSAA (and other similar counseling processes) could ask for more money after the first couple of rounds, which will be refunded progressive less as the days pass by. Also, the student should be able to delete options that s/he has filled in. So you could ask them to deposit Rs. 10,000 after the second round, and then say that if the student is not offered admission in any of his/her choices, the entire Rs. 10,000 will be refunded. On the other hand, if the student is indeed offered admission, and s/he decides not to accept it, then the refund will be based on how delayed the withdrawal has been. This will ensure that as soon as I get an offer from another good college, I go to JOSAA website, and remove some of my lower preferences which I would not want to accept in comparison with the offer that I have received. This makes the counseling process much more efficient and reduces the stress levels in the system tremendously. Coupled with a few extra admissions, this can really revolutionize the admission process and make it absolutely smooth. But it will require a small financial penalty for late decision making by students and parents.

To conclude, the seats remain vacant not because there is no one to accept those seats, but because we have a brain dead (but populist) admission process which can not identify the students who may be interested in those seats.