Understanding Educational Outcomes

In an earlier post, I had talked about the increasing emphasis on "outcomes" in several administrative processes-- especially in education. The focus on outcomes is inherently a sensible move-- as compared to some previously existing models that (for example), measured the effectiveness of education by numbers of graduates, or literacy by whether someone can sign their names. 

However, the strategy and implementation of this focus on outcomes leaves a lot of gaping holes and major concerns. Because a flawed notion of outcomes is tightly tied to appraisals and survivability in academia and industry, it has largely become yet another exercise in compliance. Like any compliance game, it is common to see the emergence of an ecosystem of "outcomes engineering", which promise  "guaranteed outcomes" if their product or methodology is used, or one becomes part of a cartel. The idea of guaranteed outcomes is rather strange. If someone guarantees the outcome of a cricket match for example, it would be a strong indication of some kind of match-fixing. 

If we ask anyone from India to quote a saying from the Bhagawad Gita, and they will most likely quote verse 2.47 that effectively says that: "we can control our actions-- but not its outcomes."

कर्मण्येवाधिकारस्ते मा फलेषु कदाचन |
मा कर्मफलहेतुर्भूर्मा ते सङ्गोऽस्त्वकर्मणि 

And then, the verse goes on to say that, let this not be a deterrent against doing what we need to do. 

This is one of the fundamental axioms of outcomes-- that the outcomes of our actions are not in our hands. The outcomes and impact of our actions is mostly a factor of external forces, rather than our own actions.  Consistent with this axiom, recently, Sundar Pichai of Google had remarked, "Reward effort, not outcomes." 

In the previous post, I had also described a common fallacy-- that of confusing outcomes with outputs. What is observable and visible, are outputs-- not outcomes. Outcomes refer to the change of state of the entity of interest-- which would be latent. Outputs are used to reason or speculate about possible outcomes that would have caused this output. They are not the outcomes themselves. 

For instance, a consider a car manufacturing company. Its outputs are described in terms of the number of cars it can produce, the quality of their cars, the mileage given by their cars, etc. Let. us say that a company has produced superlative outputs. These outputs could be a result of several possible outcomes that changed the way in which the company operates. Maybe the increased output was a result of better communication and coordination in the company, due to recent policy changes. Maybe the increased output was a result of the company embracing more automation. Maybe the increased output is a result of greater ownership by the company's employees, who took it upon themselves to make the company shine. Or maybe the increased output was a result of greater insecurity among employees in the company-- where most of them have been made to work with a sword dangling above their heads.  

While the outputs are observable and quantifiable, the outcomes (like "ownership") are often latent and hard to quantify. 

In order to design for outcomes, we need to first answer the question, what is our entity of interest? Who are we working for, and whose changes in state is it that we would like to influence? 

When it comes to curriculum design, our entity of interest is the student. We define course and program outcomes in terms of what the students should be able to quantifiably demonstrate. This in turn, places the entire emphasis of the pedagogy on the examination, rather than on the learning and empowerment of the student. The fallacy here is because we confuse outputs with outcomes. 

In order to understand student outcomes better, consider the figure below: 

This figure shows what is called as a "Learning map" for one of my courses. Each asterisk in the figure represents a student, and each blue triangle represents a learning resource. Each dotted line represents a "topic" or "competency" covered in this course. The map is semantically organised using AI algorithms to depict topical proximity between objects. Pursuing a dotted line (topic) in isolation tells us where we will end up in this semantic space had we pursued only this topic. 

The solid, magenta line shows the topical journey of the lectures I have covered in the course. In this course, in addition to passing conventional exams, students were required to make a few contributions of their own. This could be in the form of essays or term papers or seminar talks. The student contributions are also embedded in this space, which are shown by yellow triangles. 

The "proficiency" of each student is now computed by taking the centroid of all of their contributions (and the grade they received for each contribution). Based on this proficiency, students are also embedded in the learning map to show where they stand in the class. 

As is evident here, there is a huge amount of diversity in student locations based on their proficiency after the class. Very few students are actually close to the magenta solid line, which was the learning pathway adopted as part of the lectures. 

Even though each student was subject to the same set of lectures, each student has their own individual footprint in the way they have assimilated the knowledge presented in the class! This is the competency state of the student, and the way this state changes is what is the outcome of the course. 

But what the students are assessed upon is how they fare on the magenta line. We assess students based on what we covered in class. If we project each student's location onto the magenta line, we see a gross distortion. For instance, there is a student in the top-left of the map, who has shown enormous interest and proficiency in the first topic, but not so much in the others. This student would have likely fared badly in terms of the assessments based on the lectures covered in the class. However, one can well argue that this enormous inclination shown by this student to one of the topics covered in the course, has its own benefits for the student (and for the industry that the student works in, and for the society at large), all of which is lost, when we grade them only on the output they produce in our assessments. 

Yet another important characteristic about outcomes is that, regardless of what is our entity of interest, we need to realise that it is a multi-dimensional system of being, rather than a single-dimensional object. Our entity of interest can change its state (also called "move the needle") only in a holistic fashion, involving all dimensions. A student who produces superlative outputs in exams, may actually be suffering from an internal implosion of spirit. An star employee may actually be suffering from a dysfunctional family. As the tennis star Martina Navratilova once remarked: "The moment of victory is too short-lived to live only for that, and nothing else." 

We need to remember that, even with technologies like the Learning Map above, we can only observe a part of their state. We don't know what exactly made one of the students to prefer just one topic so strongly. Was that topic in some way personally appealing to them, or was it because they thought that this topic has a lot of "scope" in the industry, or maybe it was some other reason altogether? 

Hence, we need to be very careful when we make statements about outcomes. Firstly, we need to stop equating outcomes with outputs. Next, we need to become aware that the change in state that we see in our entity of interest need not be the complete state description of the entity. And third, we need to remember the Bhagawad Gita-- that we really cannot control outcomes. At the end of the day, we need to just do our best work, and hope for the best.