Enhancing the Quality of Education

By Henry Kelly, Senior Scientist, Industry Partnership Leader, Midas

Henry Kelly, Senior Scientist, Industry Partnership Leader, Midas

The modern economy is rapidly reshaping the intellectual and social skills demanded by employers. Information technology offers a rich set of tools for measuring and building these skills but traditional institutions have been remarkably slow to use them. They are in danger of finding themselves outflanked by a growing and energetic set of organizations who argue that they can provide measures of competence far superior to college transcripts and can offer more efficient, flexible, and personalized methods for building competence. Many companies are discovering that traditional degrees may not be a good measure of whether a person will succeed. Only a third of employers ask recent college graduates for their transcripts.

"One of the most thorough studies of non-degree credentials concluded that it is: ‘crowded, chaotic, and confusing"

A core problem is that the market forces that guided the technology-driven transitions in other parts of the economy are all but non-existent in education. This is, in part, because there is no agreed metric of quality that can provide a basis for competition. Without strong market incentives, it is difficult to justify the kinds of talented, multi-disciplinary teams needed to build, operate, and continuously improve the tools of a modern service enterprise. Most post-secondary institutions cling ferociously to an inflexible business model: the assumption that education is a cottage industry remains unchallenged in most parts of education–a solo-instructor, master of the classroom, and sole definer of what a student needs to know. Technology is allowed only when it serves to perpetuate this norm.

What is Post-Secondary Education

The changes underway in post-secondary education are difficult to follow since the data available to track them is astonishingly incomplete. Statistics are available for the $400 billion is spent at degree-granting colleges and universities but very little is known about the nearly $700 billion spent other post-secondary institutions (certificate and licensing programs, for example). One of the most thorough studies of non-degree credentials concluded that it is: “crowded, chaotic, and confusing”. Major changes are underway even in traditional degree-granting institutions. Nearly half of their students are more than 24 years old and many are not interested in obtaining a degree. But again, data on what is actually being taught is uneven. We know how many people get degrees in chemistry, for example, but don’t have a clear idea of how many people are taking Chemistry 101 in any year since there’s no consistent definition. And there’s no clear way to determine changes in quality (though some data suggests that the overall quality is in decline).

Defining and Measuring Competence

New tools of data analytics may change the game. A critical first step is to define a measure of success. Recent thinking about credentials suggests that we should focus on measures of output (demonstrated competence) instead of inputs (semester-hours) but this depends on finding ways to define and measure competence. There have been many recent efforts at definitions but all rely on the opinions of experts—“I know it when I see it”. Even when biases can be removed (it’s essential that everyone know what I teach in my class), there is good reason to believe that experts are often not able to explain their expertise. In a recent experiment, for example, experts were asked to write detailed description of how to assemble a laser, but their detailed description didn’t include critical steps which meant that non-experts weren’t able to perform the needed assembly. In another case a video of experts debugging a computer system was compared with the experts’ description of what they did. The expert descriptions missed 53 percent of the problem-solving steps.

A variety of new tools in “people analytics” can help us beyond opinion. Most jobs leave behind an extensive digital trail, giving companies an ability to use machine learning and other methods to explore the difference between superb and mediocre performers. Firms like Gild are using these tools to search for employees with skills best matched to available jobs, and many businesses are developing their own set of tests for applicants including “prior learning assessment” tools. Not surprisingly, this has also generated a business opportunity for companies like Jobscan that help applicants get higher rankings. A recent McKinsey study found that these new systems identified people from unexpected backgrounds who were highly successful hires. These tools now work well in well-defined skill areas like flying an airplane or programming but are increasingly being used to define competence in a wider market including surgery and nursing.

Building Competence

Given some agreement on how to define competencies, it should be easier to create a more efficient market for the many learning technology products now entering the market–particularly the highly innovative tools that go beyond simply mimicking standard “cottage industry” methods of instruction. Simulations, for example, make it possible to expose large numbers of people to complex challenges and measure their performance in practical ways. They even provide a way to explore the individual’s “soft skills” such as working well in teams and communicate effectively. Data analytics can compare novice performances with expert performances and provide detailed recommendations.

Even if a mechanism to define and measure competence is developed, innovative analytic tools would be needed to evaluate innovative approaches to learning. Unfortunately, most studies of the effect of educational interventions have involved very small samples, but progress is clearly possible. Udacity and the Kahn academy which provide a rich data trail on all their students are beginning to use sophisticated A-B testing and other tools to continuously improve their services.

Where Do we Go from Here?

There are unmissable clear signs that the landscape of post-secondary education is changing rapidly, and new tools are needed to define, measure, and build the complex set of competencies needed by modern employees. A century and a quarter ago the Carnegie brought some order to the chaotic collection of degree programs by with clearly defined set of metrics such as the credit hour, semesters, 4 year degree programs. They’ve had a good run, but it’s time for a new look.