Measuring what matters: a convening

By:

Design

All of us involved in social design (and I include education in among those as well) ask ourselves, or are asked this question:

How do we measure the impact of the work we do?

This begs the question, why measure in the first place? Lord Kelvin, one of the greatest physicists of the 19th century, provided the canonical answer to this question back in 1883, saying:

…When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.

William Thompson Kelvin
Popular Lectures, Vol. I, “Electrical Units of Measurement,” p. 73, 1883

In other words, according to Lord Kelvin “to measure is to know” (which as it happens is another of his oft-quoted remarks).

This emphasis on measurement can be seen in all aspects of our lives. We measure our children’s heights, and our GDP; we measure learning gains in students, and recidivism rates of inmates released from prison; we measure the number of steps we walk and the calories we eat.

That said, we also know that there are things, important things, that we have not yet managed to measure, or do not know how to. These are often intangible, ineffable—and yet, important.

How do we measure if students are seeing beauty in mathematics? Or the economic value of public art? Or the capacity of music to move us?

These issues are of concern for all of us engaged in social design (i.e. the application of design methods to tackle complex human problems). In a recent blog post, prompted by the COVID19 crisis and global school closures, I asked the question about the value of school (in two parts, One & Two)– and argued that measures that focus on one dimension (that of learning, via standardized tests) miss all the other values that schools bring to our lives.

The issue, of course, is that if we can’t measure the impact of our work how can we know if what we are doing is working or not? The issue is not whether we should measure but rather if we are measuring what we ought to be measuring. Often, we are limited to measuring what we can easily measure rather than what we ought to measure. Alternatively:

Do we measure what we value or do we merely value what we can measure?

These are not new questions, but they are important ones, and worth revisiting.

the words

Recently we organized a virtual convening of scholars and leaders from across ASU on this very topic. Titled: The Importance of the Ineffable: Measuring What Matters,the convening was organized by the Office of Scholarship and Innovation, the Principled Innovation Team and InnovationSpace. Participants came from a range of units within and outside of ASU, bringing their disciplinary expertise as well as their deep expertise into the conversation. (Complete list of colleges and units that participated given at the end.)

This event had initially been planned as being a face-to-face convening, till COVID19 got in the way. What this shift to an online convening meant that we had to get creative to ensure that the different approaches and perspectives could get properly and thoughtfully addressed and some synthesis reached.

To us that meant that we could not (and should not) just drop people into a zoom room and hope for good things would happen.

Jennifer Stein and Henry Borges put in some significant effort to design a genuinely powerful experience – with additional support from Clarin Collins, Ben Scragg, and Lok-Sze Wong. We moved seamlessly from whole group to smaller teams and back to whole group, toggling between Zoom and Google Slides and Google Draw, sharing ideas, collapsing them in to broad categories of questions and then digging deeper into them.

Screenshot from virtual card-sorting exercise.
Screenshot from virtual card-sorting exercise. Names have been deleted.

A range of topics were generated and were virtually categorized and then people self-selected to discuss each of these overarching topics further.

Overall it was an exciting and intellectually challenging event, providing us with lots to chew on and we continue to explore these ideas in actual projects in the future.

Note: Participants included people from the following units or organizations: The School for the Future of Innovation in Society, School of Earth and Space Exploration, College of Public Service and Community Solutions, School of Computing, Informatics, and Decision Systems Engineering, Herberger Institute for Design and the Arts, School of Mathematical and Statistical Sciences, Mary Lou Fulton Teachers College, and the Kern Family Foundation.

End note:

Finally, for two slightly different, and idiosyncratic yet relevant, takes on the whole issue of measurement see these two blog posts from a few years ago: