Wednesday, February 03, 2010

How are we doing?

It used to be so easy to measure achievement. In the days when most industries involved making widgets of one sort or another, or selling those widgets to people, you could tell that you'd made progress if your widget production numbers or your sales went up. But how do you measure the impact of your work when your product is knowledge?

That's something that my organization is just beginning to address in a systematic way. Of course there have always been measures like publication sales and meeting attendance, but those aren't really true measure of the impact of the work we do. Just as in libraries measures of circulation statistics and headcounts don't tell you whether or not the people coming through the door are really finding value in your services. I used to work in higher education and there too the concept of meaningful evaluation (beyond standardized testing) is starting to become more widespread.

So given that the product of a think tank is the thinking, how do we go about measuring the impacts of the work and using those measurements to improve for the future? One approach that my VP seems enthusiastic about is collection of citation information for our publications, on the theory that if others are citing what we write then it must be having an impact in the broader community of practice. The problem with this approach is that it is extremely time consuming, and not particularly useful for improving future work. The number of citations a book or report gets can depend on things like the flow of current events related to the topic, which we don't control.

As this evaluation project moves forward, my particular concern is finding a way to capture the results so that they can be used in a cycle of assessment and improvement. We don't have time to be reinventing the wheel for every new project that comes along.

Photo credit: Segozyme

No comments:

Post a Comment