How Do We Measure the Quality of a College or University?

That's an interesting question. Clearly there are lot's of metrics we use to gauge the performance of institutions of higher ed. As this Kevin Carey post inThe Quick and the Ed blog points out, one of the most common measures of quality is the SAT scores of incoming freshmen - a criterion that wouldn't even apply to open-enrollment colleges such as mine. The post elaborates on the flaws inherent in the SAT metric - most glaringly, how does a college get credit for a test score from a students junior year in high school? So the question becomes - "How do you actually measure the performance of a college?" Although not a definitive answer, Carey argues that perhaps growth, as measured by freshmen versus senior differentials on standardized tests, such as the Collegiate Learning Assessment, would serve as a better indicator of a college's performance.

It's interesting that I see this same dynamic in my classroom. Of course there are the students that will ace every assignment, quiz and test - the perennial A-students. For other students, one of the indicators I am always looking for is growth - students who may not have started with the best test scores, but keep working hard and show that growth or improvement over the course of the semester. Sometimes it is working with these students that provides a faculty member with the greatest satisfaction.

Colleges and universities distinguish themselves from one another in lots of different ways-- scholarly reknown, the size of the endowment, success on the athletic fields, etc. But the most commonly-used measure is probably the 'quality' of the freshman class, as measured by standardized tests like the SAT and ACT. Average incoming SAT scores at University of Texas campuses, for example, look like this:

The Austin and Dallas campuses are getting students at 1200 and above while the non-selective regional campuses like Pan American and Permian Basin are below 1000. This conforms with nearly any measure of prestige and status one could name: Austin is an internationally known, Research I, AAU institution with a multi-billion dollar endowment and a football team that was lucky enough to beat my Ohio State Buckeyes in the Fiesta Bowl last month, not that I'm bitter. (Although: 'Colt McCoy'? Really?) Permian Basin has none of these things, and probably never will.

But SAT scores leave the question of college student learning unanswered. It's odd, the way we give colleges credit for how their student did on a test they took while they were juniors in high school. Colleges argue that high SAT scores are an implicit quality signal because they reflect high demand, but the demand may just be for the prestige and the football team and the nice facilities and the chance to hang around with other students who also have high SAT scores. To really get a handle on learning, it makes more sense to test a sample of freshmen and a sample of seniors, and see how they compare. And in fact the University of Texas system has done exactly that, using the Collegiate Learning Assessment. Here's what they found:

Each block on the graph shows two data points: freshman and senior scores on the CLA. As you'd expect, freshman scores correspond fairly closely with SAT scores: Austin and Dallas have the highest, regionals like Permian Basin the lowest, and the rest are in between. Much more interesting is growth. While Austin students arrive at high levels, they don't seem to improve very much while they're in college--the difference of 53 points is less than half the national average of 111 points. This may because of some sort of 'ceiling effect,' or it may be that elite universities don't focus much on improving students who arrive in great shape to begin with. Pan American and Permian Basin have very similar freshman scores, but Permian Basin's growth is more the double that of UTPA -- 197 to 90, bringing students from well below the national average on entry to above it on completion.