Why Rankings – Including University Rankings – Are Crap

I developed CoinDesk’s Best Universities for Blockchain rankings in 2020, and have been tortured to some degree by them ever since.

AccessTimeIconSep 26, 2022 at 7:30 p.m. UTC
Updated May 11, 2023 at 4:46 p.m. UTC
AccessTimeIconSep 26, 2022 at 7:30 p.m. UTCUpdated May 11, 2023 at 4:46 p.m. UTCLayer 2
AccessTimeIconSep 26, 2022 at 7:30 p.m. UTCUpdated May 11, 2023 at 4:46 p.m. UTCLayer 2

While rankings (especially university rankings) have non-zero value, they have significantly less value than we, as a society, seem to ascribe to them. What we’ve compiled here is a collection of data (based on arbitrary criteria) that has been assigned a weight (based on arbitrary values). At best this is subjective, and at worst it’s fundamentally flawed. Ultimately, I feel strongly that any rankings generated by humans are much closer to the “fundamentally flawed” end of the spectrum. There are reasons for this that exist on both the universal scale and a more granular scale that is specific to our project.

This post is part of Education Week.

Starting with the macro, most rankings are limited simply because they force an order on things that, inherently, have no order. Blanket statements are rarely correct in all circumstances; we live in a world of edge cases. The common example leveraged in discussions about why university rankings in general should be abolished (a position that may have merit) is that the No. 1 school (say, Oxford, in the 2022 Times Higher Education rankings) ought not be everyone’s number one choice. Individual circumstances, future goals, personality types, etc. all create a complex landscape in which Oxford is not a universally good (or even mediocre) choice. What being ranked No. 1 is really signaling is that someone out there thinks that Oxford possibly (but not certainly) has more of the things they personally value than most other schools. This is an interesting data point that many people internalize as absolute truth.

For a more visceral example, it would be pretty easy for someone to create a ranking of food (or, alternatively, “insert your preferred highly subjective thing here”). If I were the one to create it, my own biases and thoughts would guide these rankings to a degree that I hope people would find unacceptable. I’d gather objective data on things such as likelihood of an allergic reaction, cost, ease of commercial availability, environmental impact and so on. I could quantify these data points into a cute little spreadsheet, run similar normalization and generate an “objective” ranking of the best food. Nobody, with few exceptions, should change their eating preferences based on this list. The underlying data might be useful (perhaps if all you care about is environmental impact, you could change your eating based on that narrow dataset), but it's highly unlikely that your specific values along every dimension of the data will be reflected in the methodology. People intuitively recognize the futility of ranking something as subjective as food. What we fail to recognize is that school rankings are no different.

The same subjective preferences that underpin thoughts about ranking foods ought to be applied to universities: Just as some humans value taste or health or allergies to varying degrees in a food context, different people value different things in a university experience. In fact, I’d go so far as to say, with a high degree of confidence, that the best, most useful rankings aren’t actually rankings at all. Instead, it would be little more than an interface into the data, allowing each person to adjust the methodology to suit their own individual needs. Unfortunately, not only does this not produce any kind of order, it’s also not particularly flashy or newsworthy.

On a micro scale, our rankings are also flawed in their own unique way. We suffer from all of the issues set out above (with the notable exception that here we know exactly whose biases are slipping into the methodology and who to blame, and that person is me, as the originator of and consultant for the CoinDesk Best Universities for Blockchain rankings since 2020), plus some new ones that are unfortunate artifacts of our limited size and scope. While it’s always a bit humbling (bordering on humiliating) to reflect on the ways in which one’s own study/analysis fell short, it’s more important that any reader keep these in mind when reading through the rankings. If nothing else, I hope this drives home some skepticism about rankings as anything more than a rough-hewn guide to what schools are doing something close to what I, personally, think they should be doing to promote blockchain education.

While not diminishing the other shortcomings of our research, there are two metrics that are causing particular concern due to data collection challenges, and I highlight these to point out the kinds of challenges that pervade research of this nature. The first is employment data. In my mind, this represents the largest delta between the relative importance of the data (here, favorable student outcomes) and the ease of gathering said data (for many reasons, not least of all the lack of tracking student outcomes with the required granularity). At the moment, we rely on a combination of school outreach, which receives about zero response rate, and manual data collection. The best way (and main way) we’ve found to get a reasonable proxy for this data is to look at large international companies on LinkedIn and see where their employees have gone to school. Whether you’re concerned about the Western bias of LinkedIn, the subjective nature of choosing companies to scrape, the skewed sample found on LinkedIn or something else entirely … you’re absolutely right. We’ve increasingly lowered the importance of this metric year after year to help combat an overrepresentation of skewed data, and have found increasingly useful proxies for this data so we’re less reliant on LinkedIn. Still, this should give a rational reader pause.

The second metric of concern is our entire qualitative section (another metric that has been rendered less and less important in our methodology every year). To some extent, this one is the most upsetting to me because we reasonably could gather strong data. The problem is a lack of creative problem solving on my part. We put out a very public survey every year to try and gather a robust view of how the public (students, academics, industry stakeholders, etc.) sees different universities and get a sense of which universities are perceived as having the most impact. All of the expected worries are highly valid (CoinDesk is skewed towards the U.S., etc.), but the biggest issue, by far, is that I can’t figure out how to properly incentivize accurate responses.

Every year, we make the same request of survey participants: Please tell us which schools you think are doing the best work in the blockchain space. And every year, the results indicate that everyone is simply sharing the schools that, if they were to be ranked highly, would most benefit the individual responder. Of course, I’m editorializing quite a bit, but the overwhelming majority of responses that we get follow the highly predictable pattern of single-school answers, clumped together by school, within 0-5 hours after a school tweets about our survey. There’s some amount of logic to the idea that students/faculty/alums think their school is doing great work (perhaps that’s why they chose that institution), but the shocking number of responses we get following this pattern, occasionally even suggesting that a school that has no discernable blockchain activity is the single best school in the space, implies to me that perhaps we are not incentivizing the right thing. In the future, I hope to experiment with other mechanics (Keynesian beauty contests, etc.), but for now, I consider this data to be highly skewed. What we’ve incentivized to date seems to be a popularity contest, not objective analysis.

That said, these problems plague rankings of all kinds. I don’t share these facts to diminish the value of our rankings, because I still believe that, properly contextualized, some combination of the rankings and the underlying data have at least some value. If not properly contextualized, however, they are crap. Ultimately, perhaps the most helpful thing we can do is remain committed to being open and transparent with our methodology, our data, our processes and, most importantly, our failures. Our rankings are not perfect; all we can do is improve upon them year after year and caution people to apply a reasonable amount of context when evaluating the list. Please reach out if you’d like to talk about these rankings, how they can be improved, or why they should be abolished.


Learn more about Consensus 2024, CoinDesk's longest-running and most influential event that brings together all sides of crypto, blockchain and Web3. Head to consensus.coindesk.com to register and buy your pass now.


Disclosure

Please note that our privacy policy, terms of use, cookies, and do not sell my personal information has been updated.

CoinDesk is an award-winning media outlet that covers the cryptocurrency industry. Its journalists abide by a strict set of editorial policies. In November 2023, CoinDesk was acquired by the Bullish group, owner of Bullish, a regulated, digital assets exchange. The Bullish group is majority-owned by Block.one; both companies have interests in a variety of blockchain and digital asset businesses and significant holdings of digital assets, including bitcoin. CoinDesk operates as an independent subsidiary with an editorial committee to protect journalistic independence. CoinDesk employees, including journalists, may receive options in the Bullish group as part of their compensation.

Author placeholder image

Lecturer/researcher, Stanford University/MIT