This project began as a post about a year ago, in which we followed the provenance of the data from a study about misinformation until, at the bottom, we found Media Bias/Fact Check. Upon further examination, it turned out that Media Bias/Fact Check is just some guy, and upon him there rests a surprising amount of academic research. We've since turned this into a study with the following abstract:
Media Bias/Fact Check (MBFC) purports to quantify the bias, credibility, and factuality of reporting for roughly 10,000 media sources, and the resulting data is commonly used in misinformation research. In the present study, we show that MBFC's methodology does not meet basic standards of rigor for academic research. Despite its widespread prevalence, studies using MBFC rarely examine it carefully, often describing it in ways that contradict its "About" page, or treating it as authoritative despite MBFC's disclaimer that it is "not a tested scientific method... [but] a simple guide to the idea of a source's bias." We identified no papers that adequately describe MBFC as the opinions of a single person or critically engage with its methodology in order to justify proceeding with its use. We argue that MBFC's data is not neutral or accurate, but a computationally legible account of hegemony, a specious dataset for uncritical research that mistakes the familiarity of the concepts it quantifies with accuracy. Our study concludes with a call for academic researchers to stop using MBFC. MBFC's data quantifies the results of political processes, including campaigns to discredit the press, and presents them as simple facts about the world, thus reproducing the crisis misinformation scholarship exists to address.
