Is FiveThirtyEight Reliable?

FiveThirtyEight is a media source known for its analysis and creation of political polls. Pew research found that polling numbers are still fairly accurate for the American general public in 2021. They demonstrated this by measuring their poll responses comparatively to the actual figures of their questions. The greatest disparity found, regardless of low participation, was only 11% off from the benchmark.

Pew Research

While FiveThirtyEight does rely on these polls which have recently been proven accurate. It does not signify that everything FiveThirtyEight claims should be taken at face value. Recognizing that while the accuracy of source material and articles is important, it is valuable to recognize that more elements are considered when determining the reliability of a source. We will discuss the reliability of FiveThirtyEight as a whole and how reliability is determined in this article.

Does Reliability Matter?

Reliability, in general, refers to how trustworthy or accurate information, or in this case, a news source is. If we consider this definition, it quickly becomes clear why reliability is important in media sources. If we can’t trust the things we read then there isn’t much of a point in continuing to consume content from that source, after all. So how exactly can we gauge the reliability of a news source anyway?

There are several potential measures of reliability to look out for when trying to determine whether a media source is reliable or not. Red flags for an unreliable article can include the presence of wild unsubstantiated claims, facts dependent on other unreliable sources, heavy use of opinionated language, and more. Some indicators of a reliable news source, on the other hand, include things like:

Absence of subjective/opinionated language in articles
Credible sources cited (e.g., neutral sources, .gov, .edu websites)
Facts and statistics backed by multiple relevant outside sources
Use of primary sources when possible (e.g., interviews, quotes)
Information that remains consistent across news sources

How Does FiveThirtyEight’s Reliability Perform?

The political reliability index developed by Biasly objectively assesses news organizations’ accuracy and trustworthiness. FiveThirtyEight’s overall Reliability Score has been rated as ‘Good’ by Biasly. This rating is a weighted average of two distinct scores: the Fact Analysis Score and the Source Analysis Score, each evaluating separate components of FiveThirtyEight’s Reliability. When computing the Average Reliability of the article the Fact Analysis score is more heavily weighted. These ratings are as follows in the next two paragraphs:

FiveThirtyEight’s Fact Analysis Score is ‘Excellent,’ which suggests readers can trust the majority of FiveThirtyEight’s content online. The Fact Analysis score focuses more on the accuracy of claims, facts, and sources presented in the article and any hints of selection and omission bias, which we will discuss further in the article.

Buzzfeed’s Source Analysis Score is ‘Fair,’ which suggests readers can trust some of the sources, links, and quotes provided by the news source. This score, which is based on A.I., focuses on assessing the quality of sources and quotes used including their number, lengths, uniqueness, and diversity.

This score indicates the average rating between articles. Our analysts found that while the articles most frequently use very accurate data to support their claims, the data presented is often one-sided favoring liberal politics. Regardless, FiveThirtyEight can be considered mostly reliable for the majority of its publications. When considering the reliability of any given article it is important to remember that there is variation depending on different elements, including the author, background, and so forth.

FiveThirtyEight’s Accuracy and Reliability

Consider the following, a news source, upon researching a particular topic, may find information that contradicts the argument they are trying to establish. This source may elect to ignore this evidence or subconsciously neglect it simply because they are looking for information supporting their bias. FiveThirtyEight has historically been found to support liberal ideologies more frequently in the past, despite their mission for generally objective poll reporting. Therefore, it is important to consider selection and omission bias in not just content developed by FiveThirtyEight, but all media organizations.

Selection bias is when stories and facts are selected or deselected, often on ideological grounds, to create a narrative in support of the new sources’ ideology. Omission bias, on the other hand, is when different opinions and political views regarding a situation are left out so that the reader is only exposed to the ideological perspective supported by the author. It’s important to keep in mind these two types of biases when trying to assess an article’s level of accuracy.

Biasly uses an average percentage of the articles’ accuracies typical of the publisher. 0% indicates inaccuracies while 100% indicates the highest possible score. Factors that contribute to the score include the amount of sources used, the quality of such sources, and the quantity of external sources. Biasly’s rating for FiveThirtyEight can be found here, with other information on our website in regards to particular articles and their unique bias and accuracy ratings.

FiveThirtyEight and liberal-leaning sources are not uniquely reliable or unreliable. For example, Forbes published an article titled “How Did Trump Secure More Than $250 Million In Bonds? Here’s What We Know.” which is considered “extremely conservative.” However, it has an “excellent” reliability score for its diversity of high-quality sources and quotes. In contrast, “Biden and Xi discuss Taiwan, AI, and fentanyl in a push to return to regular leader talks”, published by DailyMail, has a bias rating of “Moderately Conservative” but only a reliability score of “good.” Therefore, political bias does not necessarily indicate a positive or negative reliability but depends on alternate factors.

An example of this in a FiveThirtyEight article is seen in their publication “In A Unanimous Decision, The Supreme Court Delivered Another Victory For The Religious Right.” This is a piece that is considered “Somewhat Conservative” by Biasly, yet has a rating of “Excellent Reliability.” The author, Kimberly Winston, has a variety of sources discussing both sides of a Supreme Court case regarding a non-discrimination policy for same-sex couples and its impact on Catholic Social Services. While the Supreme Court ruled unanimously in favor of Catholic Social Services on grounds that it violated their First Amendment rights, Winston provides a great variety of perspectives. For example, she provides this quote from a legal scholar, Marci Hamilton:

“The real danger, and in my view, the evil that resides in this concept of religious liberty without regard to consideration of the common good is that we end up permitting religious actors to question laws necessary for all of us. It will open the floodgates to religious organizations saying they shouldn’t have to cover any medical procedures they deem against their faith, whether it is a blood transfusion, reproductive care, or covering vaccines.”

She provides the alternate perspective in the following passage with an equally qualified candidate, Howard Slugh, a council-man for the Jewish Coalition for Religious Liberty:

“‘It would give much more protection to every religious person throughout the country,’ he said. ‘It means the government will have to meet a much higher bar’ before it can infringe on the free exercise of religion.”

The bias rating is sourced from the article’s insistence on the persuasion of religion in the court system. However, the bias is smaller in consideration of the wide range of sources used and the variety of arguments for and against CSS in this particular court case. This article is considerably reliable because of the range of opinions and sources presented. If it had lacked sources depicting both perspectives, it would have been considerably more biased in favor or against the issue at hand. One article, however, does not show the overall reliability of FiveThirtyEight, hence other articles should be considered.

Analysis of Reliability in FiveThirtyEight Opinion Pieces

Opinion pieces are distinctive from other articles because of their relative subjectivity. These articles are outlets for authors’ beliefs rather than attempting to report objectively. Generally speaking, other news articles evade promoting their personal opinions and actively write to avoid inherent bias. With an understanding of the subjectivity of these pieces, readers can use these pieces to understand current arguments on particular issues from different perspectives.

While FiveThirtyEight doesn’t publish pieces overtly labeled as opinion pieces, some of their articles are more liberal than others. These pieces generally follow the same formula, using graphs and analyzing poll data to tell a story about the current political landscape, however notably lack evidence or discussion of the opposing perspective.

Quality of Sources and Facts Used

Consider the following article, “Why Many Americans Can’t See The Wealth Gap Between White And Black America,” written by Neil Lewis Jr. While the article uses graphs that showcase racial disparity and separation, similar to other FiveThirtyEight articles, it does not provide arguments that support the other side or define why this is the case. The article provides no quotes regarding the topic but does provide data. When considering data, it is important to remember the variety of sources used and the inherent bias of those who collected it. The article includes these sources:

2010 Census Block Data from Michigan (non-partisan)
2014 survey from PRRI (non-partisan)
Perspectives on Psychological Science (non-partisan)
Rhea Boyd (left-leaning)
Sage Journals (non-partisan)

While Lewis utilizes various sources for the data to support his claims, all elaboration and explanation of said data is from partisan sources or himself. The lack of quotes and external opinions from right-leaning sources makes the piece inherently more biased. Without diversity of thought, regardless of the accuracy of data used, it becomes much more liberal-leaning.

However, the data used is accurate. The census data is an example of objectively accurate information about the demographics in America, regardless of one’s political alignment. Similarly, the studies used are peer-reviewed from non-partisan sources. Such studies are generally more reliable because of the peer-review process. However, it lacks the necessary information to understand the holistic perspective of the issue of race in America, and it becomes an untrustworthy source, regardless of the quality of such sources.

Consider the following from his article:

“For Black Americans, in particular, the statistics around the inequalities of COVID-19 are numerical stand-ins for a much larger issue that permeates so many aspects of life — everything from being able to get a life-saving vaccine to the risk of being killed by police.

The reality is that our nation is still racially segregated. And it’s segregated in ways that limit our opportunities to learn about each other’s life experiences, even if our laws do not formally segregate our nation as they once did. This means that some live in a world in which they rarely encounter the conditions that bring harm to others everyday; others can’t escape those very conditions.”

He cites sources to support his factual claims, however, comes to a conclusion that is decisively liberal without discussing other possibilities for the circumstances he is highlighting. Furthermore, he discusses his conclusions as if they are facts. His biases play a clear role in interpreting the facts he is citing. In other words, the conclusion that Americans are unaware of dangerous conditions or do not face similar challenges because of segregation is biased because it jumps to conclusions not inherently evident from the evidence provided and his use of “segregation” as a legalistic term with an inherently negative connotation. He infers racial segregation yet his admittance that the laws no longer segregate by race shows bias and incorrect use of said term as Britannica defines it:

“the practice of restricting people to certain circumscribed areas of residence or to separate institutions (e.g., schools, churches) and facilities (parks, playgrounds, restaurants, restrooms) on the basis of race or alleged race.”

Britannica

Selection and Omission Bias

Another example is found in Lee Drutman’s “Why The Two-Party System Is Effing Up U.S. Democracy.” The article primarily attempts to oppose the two-party system rather than a particular political party, however, tends to put more blame on Republicans for the national division over politics. Typical of the majority of FiveThirtyEight articles, few quotes are used, but three sources are cited. This includes Levi Boxell, Matthew Gentzkow, and Jesse M. Shapiro’s work, Jonathan Rodden, and Noam Gidron, James Adam, and Will Horne’s book “American Affective Polarization in Comparative Perspective.” The sole quote provided comes from Boxell, Gentzkow, and Shapiro:

“[O]ur central conclusion — that the U.S. stands out for the pace of the long-term increase in affective polarization — is not likely an artifact of data limitations.”

They are left-leaning, with the other sources’ political affiliations unknown. While they may be partisan, the conclusion itself is not. It is simply a reflection of the data regarding political division in the United States in comparison to other countries. Regardless, republicans are found to blame for a primary reason for political division in the United States within the article as well as using vocabulary to place them in a negative light. This includes the insistence that a majority of Republicans believed Trump was still president in 2020 without discussing Republicans who had opposed these ideas.

Opinion articles as a whole are more susceptible to biased reporting of information to support the political ideology the author espouses. It is not unique to FiveThirtyEight. It is important to remember the distinction between opinion articles and more general news reporting, as it will increase the opportunity of the reader to recognize bias in its different forms across different modes of media. FiveThirtyEight, generally favoring liberal policies, will more likely have selection and omission bias favoring left-wing politics, just as a conservative publication would more likely do the opposite. The power lies in information and recognizing the truthfulness of it.