How Effective Are Fact-Checkers?

In today’s digital environment, the ability to propagate false information at scale and with lightning speed is greater than ever before, prompting our team of data scientists to ask, how effective are fact-checkers? In recent publications in collaboration with the Atlantic Council’s Disinfo Portal, Constella has outlined a broad view of disinformation in the digital ecosystem by highlighting some of the fundamental challenges and suggesting potential remedies. In this ecosystem in which fact-checkers play a critical role, challenges and solutions are much more complex than the major platforms themselves.

Article 1: An Ecosystem of Mistrust and Disinformation
Article 2: Fixing the Digital Ecosystem
Article 3: Europe’s Parliamentary Elections in the Digital Ecosystem

Within this wider context, the number of active fact-checking organizations has increased rapidly in efforts to combat challenges of digital disinformation head-on. According to the latest assessment by the Duke Reporters’ Lab, the number of fact-checking outlets around the world has grown to 188 in more than 60 countries.

In order to better understand fact-checkers’ efforts in counteracting disinformation and reaching a variety of different digital communities, Constella’s analysts conducted preliminary research based on data collected from the public digital socio-political debate ahead of the European Union Parliament elections in May 2019. The focus of that research – which is detailed here – was to identify disinformation strategies from a wide range of public digital sources, including social media, public forums, blogs, digital communities, discussion boards, news, video, wiki sites, and other sites, from mid-December 2018 to the end of May 2019 in five key countries: France, Germany, Italy, Poland, and Spain. Throughout the research project, Constella’s team identified consistent characteristics of disinformation content and the nature of its diffusion across communities of opinion in the public digital conversation.

Within this context, Constella’s preliminary research began with an analysis of Twitter as the largest source of publicly available data. The research included the following lines of inquiry:

1. 1. 1. To what extent are fact-checking domains distributed across digital communities of opinion in the public socio-political conversation?

 

1. 1. 2. Do fact-checkers achieve higher distribution in certain communities? If so, in which communities are fact-checkers generating the most interactions and penetrating most consistently?

 

1. 3. Taking into account Constella’s previous research which identifies where signals of disinformation activity are emerging most frequently, what can be concluded regarding the nature of the distribution of fact-checkers’ content? Is it shared in a silo, or is it reaching the users that tend to be exposed to disinformation more often?

Methodology

To address these questions, our analysts first identified reputable and verified fact-checkers in each of the five countries. The selection of fact-checkers was conducted by consulting verifiable fact-checking databases such as the Duke Reporters’Lab.

Next, Constella’s data scientists used Analyzer, Constella’s proprietary cloud-based analytics platform, to build and identify the largest communities of users interacting in connection with the public debate in each of the five countries. This community was identified in Twitter, and their interactions were filtered for retweets only, in order to perform a topological analysis of the propagation of messages. This resulted in a unimodal network of authors (giant component, edges being re-tweets). To determine the different communities and visually represent the resulting network, Constella’s data scientists applied clustering algorithms, such as the Louvain Method for community detection, for example. The most relevant communities of users in the public digital socio-political conversation were then used as a reference through which penetration and distribution of fact-checking content across these communities were analyzed. The analyses were conducted at distinct intervals depending on the country of focus, with the range of analyses occurring between December 2018 and May 2019 to provide a snapshot of the activity in each country across the pre-election period.

Constella’s analysis then identified and visually represented profiles interacting (retweet, reply, or mention) with fact-checkers’ content on Twitter. The penetration of each respective fact-checker in the identified communities and across the broader network was then calculated based on the number of users interacting with the content. Depending on the intensity and nature of the penetration achieved in the main communities talking about socio-political topics (the scope of reference for the analysis), the effectiveness of fact-checkers in reaching the communities which are most exposed to disinformation content can then be deduced. The communities which interact most frequently with suspicious or questionable media and content are identified and detailed in Constella’s country-specific analyses on the pre-EU elections public socio-political debate in each country.

The tables in each section indicate the percentage of penetration within the top communities and outside of the top communities. This means that the network that is visually represented in each Communities Detection Analysis is representative of the most relevant communities and users interacting within the public sphere socio-political conversation in each country. Those users and communities outside of these most relevant conversations and interactions which are core to the public digital debate are also analyzed, although they are not visually represented as their volume is marginal compared to the overall network. Nevertheless, the penetration of the fact-checkers’ within the top, most relevant communities in each country’s digital public sphere conversation is another indication of where their content is being shared, that is, within the most relevant conversations or not. Below find details on the analysis of fact-checkers’ reach within each country analyzed:

Penetration of Identified Fact-Checkers in France

 

 

Analysis of the socio-political conversation in France was conducted from December 15th, 2018 to January 20th, 2019.

In France, Constella’s analysts examined the penetration of fact-checkers including AFP’s Factuel, Liberation’s Desintox, Le Monde’s Les Decodeurs, and others as can be seen in the table above. Fact-checkers achieve a total penetration of 6.37% in the public digital sphere conversation in France, meaning that 6.37% of all users in the digital conversation interacted with pre-identified fact-checkers’ content through a reply, retweet, or mention. As can be seen in the table, 4 out of the five fact-checkers analyzed achieve a similar penetration of 29-39% in the polarized communities of La Republique en Marche and La France Insoumise.

Constella’s previous analyses on the French public sphere conversation demonstrated that disinformation signals and content were predominantly identified in the Rassemblement National community and in the La France Insoumise community to a lesser degree. Based on the results of this analysis, fact-checkers do not penetrate as effectively in the Rassemblement National community as they do in either the La Republique en Marche or La France Insoumise communities. Further context is available in Constella’s country-specific research on the public sphere digital conversations in France ahead of the EU elections.

In the table above, abbreviated titles are used to represent the most relevant communities in the French socio-political conversation. “Satirical” represents the Satirical and Humorous Media and Citizens community, “LaREM” represents the La Republique en Marche! and Several National and International News Media community, “FI” represents the La France Insoumise community, “RN” represents the Rassemblement National community, and “PSG-s” represents the Parti Socialiste and Génération.s community.

Penetration of Identified Fact-Checkers in Germany

 

Analysis of the socio-political conversation in Germany was conducted from December 15th, 2018 to March 15th, 2019.

In Germany, Constella’s analysts examined the penetration of fact-checkers including Tagesschau’s Faktenfinder, Zeit’s Fakt oder Fake, BR’s Faktencheck, Correctiv, and others as indicated in the table above. Fact-checkers achieve a total penetration of 2.2% in the public digital sphere conversation in Germany, meaning that 2.2% of all users in the digital conversation interacted with pre-identified fact-checkers’ content through a reply, retweet, or mention. All but one of the fact-checkers analyzed propagate primarily in the Left community. Zeit.com’s “Fakt oder Fake” has a penetration of at least 14% in the 3 communities of Left, CDU & SPD, and AfD supporters. Although Fakt oder Fake propagates primarily in the Left community with a penetration of 34.67%, this fact-checker demonstrates the most balanced relative distribution as compared to the others.

In Constella’s previous analyses of the German public digital sphere conversation, disinformation content and signals, such as a high proportion of abnormally high activity users and disinformation content focused on anti-immigration and anti-Islam narratives, were overwhelmingly identified in the AfD Supporters community. However, our current analysis demonstrates that fact-checkers’ penetration in Germany is primarily achieved in the community most polarized to the AfD Supporters community, the Left community. Further context is available in Constella’s research on the public sphere digital conversations in Germany ahead of the EU elections.

In the table above, abbreviated titles are used to represent the most relevant communities in the German socio-political conversation. “Left” represents the Left community, “Piraten” represents that Piraten Partei community, “CDU & SPD” represents the CDU and SPD community, and “AfD” represents that AfD Supporters community.

Penetration of Identified Fact-Checkers in Italy

 

Analysis of the socio-political conversation in Italy was conducted from January 21st, 2019 to March 7th, 2019.

In Italy, Constella’s analysts examined the penetration of fact-checkers including Open Online, Butac, David Puente, Pagella Politica and others as indicated in the table above. Fact-checkers achieve a total penetration of 3.67% in the public digital sphere conversation in Italy, meaning that 3.67% of all users in the digital conversation interacted with pre-identified fact-checkers’ content through a reply, retweet, or mention. All seven of the fact-checkers analyzed predominantly propagated in the Partito Democratico community with a penetration of over 55%. In Constella’s previous analyses of the Italian public digital sphere conversation, disinformation content and signals such as a high proportion of abnormal high activity users generating content with anti-immigration and anti-Islam narratives, were identified in the Right and Anti-Immigration community, and to a lesser degree in the Partito Democratico community. Our current analysis indicates that fact-checkers’ penetration in Italy is primarily achieved in the community most polarized to Right and Anti-Immigration community, the Partito Democratico community. Further context is available in Constella’s research on the public digital sphere conversation in Italy ahead of the EU elections.

In the table above, abbreviated titles are used to represent the most relevant communities in the Italian socio-political conversation. “PD” is used to represent the Partito Democratico community, “Salvini Criticism” represents the Criticism Against Salvini community, “Right” represents the Right and Anti-Immigration community, “M5S” represents the Movimento 5 Stelle community, and “Venezuela” represents the Venezuela Political Crisis community.

Penetration of Identified Fact-Checkers in Poland

 

Analysis of the socio-political conversation in Poland was conducted from January 21st, 2019 to March 7th, 2019.

In Poland, Constella’s analysts examined the penetration of fact-checkers including OKO.press, Gazeta Wyborcza’s Sonar, Demagog, and others as can be seen in the table below. Fact-checkers achieve a total penetration of 6.41% in the public digital sphere conversation in Poland, meaning that 6.41% of all users in the digital conversation interacted with pre-identified fact-checkers’ content through a reply, retweet, or mention. Five out of the six fact-checkers analyzed achieve more than 50% of their penetration in the political opposition community of PO. Contrary to the trend in Poland of fact-checkers achieving greater distribution in the PO community, “demaskator24.pl” achieved maximum penetration in the community of PiS, the governing political party. Further context is available in Constella’s research on the public digital sphere conversation in Poland ahead of the EU elections.

In the table below, abbreviated titles are used to represent the most relevant communities in the Polish socio-political conversation. “PiS” represents the Affinity to PiS community, “PO” represents the Affinity to PO community, “Wiosna” represents the Wiosna Biedronia community, and “Foreign Affairs” represents the Ministry of Foreign Affairs community.

Penetration of Identified Fact-Checkers in Spain

 

Analysis of the socio-political conversation in Spain was conducted from January 21st, 2019 to March 7th, 2019.

In Spain, Constella’s analysts examined the penetration of fact-checkers including Newtral, Maldito Bulo, and La Sexta’s El Objetivo. Fact-checkers achieve a total penetration of 4.94% in the public digital sphere conversation in Spain, meaning that 4.94% of all users in the digital conversation interacted with pre-identified fact-checkers’ content through a reply, retweet, or mention. In Constella’s previous analyses, disinformation content and signals were identified primarily in the communities of users interacting most heavily with affinity to the VOX and Catalan Independentists communities, with the Podemos community representing significant but lesser penetration of similar signals.

In the table above, some abbreviated titles are used to represent the most relevant communities in the Spanish socio-political conversation. “PP” is used to indicate the Partido Popular community.

Key Insights

Our analysis doesn’t conclude that the fact-checkers are not effective in the broader socio-political conversation. But it does show that the reach of fact-checkers is limited, often to those digital communities which are not targets for or are propagating disinformation. In response to the initial research questions posed, Constella’s analysis has yielded the following insights taking into account the scope and limitations of this analysis:

 

1. 1. Analyzed fact-checkers achieve a limited penetration in their respective countries, ranging from 2.2% to 6.5% across the most relevant communities of users in the public digital sphere conversation (assessed by the number of users interacting via retweets, replies, or mentions with fact-checkers’ content). Further, fact-checkers appear unable to significantly penetrate the communities which tend to be exposed most frequently to disinformation content, as identified in Constella’s previous analyses.

 

1. 1. 1. Analyzed fact-checkers demonstrate a tendency to propagate heavily in one or two communities, exhibiting characteristics of siloed content propagated by disinformation sites and less like mainstream media outlets. Accordingly, content from fact-checkers shows a tendency to be situated opposite to or polarized from disinformation content.

 

1. 1. 2. Analyzed fact-checkers are predominantly shared among the most relevant conversations in the public digital sphere debate, although the extent of their penetration across the entire network is limited and isolated to communities where disinformation content and signals are least prevalent, based on Constella’s previous analyses.

 

1. 1. 3. Among the fact-checkers analyzed, those in France achieve the greatest spread of distribution across several communities. In the rest of the countries, most of the fact-checkers’ distribution has a higher concentration in one community. The scope of this analysis does not conclude the underlying reasons for the increased distribution of French fact-checkers across several communities in France, and further investigation would be required to establish any causality.

 

1. 1. 4. Fact-checking initiatives that are off-shoots of media outlets (Le Monde, AFP, Tagesschau, Zeit, BR, Liberation, TVN24, etc.) exist alongside organizations almost solely dedicated to fact-checking as a reporting strategy (Correctiv, OKO.press), and their effectiveness varies by country and does not appear to be correlated with this distinction.

 

1. 5. In terms of relative penetration, Germany has the least penetration (2.2%) followed by Italy (3.7%) in the public digital sphere conversation in their respective countries.

Further Research

There are many possible lines of investigation to build on the insights offered, and the findings here are limited to the timeframe, a single platform (Twitter) and context of the social-political debate in the five countries ahead of the European elections. In no way should they be seen as downgrading the importance of fact-checkers. Rather the research is aimed at adding to a broader understanding of their effectiveness in certain contexts. Further research could incorporate the following:

 

1. 1. Fact-checkers in France are shown to have the greatest distribution across several communities. Further research is needed into the reasons why France is different to the other countries analyzed.

 

1. 2. Search engines are an important source of discovering fact-checkers’ stories. Ranking fact-checkers’ content on search engines with relation to the news stories they are aiming to debunk would be a useful exercise. A comparison of relevance would yield insights into the ability of fact-checkers to be where the disinformation stories are, and increase the chances of reaching those digital communities where disinformation is most prolific.

 

1. 3. Extending this type of analysis to incorporate different platforms and digital communities, including Facebook and YouTube, could potentially yield a richer understanding of the effectiveness of fact-checkers across other networks.

 

1. 4. Another line of research to expand on the insights offered in this preliminary analysis could include a detailed content analysis – including memes, videos, articles, images, and others – to better understand which types of content tend to achieve the greatest penetration across several digital communities.

 

Finally, the scope of this analysis does not conclude the underlying reasons for the variations in either the total penetration or the spread of distribution among the countries analyzed. Additional analyses would be necessary in order to establish any causality or relationship between the fact-checkers’ effectiveness and the nature of their activity.

Article written by Jonathan Nelson. Originally published on July 12, 2019

Interested in our work? Please contact us at info@constellaintelligence.com. To learn more about Constella, subscribe to our newsletter below.