The Story of a Twitter API that Went Awry: Deciphering the Politwoops Shutdown

Photo Credit: goodbye via photopin (CC BY 2.0)

A month ago, Politwoops was a ragingly popular operation, tracking and publicly archiving politicians' deleted tweets in over 30 countries. It still is, but in May 2015, something odd happened. The American version of the Politwoops site unexpectedly stopped working, while its counterparts abroad experienced no tech problems and continued to archive deleted tweets seamlessly. When the Sunlight Foundation (SF), the creators of Politwoops-USA, called Twitter's support desk, the company eventually stated that the site violated its Developer Agreement & Policy.

This wasn't the first time that Twitter had told the organization to stop collecting the deleted tweets of American politicians. Back in 2012, Twitter informed the SF that Politwoops was in violation of its Application Program Interface (API) Terms of Service; at that juncture, though, the SF explained its mission, goals, and review process to Twitter, and after some back and forth, the two parties agreed that Politwoops-USA could continue to access and republish parts of Twitter's data stream.

This time around, however, it appears that Twitter isn't planning to restore Politwoops’ access to the API. It's worth noting that deleted tweets have posed a problem to Twitter in one way or another almost since its inception, and in the interim, users have launched a variety of projects to deal with the controversial content. Tom Scott's "Tweleted" recovered deleted tweets by comparing a person's Twitter history to Twitter search; he shut it down after Twitter stopped deleted messages from showing up in the Twitter Search. Bradley Griffith's Undetweetable took another approach and archived deleted tweets by anyone entered into its database. His project was much far-reaching than Politwoops, as it could collect deleted tweets by Alec Baldwin, Miley Cyrus, rambling professors, employees of publicly traded stocks, and completely anonymous Twitter users. Citing API concerns, Twitter shut it down almost immediately.

The recent Politwoops case has led many to problematize the social architecture of Twitter. Responses have greatly varied, ranging from the outraged to the elated. Some, like Joshua Benton of Harvard's Nieman Lab, have eulogized the project. Others, like Nathaniel Mott of Pando Daily , have come to Twitter's defense, echoed earlier concerns about the service, and praised the company’s desire to protect users' privacy.

Today, other questions remain. Is Twitter becoming more of a private platform than a public square? What might happen to other branches of Politwoops in India, Argentina, Bermuda, Tunisia, the European Union, and elsewhere? What sort of mechanisms might be used in the future to preserve deleted tweets? Who, ultimately, has control over tweets? What extra-judicial norms, laws, and community standards dictate Twitter users' behavior? Will Twitter's global public policy team adopt a standard approach towards all branches of Politwoops in the coming weeks?

For insight, Muira McCammon talked to Nicko Margolies (Politwoops' Project Lead at the Sunlight Foundation), Arjan El Fassed (Director of the Open State Foundation, which founded Politwoops in the Netherlands), and Prof. Michael Beurskens (a Twitter researcher and intellectual property law lecturer based at the University of Bonn in Germany).

Photo Credit: Arjan El Fassed by Bob Bronshoff (CC BY 3.0) 

Part I: Micro-Interview with Arjan El Fassed, Director of Open State Foundation 

Muira McCammon: How did Politwoops initially decide to expand? It seems to be active in 35 countries, but I'm curious to know more about how that network developed.

Arjan El Fassed: Politwoops originated in the Netherlands, as an Open State Foundation project to follow Dutch politicians, as part of a greater effort towards transparency and accountability in government. We wanted to track elected politicians of parliaments in the UK, France, Germany, Canada, Tunisia, Argentina, Egypt, Turkey and other countries, including all members of the European Parliament. Politwoops started in the Netherlands in 2010 with first parliamentarians of both the First and Second Chambers, and later on, we also included municipal and regional council members. After the Netherlands, we first expanded Politwoops to the UK, Canada and other countries before we helped Sunlight Foundation launch the US version of Politwoops. So, before the US, there were already 12 versions running. 

MM: Is Politwoops still expanding to other countries?

AEF: In the past five years, we have expanded and assisted the launch of various sites and versions in over 30 countries. The last new country was Chile.

MM: How did you initially go about negotiating the deal with Twitter?

AEF: We as the Open State Foundation have not negotiated with Twitter. Days after Politwoops was launched in the US in 2012, Twitter contacted the Sunlight Foundation and told them, "Your service violates our API Terms of Service on a fundamental level." Sunlight Foundation explained the goals of the project and agreed to create [a system of] human curation in order to ensure that the site screened out corrected low-value tweets like typos, links and Twitter handles. Sunlight Foundation implemented this layer of curation and the site continued.

MM: Do you feel that Politwoops is "safe" in any country given the latest changes in the States?

AEF: We do not know. Usually Twitter takes action only when there are complains. [The Dutch] Politwoops is especially interesting in times of elections. So far all other versions are still running. The notion that it is a violation of Terms of Service was known for the past five years. Twitter continued to allow this for the past five years. We were never contacted about this or automatically closed down. It is not black and white. In Europe, there is this notion of the right to be forgotten which applies only to users who are not public figures, such as politicians. Since most public figures don't start as public figures, we start following politicians, when they're elected. We do not follow private people but rather, public figures posting public messages.

Photo Credit: Headshot supplied by Nicko Margolies.

Part II: Micro-Interview with Nicko Margolies, Politwoops Project Lead at the Sunlight Foundation

MM: At a really technical level, can you explain to those of us who aren't tech-savvy how the Sunlight Foundation interacted with Twitter's API?

Nicko Margolies: We identified the accounts of politicians, and whenever Twitter sent a deletion notification through their API (a data stream for software to communicate with other software), Politwoops would highlight that removed tweet to Sunlight staff. From there, the person would compare it to the public feed of the politician to see if it was a minor error that was corrected and if not, it would be approved to appear in Politwoops' archive. Like everything the Sunlight Foundation does, the code for Politwoops is open source and available on Github here.

MM: Were there any tweets that were reviewed and ultimately not approved by the Sunlight Foundation?

NM: Absolutely! Deletions were hand curated by Sunlight staff to ensure that we weren't surfacing tweets deleted for minor typos or things like corrected Twitter handles. We did this to ensure that there was a journalistic value to the project. Less than one third of the deleted tweets of politicians Politwoops caught appear on the public archive because Sunlight screened out these minor errors.

MM: What has surprised you - personally - the most about Twitter's decision to change its policy towards Politwoops?

NM: Twitter's decision reminds us that the platform is less a town square and more of a walled garden. The communications that happen there aren't truly public, even when they're made by public officials. We're disappointed that Twitter changed its mind and decided to pull the plug on a valuable tool for accountability.

Photo Credit: Headshot supplied by Prof. Michael Beurskens.

Part III: Micro-Interview with Prof. Michael Beurskens, intellectual law expert and contributor to Twitter and Society

MM: As a Twitter user, how does this recent policy move make you feel? 

Prof. Michael Beurskens: I was actually not entirely surprised by Twitter's "boycott" of Politwoops.

MM: Do you think that Politwoops, given current EU laws and cultural norms, should be shut down in the Netherlands, then? 

MB: The question of whether Twitter's conduct is "right" is not so much a legal question but rather a political and moral one.  This is probably also why [Twitter] does not go after Politwoops on a global basis. A public uproar would not be in their best interest.

Indeed, one might make a case that due to [Twitter's] monopoly rights over the pool of "source tweets" that they should act as a steward of public interest or an [archivist] of historic events. You could argue that Twitter should prevent the deletion of public figures' tweets, archive such tweets themselves, or grant access to third parties.

The relevant clause in their Terms of Use has no commercial value to them in the current context (they do not sell deleted tweets), which means they themselves have little interest. However, the public has rights guaranteed under many constitutions or fundamental rights charters to access information that was made at least temporarily available to the public.

The European Court of Justice's case law only deals with the question of a duty of Google (or in this case Twitter) to remove data. It does not provide any guidance regarding the public's right to access, copy, and reuse the data.

It may therefore well be within Twitter's right to limit access to deleted tweets.

These interviews have been edited and condensed.