Google's Jigsaw open sources Altitude to help online platforms weed out extremist content

Google’s incubator subsidiary Jigsaw has open sourced a counterterrorism tool designed to help smaller companies address extremist content shared through their online platforms.

While most of the big tech companies, including Meta, X and Google itself, are well-versed in moderating content to weed out illegal or toxic content, the resources required to do this are beyond the means of many smaller digital players. This is particularly problematic at a time when legislation such as Europe’s Digital Services Act (DSA) and terrorism-specific content rules have placed companies under increasing pressure to remove illicit content post-haste.

Formerly known as Google Ideas ahead of a rebrand in 2016, Jigsaw is focused on developing and experimenting new products that tackle societal issues such as disinformation, censorship and the spread of toxicity and extremist content online. Last November, Jigsaw announced a new online content moderation tool called Altitude, which helps moderators verify whether content on their platforms matches that which has been flagged on third-party databases, rather than having to check against each source individually.

Altitude was developed in partnership with Tech Against Terrorism, an initiative launched in 2017 by the United Nations’ Counter-Terrorism Committee Executive Directorate (CTED). Tech Against Terrorism’s remit is to support the tech industry in developing more effective tools to tackle terrorists’ use of the internet to spread harmful content. The tool was also developed in partnership with industry-led outfit Global Internet Forum to Counter Terrorism (GIFCT), which counts most of the big tech companies as members.

Nine months on, and Jigsaw is now formally handing Altitude over to Tech Against Terrorism, which will continue its development and maintenance. The code is also now available on GitHub under an Apache 2.0 license, meaning that companies are able to self-host the tool, and third-parties are able to inspect the code.

How Altitude works

Altitude leans on “trusted” data sources provided by the likes of GIFCT, whose members pool their proprietary information for the purpose of powering automated content moderation tools (such as Altitude), as well as Tech Against Terrorism’s own Terrorist Content Analytics Platform, a database of real-time, verified terrorist content. Trusted sources identify and flag this type of content using their own methods.

By funneling into these databases, Altitude consolidates what it calls “terrorist and violent extremist content” (TVEC) into a single view, with moderators able to view, review and remove content swiftly and adhere to their obligations under whatever jurisdiction they operate within. It’s worth noting that no content is automatically removed, it’s all about flagging and providing alerts to potential extremist content for humans to assess.

Neither Jigsaw nor Tech Against Terrorism has revealed any of the platforms that have used Altitude through its initial testing period, however the kinds of applications that are most likely to use it include messaging apps, URL-shortening services, video-hosting sites, online forums, social networks and pastebins.

Jigsaw wrote in a blog post Monday:

Over the course of a year, Jigsaw conducted interviews with founders, trust and safety leaders, and TVEC content reviewers at eleven platforms, ranging in size. Platforms included image-sharing platforms, social media sites, url shortening services and file and text storage sites. We found that platforms existed along a scale of resourcing and preparedness to manage TVEC, and that that level of preparedness was not neatly related to the size of their organization or user base. Even relatively mature services still sometimes lacked signals to proactively identify harms on their platform. We also saw that, while they may eventually prefer to integrate all tools into an in-house platform, a separate interface for dedicated harms could provide a way to get started in an under-resourced environment.

As an open source project, any company can integrate Altitude into their own platforms, or they can work with Tech Against Terrorism to access “bespoke onboarding support.”

Jigsaw and Tech Against Terrorism say that they may add “specialized databases” to the data sources in the future, and they plan to expand the tool beyond English.