Google's call-scanning AI could dial up censorship by default, privacy experts warn

A feature Google demoed at its I/O confab yesterday, using its generative AI technology to scan voice calls in real time for conversational patterns associated with financial scams, has sent a collective shiver down the spines of privacy and security experts who are warning the feature represents the thin end of the wedge. They warn that, once client-side scanning is baked into mobile infrastructure, it could usher in an era of centralized censorship.

Google’s demo of the call scam-detection feature, which the tech giant said would be built into a future version of its Android OS — estimated to run on some three-quarters of the world’s smartphones — is powered by Gemini Nano, the smallest of its current generation of AI models meant to run entirely on-device.

This is essentially client-side scanning: A nascent technology that’s generated huge controversy in recent years in relation to efforts to detect child sexual abuse material (CSAM) or even grooming activity on messaging platforms.

Apple abandoned a plan to deploy client-side scanning for CSAM in 2021 after a huge privacy backlash. However, policymakers have continued to heap pressure on the tech industry to find ways to detect illegal activity taking place on their platforms. Any industry moves to build out on-device scanning infrastructure could therefore pave the way for all-sorts of content scanning by default — whether government-led or related to a particular commercial agenda.

Responding to Google’s call-scanning demo in a post on X, Meredith Whittaker, president of the U.S.-based encrypted messaging app Signal, warned: “This is incredibly dangerous. It lays the path for centralized, device-level client side scanning.

“From detecting ‘scams’ it’s a short step to ‘detecting patterns commonly associated w[ith] seeking reproductive care’ or ‘commonly associated w[ith] providing LGBTQ resources’ or ‘commonly associated with tech worker whistleblowing.’”

Cryptography expert Matthew Green, a professor at Johns Hopkins, also took to X to raise the alarm. “In the future, AI models will run inference on your texts and voice calls to detect and report illicit behavior,” he warned. “To get your data to pass through service providers, you’ll need to attach a zero-knowledge proof that scanning was conducted. This will block open clients.”

Green suggested this dystopian future of censorship by default is only a few years out from being technically possible. “We’re a little ways from this tech being quite efficient enough to realize, but only a few years. A decade at most,” he suggested.

European privacy and security experts were also quick to object.

Reacting to Google’s demo on X, Lukasz Olejnik, a Poland-based independent researcher and consultant for privacy and security issues, welcomed the company’s anti-scam feature but warned the infrastructure could be repurposed for social surveillance. “[T]his also means that technical capabilities have already been, or are being developed to monitor calls, creation, writing texts or documents, for example in search of illegal, harmful, hateful, or otherwise undesirable or iniquitous content — with respect to someone’s standards,” he wrote.

“Going further, such a model could, for example, display a warning. Or block the ability to continue,” Olejnik continued with emphasis. “Or report it somewhere. Technological modulation of social behaviour, or the like. This is a major threat to privacy, but also to a range of basic values and freedoms. The capabilities are already there.”

Fleshing out his concerns further, Olejnik told TechCrunch: “I haven’t seen the technical details but Google assures that the detection would be done on-device. This is great for user privacy. However, there’s much more at stake than privacy. This highlights how AI/LLMs inbuilt into software and operating systems may be turned to detect or control for various forms of human activity.

“So far it’s fortunately for the better. But what’s ahead if the technical capability exists and is built in? Such powerful features signal potential future risks related to the ability of using AI to control the behavior of societies at a scale or selectively. That’s probably among the most dangerous information technology capabilities ever being developed. And we’re nearing that point. How do we govern this? Are we going too far?”

Michael Veale, an associate professor in technology law at UCL, also raised the chilling specter of function-creep flowing from Google’s conversation-scanning AI — warning in a reaction post on X that it “sets up infrastructure for on-device client side scanning for more purposes than this, which regulators and legislators will desire to abuse.”

Privacy experts in Europe have particular reason for concern: The European Union has had a controversial message-scanning legislative proposal on the table since 2022, which critics — including the bloc’s own Data Protection Supervisor — warn represents a tipping point for democratic rights in the region as it would force platforms to scan private messages by default.

While the current legislative proposal claims to be technology agnostic, it’s widely expected that such a law would lead to platforms deploying client-side scanning in order to be able to respond to a so-called detection order demanding they spot both known and unknown CSAM and also pick up grooming activity in real time.

Earlier this month, hundreds of privacy and security experts penned an open letter warning the plan could lead to millions of false positives per day, as the client-side scanning technologies that are likely to be deployed by platforms in response to a legal order are unproven, deeply flawed and vulnerable to attacks.

Google was contacted for a response to concerns that its conversation-scanning AI could erode people’s privacy but at press time it had not responded.

We’re launching an AI newsletter! Sign up here to start receiving it in your inboxes on June 5.