Best Infosec-Related Long Reads for the Week, 10/28/23

The scientist who invented captcha, AI is poisoning web search, New tool can break AI models, Biden's AI EO addresses cyberweapons, Stopping unlawful location disclosures, ICE scans social media

Metacurity is pleased to offer our free and paid subscribers this weekly digest of the best long-form infosec-related pieces we couldn’t properly fit into our daily crush of news. So tell us what you think, and feel free to share your favorite long reads via email at info@metacurity.com. We’ll gladly credit you with a hat tip. Happy reading!

How this Turing Award-winning researcher became a legendary academic advisor

In MIT Technology Review, Sheon Han profiles Nobel prize-winning theoretical computer scientist emeritus profess at Carnegie Mellon University and lauded teacher Manuel Blum, who, among other things, created the Completely Automated Public Turing test to tell Computers and Humans Apart, better known as the captcha.

The kinds of questions Blum poses read like paradoxes and have a somewhat playful quality, making complexity theory and cryptography sound almost like a subgenre of sci-fi. “He is completely original and goes off and does what he thinks is interesting and important. And often it turns out to be something really significant,” Sipser told me.

In his seminal paper “Coin Flipping by Telephone,” the question that he poses is: “Alice and Bob want to flip a coin by telephone. (They have just divorced, live in different cities, and want to decide who gets the car.)” Let’s say that Alice calls “heads” and Bob says she lost; how does she trust that he is being truthful? And how could Bob trust Alice if the situation were reversed?

What sounds like a riddle addresses a fundamental problem in cryptography: How can two parties engage in trustworthy exchanges over a communication channel in such a way that neither party can cheat?

Blum showed that this can be achieved using the concept of “commitment.” In a simplified analogy, the idea is that Alice gives Bob a locked box with her prediction inside, but without the key. This prevents Alice from altering her prediction and stops Bob from discovering Alice’s guess prematurely. Once Bob tosses the coin, Alice hands over the key to open the box.

Chatbot Hallucinations Are Poisoning Web Search

Wired’s Will Knight illustrates how “the age of generative AI threatens to sprinkle epistemological sand into the gears of web search” by walking through how Ph.D. researcher Daniel Griffin fictionalized work by Claude Shannon, the mathematician and engineer known for his work on information theory in the 1940s in AI chatbots, only to have them show up in Bing’s AI-powered search results.

This generative-AI trap that caused Bing to offer up untruths was laid—purely by accident—by Daniel Griffin, who recently finished a PhD on web search at UC Berkeley. In July he posted the fabricated responses from the bots on his blog. Griffin had instructed both bots, “Please summarize Claude E. Shannon’s ‘A Short History of Searching’ (1948)”. He thought it a nice example of the kind of query that brings out the worst in large language models, because it asks for information that is similar to existing text found in its training data, encouraging the models to make very confident statements. Shannon did write an incredibly important article in 1948 titled “A Mathematical Theory of Communication,” which helped lay the foundation for the field of information theory.

Last week, Griffin discovered that his blog post and the links to these chatbot results had inadvertently poisoned Bing with false information. On a whim, he tried feeding the same question into Bing and discovered that the chatbot hallucinations he had induced were highlighted above the search results in the same way as facts drawn from Wikipedia might be. “It gives no indication to the user that several of these results are actually sending you straight to conversations people have with LLMs,” Griffin says. (Although WIRED could initially replicate the troubling Bing result, after an enquiry was made to Microsoft it appears to have been resolved.)

Griffin’s accidental experiment shows how the rush to deploy ChatGPT-style AI is tripping up even the companies most familiar with the technology. And how the flaws in these impressive systems can harm services that millions of people use every day.

This new data poisoning tool lets artists fight back against generative AI

MIT Technology Review’s Melissa Heikkilä walks through a new tool called Nightshade that allows artists to fight back against the theft of their work by adding invisible pixels that break AI models in chaotic and unpredictable ways.

Nightshade exploits a security vulnerability in generative AI models, one arising from the fact that they are trained on vast amounts of data—in this case, images that have been hoovered from the internet. Nightshade messes with those images.

Artists who want to upload their work online but don’t want their images to be scraped by AI companies can upload them to Glaze and choose to mask it with an art style different from theirs. They can then also opt to use Nightshade. Once AI developers scrape the internet to get more data to tweak an existing AI model or build a new one, these poisoned samples make their way into the model’s data set and cause it to malfunction.

Poisoned data samples can manipulate models into learning, for example, that images of hats are cakes, and images of handbags are toasters. The poisoned data is very difficult to remove, as it requires tech companies to painstakingly find and delete each corrupted sample.

The researchers tested the attack on Stable Diffusion’s latest models and on an AI model they trained themselves from scratch. When they fed Stable Diffusion just 50 poisoned images of dogs and then prompted it to create images of dogs itself, the output started looking weird—creatures with too many limbs and cartoonish faces. With 300 poisoned samples, an attacker can manipulate Stable Diffusion to generate images of dogs to look like cats.

Sweeping new Biden order aims to alter the AI landscape

Politico’s Mohar Chatterjee and Rebecca Kern got a hold of the draft of the wide-ranging AI executive order that President Biden is slated for release as soon as Monday, concluding that, among many other things, it will create “extensive new checks on the technology, directing agencies to set standards to ensure data privacy and cybersecurity, prevent discrimination, enforce fairness and also closely monitor” the industry.

The order indicates the White House sees the rapid development of advanced cyberweapons as one of the most significant risks posed by artificial intelligence.

To prevent powerful AI models from falling into the hands of foreign adversaries, the order would require companies developing powerful AI models to provide regular reports to the Commerce Department outlining how they plan to protect their technology from espionage or digital subversion and mandate that large cloud services providers like Amazon and Microsoft notify the government each time foreigners rent server space to train large AI models.

The AI mandate would give federal agencies three months to identify the risks of AI use within the sectors they oversee and six months to develop guidelines for how private companies within those industries should incorporate the White House’s new AI Risk Management Framework. The Treasury Department specifically is required, within 150 days, to submit a public report on ways the banking sector can manage cyber risks involved in the use of AI technologies.

While AI poses a wide array of new cyber risks, the order also addresses its potential benefits. DHS and the Department of Defense are required to put together plans to carry out an “operational pilot program” within six months to test the use of AI technologies in discovering vulnerabilities in U.S. government networks.

Finding YouThe Network Effect of Telecommunications Vulnerabilities for Location Disclosure

Gary Miller and Christopher Parsons of Citizen Lab wrote an in-depth paper on how unlawful location disclosures depend on the vulnerabilities in the protocols used by 3G, 4G, and 5G network operators, offering case studies and statistics on how these kinds of threats are proliferating and outlining how policymakers should move to adopt security features in 5G networks to protect against the threats.

The ability for foreign networks to target international users with signaling messages to reveal geolocation constitutes the most prevalent known attacks on mobile networks. Despite this being well known within the telecommunication industry the question remains as to whether operators are protecting their customers from these threats.

In fully-compliant, cloud-native 5G deployments, international roaming signaling messages transit foreign networks with a new interface called N32 and use a network function called the Security Edge Protection Proxy (SEPP). This function was introduced into the 5G network architecture to add protection to the historically vulnerable communication between foreign network operators. The SEPP provides much needed encryption, integrity, and authentication at the border edge between roaming networks.

However, to provide privacy protection, networks on both ends of the roaming interface must implement the SEPP function. Getting all roaming partners to implement SEPP may be extremely challenging; of the 351 network operators reported to have launched 5G services, only 41 have launched 5G cloud-native architectures according to the Global Mobile Suppliers Association (GSA) as of April 2023. The remaining 310 operators are still using the Non-Standalone Architecture (NSA) for 5G, which lets mobile operators bypass the SEPP feature in 5G roaming while still providing the improved speed and reduced latency benefits of the 5G radio access network.

According to interviews with telecommunications security vendors at the Mobile World Congress (MWC) conference in March 2023, only a handful of operators have deployed SEPP, let alone are actually using it. The effect is that many operators are not integrating the security and privacy benefits of the 5G standards when they are deploying 5G networks.

Many network vulnerabilities are specific to a given mobile network operator’s implementation of telecommunications standards. However, given that many operators have shown a willingness to sell access to third-parties, there is a serious concern that surveillance actors will have software code in place to probe and test the integrity of foreign 5G networks. This will let surveillance actors adjust their tactics, techniques, and procedures for various network type vulnerabilities across each target network implementation. Historically, surveillance actors have quickly learned to modify their attacks to disguise traces and circumvent firewalls, and the slow pace of operator security deployments reduce the challenge that such actors will have in finding and exploiting obvious vulnerabilities.

Inside ICE’s Database for Finding ‘Derogatory’ Online Speech

404 Media’s Joseph Cox reveals how, through FOIA requests, the ACLU discovered that the Immigration and Customs Enforcement (ICE) has used a system called Giant Oak Search Technology (GOST) to help it scrutinize social media posts, determine if they are “derogatory” to the US, and then use that information as part of immigration enforcement.

GOST’s catchphrase included in one document is “We see the people behind the data.” A GOST user guide included in the documents says GOST is “capable of providing behavioral based internet search capabilities.” Screenshots show analysts can search the system with identifiers such as name, address, email address, and country of citizenship. After a search, GOST provides a “ranking” from zero to 100 on what it thinks is relevant to the user’s specific mission.

The documents further explain that an applicant’s “potentially derogatory social media can be reviewed within the interface.” After clicking on a specific person, analysts can review images collected from social media or elsewhere, and give them a “thumbs up” or “thumbs down.” Analysts can also then review the target’s social media profiles themselves too, and their “social graph,” potentially showing who the system believes they are connected to.

DHS has used GOST since 2014, according to a page of the user guide. In turn, ICE has paid Giant Oak Inc., the company behind the system, in excess of $10 million since 2017, according to public procurement records. A Giant Oak and DHS contract ended in August 2022, according to the records. Records also show Customs and Border Protection (CBP), the Drug Enforcement Administration (DEA), the State Department, the Air Force, and the Bureau of the Fiscal Service which is part of the U.S. Treasury have all paid for Giant Oak services over the last nearly ten years.