Snapshot of generative AI-based threats, and risks to New Zealand: A capture of research from 2023 onwards

The research I lead at The Disinformation Project revolves around a daily situation report looking at Aotearoa New Zealand’s truth decay. It’s something I’ve done without a break since July 2021. In the last quarter of 2022, I began to look at generative AI (starting with OpenAI’s ChatGPT), in what remains an endeavour to determine how the misuse of this technology can adversely impact democracy. This involves looking at various generative AI products, platforms, apps, and services to see how guardrails (to extent they exist) can be manipulated to produce harmful content, as well as the study of how others promote the misuse of generative AI, including those in the country’s far-right online networks who are now imbricated with leading disinformation, anti-government, anti-establishment online networks that were created or strengthened during the Covid-19 pandemic.

The following was written for, and included in today’s situation report.

###

Neo-Nazis Are All-In on AI, a compelling Wired magazine article by David Gilbert referenced a new report by The Middle East Media Research Institute. As Gilbert notes,

Extremists across the US have weaponised artificial intelligence tools to help them spread hate speech more efficiently, recruit new members, and radicalise online supporters at an unprecedented speed and scale, according to a new report from the Middle East Media Research Institute (MEMRI), an American non-profit press monitoring organisation. The report found that AI-generated content is now a mainstay of extremists’ output: They are developing their own extremist-infused AI models, and are already experimenting with novel ways to leverage the technology, including producing blueprints for 3D weapons and recipes for making bombs. Researchers at the Domestic Terrorism Threat Monitor, a group within the institute which specifically tracks US-based extremists, lay out in stark detail the scale and scope of the use of AI among domestic actors, including neo-Nazis, white supremacists, and anti-government extremists.

His article, and MEMRI’s research builds on the challenges, and concerns around the instrumentalisation of (generative) AI by right-wing extremists noted in ‘Weapons of Mass Hate Dissemination: The Use of Artificial Intelligence by Right-Wing Extremists‘ – a report by ISD earlier this year, penned by Federico Borgonovo, Silvano Rizieri Lucini, and Giulia Porrino.

The report ended by noting,

According to recent reports detailing extremist networks on Telegram, online RWEs exploit existing generative AI models for the production of visual propaganda and even the development of explosives used in kinetic attacks. However, there is a looming concern that with the increasing IT capabilities among far-right groups, a scenario could emerge where AI is harnessed to generate more sophisticated and targeted propaganda, as well as to carry out cybercrime campaigns targeting online infrastructures. For this reason, it is imperative to intensify research efforts within these ecosystems to prevent and counter the use of AI by extremists and to adopt a proactive approach to prevent future threats. Tech companies must remain vigilant in monitoring the development of novel, extremist-owned models which may be misused for nefarious purposes. Implementing internal threat assessment teams and devising terrorism-focused procedures are crucial steps to identifying and addressing potential threats posed by RWEs using AI technologies.

ISD’s research was supported by ‘Propaganda for Fun:” How Extremists Use GAI to Camouflage Hate‘ – a report published by ADL in March this year looking at how “Disturbing memes and images created using Generative Artificial Intelligence (GAI) are polluting social media with camouflaged hate and extremist propaganda.” As the ADL’s report noted,

In general, mitigating harmful GAI content comes with a unique set of challenges. Open-source GAI models allow users to essentially build their own chatbots or image generators, making such systems harder to regulate than those created with closed-source models. Also, by hiding calls for violence in these images, extremists are better able to evade content moderation on social media, because embedded text in an image is traditionally more difficult to detect than actual text in a post. This also means that more direct, targeted threats may go unnoticed, and therefore uninterrupted.

Findings, highlights, and evidence-based concerns noted across all four of these sources underscore the growing sophistication, strategy, and scale of extremists’ use of AI technologies that are freely accessible, posing significant challenges for content moderation, counter-extremism efforts, and online safety. These four sources alone flag,

1. Widespread adoption: Neo-Nazis, white supremacists, and other extremist groups have rapidly adopted AI tools for propaganda creation and dissemination.
2. Image generation: Extremists are using AI image generators to create racist, antisemitic, and hateful memes, posters, and visuals that often mimic popular styles like Pixar movies to increase virality.
3. Video and audio manipulation: AI is being used to create deepfake videos and audio, including fake speeches by political figures or celebrities expressing extremist views.
4. Translation and reach: AI translation tools are helping extremists reach broader international audiences by translating content like Hitler’s speeches into multiple languages.
5. Custom AI models: Some extremist groups are developing their own AI models without content moderation, allowing for more explicit hate speech and dangerous content generation.
6. Weaponisation: There are concerns about extremists using AI to generate weapon blueprints, bomb-making instructions, and malicious code.
7. Rapid response: AI allows extremists to quickly produce large quantities of propaganda in response to current events.
8. Evasion tactics: Extremists are finding ways to bypass AI content filters, such as the “grandma loophole” to obtain information on creating weapons or explosives.
9. Music generation: AI is being used to create racist and extremist songs, sometimes imitating popular artists’ voices.
10. Coordination and fundraising: AI chatbots and other tools are being used to enhance recruitment, radicalisation, and fundraising efforts.
11. Camouflaged content: Extremists are using AI to embed hate symbols or messages within seemingly innocuous images, making detection more difficult.
12. Increased scale and speed: AI significantly enhances extremists’ ability to produce and disseminate content at an unprecedented scale and speed.
13. Cross-platform spread: AI-generated extremist content is being shared across various platforms, including mainstream social media and niche far-right sites.
14. Election concerns: There are worries about the potential impact of AI-generated extremist content on upcoming elections.
15. Erosion of epistemic authority, and trust: The proliferation of AI-generated deepfakes and misinformation is contributing to a general erosion of trust in online information.

Every one of these challenges is present, and growing in Aotearoa New Zealand. They reflect developments studied by TDP since late 2022, and especially after early 2023. Situation reports on the country’s complex, and evolving disinformation landscapes, penned daily, have studied in significant detail key global developments related to the instrumentalisation of (generative) AI by malevolent actors including the far-right, and neo-Nazis alongside endogenous experimentation, content production, and related commentary by domestic far-right networks, and leading neo-Nazis.

In January 2023, and based on the study of what even then was a significant body of evidence around the rapid adoption, and adaptation of generative AI by the country’s far-right networks, I wrote ten key points around the risks, and direct threats to social cohesion, and national security posed as a consequence of this technology’s weaponisation.

This research fed into a presentation at 2023’s Otago National Security School by TDP’s Director of Research, Dr Sanjana Hattotuwa, on generative AI’s growing threat in the country. The fact that generative AI was “experimented with, promoted, talked about most by disinfo, & far-right” was stressed.

Comparing the research, and analysis in just the first quarter of 2023 to the four sources noted above from Wired magazine, MMRI, ADL, and ISD, several parallels emerge.

• Early Adoption and Experimentation: TDP’s analysis shows that far-right individuals and groups in New Zealand were among the first to experiment with and adopt AI tools like ChatGPT and image generation platforms. This aligns with the historical pattern of extremists being early adopters of new technologies, noted in the MEMRI report.
• Diverse Applications: Our research highlights how extremists were using AI for various purposes, including generating propaganda, creating memes, and producing text-based content. This prefigured the global trend observed by ISD and ADL, where right-wing extremists, and neo-Nazis are leveraging AI for multiple aspects of their operations.
• Bypassing Content Moderation: TDP’s research shows domestic extremists attempting to bypass AI content moderation systems through prompt engineering, and other techniques. They were actively sharing findings, so as to make jailbreaking respective platforms (mostly LLMs at the time), a shared challenge, and endeavour.  This reflects similar observations made globally, where extremists are finding ways to circumvent safeguards built into AI systems.
• Spreading Disinformation: Our research highlights how AI-generated content was used to spread disinformation and conspiracy theories, particularly around topics like COVID-19, and mRNA vaccines. This aligns with what are now even more challenging global concerns about AI’s potential to supercharge disinformation campaigns.
• Holocaust Denial, and Distortion: TDP observed several instances of AI being used for Holocaust distortion, and denial in Aotearoa New Zealand, which is consistent with concerns raised by ADL et al about AI’s potential to amplify harmful historical revisionism.
• Normalisation of Extreme Content: Our study shows how AI-generated content was used to normalise white supremacist, far-right ideologies, violent extremism, and hateful rhetoric, mirroring widespread concerns today about AI’s potential to make extremist content more palatable and widespread (including through radicalisation pathways created through synthetic media’s appeal, and engagement).
• Cross-Platform Dissemination: TDP’s analysis reveals how AI-generated content is being shared across multiple platforms within Aotearoa New Zealand’s information, and media ecologies, reflecting a trend of extremists using AI to create content for a wide spectrum of social media, and messaging apps so as to maximise reach, and evade platform governance (where it exists).
• Erosion of Trust: Our research highlights concerns about AI’s potential to erode trust in official sources (i.e., a truth decay, and an epistemic crisis), which aligns with worries about AI’s impact on the information ecosystem, and democratic processes.
• Low-Cost, High-Output Propaganda: TDP’s findings showed, as early as the first quarter of 2023, how AI enabled the production of large amounts of propaganda at low cost in Aotearoa New Zealand, reflecting observations about AI’s potential to dramatically increase the scale of extremist content production, and sustain it.
• Multimodal Capabilities: Our research flagged the increasing use of AI for generating not just text, but also images, audio, and video content by members of the far-right in the country. This aligns with trends in the exploitation of AI’s multimodal capabilities by extremist groups globally, evidenced by the four sources above.

Comparing TDP’s 2024 research output around generative AI, with findings from the first quarter of 2023 surface several significant changes, in addition to trends that remain as core threats to the country’s liberal democratic firmament.

Key differences year-on-year included,

• Increased Sophistication: This year’s research (to date) shows a marked increase in the sophistication of AI tools being used by extremists, and far-right networks in the country. While early 2023 content, and commentary focused on text generation, and simple image creation, 2024 sees the adoption of more advanced technologies like voice cloning and the instrumentalisation of multimodal AI systems.
• Scale of Adoption: 2024’s research indicates a broader adoption of (generative) AI tools. What was experimental or embryonic in early 2023 has become more mainstream, and entrenched within the country’s far-right networks including all of the disinformation ecologies studied, in addition to anti-vaxx, anti-government, and anti-establishment networks that now seamlessly exchange content, and are inextricably entwined with neo-Nazi, accelerationist, fascist, and far-right networks.
• Targeted Disinformation: This year’s research highlights more targeted use of AI for disinformation, particularly in relation to specific events like the official COVID-19 inquiry. This represents a shift from the more general experimentation seen in early 2023, with no specific individual or institutional target (though there were, even then, exceptions to this).
• Voice and Audio Manipulation: 2024’s research introduces more significant, grounded concerns about voice cloning and audio manipulation, which were not prominent in the early 2023 findings because the technology wasn’t as advanced. This represents new frontiers in synthetic media creation, rendering as front-door challenges (for Aotearoa New Zealand) that in 2023’s analysis was projected as risks, and threats that wouldn’t be realised for some years.
• Impact on Democratic Integrity: While early 2023 research hinted at potential impacts on democratic processes, the 2024 research provides concrete examples and analysis of how AI-generated content has contributed to undermining public consultation mechanisms.
• Regulatory Concerns: This year’s research more explicitly addresses the lack of regulatory frameworks in New Zealand to deal with AI-generated disinformation, reflecting a growing awareness of this gap, but no visible movement in domestic, fit-for-purpose policymaking.
• Cross-Platform Strategies: 2024’s study shows more sophisticated cross-platform strategies for disseminating AI-generated content, compared to the more platform-specific approaches noted in early 2023. There are more involved, and those who are show a significantly higher awareness of how (generative) AI can aid the production, and dissemination of content – which is to say, the seed, and spread of violent extremist material.

More general year-on-year trends include the fact that extremist and far-right groups in New Zealand continue to be early adopters of new AI technologies. They are the first to notice new platform developments, and product releases. The more tech savvy individuals in networks present prompts, and resulting content for others to build on, and emulate. The rapid evolution of generative AI also means that the resulting content is at a higher fidelity, even in its most violative forms. While new forms of AI-generated content have emerged, the creation of text and images remains a significant focus, as it was in early 2023. The use of open-source AI models (without any guardrails or safety measures), noted in early 2023, continues to be a trend in 2024, allowing for greater customisation, and evasion of content moderation. The creation of synthetic media targeting public figures, particularly politicians, remains a consistent theme from the first quarter of last year, to this year. The use of generative AI (and especially LLMs) to propagate conspiracy theories, and disinformation, particularly around COVID-19, remains a persistent issue from 2023 to 2024. Alongside the evolution of platforms like ChatGPT, and a plethora of open-source models on Hugging Face, the sophistication of far-right narratives based on or enhanced by generative AI shows clear growth. Individuals with poor native English language skills, and grammar – evidenced by their interactions on domestic disinformation ecologies studied by TDP – go on to produce polished texts defined by dangerous speech, and targeted harms. Telegram, and other alternative platforms continue to be favoured by extremist groups for sharing AI-generated content, as noted in both periods of research.

2024 registers an even greater potential for AI-generated content to erode public trust in official information sources, including the mainstream media. Finally, the difficulty in distinguishing AI-generated content from human-created content, highlighted in early 2023, is an even more significant challenge in 2024.

Some significant, and deeply worrying developments included,

• Voice Cloning Technology: TDP identified the use of advanced voice cloning technology, such as Prime Voice AI, by some of Aotearoa New Zealand’s leading neo-Nazis, and far-right activists, raising concerns about the potential for highly convincing audio-based disinformation.
• Holocaust Denial, Distortion and AI: The research uncovered unprecedented, multiple instances of AI being used to generate content that distorted, denied or minimised the Holocaust. Specific generative AI platforms, and examples distributed on Telegram were found to produce text that trivialises the Holocaust, with the weaponisation of historical revisionism celebrated by the country’s far-right on disinformation ecologies. Content now regularly features visual imagery, and shows forays into video.
• Targeted Synthetic Media: TDP observed an increase in the creation and distribution of AI-generated images targeting specific political figures, particularly former PM Jacinda Ardern. These dehumanising, denigrating images, and memes, often shared on platforms like Facebook and Telegram, are designed to provoke, promote, and perpetuate extreme harms, abusive narratives, and violative commentary. Extremists are increasingly using AI to generate content across multiple modalities – text, image, audio, and potentially video – creating more immersive, and convincing disinformation narratives.
• AI-Generated Submissions to Public Inquiries: The research highlighted the potential for AI to be used to flood public consultation processes, such as the Royal Commission of Inquiry into the COVID-19 response, with seemingly unique but artificially generated submissions. This poses a significant threat to the integrity of democratic processes, including public consultations run/hosted by Parliament’s public consultation, and petition platforms.
• Cross-Platform Dissemination: Telegram is a key hub for the distribution of extremist AI-generated material. TDP identified platforms like Gab’s AI tools, which operate without content moderation safeguards, as new vectors for the creation and spread of extremist content, including neo-Nazi propaganda.
• Sophisticated Prompt Engineering: TDP observed that extremists are becoming more adept at prompt engineering, allowing them to bypass content moderation systems and generate more nuanced, pointed, and persuasive disinformation.
• AI-Enhanced DaaS (Disinformation as a Service): The research suggests that AI is enabling more sophisticated and scalable disinformation campaigns, potentially leading to the evolution of DaaS models tailored to country, community, and context. This dovetails into parallel research looking at growing signatures of foreign malign influence operations (FMIO), and influence operations (IO) in general, most evident in the run-up to 2023’s general election.
• Erosion of Trust in Public Institutions: We bring out how the proliferation of AI-generated content, engaged with every day on disinformation ecologies, is contributing to a broader, and faster erosion of trust in public institutions, government, official information sources (like NIWA), and mainstream media in Aotearoa New Zealand.
• Lack of Regulatory Preparedness: Over 2024, we’ve more consistently highlighted the absence of specific regulatory frameworks to address the challenges posed by AI-generated disinformation, leaving the country increasingly vulnerable to its effects. These effects, read in the context of our other research, have intergenerational consequences.
• Impact on Māori and Pasifika Communities: Since 2023, but especially in 2024 – in the context of vaulting anti-Māori, and anti-Te Tiriti sentiment – TDP has raised concerns about the potential for AI-generated disinformation to disproportionately affect Māori, and Pasifika communities, exacerbating structural inequalities, and inequity. AI generated visual material (images, and memes) targeting Māori in general, and especially wāhine Māori in positions of power, or public authority (like academics, and journalists) has been especially horrific, and widespread.

TDP’s research, while focused on Aotearoa New Zealand’s socio-political/socio-technological context, has consistently identified trends and concerns that have later been echoed in global reports, including the four above. By providing detailed, grounded, and gendered analysis of generative AI platform, and product affordances in the hands of a highly motivated, inter-connected, and extremely agitated far-right in Aotearoa New Zealand, TDP’s research insights both anticipates, and complements the broader international research landscape on the dangers of AI to democratic, institutional, and electoral integrity, aside from growing threats to national security, and social cohesion.

To wit, as noted in our situation report from 9 January 2023,

Instead of looking at the simplistic copy/paste model of AI-generated text, what TDP sees as an unprecedented risk to Aotearoa NZ’s democratic fabric (in the first instance, and over 2023) is the use of ChatGPT (and related AI-tools, captured in last week’s situation reporting) to create seed content which disinformation producers can then use to hone their output, from scripts informing video productions, to speeches, press releases, photos, memes, and podcasts. ChatGPT generated text can provide narrative frames, ideas, new words, and a turn of phrase which can augment existing content production, and a more limited of subject matter expertise, and the English language. AI-generated texts (written, and visual) are also free, or cost very little money to produce, and iterate on at-scale. From late-2021 (after the L4 lockdown), and in a more sustained, and significant manner over 2022, we have studied the development, and promotion of deep-fakes anchored to PM Ardern, and manipulated media around others in government, academia, mainstream media, and civil society. Though not all domestic disinformation producers are convinced of ChatGPT’s potential, those who are signalling, and using it, are growing (as flagged in our situation reports).

Our in-depth captures of these developments – which have already contributed to significant harms including rampant misogyny, and the normalisation of hate against specific individuals, and community – have also spelt out the path-dependency of harms, which will be accelerated by the development, adoption, and adaptation of ChatGPT, and other AI-tools.

The resulting threats to Aotearoa NZ’s shared reality, which is the basis of social cohesion, and in turn, the foundation of, inter alia, democratic stability, and electoral integrity are historically unprecedented. It is unclear to TDP the degree to which domestic institutions, including but not limited to Crown entities, are aware of these new challenges. For example, TDP foresees significant, sustained harms against Māori, Pasifika and other minority peoples, the LGBTIQ+ community, and women because of AI-generated harms which will be virtually indistinguishable, and arguably even more compelling, viral, and widespread than human-generated content, and digital vitriol. What we are on the cusp of (globally, and domestically) is the industrial production of disinformation in forms, frames, and formats indistinguishable from factual, robust, evidence-based, trusted, and truthful content, narratives, and even scientific analysis.

No other official, academic, or civil society entity in Aotearoa New Zealand at the time flagged these risks, trends, and content signatures which, over the past year and a half, have significantly grown in scope, sophistication, and scale.

Everything Gilbert noted in the Wired magazine article based on MEMRI’s research, that in turn maps on to research by ADL, and ISD is present in Aotearoa New Zealand. The realisation of this, however, shows significant variance even within government, even as adversarial innovation threatening the country’s democracy grows at pace.

###

Image courtesy Freedom House, and The Repressive Power of Artificial Intelligence.