arXiv:2603.19127v2 Announce Type: replace Abstract: As Spoken Language Models (SLMs) integrate speech and text modalities, they inherit the safety vulnerabilities of their LLM backbone while introducing an expanded attack surface. SLMs have been previously shown to be susceptible to jailbreaking, where adversarial prompts induce harmful responses. Yet existing attacks largely remain unimodal, optimizing either text or audio in isolation. We explore gradient-based multimodal jailbreaks by introducing JAMA (Joint Audio-text Multimodal Attack), a joint multimodal optimization framework combining
The rapid deployment and increasing sophistication of Multimodal Large Language Models (LLMs) and Spoken Language Models (SLMs) necessitates a proactive understanding of their vulnerabilities.
Sophisticated actors could exploit these vulnerabilities to manipulate information, bypass safety protocols, or extract sensitive data, posing significant risks to AI systems' integrity and trustworthiness.
The research introduces a 'Joint Audio-text Multimodal Attack' (JAMA) framework, demonstrating a more effective methodology for jailbreaking SLMs by co-optimizing across modalities, which was previously underexplored.
- · Cybersecurity researchers
- · AI safety and ethics developers
- · Organizations developing robust AI defenses
- · Spoken Language Model developers
- · Users relying on SLMs for sensitive tasks
- · Organisations with inadequate AI security protocols
Increased pressure on SLM developers to implement more robust multimodal safety mechanisms against sophisticated adversarial attacks.
Development of new AI red-teaming tools and methodologies specifically designed for multimodal systems.
A potential arms race between AI security and AI exploitation, leading to more complex and resilient AI systems on both sides.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.LG