arXiv:2605.30365v1 Announce Type: cross Abstract: Retrieval-augmented text-to-music (TTM) systems augment underspecified user prompts using captions retrieved from a music caption dataset. This design introduces an integrity dependency on the music knowledge database. We show that an attacker can poison the database by injecting a small number of crafted music captions, causing the system to retrieve malicious captions that bias prompt augmentation and steer generation away from the user's intended function, without modifying the user prompt, retriever, or generator. To achieve the music capti
The proliferation of retrieval-augmented generation (RAG) models makes them increasingly susceptible to data poisoning attacks on their external knowledge bases, as demonstrated by this new research.
This highlights a significant vulnerability in emergent AI systems that rely on external data, posing integrity and control risks that could lead to widespread system manipulation or malfunction.
The integrity of the underlying data used by retrieval-augmented AI systems becomes a critical attack surface, demanding new security paradigms beyond prompt manipulation.
- · Cybersecurity firms
- · AI ethicists
- · Data verification services
- · AI developers not prioritizing data integrity
- · Users of poisoned AI systems
- · Generative AI platforms
Retrieval-augmented AI systems become untrustworthy or unusable if their knowledge bases are compromised.
This could lead to a 'data integrity crisis' where the provenance and quality of training and augmentation data become paramount.
Nations or strategic actors might weaponize data poisoning to undermine foreign AI infrastructure or subtly steer narratives.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI