Preethi Lahoti, Kevin Robinson, Bhaktipriya Radharapu, L. Aroyo
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
Adversarial testing of large language models (LLMs) is crucial for their safe and responsible deployment. We introduce a novel approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. We call it AART AI-assisted Red-Teaming -an automated alternative to current manual red-teaming efforts. AART offers a data generation and augmentation pipeline of reusable and customizable recipes that reduce human effort significantly and enable integration of adversarial testing earlier in new product development. AART generates evaluation datasets with high diversity of content characteristics critical for effective adversarial testing (e.g. sensitive and harmful concepts, specific to a wide range of cultural and geographic regions and application scenarios). The data generation is steered by AI-assisted recipes to define, scope and prioritize diversity within the application context. This feeds into a structured LLM-generation process that scales up evaluation priorities. Compared to some stateof-the-art tools AART shows promising results in terms of concept coverage and data quality. Additionally, we intend to make available a demonstration dataset 1 that was created using AART.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2311.08592")
get_code_for_paper("2311.08592")
have("2311.08592")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2311.08592.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2311.08592)
Connect an agent — have() is free.