SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2408.03837 · EMNLP · 2024

WALLEDEVAL: A Comprehensive Safety Evaluation Toolkit for Large Language Models

Soujanya Poria, Rishabh Bhardwaj, Prannaya Gupta, Le Yau, Hao Low, I-Shiang Lee, Hugo Lim, Yu Teoh, Jia Koh, Win Dar, Liew, Rajat Bhardwaj, and 1 more

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

WALLEDEVAL is a comprehensive AI safety testing toolkit designed to evaluate large language models (LLMs). It accommodates a diverse range of models, including both open-weight and API-based ones, and features over 35 safety benchmarks covering areas such as multilingual safety, exaggerated safety, and prompt injections. The framework supports both LLM and judge benchmarking and incorporates custom mutators to test safety against various text-style mutations, such as future tense and paraphrasing. Additionally, WALLEDEVAL introduces WALLEDGUARD, a new, small, and performant content moderation tool, and two datasets: SGXSTEST and HIXSTEST, which serve as benchmarks for assessing the exaggerated safety of LLMs and judges in cultural contexts. We make WALLEDEVAL publicly available at https: //github.com/walledai/walledeval.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2408.03837")
get_code_for_paper("2408.03837")
have("2408.03837")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2408.03837.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2408.03837.svg)](https://syntology.ai/paper/2408.03837)

Connect an agent — have() is free.