Evaluating Large Language Models for Antisemitic Incident Classification

The Problem

Most automated tools for detecting antisemitism online have been built to spot hateful language — slurs, threatening rhetoric, coded expressions. The authors argue that this leaves a serious gap. A large share of real-world antisemitic harm does not show up in speech alone. Incidents recorded in civil society reports, news articles, and official records — vandalism, harassment, exclusion from campus events — often contain no recognizable hate terms, but they are exactly the events practitioners need to track.

Halevy et al. propose a different task, which they call hateful event detection. Instead of asking whether the language in a post is hateful, the task asks whether a written report describes a real-world incident of hate-motivated harm, and if so, what kind. The authors illustrate the distinction with a sample text that a popular toxicity-scoring tool rates as only 6.7% toxic — close to "clean" — even though it describes clearly antisemitic graffiti. They argue that monitoring harm at scale requires moving beyond language-level signals to structured classification of what actually happened.

The practical relevance is considerable. Organizations responsible for tracking and responding to antisemitic incidents now receive more reports than human analysts can process.

Approach and Findings

The authors test two large language models — OpenAI's GPT-4o and Meta's Llama-3.2-3B-Instruct — across four datasets:

• AMCHA: 4,410 reports of antisemitic incidents on US college campuses, curated by the AMCHA Initiative. Each is labeled with one of two broad categories (Targeting or Expression) and one of nine more specific types, including Physical Assault, Discrimination, Genocidal expression, Bullying, Denigration, Historical Antisemitism, and Condoning Terrorism.
• ADL-HEAT: 4,522 incidents from across the US, labeled as Harassment, Vandalism, or Assault.
• Synthetic: AI-generated descriptions of non-antisemitic but Jewish-related events (e.g., a Passover seder, a Jewish folk dance class), used to test whether the models wrongly flag harmless content.
• Campus-News: 5,275 articles from five university newspapers (Oct 2022 – Dec 2024), with a 225-article subset hand-coded by experts.

The authors also varied how they asked the models the question — sometimes just providing the incident description, sometimes adding definitions of the categories, sometimes including a worked example. This lets them see which kinds of additional information actually help, and for which kinds of incidents.

On the simple "is this antisemitic?" question, both models perform well. GPT-4o correctly identifies between roughly 91% and 98% of antisemitic incidents (depending on dataset and prompt) and never wrongly flags items from the Synthetic dataset. Llama's false-positive rate is higher and rises from 1.56% to 13.36% when definitions are added — a gap the authors flag as material for real deployment.

The paper's central finding concerns fine-grained classification: the optimal prompting strategy depends on the kind of incident. 

• For rhetoric-oriented incidents (classical tropes, genocidal imagery, conspiracy invocations), adding definitional context yields the strongest gains, lifting the weighted F1 score — a standard measure that balances correct detections against false positives and false negatives — from 56% to 67% (Historical 61.8% → 74.7%; Genocidal 57.2% → 73.6%).
• For action-oriented incidents (physical assault, vandalism, suppression of assembly), in-context examples outperform definitions, improving weighted F1 from 56% to 66%.

The authors read this as reflecting what each kind of prompt activates: stable cultural knowledge for rhetorical categories, situational specificity for action-oriented ones.

The authors note one interpretive caveat. AMCHA treats Targeting and Expression as mutually exclusive and prioritizes Targeting when both apply. This means some apparent model "errors" may reflect a taxonomy choice. For instance, models often label swastika graffiti as Expression (Historical + Genocidal), while AMCHA codes it as Targeting.

Some failure modes persist. Bullying and Discrimination stay below 50% accuracy under all conditions, and the models struggle with indirect references that rely on cultural background knowledge the models fail to draw on. When applied to the full Campus-News corpus, the best-performing setup flags 6.9% of articles as describing antisemitic events — with substantial variation from 0.9% (Daily Illini) to 13.8% (Columbia Spectator). The mix of Expression versus Targeting also varies: 80.6% of flagged incidents in the Michigan Daily are coded as Expression, while the Columbia Spectator is closer to evenly split. On the human-annotated validation subset, the model catches most antisemitic incidents (high recall) but produces a meaningful share of false positives (lower precision), with an overall binary F1 of 35.7%. The authors take this to support using the system as a first-pass filter for human reviewers rather than as a standalone classifier.

Implications

The authors argue that specificity in definitions and annotation guidelines is central to building systems that detect social harm reliably — a claim they extend beyond AI to the human annotation practices that produce training and evaluation data. For practitioners, the implication is that the prompting strategy should be matched to operational purpose: definitional context for monitoring rhetorical content, labeled examples for tracking documented physical incidents.

The per-outlet variation in flagging rates and Expression/Targeting distributions is itself informative. Targeting-heavy reporting may indicate where on-the-ground interventions are needed; Expression-heavy reporting may point toward education-oriented responses. For researchers, the formalization of hateful event detection as a distinct task — together with the public release of AMCHA and Campus-News — provides a foundation for further benchmarking. The authors are explicit that the work does not advocate a normative stance on contested definitions of antisemitism: they restrict their taxonomy to categories on which both the IHRA and Jerusalem Declaration definitions agree, and note that future researchers should adapt categories to their own frameworks. They also identify a generative extension: distinguishing descriptions of hateful events from descriptions of responses to those events (counterspeech, policy actions, solidarity demonstrations).