A new Arabic meme benchmark found a gap between spotting hate and assigning more specific labels. The top macro-F1 scores, which average results across labels, were 0.823 for a hateful-or-not task and 0.419 for a separate ten-label task.
ArGuard’s overview paper reports 5,000 manually annotated Arabic memes, including a 500-item blind test set. Trained native Arabic speakers used bilingual guidance that accounted for dialect and cultural cues.
In the first meme task, systems marked each example hateful or not hateful. The second required one or more of ten labels, including types of attack as well as non-hateful humor and sarcasm. Both leaderboards used macro-F1, which averages performance across labels so rare categories count as much as common ones. The figures are therefore not accuracy rates.
The authors point to sparse labels and a shift in the test set as reasons the detailed task was harder. One category, Exclusion, appeared only 17 times in the full dataset and three times in the blind test. Compared with training, that test also contained fewer hateful memes and more examples labeled Humor, Sarcasm and Contempt.
For teams evaluating Arabic moderation, the result argues for testing more than a binary hateful-or-not decision. The benchmark measures systems on one fixed test, however; it does not establish how a deployed service would perform across Arabic dialects, platforms or changing events.
Sources
- ArGuard Shared Task paper (v2, September 25, 2026)
- ArGuard task website
- ArGuard data and evaluation repository
- Photo: Binter, Wikimedia Commons; released to the public domain worldwide.
