CulturalRiddles 以 6,200 道跨文化謎語評估 28 種大語言模型

Authors

My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego MMS Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy, Akhil Eppa, Silvia Fernandez, Alex Gimei, Arya Hariharan, Christian Hoang, RamMohanRao Kadiyala, Diana Kozachek, Peerat Limkonchotiwat, Ankita Maity, Di Mo, Ayush Munot, Aylin Naebzadeh, Anastasiia Nikitina, Dina Pisarevskaya, Divas Poudel, Jebish Purbey, Md Rabius Sany Apu, Ahmad Mufli Ramadhan, Ralitza Nikolova Rizova, Diogo M. Serrano-Miranda, Vivek Gopalkrishna Silimkhan, Luis Fernando Solis Navarro, Kaiser Sun, Arya Suneesh, Manaswitha Sunkara, Tejaswi Sunkara, Jonathan E Swindell, Hanna Yukhymenko, Jingyi Zheng, Esther Adenuga, Ilia Afanasev, Wafa Aissa, Lisa Alazraki, Lauren C Altomare, Priscilla Amondi Amuok, Emile Anand, Gabriel Assis, José Matías Avalos, Silvia Lorena Avila, Zakaria Baannou, Niyati Bafna, Yash Bagla, Sarah Bahrami, Bontu F Balcha, Vanya Bannihatti Kumar, Irene Baucells, Abdumutalib Begmuratov, Naman Bhatia, Souvik Bhattacharyya, Isabel Blanco Ferreiro, Felipe Bochini, Sayuru Bopitiya, Angana Borah, Tulika Bose, Olena Bugaiova, Juliana Carvalho, Nancy F Chen, Stephy Chen, Rochelle Choenni, Jayanth Krishna Chundru, Leandro A Cossio, Nebojša Damnjanović, Ona de Gibert, Marco De Santis, Pedro Delgado, Kaustubh Dholé, Adnan El Assadi, Wesam El-Sayed, Nadine M Elframawy, Oumayma Essarhi, Naome A Etori, Reuben Chagas Fernandes, Saanidhi Pravin Gade, Javier García Gilabert, Álvaro González-Escapa, Bin Hu, Kaiyu Huang, Abderrahmane Issam, Danka Jokić, Roy Ka-Wei Lee, Haichang Li, Chen Cecilia Liu, Zhengyuan Liu, Cedric Lothritz, Houman Mehrafarin, Antonio Miranda-Escalada, Fengran Mo, Mehrnaz Mofakhami, Wafaa Mohammed, Felipe Moreno-Vera, Prahitha Movva, Jannatul Nayem, Van Ngo, Joy Olusanya, Marc Pàmies, Avinash Kuney Pandey, Yurii Paniv, Fred Philippy, Vladislav Poritski, Dikshya Poudel, Joanna Radoła, Sara Rajaee, Swati Rajwal, Prerana Rane, Muhammad Ravi Shulthan Habibi, Lucas Resck, Shubhashis Roy Dipta, Ananya Sahu, Eduardo Sánchez, Borja Sánchez Clemente, Aleix Sant, Nobin Sarwar, Istiak Shihab, Santhoshini Pranathi Singaraju, Jane Smith, David Stap, Emily Z Su, Ananth K Suresh, Sajag Swami, Ruiyang Tan, Nazia Tasnim, Khalid Tounoussi, Patteera Triamamornwooth, Muhammad Saad Umar, Shankar Venkitachalam, Oksana Volchek, Lingbo Wang, Debela Desalegn Yadeta, Rushikesh Zawar, Jinghan Zhang, Mike Zhang, Weihua Zheng, Marzieh Fadaee, Akhil Arora, Károly Boczka, Julia Kreutzer

Abstract

As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures(45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities.
We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.

RESEARCH BLOG

I Speak Every Language but Know No Culture. What Am I?

The art of riddles

We can probably say that riddles are a universal art. In Tamil Nadu, India, children grow up with tales of King Vikramadithan and the Vetalam, a spirit that poses riddles as the king carries it on his back. In Europe, Greek mythology offers the Sphinx's famous riddle. In China, families have long gathered to solve lantern riddles during the Lantern Festival. In Uganda, Luganda riddling begins with the call “Kooyoo!”, to which everyone responds “Lya!” (“Bring it on!”).

Yet riddles take remarkably different forms across cultures, and understanding them often requires familiarity with the traditions that shape them. Knowing a language without its culture is like reading a restaurant menu without ever tasting the food: you can name every ingredient, yet miss what the meal means to the people eating it. AI models are very good at reading the menu, but how well do they understand the meal?

Try this one, written by our co-author Julia: “Name of the German 9/11.” (original: “Name des deutschen 9/11”). If you grew up in Germany, you know that Germans write the day before the month, so 9/11 is the 9th of November, a date heavily tied to German history, including the night the Berlin Wall fell. Germans have a name for it: Schicksalstag, the day of fate. If you didn’t grow up there, you probably thought of New York. We gave this riddle to 28 AI models: Seven of them did think of New York, six got as far as 9 November but not its name, only one got it right.

How it started

It began with one language. Hungarian diplomat, Boczka Károly, noticed that in spite of claims to speak his language, AI models would often not capture his culture well, so he set out to systematically study this shortcoming. He wrote original riddles in Hungarian that lean on local culture, references, and colloquial knowledge, and tested them on a range of AI models. The models struggled and tended to make random guesses or hallucinate, rather than admit their uncertainty. That raised a bigger question: was this simply a Hungarian phenomenon, or a global one?

To find out, we assembled a global open science collaboration of 194 volunteers representing 62 cultural communities and spent six months collecting riddles, passing them to AI models to answer, and grading their answers.

A benchmark built on lived experience

Many existing AI benchmarks assess cultural knowledge, from everyday breakfast foods and historical events to traditional holiday customs. Yet they often treat culture as a collection of static facts: things you can memorize or look up with a quick web search.

But knowing facts about a culture is not the same as understanding how people within it make sense of the world, just as memorizing mathematical formulas does not mean knowing how to solve a math problem. We therefore needed a test that asks models to connect seemingly unrelated pieces of cultural knowledge and put them to use. This led us to riddles, and to a different kind of cultural benchmark: one rooted in lived experience.

To build it, we asked open science collaborators to write original riddles about their own cultures, in their own languages. The result: 6,200 riddles in 51 languages from 45 countries. For some countries, multiple communities formed to write about their cultures each in their preferred language, e.g. 10 different communities for India. At the same time, some languages are shared by communities across different countries or even continents, such as riddles written in Spanish from Spain, Cuba, Peru or Argentina. The benchmark also includes languages absent from earlier cultural AI benchmarks, such as Buginese, Mongondow, Konkani, and Sudanese Arabic.

Each community covered eight shared topics, e.g., Everyday Life, History, Art, allowing comparisons across cultures, and chose two “wildcard” topics of its own, allowing us to capture both shared interests across cultures and themes distinctive to each community, with local themes ranging from public bath etiquette in Japan to hawker food and Singlish in Singapore. Culture also shaped the clues themselves: Tamil riddles drew on the emotional landscapes of ancient poetry; Persian riddles used visual wordplay; Lumasaba riddles reflected the slopes of Mount Elgon; and Bangladeshi and Syrian riddles drew on historical memory and survival.

We sent these riddles to 28 AI models, including proprietary, open, and regional-focused models of various sizes. We collected their answers and shared them back to our community of native speakers to rate their answers. Altogether, they reviewed around 68k unique model answers, recognizing valid responses that rigid word matching would have missed, and particularly looking out for hallucinations. These expert annotations were then also used to carefully tune and validate a LLM jury to judge quality where expert labels were missing.

Where models fall short

Good overall scores hide big gaps: The best model got 79% of all the riddles right. But that number hides several regional gaps. Among the riddles from Luxembourg, many models easily identified “the black party” as the Christian Social People’s Party. On riddles in Lumasaba from Uganda, and in Mongondow from Indonesia, though, no model got more than 22% right. And none of the 28 models could solve this Afaan Oromo riddle from Ethiopia: “A narrow path on top of a mountain,” for which the correct answer is “an eyebrow”.

Translation helps, but not much: To test whether the limitation just lies in the fact that models might not cover the respective languages sufficiently well, we gave five of the models English translation of riddles from five of the hardest-to-solve communities. Their score went up from an average of 22% to 32%, an improvement, but most of what makes these riddles challenging survives translation. The language barrier thus explains a small part of the gap, not most of it.

Models are more likely to guess than say “I don’t know”: We told the models they could answer “I don’t know” if they didn’t have the answer. Nevertheless, they did so just 0.7% of the time. In contrast, about 31% of the answers our reviewers checked were simply made up, with various levels of creativity. This confirmed that the hallucination phenomenon we had seen for the original Hungarian riddles remains across cultures.

They reach for the familiar answer: A Kenyan riddle asks for the “home of the Lunatic Express”. Several models named the railway that carries that nickname. None named the answer: the Railway Museum in Nairobi. Similarly, asked as part of the German riddles for a game where people are “fighting for a ball, but in another element”, 18 of the 28 models named “water polo”. The answer was underwater rugby, a sport invented in Germany. Even though we gave the models the context of the culture to ground their answers in, they often defaulted to answers that were grounded in other cultures, those more “available” to them.

A new frontier

AI models keep getting better at representing facts that are available in writing on the internet. These riddles test something else: what people know from growing up in a particular place. This is a frontier current models have not yet crossed, and this benchmark shows where they fall short, community by community.

It’s also built to keep growing. The riddles, code, demo, and an interactive dashboard are open, and anyone can add riddles from their own culture through a checked contribution process. Read our full technical report to get all the details of the benchmark and how models scored on it.

If your culture isn't there yet, we’d love your riddles. We’re betting the AI won’t get them.

Explore the dataset

Join our talk

Research Blog Authors: Vinod Anbalagan, Tianyi Hu, Madeline Smith, Julia Kreutzer