June 2025 – Welo Data has published new research presenting a bilingual evaluation framework aimed at measuring how large language models (LLMs) perform in complex causal reasoning tasks across multiple languages. The study, titled “Diagnosing Performance Gaps in Causal Reasoning via Bilingual Prompting in LLMs,” highlights model behavior under more realistic multilingual conditions, using blended-language prompts that better reflect real-world applications.

The evaluation covered eight LLMs developed by four major companies, using over 70,700 prompts across six languages—English, Spanish, Japanese, Korean, Arabic, and Turkish—and four causal question types: causal discovery, confounder identification, language variation, and norm violation.

Expanding Beyond Monolingual Benchmarks

Welo Data researchers note that existing causal reasoning benchmarks often lack linguistic diversity and task complexity. To address this, they constructed a dataset of narrative-based prompts developed by professional analysts with advanced degrees and domain experience, aiming to create more representative testing conditions.

Key Observations

• Bilingual prompts led to a consistent accuracy decrease, with performance dropping an average of 4.6% compared to monolingual prompts.

• Models exhibited a recency bias, prioritizing the language of the question over that of the story, which affected both accuracy and reasoning language.

• A negative response bias was noted in binary causal tasks, where models tended to reject causal claims, especially when the correct answer was “yes.”

• Larger LLMs outperformed smaller ones in complex scenarios, particularly in identifying confounders and evaluating norm violations.

Implications for Real-World Use

“Multilingual evaluation typically relies on monolingual prompts translated into multiple languages,” said Dr. Abigail Thornton, Head of Research at Welo Data. “But actual use cases often involve bilingual or multilingual inputs. Our findings show that structure and language pairing can meaningfully influence model behavior.”

Dr. David Harper, lead author of the study, added: “This kind of evaluation goes beyond surface-level performance and reveals how LLMs respond to complexity—an important factor for global applications.”

According to the researchers, bilingual prompts may expose reasoning weaknesses that are less visible in standard benchmarks. These findings could help developers improve model consistency and generalization across multilingual deployments.

The research is part of Welo Data’s broader initiative to support transparent evaluation practices through its Model Assessment Suite, which tests LLMs using domain-specific scenarios in multiple languages.

The full study is available at welodata.ai.

About Welo Data

Welo Data, a division of Welocalize, specializes in AI training data solutions. With a global network of over 500,000 contributors and a focus on quality, cultural relevance, and bias reduction, the company supports AI development through services such as data annotation, model enhancement, and relevance assessment. Its proprietary NIMO framework helps ensure high-quality outputs through workforce assurance and data validation protocols.