Samsung Electronics has announced the launch of TRUEBench, a new benchmark developed by Samsung Research to assess AI productivity in real-world scenarios. The tool is designed to evaluate how large language models (LLMs) perform in workplace applications, using a set of metrics that reflect actual enterprise tasks.
TRUEBench measures AI performance across 10 categories and 46 sub-categories, focusing on tasks such as content generation, data analysis, summarization, and translation. It incorporates diverse dialogue scenarios and supports multilingual conditions to provide realistic assessments.
Paul (Kyungwhoon) Cheun, CTO of the DX Division at Samsung Electronics and Head of Samsung Research, stated: “Samsung Research brings deep expertise and a competitive edge through its real-world AI experience. We expect TRUEBench to establish evaluation standards for productivity and solidify Samsung’s technological leadership.”
Existing benchmarks for LLMs have been criticized for focusing mainly on overall performance, being limited to English-language evaluations, and relying on single-turn question-answer formats. These limitations reduce their ability to reflect actual work environments. TRUEBench addresses these issues by including 2,485 test sets across 10 categories in 12 languages—Chinese, English, French, German, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish and Vietnamese—and supporting cross-linguistic scenarios.
The benchmark uses test sets ranging from brief prompts of eight characters to document summarizations exceeding 20,000 characters. This range allows it to evaluate both simple requests and complex tasks typical in workplaces.
To ensure accurate assessment of AI model responses, TRUEBench uses criteria developed collaboratively by human annotators and AI systems. Human experts create initial evaluation standards; then AI reviews them for errors or unnecessary constraints before humans refine them further. This process is repeated until precise standards are achieved. The resulting criteria enable automatic evaluation that minimizes subjective bias and ensures consistency.
All conditions must be met for an AI model to pass each test set within TRUEBench. This approach enables detailed scoring across different tasks.
TRUEBench’s data samples and leaderboards are available on Hugging Face’s global open-source platform at https://huggingface.co/spaces/SamsungResearch/TRUEBench. Users can compare up to five models simultaneously and view data on average response lengths alongside performance scores.