跳到正文
LMSYS Blog· LMSYS Arena Team·· 2024-05-02AI 評分61

LMSYS 與 Kaggle 推出人類偏好預測競賽,總獎金 $100,000

LMSYS Kaggle Competition – Predicting Human Preference with $100,000 in Prizes

AI 導讀

LMSYS 與 Kaggle 推出人類偏好預測競賽,參賽者須預測用戶在大型語言模型對戰中的偏好,總獎金為 $100,000。訓練集包含超過 55,000 段真實用戶與 LLM 對話及偏好資料,並已移除個人識別資料;提交方案會在含 25,000 個樣本的隱藏測試集上評估,數據涵蓋超過 70 個前沿 LLM。

正文

Overview

LMSYS and Kaggle are launching a human preference prediction competition! You are challenged to predict which responses users will prefer in head-to-head battles between Large Language Models (LLMs). You’ll work with a dataset from the Chatbot Arena, containing conversations and user preferences across various LLMs. By developing a model that accurately predicts human preferences, you’ll contribute to improving chatbot performance and alignment with user expectations. The training dataset includes over 55,000 real-world user and LLM conversations and user preferences, with personally identifiable information removed. Your solution submission will be tested on a hidden test set of 25,000 samples. The dataset includes real-world conversations with over 70 state-of-the-art LLMs, such as GPT-4, Claude 2, Llama 2, Gemini, and Mistral models. Click here to join the competition and download the dataset!

Background

Current LLM benchmarks often fail to capture real-world LLM usage, resulting in a discrepancy between model performance and user satisfaction. Platforms like Chatbot Arena allow users to submit questions and vote on preferred responses; however, the potential of this data has been largely untapped in developing models that predict and optimize for user preferences at scale. Predicting user preferences is essential for creating human-aligned conversational AI that delivers a satisfying user experience. Successful models could enable language models to dynamically adapt their output based on individual preferences across different contexts and use cases. Moreover, this competition aims to uncover the factors that drive user preferences beyond objective correctness. Many user questions are open-ended, and we have already found a correlation between user preference and subjective qualities like conversationality. This could also be one of the best testbeds for reward modeling in your RLHF algorithms.

Competition Details

The competition will run until August 5th, with a total prize of $100,000, featuring a $25,000 prize for 1st place, $20,000 prizes for 2nd through 4th places, and a $15,000 prize for 5th place. This is your opportunity to contribute to the advancement of human-aligned language models while gaining valuable insights into human preferences and decision-making. These insights could provide value to both the computer science and psychology communities, shedding light on the factors that shape human preferences in conversational AI.

來源:LMSYS Blog · lmsys.org