Code for the Paper "CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery".

For more details, please refer to the project page with dataset exploration and key results: https://csbench.github.io/.

🔔 If you have any questions or suggestions, please don't hesitate to let us know. You can comment on the Email, or post an issue on this repository.

[Webpage] [Paper] [Huggingface Dataset] [Leaderboard] [Result Explorer]

If you find CS-Bench useful for your your research and applications, please kindly cite using this BibTeX:

@article{song2024cs,
  author       = {Xiaoshuai Song and
                  Muxi Diao and
                  Guanting Dong and
                  Zhengyang Wang and
                  Yujia Fu and
                  Runqi Qiao and
                  Zhexu Wang and
                  Dayuan Fu and
                  Huangxuan Wu and
                  Bin Liang and
                  Weihao Zeng and
                  Yejie Wang and
                  Zhuoma Gongque and
                  Jianing Yu and
                  Qiuna Tan and
                  Weiran Xu},
  title        = {CS-Bench: {A} Comprehensive Benchmark for Large Language Models towards
                  Computer Science Mastery},
  journal      = {CoRR},
  volume       = {abs/2406.08587},
  year         = {2024},
  url          = {https://doi.org/10.48550/arXiv.2406.08587},
  doi          = {10.48550/ARXIV.2406.08587},
  eprinttype    = {arXiv},
  eprint       = {2406.08587},
  timestamp    = {Tue, 09 Jul 2024 17:23:21 +0200},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2406-08587.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

• 💥 News 💥
• 👀 About CS-Bench
• 🏆 Leaderboard on CS-Bench (English) 🏆
• 🏆 Leaderboard on CS-Bench (Chinese) 🏆
• 📊 CS-Bench Dataset
• 📝 Evaluation on CS-Bench
• 📜 License
• 🤝 Contributors

• [2023.7.11] We contribute our dataset to OpenCompass at https://hub.opencompass.org.cn/dataset-detail/CS-Bench.
• [2023.6.14] Our paper is now accessible at https://arxiv.org/abs/2406.08587.
• [2024.6.13] Our dataset is now accessible at Huggingface Datasets.
• [2024.6.12] Our project homepage can be accessed at https://csbench.github.io/.

Computer Science (CS) stands as a testament to the intricacies of human intelligence, profoundly advancing the development of artificial intelligence and modern society. However, the current community of large language models (LLMs) overly focuses on benchmarks for analyzing specific foundational skills (e.g. mathematics and code generation), neglecting an all-round evaluation of the computer science field. To bridge this gap, we introduce CS-Bench, the first bilingual (Chinese-English) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 5K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of over 30 mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvements, including knowledge supplementation and CS-specific reasoning. Further cross-capability experiments show a high correlation between LLMs' capabilities in computer science and their abilities in mathematics and coding. Moreover, expert LLMs specialized in mathematics and coding also demonstrate strong performances in several CS subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM applications in the CS field and paving new avenues in assessing LLMs' diverse reasoning capabilities.

Overview diagram and statistics of CS-Bench.

For more details, you can find our project page here and our paper here.

The evaluation instructions are available at 📝 Evaluation on CS-Bench.

To submit your results to the leaderboard, please send to this email with your result json files.

The leaderboard of LLMs on CS-Bench (EN) .

Some notations in the table:

• Domains

  • DSA: Data Structure and Algorithm
  • CO Computer Organization
  • CN: Computer Network
  • OS: Operating System

• Types:

  • Klg: knowledge-type
  • Rng: reasoning-type
  • Avg: Average

The leaderboard of LLMs on CS-Bench (CN) .

Option 1: Use Step 1 to construct the reasoning prompt, replace Step 2.1 with your own inference method to obtain the model's output, and use Steps 3 and 4 to get the model's scores.

Option 2: Use Step 1 to construct the reasoning prompt, use the vllm reasoning we provide in Step 2.1 (requires environment setup) to obtain the model's output, and use Steps 3 and 4 to get the model's scores.

git clone https://github.com/csbench/csbench

Fill in your file path in create_input.py and create English(default) or Chinese prompt by running the functions create_en_prompt and create_cn_prompt.

You may use inference engine such as vLLM or SGLang to generate your model answers. We will provide our inference code in the near future.

Please ensure that your answer is saved in JSONL format and retains all keys from the original dataset.

vLLM is a fast and easy-to-use library for LLM inference and serving.

Visit our documentation to get started.

• Installation
• Quickstart
• Supported Models

# (Recommended) Create a new conda environment.
conda create -n myenv python=3.9 -y
conda activate myenv
# Install vLLM with CUDA 12.1.
pip install vllm

Fill in your model path, data save path and other parameters in run_csbench.sh and run this script.

bash run_csbensh.sh

If you want to evaluate questions in all formats.Fill in your API in test_call_llm.py Run the command to generate judgments with GPT:

python gen_judgment.py --judge_with_gpt 1 your_file_path

If you only want to evaluate questions in 'Multiple-choice' and 'Assertion'. Run the command to generate judgments without GPT:

python gen_judgment.py --judge_with_gpt 0 your_file_path

Output model win scores. Run the command to generate judgments without GPT:

python show_result.py your_file_path

Our dataset are distributed under the CC BY-NC 4.0 license.

Here are the key contributors to this project:

Xiaoshuai Song, Muxi Diao, Guanting Dong, Zhengyang Wang, Yujia Fu, Runqi Qiao, Zhexu Wang, Dayuan Fu, Huangxuan Wu, Bin Liang, Weihao Zeng, Yejie Wang, Zhuoma GongQue, Jianing Yu, Qiuna Tan, Weiran Xu.

PRIS-NLP Research Group , Beijing University of Posts and Telecommunications.