Mastering games is a hard task, as games can be extremely complex, and still fundamentally different in structure from one another. While the AlphaZero algorithm has demonstrated an impressive ability to learn the rules and strategy of a large variety of games, ranging from Go and Chess, to Atari games, its reliance on extensive computational resources and rigid Convolutional Neural Network (CNN) architecture limits its adaptability and scalability. A model trained to play on a $19\times 19$ Go board cannot be used to play on a smaller $13\times 13$ board, despite the similarity between the two Go variants. In this paper, we focus on Chess, and explore using a more generic Graph-based Representation of a game state, rather than a grid-based one, to introduce a more general architecture based on Graph Neural Networks (GNN). We also expand the classical Graph Attention Network (GAT) layer to incorporate edge-features, to naturally provide a generic policy output format. Our experiments, performed on smaller networks than the initial AlphaZero paper, show that this new architecture outperforms previous architectures with a similar number of parameters, being able to increase playing strength an order of magnitude faster. We also show that the model, when trained on a smaller $5\times 5$ variant of chess, is able to be quickly fine-tuned to play on regular $8\times 8$ chess, suggesting that this approach yields promising generalization abilities. Our code is available at https://github.com/akulen/AlphaGateau.

Poster

Reinforcement Learning Deep learning · graph neural networks

A Benchmark Suite for Systematically Evaluating Reasoning Shortcuts

Samuele Bortolotti, Emanuele Marconato, Tommaso Carraro, Paolo Morettin, Emile van Krieken, Antonio Vergari, Stefano Teso, Andrea Passerini

Poster

Reasoning Models Multimodal LLMs

Benchmarking Trustworthiness of Multimodal Large Language Models: A Comprehensive Study

Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Yifan Wang, Huanran Chen, Xiao Yang, Xingxing Wei, Hang Su, Yinpeng Dong, Jun Zhu

Poster

Multimodal LLMs

Through the Looking-Glass: Tracing Shifts in AI Data Consent across the Web

Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Anis, An Dinh, Caroline Shamiso Chitongo, Da Yin, Damien Sileo, Deividas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kushagra Tiwary, Lester James V. Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Minh Chien Vu, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, Alex Pentland

Poster

Click-Through Prediction

Image2Struct: A Benchmark for Evaluating Vision-Language Models in Extracting Structured Information from Images

Josselin Roberts, Tony Lee, Chi Heem Wong, Michihiro Yasunaga, Yifan Mai, Percy Liang

Poster

Multimodal LLMs Information Extraction

WikiDO: Evaluating Out-of-Distribution Generalization of Vision-Language Models in Cross-Modal Retrieval

Pavan Kalyan Tankala, Piyush Pasi, Sahil Dharod, Azeem Motiwala, Preethi Jyothi, Aditi Chaudhary, Krishna Srinivasan

Poster

Multimodal LLMs Applications · computer vision

SETBENCH: Assessing the Analytical and Semantic Robustness of Language Models

Nicholas Dronen, Bardiya Akhbari, Manish Digambar Gawali

Poster

Alignment and Safety Social aspects · accountability transparency and interpretability

Comprehensive Framework for Curating Speech Datasets and Evaluating ASR Systems: A Case Study for the Polish Language

Michał Junczyk

Poster

Dialogue Systems Applications · speech and audio

A Novel Benchmark for Decision-Making in Uncertain and Competitive Games

Kefan Su, Yusen Huo, Zhilin Zhang, Shuai Dou, Chuan Yu, Jian Xu, Zongqing Lu, Bo Zheng

Poster

Alignment and Safety Multimodal LLMs

SurgicAI: A Fine-grained Platform for Data Collection and Benchmarking in Surgical Policy Learning

Jin Wu, Haoying Zhou, Peter Kazanzides, Adnan Munawar, Anqi Liu

Poster

Multimodal LLMs Healthcare Policy Modeling

Quantifying the Bitter Lesson: How Safety Benchmarks Measure Capabilities Instead of Safety

Richard Ren, Steven Basart, Adam Khoja, Alexander Pan, Alice Gatti, Long Phan, Xuwang Yin, Mantas Mazeika, Gabriel Mukobi, Ryan Kim, Stephen Fitz, Dan Hendrycks

Poster

Alignment and Safety Social aspects · safety

There is No Silver Bullet: Benchmarking Methods in Predictive Combinatorial Optimization

Haoyu Geng, Hang Ruan, Runzhong Wang, Yang Li, Yang Wang, Lei Chen, Junchi Yan

Poster

theory->optimization Deep learning · optimization

SubjECTive-QA: A dataset for the subjective evaluation of answers in Earnings Call Transcripts (ECTs)

Huzaifa Pardawala, Siddhant Sukhani, Veer Kejriwal, Rohan Bhasin, Abhishek Pillai, Dhruv Adha, Tarun Mandapati, Andrew DiBiasio, Agam Shah, Sudheer Chava

Poster

General ML · evaluation Datasets and Benchmarks

Norms for Managing Datasets: A Systematic Review of NeurIPS Datasets

Yiwei Wu, Leah Ajmani, Shayne Longpre, Hanlin Li

Poster

Datasets and Benchmarks Simulation for Driving

OpenCDA-Loop: A Closed-loop Benchmarking Platform for End-to-end Evaluation of Cooperative Perception

Chia-Ju Chen, Runsheng Xu, Wei Shao, Junshan Zhang, Zhengzhong Tu

Poster

General ML · evaluation End-to-End Driving

CryoBench: Datasets and Benchmarks for Heterogeneous Cryo-EM Reconstruction

Minkyu Jeon, Rishwanth Raghu, Miro Astore, Geoffrey Woollard, J. Feathers, Alkin Kaz, Sonya Hanson, Pilar Cossio, Ellen Zhong