Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusions

Abstract

The effectiveness of in-context learning (ICL) varies significantly across models and tasks, posing challenges for practitioners to determine when ICL reliably improves performance. Current evaluation approaches, reliant on performance changes after applying ICL, suffer from low reliability, poor attribution, and impracticality in data-insufficient scenarios. We propose the Learning-to-Context Slope (LCS), a novel metric that quantifies ICL effectiveness by modeling the slope between learning gain (loss decrease from demonstrations) and contextual relevance (demonstration-input relevance). LCS addresses key limitations of performance-based metrics: (i) it captures loss changes even when outputs are incorrect; (ii) its formulation attributes ICL failures to weak contextual alignment or strong output calibration; (iii) it minimizes reliance on labeled data via synthetic evaluation. Experiments show that LCS can better reflect ICL effectiveness when performance change fails, demonstrating its effectiveness.