CosmicAI Researchers and team Aditya Thimmaiah, Jiyang Zhang, Jayanth Srinivasa, Junyi Jessy Li, and Milos Gligoric explored the question “Can large language models (LLMs) condition their reasoning on explicitly supplied formal rules even when those rules conflict with their training priors?”
What the team did
We created the first counterfactual dataset for understanding LLMs’ reasoning capabilities on programming language semantics. The counterfactuals include two variants. In the first variant, semantic meanings of familiar symbols such as “+” and “-” are swapped. So, the derived programs keep syntactic familiarity but with semantics redefined. In the second variant, we removed syntactic familiarity altogether by assigning semantics of addition, subtraction, etc., to new symbols from the rarely encountered Caucasian-Albanian script. 11 frontier models spanning non-reasoning and reasoning capabilities, were then evaluated on their abilities to condition their reasoning on these explicitly provided counterfactual semantic variants to solve three kinds of program execution related tasks: program final state prediction (PredState), program semantic rule prediction (PredRule), and program execution trace prediction (PredTrace).
What researchers found
We found that rule-conditioned reasoning, i.e, models condition their reasoning on explicitly provided formal semantic rules, is a rare capability axis. Out of the 11 evaluated models, only one (Gemini-2.5-Pro) performed reliably and consistently well across the three tasks. But even the best performing model failed when the structural complexity (nested loops, complex control flow) of the programs for the tasks was increased. This shows that rule-conditioned reasoning based on formal semantics is a largely unsolved capability axis and addressing it could help build future models that reason more faithfully about formal specifications.
Why the work matters
Mainstream languages such as Python, C++, Scala, etc., support operator overloading; a feature where the semantics of commonly used operators such as “+”, “-”, etc., can be redefined, or novel semantics can be assigned to new symbols that are not part of the language conventionally. However, such scenarios may be out-of-distribution for LLMs as they may have not seen them during pre-training, yet are expected to generalize. Our work fills this gap by constructing the first dataset for reasoning about programming language semantics that incorporates counterfactuals to understand how, where, and why LLMs fail to generalize to redefined or novel semantics.
The paper was published in the 43rd International Conference on Machine Learning (ICML 2026) and the 2nd International Workshop on Language Models and Programming Languages (LMPL 2026)