Below the Ice — When a 1970s Logic Language Explains a 2020s Neural Network
Una política de aprendizaje por refuerzo puede funcionar de maravilla, pero nadie puede leerla. Una nueva técnica reescribe esos pesos opacos como reglas de Prolog: lógica que una persona puede leer, que un motor lógico puede ejecutar, y que un optimizador puede editar. Esta noche trazamos cada paso de esa transformación.
This is the print twin of tonight's Below the Ice episode — one topic, told properly. Pour something warm and read slowly.
The headline, and what sits under it
A reinforcement learning agent learns by trial and error, millions of times over, until it finds a strategy that scores points or solves the task. When it works, it works uncannily well — better than any human-authored rule set would. But ask someone what rule the agent is actually following and the honest answer is: nobody knows. There are weights, there are layers, there are billions of multiplied numbers. There is no sentence you can write down and say this is what it learned.
That gap between capability and comprehensibility is not just philosophically uncomfortable. It is becoming a practical problem. Regulators want auditable decisions. Engineers want to debug agents that fail. Users want to trust systems that act on their behalf. "It works" is no longer enough.
Tonight's deep dive sits in the space between those two facts. A recent research paper — From Black Box to Executable Logic — proposes a three-stage pipeline that takes a frozen, trained RL policy and rewrites it as Prolog clauses. Not a summary. Not a diagram. Actual executable logic that a Prolog interpreter can run and produce the same decisions the neural network would.
The paper is one data point. But the problem it points at — and the broader wave of neuro-symbolic thinking it joins — is worth understanding from first principles.
What it is
Explainable AI (XAI) is a family of techniques for making machine learning decisions legible to humans. Most XAI methods are approximations: LIME, SHAP, saliency maps. They answer "what influenced this prediction?" but they do not give you a program you can run independently.
The approach in tonight's paper is different. It aims for post-hoc symbolic extraction — after training is complete, a process extracts the policy's behavior as symbolic logic rules that are both human-readable and machine-executable.
The target representation is Prolog — a programming language from 1972 built on formal logic. A Prolog program is a set of clauses. Each clause says: if these conditions are true, then this conclusion follows. A Prolog interpreter reads those clauses and reasons forward or backward over them. The language has been used for decades in expert systems, planning, and constraint solving.
Pressing Prolog into service to explain deep RL in 2026 is the core juxtaposition the research offers. Logic programming from the era of bell-bottoms; deep learning from the era of foundation models. The bet is that the 1970s idea can make the 2020s idea readable.
How it actually works — from first principles
Imagine a trained RL agent that plays a simplified grid-world game. It has learned, through millions of episodes, exactly which direction to move given the current state of the board. If you probe it — call it with every possible state and record the action it outputs — you get a large table of (state → action) pairs. This table is the agent's behavior, externalized.
Stage one: behavior extraction. The first stage does exactly this. It runs the frozen policy across a systematic sample of states, recording every (state, action) pair. This creates an empirical dataset of what the agent does — not what it is internally.
Think of this like reverse-engineering a colleague's decision-making by shadowing them for a month and writing down every choice they make. You don't understand why yet; you just have the observations.
Stage two: logic induction. The second stage takes that observation dataset and applies inductive logic programming (ILP) — algorithms that search for Prolog clauses that cover the observed examples. ILP has existed for decades; the paper uses it as the extraction engine. The output is a set of Prolog rules of the form:
action(move_right) :- state_feature(X, wall_left), state_feature(X, goal_right).
Which reads: move right if there is a wall to the left and the goal is to the right. A logic engine can now evaluate that clause against any new state and produce a decision — without the neural network.
Stage three: optimization and verification. The extracted clauses are not guaranteed to match the neural policy perfectly — ILP is searching over a space of possible rules and finds an approximation. Stage three applies optimization to improve coverage and fidelity, then checks the resulting Prolog program against the original policy's outputs on a held-out test set. The goal: get the Prolog program's decisions to match the neural network's decisions as closely as possible.
If the match is high enough, you now have a representation of the policy that is:
- Readable. A developer can open the Prolog file and read the rules.
- Executable. A Prolog interpreter can run it without the original neural network.
- Editable. An engineer can change a rule and see the effect — something you cannot do inside a weight matrix.
Why it matters now
Three forces are converging.
Regulatory pressure is real. The EU AI Act, which entered application phases in 2025, requires high-risk AI systems to be transparent and auditable. Article 13 demands that providers give deployers "sufficiently detailed" technical documentation to understand what the system does and why. "The model said so" does not satisfy that requirement. Symbolic representations that can be written down and explained to a regulator do.
Debugging agents is getting harder. As RL and LLM-based agents take more autonomous actions in the world, the cases where they fail become higher-stakes and harder to trace. An agent that fails in production, and whose policy lives only in weights, is nearly impossible to debug from first principles. A Prolog representation gives an engineer a place to look — a set of rules that can be tested, contradicted, or patched.
Neuro-symbolic AI is having a quiet revival. The pure-data, pure-gradient era of deep learning is maturing. Researchers are asking: what if you paired learned behavior — which is powerful but opaque — with symbolic structure — which is readable but brittle? Neuro-symbolic hybrids appear in a growing number of papers and system designs. The Prolog extraction paper is one instance of this wave. Research on combining logic and optimization for AI published alongside it makes the broader trend explicit: logic-based components are being proposed as a path toward more trustworthy, auditable AI systems.
What is overhyped
Being honest is part of the show.
Explainable does not mean simple. ILP can produce Prolog programs that are technically readable but span hundreds of clauses. A 400-rule Prolog program is not, in practice, something a human can reason about holistically. The form is interpretable; the complexity may not be.
The technique has only been demonstrated on small, frozen policies. The paper's experiments use compact RL environments. Whether post-hoc logic extraction scales to a modern LLM-based agent acting over a complex action space is an open question — and a significant one. Extraction requires probing the policy across its state space; for policies with enormous state spaces, that sampling problem becomes intractable.
Matching is not identity. A Prolog program that matches the neural policy 97% of the time is still wrong 3% of the time. In any high-stakes application, those 3% cases might be exactly the cases that matter most — edge conditions, adversarial inputs, distribution shift.
ILP is not new, and it's not magic. Inductive logic programming has been an active research area for thirty years. The paper's contribution is applying it to this specific extraction problem, not inventing the underlying engine. The combination is novel; neither piece is.
What to watch
Three concrete things worth tracking as this area develops.
1. Scaling post-hoc extraction to larger networks. The open question is whether symbolic extraction can work on policies that are orders of magnitude more complex than what the paper demonstrated. Research groups in XAI and neuro-symbolic AI are actively testing the boundaries. Progress here would make this technique relevant to production systems; failure to scale would limit it to constrained domains.
2. What AI transparency regulations actually require from practitioners. The EU AI Act is the most codified framework, but similar requirements are developing in the US, UK, and Singapore. The gap between what regulators write and what engineers know how to deliver is significant. Following that space — and the practical compliance tooling that emerges — matters for anyone building systems that touch regulated decisions: hiring, credit, healthcare, autonomous action.
3. The broader logic-plus-optimization wave. The Prolog extraction paper sits next to a growing body of work on combining learned and symbolic components. Watch for papers and engineering reports at venues like NeurIPS, IJCAI, and systems-focused venues like OSDI and SOSP that report production experience with neuro-symbolic hybrids. When that wave produces systems that go into production at scale, it will signal something real about where the field is heading.
Sources:
- Zhang et al. (2026) — From Black Box to Executable Logic: Explainable Reinforcement Learning through Prolog Expert Systems, arXiv:2607.15459
- Hook & Hooker (2026) — Logic, Optimization, and Artificial Intelligence, arXiv:2607.15532
- European Union — EU AI Act, Article 13: Transparency obligations for certain AI systems
- EU AI Act transparency requirements — EUR-Lex full text
Listen to the full evening dive on Below the Ice. Every weeknight at 7 PM Monterrey time, from The Penguin Alley.