RP-007: Deterministic vs. Generative Approaches to Structural Model Review
Executive Summary
Key Takeaways
- ✓ Reproducibility is the defining methodological distinction between deterministic and generative approaches to structural model review — the same input should, and for a deterministic engine does, produce the same output every time.
- ✓ A deterministic rule engine applies a fixed, disclosed methodology consistently to every formula in a model; a general-purpose language model's coverage and consistency across a large model's full formula set is not guaranteed in the same way.
- ✓ Both approaches are describable as involving AI in a loose sense, but only the deterministic approach produces the specific property (repeatable, explainable, evidence-backed output) typically required to support a material financial decision.
- ✓ This is not a claim that generative review has no value — it can be a useful exploratory, supplementary check — but a claim about which property is required for the specific use case of evidence behind a material decision.
Institutional publication. Not peer-reviewed.
Abstract¶
This paper compares deterministic, rule-based structural review and generative large-language-model review of financial models, isolating reproducibility as the property that most clearly and consequentially distinguishes the two approaches for the specific use case of evidence behind a material financial decision.
1. Two Approaches, Both Describable as "AI"¶
Both approaches can loosely be described as applying artificial intelligence to financial model review. A deterministic rule engine is software — code implementing a fixed, disclosed set of structural checks. A generative review uses a large language model's learned, probabilistic reasoning to read a model and offer commentary. The looseness of the shared label ("AI") is itself part of why the methodological distinction between the two needs to be stated explicitly rather than assumed from the label.
2. Reproducibility as the Defining Property¶
A deterministic engine, such as the one documented throughout this Technical Documentation library, applies the same fixed methodology to every formula in a workbook, every time it runs. As documented on the Architecture Overview page, no stage of such a pipeline calls a language model or involves probabilistic inference — the same input workbook produces the same findings and the same score on every run, by construction.
A generative review, by contrast, uses a model whose output is not guaranteed to be identical across repeated invocations, even given the same input, and whose depth of scrutiny across a large model's full formula set is not guaranteed to be consistent from run to run or reviewer to reviewer. This is not necessarily a criticism of the underlying language model's capability — it is a structural consequence of how such models generate output.
3. Why This Property Matters for Evidence¶
A material financial decision — a lending decision, an investment committee approval — often needs to be defensible after the fact: a specific finding needs to be traceable to a specific, checkable cause, and ideally reproducible if the same check were run again. FMAE's own Finding Data Model & Confidence page documents exactly this property for a deterministic engine's own findings — each carries a rule snapshot, evidence, and a deterministic (not probabilistic) confidence label. A generative review's commentary does not carry the same guarantee: the same model reviewed twice by the same generative process is not guaranteed to surface the same issues in the same way, which limits its suitability as the primary, defensible evidentiary basis for the decision, whatever its exploratory value.
4. Coverage Consistency¶
A related, secondary distinction is coverage. A deterministic engine's rules are applied to every relevant cell in a workbook, exhaustively, by construction — the same structural check either runs against every formula meeting its criteria or it does not run at all, per the all-or-nothing execution model FMAE's own rule engine implements. A generative review's attention across a large model's full formula set is not governed by the same exhaustive, rule-by-rule guarantee.
5. Scope of the Claim¶
This paper's claim is specifically about reproducibility and coverage consistency as properties relevant to evidentiary use. It is not a general claim that generative review has no analytical value — a generative review can surface useful exploratory observations a rule-based engine, limited to its declared rule set, would not. The claim is narrower: for the specific purpose of producing defensible, repeatable evidence behind a material decision, reproducibility is the property that matters, and it is the property a deterministic engine provides by construction.
Related Reading¶
- Deterministic Audit vs Generative AI Review — the Knowledge Centre's buyer-facing treatment of this same distinction.
- FMAE Audit Engine — Architecture Overview — the deterministic pipeline this paper's comparison is grounded in.
- Finding Data Model & Confidence — the deterministic, non-probabilistic confidence model referenced in Section 3.
How OXXON tests thisRun a free structural check with FMAE
Frequently Asked Questions
What is the key methodological difference this paper focuses on?
Reproducibility — whether the same model, reviewed twice, produces the same findings both times. A deterministic rule engine guarantees this by construction; a generative language model's output is not guaranteed to be identical, or even substantively equivalent, across repeated runs.
Does this mean generative AI review has no value in financial model review?
No. This paper's claim is narrower and specific to reproducibility as a property required for evidence behind a material decision. A generative review can still be a useful exploratory or supplementary check; the paper's point is that it does not, on its own, provide the same property a deterministic engine provides.
Why does reproducibility specifically matter for a material financial decision?
Because a decision's defensibility often depends on being able to show, after the fact, exactly what was checked and why a specific finding was raised, in a way that would be reproduced identically if the same check were repeated. A non-reproducible review cannot offer this same evidentiary property, regardless of how insightful any single run's commentary happens to be.
Related Articles
FMAE Rule Taxonomy
Every rule in the FMAE structural rule pack declares a category attribute at the point it is defined in source — this is not a classification imposed on the rules afterward for documentation purposes, it is the classification the engine itself uses. Six categories cover all 26 rules — Structural (18 rules), Assumptions Governance (1), Integrity Controls (2), Structural Hygiene (1), Aggregation Logic (1), and Model Governance (3). This page publishes that taxonomy as the FMAE equivalent of a control catalog's classification scheme, cross-linked to the Rule Reference page for each member rule.
Rule Engine & Rule Packs
FMAE's 26 structural rules are not called individually. They are registered into a single named, versioned rule pack — fmae.structural, version 1.0.0 — and executed as a unit through a narrow public interface (register_rule_pack / execute_rule_pack), never through direct access to individual rule classes from outside the engine. Every finding the pack produces is stamped with exactly which engine version and rule pack version produced it, so a finding from any past audit remains attributable to the exact rule logic that generated it.
Finding Data Model & Confidence
Every FMAE finding carries the same fixed structure — which rule triggered it and that rule's identity at the time it ran, its evidence, a narrative explanation, and a confidence label of high, medium, or low. Confidence is a deterministic categorical label derived from how strongly a row's cells agree on a pattern, not a machine-learning probability estimate. Stating this plainly matters, since describing FMAE's confidence as AI-derived or probabilistic would overstate what the engine actually does.