Model Risk Score
Executive Summary
Key Takeaways
- ✓ A model risk score is a numeric summary of a model's structural findings, calculated by weighting each triggered issue against a fixed, disclosed basis.
- ✓ A score alone does not tell the full story — a small number of critical-severity findings can be weighted heavily enough to dominate an otherwise clean result.
- ✓ Under FMAE's SM-2.0 methodology, five critical-override rules can cap a model's letter grade below A or B regardless of its numeric score, described in full on the Scoring Engine technical documentation.
- ✓ A model risk score reflects structural and mechanical integrity, not commercial assumption reasonableness.
Definition¶
A model risk score is a numeric output summarizing the structural findings identified during a financial model audit, calculated by weighting each triggered rule or issue against a fixed, disclosed methodology. It is typically presented alongside, or translated into, a letter grade for ease of interpretation by a lender, investment committee, or board.
A model risk score reflects structural and mechanical integrity, circular references, hardcoded values, broken links, formula inconsistencies, and related issues, not whether the model's underlying business assumptions are commercially reasonable, which remains a separate judgement exercise.
Why It Matters¶
A single numeric score offers a fast, comparable summary of a model's structural condition, but it can be misinterpreted if read in isolation from the methodology that produced it. Two models can carry the same numeric score while differing materially in risk, if one model's score is driven by several minor findings and the other's by a single critical, override-triggering issue. Understanding how a specific methodology weights and, where applicable, caps its score is necessary to interpret the number correctly rather than treating it as a simple, self-explanatory figure.
Technical Background¶
FMAE's SM-2.0 Methodology¶
Under FMAE's active scoring methodology, SM-2.0, a model's numeric score is calculated as:
score = 100 − (triggered rule weight ÷ 207.0 × 100)
where 207.0 is the sum of the declared weights of all 26 active rules in the current rule set. Each triggered rule contributes its own declared weight to the total subtracted from 100; rules are not weighted equally, so two models with the same number of triggered findings can still receive different scores depending on which specific rules triggered.
Critical-Override Rules¶
A small set of rules, five under SM-2.0, are designated critical-override rules. If any one of them triggers, the resulting letter grade is capped below A or B regardless of the numeric score the weighted formula produces. This means a model can score in the high 70s or 80s numerically and still receive a C grade, not a B, if one of the triggered findings is a critical-override rule. The override exists specifically so a high numeric score cannot be read as certifying a model that carries one specific, structurally serious defect.
Versioning¶
A scoring methodology is typically versioned as the underlying rule set changes. FMAE's original methodology, SM-1.0, covered a smaller rule set against a different basis and is retained as a frozen historical reference rather than deleted when SM-2.0 was introduced to cover the full, current rule set. See the Scoring Engine technical documentation for the complete, current methodology and a fully worked example.
Common Misconceptions¶
"A high numeric score means the model has no serious issues." Not necessarily, under a methodology that includes critical-override rules, a single serious structural finding can cap the letter grade regardless of the numeric score.
"A model risk score reflects whether the model's assumptions are reasonable." It does not. It reflects structural and mechanical integrity as defined by the specific methodology's rule set; assumption reasonableness is a separate, human judgement exercise.
"Scores from different providers are directly comparable." Only if the underlying rule set and weighting basis are equivalent or explicitly reconciled; otherwise the same numeric score can represent materially different underlying findings.
Continue Reading¶
Related Pillars¶
Related Technical Documentation¶
Related Glossary¶
How OXXON tests thisRun a free structural check with FMAE
Frequently Asked Questions
What is a model risk score?
A numeric output summarizing a financial model's structural audit findings, calculated by weighting each triggered rule or issue against a fixed, disclosed methodology, and typically translated into a letter grade for ease of interpretation.
How is a model risk score calculated?
Methodologies vary by provider. Under FMAE's active SM-2.0 methodology, every triggered rule's declared weight is summed, and the score is 100 minus that sum divided by a fixed basis, currently 207.0, times 100, described in full on the Scoring Engine technical documentation.
Can a model score highly and still receive a poor letter grade?
Yes, under methodologies that include critical-override rules. Under SM-2.0, if any of five specific critical-override rules triggers, the resulting letter grade is capped below A or B regardless of how high the numeric score is, so a numeric score alone cannot certify a structurally broken model as sound.
Does a model risk score assess whether the model's assumptions are commercially reasonable?
No. A model risk score, as used in structural audit, reflects mechanical and structural integrity, circular references, hardcoded values, broken links, and related issues, not the reasonableness of the model's underlying business assumptions.
Is a model risk score comparable across different providers?
Only if the underlying methodology, rule set, and weighting basis are the same or explicitly reconciled. A score produced by one provider's methodology is not directly comparable to a score from a different provider's methodology without understanding both.
Why does a scoring basis change over time?
When a new rule is added to an active rule set, its weight is added to the basis and the basis is re-pinned to the new total, which is why a scoring methodology is typically versioned, described on the Scoring Engine technical documentation.
Related Articles
Structural Risk
Structural risk in the context of financial modelling is the risk of model failure arising from errors, inconsistencies, or weaknesses in the model's design, architecture, and internal logic — as distinct from the risk arising from incorrect input assumptions or adverse external outcomes. Structural risk exists within the model itself, regardless of the accuracy of the assumptions fed into it. A model with high structural risk will produce incorrect outputs even when its inputs are correct. This makes structural risk particularly dangerous: it cannot be remediated by revising assumptions or updating market data. It requires identifying and correcting the model's internal logic.
Model Materiality
Model materiality is the threshold at which an error, deviation, limitation, or uncertainty in a financial model is considered significant enough to affect a decision, require remediation, or warrant disclosure. A finding is material if, had it been known, it would or could have changed a decision made using the model's outputs. Model materiality is a judgement — it depends on the purpose of the model, the magnitude of the finding, and the sensitivity of the key outputs to the finding. The same error may be material in one context and immaterial in another.
Deterministic Audit
A deterministic audit is a financial model audit performed by applying a fixed, disclosed rule set systematically to a model's formulas and structure, such that running the same audit against the same model produces the same findings every time. It is distinguished from both manual, judgement-based review and generative AI-based review, neither of which is guaranteed to be repeatable in this sense.
Model Tiering
Model tiering is the process of classifying financial models into risk-based categories — tiers — that determine the level of governance oversight, validation rigour, documentation standards, and review frequency applied to each model. Higher-tier models, which are more complex, more material to decision-making, or more difficult to verify, receive more intensive governance than lower-tier models. Model tiering allows organisations to apply governance resources proportionately. Without tiering, an organisation must either apply heavy governance to every model (impractical) or apply light governance to every model (insufficient for high-risk models). Tiering resolves this by concentrating oversight where it matters most.