RP-008: Materiality and Severity — Distinguishing Two Axes of Model Risk

Institutional publication. Not peer-reviewed.

Abstract¶

This paper distinguishes two axes along which a structural finding can be assessed — severity and materiality — and formalizes the distinction using a concrete mechanism already implemented in a deployed scoring methodology: the critical-override rule.

1. Two Axes, Stated Generally¶

Severity is a property of a single finding, considered on its own terms: how serious is this specific defect, independent of anything else found in the model?

Materiality is a property of a finding's role within an overall assessment: how much weight should this one finding carry when forming an overall judgement about the model's reliability, considered alongside every other finding?

These are related — a more severe finding is generally also more material — but they are not identical, and collapsing them into a single number can misrepresent overall risk in a specific, correctable way.

2. The Failure Mode of Pure Additivity¶

A purely additive scoring model — summing every finding's weight and normalizing against a fixed basis — treats every finding's contribution as fungible: numerically, a handful of low-weight findings could in principle sum to the same aggregate deduction as a single severe one, and a large enough set of low-weight findings could dilute a severe finding's proportional contribution to an overall score, producing a passing-looking aggregate figure despite the presence of one seriously defective calculation.

3. The Critical-Override Mechanism as a Resolution¶

FMAE's own SM-2.0 Scoring Methodology resolves this specific failure mode with a critical-override mechanism: five designated rules, if triggered even once, cap the model's overall letter grade below the top two grade bands, regardless of how high the aggregate numeric score otherwise computes to. This is a direct, concrete operationalization of the severity/materiality distinction: these five findings are treated as materially disqualifying on their own terms, not merely as contributing their proportional numeric weight to an aggregate figure that other, lesser findings could dilute.

The worked example already published on the Scoring Engine page illustrates the mechanism precisely: a model with triggered findings summing to a numeric score that would ordinarily grade in the B band is capped to a C, specifically because two of the triggered findings are critical-override rules — the override exists precisely so a hardcoded override or an internally inconsistent formula can never be graded as though the model were merely "pretty good."

4. Generality of the Framework¶

This distinction is stated here independently of any one engine's specific implementation, though FMAE's SM-2.0 methodology is used as the paper's concrete, checkable grounding instance. Any structural-risk scoring system facing the same purely-additive failure mode — a large enough set of minor findings offsetting one severe finding's proportional contribution — faces the same underlying design question this paper formalizes: whether, and how, to designate certain findings as materially disqualifying independent of the aggregate numeric calculation.

5. Boundary of the Claim¶

This framework describes a design pattern for reconciling severity and materiality, not a claim about which specific findings should be designated as override-worthy in any given system — that is a methodology-specific, empirically grounded decision (as FMAE's own critical-override rule set is), not a universal, engine-independent list.

Related Reading¶

• FMAE Scoring Engine — SM-2.0 Methodology — the concrete critical-override mechanism this framework is formalized from.
• FMAE Rule Taxonomy — where the five critical-override rules sit within the engine's broader classification.