Predictions and attributions | Umnai

Predictions and attributions are the core signals in a local explanation.

A prediction is the model output for a query. Attributions explain how that prediction was built by showing how much each module, feature, or feature interaction contributed.

Prediction

A prediction is the output of the model for a query.

For regression problems, the prediction is the sum of all module attributions plus the model intercept.

For classification problems, the sum of all module attributions plus the intercept is compared with a threshold to determine the predicted class.

In both cases, the prediction is built from attributions. This is the first step in explaining how the model produced the output.

Module attributions

Module attributions explain how much each feature or feature interaction contributed to a prediction.

An XNN is composed of modules. Each module represents a feature or a feature interaction. When a query is run through the model, it activates one partition in every module.

The local neural model in the rule of each activated partition calculates the attribution of the feature or interaction represented by that module.

Module attributions can be positive or negative.

Direction	Meaning
Positive attribution	The module increased the prediction.
Negative attribution	The module decreased the prediction.

The sum of all module attributions is a direct representation of the prediction.

Module attribution plot for a prediction.

Module attribution table (top 15 modules only) showing attributions and activated partitions and rules.

Feature attributions

Feature attributions explain how much each individual feature contributed to a prediction.

A feature can contribute directly through its own module and indirectly through interaction modules. To produce feature attributions, interaction module attributions are decomposed and grouped back to the features involved.

Like module attributions, feature attributions can be positive or negative.

Direction	Meaning
Positive attribution	The feature increased the prediction.
Negative attribution	The feature decreased the prediction.

Feature attributions provide a feature-level explanation of the prediction across all modules.

Feature attribution plot for a prediction.

Feature attribution table for a prediction.

Activations

Activations explain the path taken by a query through a model.

For a given query, activation data identifies which partition and embedded rule were activated in each module.

Activation data connects the prediction and attribution values back to the model structures that produced them.

Activations can also be aggregated across a segment, across the entire dataset, or monitored in real time when a model is deployed for inference.