How it Works

The column_anomalies test from the Elementary dbt package monitors a specific column for statistical deviations from its historical baseline. It can track a variety of column-level metrics depending on the column type — including null rates, zero rates, distinct value counts, min/max values, and string length statistics. When a metric deviates beyond a configurable threshold, the test fails.

This test uses a training period of historical data to establish what 'normal' looks like, making it adaptive rather than threshold-based.

Steps and Conditions

1. Column Selection: Apply the test to the specific column(s) to monitor.

2. Select Monitors: Choose which metrics to track (e.g., null_count, distinct_count, average).

3. Training Period: Historical data is used to establish baseline norms.

4. Execution: Current metric values are compared to expected ranges.

5. Outcome: Pass if all metrics are within normal bounds; fail if an anomaly is detected.

Example Usage: Subscription Analytics

A subscription analytics team wants to detect unexpected changes in the mrr (Monthly Recurring Revenue) column, such as sudden spikes or drops that could indicate billing errors.

models:
  - name: subscriptions
    columns:
      - name: mrr
        tests:
          - elementary.column_anomalies:
              column_anomalies_monitors

models:
  - name: subscriptions
    columns:
      - name: mrr
        tests:
          - elementary.column_anomalies:
              column_anomalies_monitors

models:
  - name: subscriptions
    columns:
      - name: mrr
        tests:
          - elementary.column_anomalies:
              column_anomalies_monitors

If the average MRR unexpectedly drops or null values spike, Elementary raises an alert before it impacts reporting.