How it Works

The dimension_anomalies test from the Elementary dbt package monitors the distribution of values within a categorical column over time. Rather than tracking total volume, it watches the relative share of each category and raises an alert if any dimension's proportion shifts unexpectedly. This is particularly useful for detecting silent data quality issues like one source suddenly dominating a feed or a category disappearing entirely.

Steps and Conditions

1. Column Selection: Specify the categorical column to monitor using dimensions.

2. Baseline Period: Elementary uses historical distributions to establish expected proportions.

3. Execution: The current distribution is compared against historical baselines.

4. Outcome: Pass if proportions are within normal variance; fail if a significant shift is detected.

Example Usage: E-commerce

An e-commerce platform monitors the distribution of payment_method values. A sudden drop in credit card payments or spike in gift card usage could indicate a payment integration issue.

models:
  - name: orders
    tests:
      - elementary.dimension_anomalies:
          dimensions

models:
  - name: orders
    tests:
      - elementary.dimension_anomalies:
          dimensions

models:
  - name: orders
    tests:
      - elementary.dimension_anomalies:
          dimensions

If one payment method's share shifts dramatically overnight, the test raises an alert for investigation.