How it Works

The volume_anomalies test from the Elementary dbt package monitors a model's row count over time and uses historical patterns to detect statistically unusual changes. Unlike a static row count threshold, this test adapts to trends and seasonality, making it well-suited for production data that grows or fluctuates predictably over time.

Steps and Conditions

1. Timestamp Column: Specify the time partitioning column.

2. Baseline Period: Elementary uses historical volume data to establish expected ranges.

3. Execution: The current period's row count is compared to the expected range.

4. Outcome: Pass if the count is within the expected range; fail if an anomaly is detected.

Example Usage: Data Pipeline

A data pipeline ingests daily customer transactions. The team wants to detect significant drops or spikes in volume that could indicate a source failure or data duplication.

models:
  - name: customer_transactions
    tests:
      - elementary.volume_anomalies:
          timestamp_column

models:
  - name: customer_transactions
    tests:
      - elementary.volume_anomalies:
          timestamp_column

models:
  - name: customer_transactions
    tests:
      - elementary.volume_anomalies:
          timestamp_column

If transaction volume drops by 80% compared to a normal day, Elementary raises an alert, catching pipeline failures before they affect business reporting.