How it Works

The freshness_anomalies test from the Elementary dbt package monitors data freshness patterns over time and flags deviations from the historical norm. Unlike threshold-based freshness tests, it learns from past update intervals and raises an alert when the current update lag is statistically unusual compared to historical behavior.

Steps and Conditions

1. Timestamp Column: Specify the field to monitor using timestamp_column.

2. Baseline Period: Elementary uses historical update intervals to learn normal freshness patterns.

3. Execution: The current time since last update is compared to the expected pattern.

4. Outcome: Pass if the update lag is within the normal range; fail if it represents an anomaly.

Example Usage: CRM Data Sync

A CRM data sync updates a contacts table at irregular but historically consistent intervals. The team wants to detect unusual delays without hardcoding a fixed threshold.

models:
  - name: contacts
    tests:
      - elementary.freshness_anomalies:
          timestamp_column

models:
  - name: contacts
    tests:
      - elementary.freshness_anomalies:
          timestamp_column

models:
  - name: contacts
    tests:
      - elementary.freshness_anomalies:
          timestamp_column

If the sync unexpectedly pauses for far longer than historically observed, Elementary raises an alert before the stale data reaches downstream dashboards.