How it Works

The event_freshness_anomalies test from the Elementary dbt package detects anomalies in the time gap between when events occur and when they arrive in the data warehouse. Unlike freshness_anomalies which looks at the last update time, this test focuses on the lag between event time and ingestion time, catching delays that would affect real-time or near-real-time analytics.

Steps and Conditions

1. Event Timestamp: Specify the column representing when the event occurred.

2. Update Timestamp: Specify the column representing when the record was ingested.

3. Baseline Period: Historical event-to-ingestion lag patterns form the baseline.

4. Execution: The current lag distribution is compared to historical norms.

5. Outcome: Pass if the lag is within normal bounds; fail if anomalous delays are detected.

Example Usage: Real-time Analytics

A real-time analytics platform processes clickstream events. The team wants to detect when the pipeline is introducing unusual delays between user actions and data availability.

models:
  - name: clickstream_events
    tests:
      - elementary.event_freshness_anomalies:
          event_timestamp_column: event_time
          update_timestamp_column

models:
  - name: clickstream_events
    tests:
      - elementary.event_freshness_anomalies:
          event_timestamp_column: event_time
          update_timestamp_column

models:
  - name: clickstream_events
    tests:
      - elementary.event_freshness_anomalies:
          event_timestamp_column: event_time
          update_timestamp_column

An unusual spike in event-to-ingestion lag signals a pipeline bottleneck before it impacts dashboards relying on near-real-time data.