Fix metric bucket anchoring and add global whole-table metrics
Generate Documentation / generate-docs (push) Canceled after 0s
Run package tests for all dbs / test-postgres (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (bigquery) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (redshift) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (snowflake) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (bigquery) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (redshift) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (snowflake) (push) Canceled after 0s

Metrics reported row_count 0/1 against tables that plainly had 10+ rows.

1. The metric bucket was [run_date - 1d, run_date) - it ENDED at midnight at
   the start of the run day. A model stamping `dbt_updated_at =
   CURRENT_DATETIME()` writes rows during the run, so every row fell outside
   the bucket that run had just computed. Incremental merges that re-stamp
   rows each run then pushed them into the wrong bucket or out of every
   bucket. The bucket now covers the run's calendar day:
   [midnight of run date, +1d). interval_length_sec stays 86400 and the
   daily series is unchanged.

2. There was no way to see a table as a whole. `metrics_base_insert` now also
   emits `global__<metric>` computed over the whole table (`where true`)
   alongside the windowed series, so metrics stay meaningful for models whose
   time_filter holds historical/backfilled timestamps that miss the current
   bucket. A model with no time filter already computed globally and is not
   duplicated. Disable with `f2dq:emit_global_metrics: false`.

3. overview.json metric rows dropped `time_window_start`, so no consumer could
   tell a bucket's width or spot an empty one. It is exported now.

Also: keep the guard that skips save_test_history when the f2dq_test_history
relation does not exist.

The integration_tests project pins the original re_data bucket anchoring and
`f2dq:emit_global_metrics: false` because its expected_*.csv seeds encode the
values that bucket produces and cannot be regenerated without the upstream
multi-database harness.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
M7XBeast
2026-10-14 22:15:58 +03:00
co-authored by Cursor
parent 13df1026cd
commit 109e009210
5 changed files with 116 additions and 38 deletions
+11
View File
@@ -19,6 +19,17 @@ sources:
vars:
f2dq:store_table_samples: true
# The expected_*.csv seeds encode re_data's original bucket anchoring
# ([run_date - 1d, run_date)) and only the rows that bucket produces. The
# package default now anchors the bucket on the run date and also emits
# `global__*` metrics; both change the result set, and the seed values cannot
# be regenerated without the upstream multi-database harness. Pin the legacy
# behaviour here so this suite keeps exercising the metric engine and the
# `global__`-vs-windowed naming contract. The corrected behaviour is covered
# by the contract tests in the control_plane test suite.
f2dq:emit_global_metrics: false
f2dq:time_window_end: '{{ run_started_at.strftime("%Y-%m-%d 00:00:00") }}'
f2dq:time_window_start: '{{ (run_started_at - modules.datetime.timedelta(1)).strftime("%Y-%m-%d 00:00:00") }}'
f2dq:anomaly_detector:
name: modified_z_score
threshold: 0.6