Fix metric bucket anchoring and add global whole-table metrics
Generate Documentation / generate-docs (push) Canceled after 0s
Run package tests for all dbs / test-postgres (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (bigquery) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (redshift) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (snowflake) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (bigquery) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (redshift) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (snowflake) (push) Canceled after 0s
Generate Documentation / generate-docs (push) Canceled after 0s
Run package tests for all dbs / test-postgres (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (bigquery) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (redshift) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (snowflake) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (bigquery) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (redshift) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (snowflake) (push) Canceled after 0s
Metrics reported row_count 0/1 against tables that plainly had 10+ rows. 1. The metric bucket was [run_date - 1d, run_date) - it ENDED at midnight at the start of the run day. A model stamping `dbt_updated_at = CURRENT_DATETIME()` writes rows during the run, so every row fell outside the bucket that run had just computed. Incremental merges that re-stamp rows each run then pushed them into the wrong bucket or out of every bucket. The bucket now covers the run's calendar day: [midnight of run date, +1d). interval_length_sec stays 86400 and the daily series is unchanged. 2. There was no way to see a table as a whole. `metrics_base_insert` now also emits `global__<metric>` computed over the whole table (`where true`) alongside the windowed series, so metrics stay meaningful for models whose time_filter holds historical/backfilled timestamps that miss the current bucket. A model with no time filter already computed globally and is not duplicated. Disable with `f2dq:emit_global_metrics: false`. 3. overview.json metric rows dropped `time_window_start`, so no consumer could tell a bucket's width or spot an empty one. It is exported now. Also: keep the guard that skips save_test_history when the f2dq_test_history relation does not exist. The integration_tests project pins the original re_data bucket anchoring and `f2dq:emit_global_metrics: false` because its expected_*.csv seeds encode the values that bucket produces and cannot be regenerated without the upstream multi-database harness. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
+17
-3
@@ -1,5 +1,5 @@
|
||||
name: "f2dq"
|
||||
version: "1.0.1" # dbt_f2dq: modernized fork (dbt_utils 1.4.1); dual-read re_data_* keys
|
||||
version: "1.0.2" # dbt_f2dq: modernized fork (dbt_utils 1.4.1); dual-read re_data_* keys
|
||||
config-version: 2
|
||||
|
||||
require-dbt-version: [">=1.0.0", "<2.0.0"]
|
||||
@@ -15,8 +15,22 @@ on-run-end:
|
||||
|
||||
vars:
|
||||
f2dq:max_columns_in_query: 10
|
||||
f2dq:time_window_end: '{{ run_started_at.strftime("%Y-%m-%d 00:00:00") }}'
|
||||
f2dq:time_window_start: '{{ (run_started_at - modules.datetime.timedelta(1)).strftime("%Y-%m-%d 00:00:00") }}'
|
||||
# The metric bucket is the run's calendar day: [midnight of the run date, +1d).
|
||||
#
|
||||
# re_data used [run_date - 1d, run_date) - i.e. the bucket ENDED at midnight at
|
||||
# the start of the run day. Any row stamped during the run (models that set
|
||||
# `dbt_updated_at = CURRENT_DATETIME()`) therefore fell outside the bucket the
|
||||
# run had just written it into, and incremental merges that re-stamp rows every
|
||||
# run made those rows land in the wrong bucket or never. row_count came out 0/1
|
||||
# against tables that plainly had rows. Anchoring the bucket on the run date
|
||||
# instead of the day before keeps `interval_length_sec` at 86400 and the daily
|
||||
# series intact, but includes everything written on the day being measured.
|
||||
f2dq:time_window_start: '{{ run_started_at.strftime("%Y-%m-%d 00:00:00") }}'
|
||||
f2dq:time_window_end: '{{ (run_started_at + modules.datetime.timedelta(1)).strftime("%Y-%m-%d 00:00:00") }}'
|
||||
# Also emit `global__<metric>` (whole-table) alongside the windowed series, so
|
||||
# metrics stay meaningful for models whose time_filter holds historical
|
||||
# timestamps that miss the current bucket.
|
||||
f2dq:emit_global_metrics: true
|
||||
f2dq:anomaly_detection_look_back_days: 30
|
||||
f2dq:select: null
|
||||
f2dq:anomalies_filtered: f2dq_anomalies
|
||||
|
||||
Reference in New Issue
Block a user