Fix metric bucket anchoring and add global whole-table metrics
Generate Documentation / generate-docs (push) Canceled after 0s
Run package tests for all dbs / test-postgres (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (bigquery) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (redshift) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (snowflake) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (bigquery) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (redshift) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (snowflake) (push) Canceled after 0s
Generate Documentation / generate-docs (push) Canceled after 0s
Run package tests for all dbs / test-postgres (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (bigquery) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (redshift) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (snowflake) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (bigquery) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (redshift) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (snowflake) (push) Canceled after 0s
Metrics reported row_count 0/1 against tables that plainly had 10+ rows. 1. The metric bucket was [run_date - 1d, run_date) - it ENDED at midnight at the start of the run day. A model stamping `dbt_updated_at = CURRENT_DATETIME()` writes rows during the run, so every row fell outside the bucket that run had just computed. Incremental merges that re-stamp rows each run then pushed them into the wrong bucket or out of every bucket. The bucket now covers the run's calendar day: [midnight of run date, +1d). interval_length_sec stays 86400 and the daily series is unchanged. 2. There was no way to see a table as a whole. `metrics_base_insert` now also emits `global__<metric>` computed over the whole table (`where true`) alongside the windowed series, so metrics stay meaningful for models whose time_filter holds historical/backfilled timestamps that miss the current bucket. A model with no time filter already computed globally and is not duplicated. Disable with `f2dq:emit_global_metrics: false`. 3. overview.json metric rows dropped `time_window_start`, so no consumer could tell a bucket's width or spot an empty one. It is exported now. Also: keep the guard that skips save_test_history when the f2dq_test_history relation does not exist. The integration_tests project pins the original re_data bucket anchoring and `f2dq:emit_global_metrics: false` because its expected_*.csv seeds encode the values that bucket produces and cannot be regenerated without the upstream multi-database harness. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -9,20 +9,33 @@
|
||||
-- depends_on: {{ ref('f2dq_test_history') }}
|
||||
{% set command = flags.WHICH %}
|
||||
{% if execute and results and command in ('test', 'build') %}
|
||||
{% set tests = [] %}
|
||||
{% for el in results %}
|
||||
{% if el.node.resource_type.value == 'test' %}
|
||||
{% do tests.append(f2dq.test_data_dict(el)) %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
|
||||
{% if tests %}
|
||||
{% do f2dq.insert_list_to_table(
|
||||
ref('f2dq_test_history'),
|
||||
tests,
|
||||
['table_name', 'column_name', 'test_name', 'status', 'execution_time', 'message', 'failures_count', 'failures_json', 'failures_table', 'severity', 'compiled_sql', 'run_at'],
|
||||
{ 'run_at': timestamp_type() }
|
||||
{% set history_rel = ref('f2dq_test_history') %}
|
||||
{% set history_exists = adapter.get_relation(
|
||||
database=history_rel.database,
|
||||
schema=history_rel.schema,
|
||||
identifier=history_rel.identifier
|
||||
) is not none %}
|
||||
{% if not history_exists %}
|
||||
{% do log(
|
||||
"f2dq_test_history relation missing (" ~ history_rel ~ "); skipping save_test_history",
|
||||
info=True
|
||||
) %}
|
||||
{% else %}
|
||||
{% set tests = [] %}
|
||||
{% for el in results %}
|
||||
{% if el.node.resource_type.value == 'test' %}
|
||||
{% do tests.append(f2dq.test_data_dict(el)) %}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
|
||||
{% if tests %}
|
||||
{% do f2dq.insert_list_to_table(
|
||||
history_rel,
|
||||
tests,
|
||||
['table_name', 'column_name', 'test_name', 'status', 'execution_time', 'message', 'failures_count', 'failures_json', 'failures_table', 'severity', 'compiled_sql', 'run_at'],
|
||||
{ 'run_at': timestamp_type() }
|
||||
) %}
|
||||
{% endif %}
|
||||
{% endif %}
|
||||
|
||||
{% endif %}
|
||||
|
||||
Reference in New Issue
Block a user