Fix metric bucket anchoring and add global whole-table metrics
Generate Documentation / generate-docs (push) Canceled after 0s
Run package tests for all dbs / test-postgres (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (bigquery) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (redshift) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (snowflake) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (bigquery) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (redshift) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (snowflake) (push) Canceled after 0s
Generate Documentation / generate-docs (push) Canceled after 0s
Run package tests for all dbs / test-postgres (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (bigquery) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (redshift) (push) Canceled after 0s
Run package tests for all dbs / test-other-dbs (snowflake) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (bigquery) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (redshift) (push) Canceled after 0s
Run package tests for all dbs / clean-up-schemas (snowflake) (push) Canceled after 0s
Metrics reported row_count 0/1 against tables that plainly had 10+ rows. 1. The metric bucket was [run_date - 1d, run_date) - it ENDED at midnight at the start of the run day. A model stamping `dbt_updated_at = CURRENT_DATETIME()` writes rows during the run, so every row fell outside the bucket that run had just computed. Incremental merges that re-stamp rows each run then pushed them into the wrong bucket or out of every bucket. The bucket now covers the run's calendar day: [midnight of run date, +1d). interval_length_sec stays 86400 and the daily series is unchanged. 2. There was no way to see a table as a whole. `metrics_base_insert` now also emits `global__<metric>` computed over the whole table (`where true`) alongside the windowed series, so metrics stay meaningful for models whose time_filter holds historical/backfilled timestamps that miss the current bucket. A model with no time filter already computed globally and is not duplicated. Disable with `f2dq:emit_global_metrics: false`. 3. overview.json metric rows dropped `time_window_start`, so no consumer could tell a bucket's width or spot an empty one. It is exported now. Also: keep the guard that skips save_test_history when the f2dq_test_history relation does not exist. The integration_tests project pins the original re_data bucket anchoring and `f2dq:emit_global_metrics: false` because its expected_*.csv seeds encode the values that bucket produces and cannot be regenerated without the upstream multi-database harness. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -11,7 +11,7 @@
|
||||
|
||||
{% for column in model.columns %}
|
||||
{% set column_name = f2dq.row_value(column, 'column_name') %}
|
||||
|
||||
|
||||
{% if should_compute_metric(model, column_name) %}
|
||||
{% do columns_to_query.append(column) %}
|
||||
{% endif %}
|
||||
@@ -37,39 +37,79 @@
|
||||
{% endmacro %}
|
||||
|
||||
{% macro metrics_base_insert(model, ref_model, columns, table_level=False) %}
|
||||
|
||||
{% set col_exprs = f2dq.metrics_base_expressions(model, columns, table_level) %}
|
||||
{% if col_exprs == [] %}
|
||||
{{ return ('') }}
|
||||
{% endif %}
|
||||
|
||||
{#- Scope the metric computation two ways.
|
||||
|
||||
'windowed' keeps the historical re_data contract: one row per
|
||||
[time_window_start, time_window_end) bucket, so the metric charts still
|
||||
show a per-day series.
|
||||
|
||||
'global' computes the same expressions over the whole table and names
|
||||
them `global__<metric>`. Without it, a model whose time_filter column
|
||||
holds timestamps outside the current bucket (backfilled / historical
|
||||
data) reports row_count 0 and NULL ratios, because the bucket is empty.
|
||||
`global__row_count` then describes the table as it actually is.
|
||||
|
||||
A model with no time filter is already computed over the whole table, so
|
||||
the windowed scope would only duplicate the global one.
|
||||
|
||||
Set `f2dq:emit_global_metrics` to false to restore windowed-only. -#}
|
||||
{% set emit_global = f2dq.f2dq_var('emit_global_metrics', true) %}
|
||||
{% if model.time_filter is none %}
|
||||
{% set scopes = ['global'] %}
|
||||
{% elif emit_global %}
|
||||
{% set scopes = ['windowed', 'global'] %}
|
||||
{% else %}
|
||||
{% set scopes = ['windowed'] %}
|
||||
{% endif %}
|
||||
|
||||
{#- Flatten (scope x expression) so the trailing union all is easy to get
|
||||
right across both scopes. -#}
|
||||
{% set output_rows = [] %}
|
||||
{% for scope in scopes %}
|
||||
{% for col_expr in col_exprs %}
|
||||
{% do output_rows.append({
|
||||
'scope': scope,
|
||||
'col_name': col_expr.col_name,
|
||||
'metric': col_expr.metric,
|
||||
'expr': col_expr.expr,
|
||||
}) %}
|
||||
{% endfor %}
|
||||
{% endfor %}
|
||||
|
||||
insert into {{ ref(ref_model) }}
|
||||
with temp_table_metrics as (
|
||||
select
|
||||
with
|
||||
{%- for scope in scopes %}
|
||||
temp_table_metrics_{{ scope }} as (
|
||||
select
|
||||
{%- for col_expr in col_exprs %}
|
||||
( {{ col_expr.expr }} ) as {{ f2dq.quote_column_name(col_expr.col_name + '___' + col_expr.metric) }}
|
||||
{%- if not loop.last %},{%- endif %}
|
||||
{% endfor %}
|
||||
from
|
||||
{{ model.table_name }}
|
||||
where
|
||||
{{ in_time_window(model.time_filter) }}
|
||||
)
|
||||
|
||||
{%- for col_expr in col_exprs %}
|
||||
{% set final_metric_name = get_final_metric_name(col_expr.metric, model.time_filter) %}
|
||||
|
||||
select '{{model.table_name}}' as table_name, '{{ col_expr.col_name }}' as column_name, '{{ final_metric_name }}' as metric, {{ f2dq.quote_column_name(col_expr.col_name + '___' + col_expr.metric) }} as value
|
||||
from temp_table_metrics
|
||||
{% if not loop.last %}union all{% endif %}
|
||||
{% endfor %}
|
||||
{%- endfor %}
|
||||
from {{ model.table_name }}
|
||||
where {% if scope == 'global' %}true{% else %}{{ in_time_window(model.time_filter) }}{% endif %}
|
||||
){% if not loop.last %},{% endif %}
|
||||
{%- endfor %}
|
||||
|
||||
{%- for row in output_rows %}
|
||||
select
|
||||
'{{ model.table_name }}' as table_name,
|
||||
'{{ row.col_name }}' as column_name,
|
||||
'{{ f2dq.get_final_metric_name(row.metric, model.time_filter, row.scope) }}' as metric,
|
||||
{{ f2dq.quote_column_name(row.col_name + '___' + row.metric) }} as value
|
||||
from temp_table_metrics_{{ row.scope }}
|
||||
{%- if not loop.last %} union all{%- endif %}
|
||||
{%- endfor %}
|
||||
{% endmacro %}
|
||||
|
||||
{% macro get_final_metric_name(metric_name, time_filter) %}
|
||||
{% if time_filter is none %}
|
||||
{% macro get_final_metric_name(metric_name, time_filter, scope='windowed') %}
|
||||
{% if scope == 'global' or time_filter is none %}
|
||||
{{ return ('global__' + metric_name) }}
|
||||
{% else %}
|
||||
{{ return (metric_name) }}
|
||||
{% endif %}
|
||||
{% endmacro %}
|
||||
{% endmacro %}
|
||||
|
||||
Reference in New Issue
Block a user