diff --git a/docs/signals/attributes/attribute-groups/index.md b/docs/signals/attributes/attribute-groups/index.md index 090a1b228..2fffc0c2d 100644 --- a/docs/signals/attributes/attribute-groups/index.md +++ b/docs/signals/attributes/attribute-groups/index.md @@ -89,10 +89,10 @@ Stream attribute groups only calculate attributes from the moment they are publi :::note[Warehouse connection] -A warehouse connection is required to use the backfill option. Only Snowflake and BigQuery are supported currently. +A warehouse connection is required to use the backfill option. Snowflake, BigQuery, and Databricks are supported. ::: -Enable **Backfill attributes** when creating the group. A date picker appears — select the date from which Signals should backfill attribute values from your Snowplow `atomic` events table. On publish, Signals backfills all events from that date up to the publish timestamp using your warehouse. +Enable **Backfill attributes** when creating the group. A date picker appears — select the date from which Signals should backfill attribute values from your Snowplow `atomic` events table. On publish, Signals backfills all events from that date up to the publish timestamp using your warehouse, based on the `load_tstamp` column. Backfill operates within a defined period: from your selected start date to the publish timestamp. This can take some time depending on data volume. The streaming engine starts immediately on publish and processes all new events in real time. Once backfill completes its defined period, only the streaming engine is active. You can monitor progress with the status bar in Snowplow Console. @@ -101,7 +101,7 @@ Backfill operates within a defined period: from your selected start date to the -Set `backfill_since_tstamp` on your `StreamAttributeGroup` to specify the earliest date to backfill from. +Set `backfill_since_tstamp` on your `StreamAttributeGroup` to specify the earliest date to backfill from. This date is matched against the `load_tstamp` column of your Snowplow `atomic` events table, the time each event was loaded into your warehouse. ```python from datetime import datetime, timezone diff --git a/docs/signals/attributes/warehouse-config/index.md b/docs/signals/attributes/warehouse-config/index.md index 064358a87..2ec6e7683 100644 --- a/docs/signals/attributes/warehouse-config/index.md +++ b/docs/signals/attributes/warehouse-config/index.md @@ -17,7 +17,7 @@ To sync existing, pre-calculated attributes from your warehouse to Signals, conf If your source data comes from Snowplow events, consider using a stream attribute group with the backfill option enabled instead. It gives you real-time updates from your event stream alongside historical backfill, without needing to maintain a separate warehouse table. Attributes that are fetched from a warehouse are best suited for pre-calculated values from non-Snowplow sources, or tables that already exist independently of your event pipeline. ::: -## Provide source table details +## Configure a warehouse source @@ -26,14 +26,36 @@ When creating an attribute group, select **Warehouse** as the data source. ![Create attribute group form showing where to select the Warehouse data source in Snowplow Console](../../images/warehouse.png) -Then provide the warehouse and table details, and select which fields you want to send to Signals. +### Provide the basic configuration + +Select the **Attribute key** you have data for. The [attribute key](/docs/signals/attributes/attribute-keys/index.md) for the group must correspond to a column in the table that contains the key values. + +Set the **TTL** to control how long the data persists in the Profiles Store. + +Updates run hourly by default. Use the **Sync frequency** selector to sync less often (6h, 12h, or 24h). + +You must also specify a **Backfill start date**: the earliest date from which Signals should read rows on the first sync. + +![Create attribute group form showing where to add basic warehouse configuration in Snowplow Console](../../images/warehouse-config.png) + +### Define which fields to sync + +Select which **Table** to sync from. + +Choose the **Modified date** column: the column that tells when a particular row was last written or updated in the table (for example, a `load_tstamp` or `updated_at` column populated by your ETL process on every write). The **Modified date** column must be in UTC. A column in local time causes rows to be silently matched against the wrong sync window, with no error raised. If multiple rows exist for the same attribute key within a sync period, the engine uses the row with the latest **Modified date** value. + +:::warning[Use a last-modified timestamp] +Don't use a business or event timestamp that doesn't change when a row is reloaded or updated. The batch engine won't be able to tell the row has changed, and the data may be missed. +::: + +Finally, select which **Table fields** you want to send to Signals. ![Warehouse source configuration showing warehouse table and field mapping options](../../images/attribute-group-warehouse-fields.png) -Configure which table to sync by specifying a `BatchSource` object. +Start by configuring which table to sync by specifying a `BatchSource` object. ```python from snowplow_signals import BatchSource @@ -57,25 +79,18 @@ The table below lists all available arguments for a `BatchSource`: | `database` | The database where the attributes are stored | `string` | ✅ | | `schema` | The schema for the table of interest | `string` | ✅ | | `table` | The table where the attributes are stored | `string` | ✅ | -| `timestamp_field` | Primary timestamp of the attribute value, indicating data freshness | `string` | ❌ | +| `timestamp_field` | Primary timestamp of the attribute value, indicating data freshness | `string` | ✅ | | `owner` | The owner of the source | `string` | ❌ | -The batch engine only sends rows with a newer timestamp to the Profiles Store, based on the `timestamp_field`. If multiple rows exist for the same attribute key within a sync period, the engine uses the row with the greatest `timestamp_field` value. +The batch engine uses `timestamp_field` as the change-watermark for incremental syncs — it must reflect when a row was actually last written or updated in the table (for example, a `load_tstamp` or `updated_at` column populated by your ETL process on every write), not a business or event timestamp that stays fixed once set. `timestamp_field` must be in UTC. A column in local time causes rows to be silently matched against the wrong sync window, with no error raised. - - - -## Define which fields to sync - - - - -Select the fields (columns) from your warehouse table that you want to sync to the Profiles Store. You must also specify a backfill start date — the earliest date from which Signals should read rows on the first sync. +If multiple rows exist for the same attribute key within a sync period, the engine uses the row with the greatest `timestamp_field` value. -The [attribute key](/docs/signals/attributes/attribute-keys/index.md) for the group must correspond to a column in the table that contains the key values. +:::warning[Use a last-modified timestamp] +Don't use a business or event timestamp that doesn't change when a row is reloaded or updated. The batch engine won't be able to tell the row has changed, and the data may be missed. +::: - - +### Define which fields to sync Use `ExternalBatchAttributeGroup` to define which source table and fields to use. Instead of `attributes`, this class uses `fields` — abstractions over the warehouse columns. @@ -84,7 +99,7 @@ You must set `backfill_since_tstamp` to tell the batch engine the earliest times The attribute key must correspond to a column in the source table. The example below uses the built-in `domain_userid` key, so the table must have a `domain_userid` column. To key on a different column, [define a custom attribute key](/docs/signals/attributes/attribute-keys/index.md#custom-attribute-keys) with `external_column` set to the column name. ```python -from datetime import datetime, timezone +from datetime import datetime, timedelta, timezone from snowplow_signals import ExternalBatchAttributeGroup, domain_userid, Field attribute_group = ExternalBatchAttributeGroup( @@ -94,6 +109,7 @@ attribute_group = ExternalBatchAttributeGroup( owner="user@company.com", batch_source=data_source, backfill_since_tstamp=datetime(2026, 6, 1, tzinfo=timezone.utc), + refresh_rate=timedelta(hours=6), fields=[ Field(name="TOTAL_TRANSACTIONS", type="int32"), Field(name="TOTAL_REVENUE", type="int32"), @@ -111,6 +127,7 @@ The table below lists all available arguments for `ExternalBatchAttributeGroup`: | `attribute_key` | The key used to identify profiles. Its name must match a column in the source table | `AttributeKey` | ✅ | | `batch_source` | The `BatchSource` defining the warehouse table to sync from | `BatchSource` | ✅ | | `backfill_since_tstamp` | The earliest timestamp from which to read rows on the first sync. Accepts tz-aware or naive UTC `datetime`; tz-aware is recommended. | `datetime` | ✅ | +| `refresh_rate` | How frequently the attribute group should be synced (e.g. `timedelta(hours=1)`, `timedelta(hours=6)`, `timedelta(days=1)`). Defaults to hourly. | `timedelta` | ❌ | | `fields` | The list of `Field` objects defining which columns to sync | `list` | ✅ | | `description` | A description of the attribute group | `string` | ❌ | | `owner` | The owner of the attribute group | `string` | ❌ | diff --git a/docs/signals/concepts/index.md b/docs/signals/concepts/index.md index e814866c1..09b9f99d4 100644 --- a/docs/signals/concepts/index.md +++ b/docs/signals/concepts/index.md @@ -115,7 +115,11 @@ Real-time stream flow: Stream attribute groups by default only calculate attributes from the moment they are published. If you would like to pre-populate your attributes with values from your warehouse, you can optionally enable backfill when creating the group. Backfilling stream attributes requires a warehouse connection. -When backfill is enabled, you select a start date. On publish, Signals uses your Snowplow `atomic` events table to calculate attribute values for all events from that date up to the publish timestamp. The streaming engine starts immediately and processes all new events in real time. Backfill operates within this defined period and can take some time depending on data volume. Once backfill completes, only the streaming engine remains active. +When backfill is enabled, you select a start date. On publish, Signals uses your Snowplow `atomic` events table to calculate attribute values for all events from that date up to the publish timestamp, based on the `load_tstamp` column. The streaming engine starts immediately and processes all new events in real time. Backfill operates within this defined period and can take some time depending on data volume. Once backfill completes, only the streaming engine remains active. + +:::info[Backfill query performance] +Signals filters the events table on the `load_tstamp` column, the time an event was loaded into your warehouse. This column is typically the partition or cluster key on Snowplow-loaded event tables, so filtering on it lets queries prune partitions efficiently. +::: You can monitor progress in Snowplow Console on the attribute group's details page. While historical values are syncing, a status bar labeled `Backfill in progress` is shown. This status remains visible until the backfill period has completed. diff --git a/docs/signals/get-started/index.md b/docs/signals/get-started/index.md index fff0c6548..d73277b62 100644 --- a/docs/signals/get-started/index.md +++ b/docs/signals/get-started/index.md @@ -31,7 +31,7 @@ If you don't have a Snowplow account yet, sign up for a [Snowplow free trial](ht You will need a [Snowplow pipeline](/docs/get-started/index.md) to use Signals. Your Signals infrastructure is deployed into the same cloud as your pipeline. :::note[Warehouse support] -Only Snowflake and BigQuery are supported currently. However, you can also use Signals without the warehouse functionality. +Snowflake, BigQuery, and Databricks are supported. However, you can also use Signals without the warehouse functionality. ::: Signals consists of several new infrastructure components. When running Signals, your Snowplow pipeline will continue to process events as usual. diff --git a/docs/signals/images/warehouse-config.png b/docs/signals/images/warehouse-config.png new file mode 100644 index 000000000..f50592635 Binary files /dev/null and b/docs/signals/images/warehouse-config.png differ diff --git a/docs/signals/setup/index.md b/docs/signals/setup/index.md index 7f0a078b9..771cb192e 100644 --- a/docs/signals/setup/index.md +++ b/docs/signals/setup/index.md @@ -20,7 +20,7 @@ Log in to Console and navigate to the **Signals** section. Click **Enable** to s ![Snowplow Console Signals section showing Enable button to set up connection](../images/console-no-connection.png) :::note[Warehouse connection] -Signals can also be deployed without connecting to a warehouse. Only Snowflake and BigQuery are supported currently. +Signals can also be deployed without connecting to a warehouse. Snowflake, BigQuery, and Databricks are supported. ::: You'll need to: