Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/signals/attributes/attribute-groups/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,10 @@ Stream attribute groups only calculate attributes from the moment they are publi
<TabItem value="console" label="Console" default>

:::note[Warehouse connection]
A warehouse connection is required to use the backfill option. Only Snowflake and BigQuery are supported currently.
A warehouse connection is required to use the backfill option. Snowflake, BigQuery, and Databricks are supported.
:::

Enable **Backfill attributes** when creating the group. A date picker appears — select the date from which Signals should backfill attribute values from your Snowplow `atomic` events table. On publish, Signals backfills all events from that date up to the publish timestamp using your warehouse.
Enable **Backfill attributes** when creating the group. A date picker appears — select the date from which Signals should backfill attribute values from your Snowplow `atomic` events table. On publish, Signals backfills all events from that date up to the publish timestamp using your warehouse, based on the `load_tstamp` column.

Backfill operates within a defined period: from your selected start date to the publish timestamp. This can take some time depending on data volume. The streaming engine starts immediately on publish and processes all new events in real time. Once backfill completes its defined period, only the streaming engine is active. You can monitor progress with the status bar in Snowplow Console.

Expand All @@ -101,7 +101,7 @@ Backfill operates within a defined period: from your selected start date to the
</TabItem>
<TabItem value="sdk" label="Python SDK">

Set `backfill_since_tstamp` on your `StreamAttributeGroup` to specify the earliest date to backfill from.
Set `backfill_since_tstamp` on your `StreamAttributeGroup` to specify the earliest date to backfill from. This date is matched against the `load_tstamp` column of your Snowplow `atomic` events table, the time each event was loaded into your warehouse.

```python
from datetime import datetime, timezone
Expand Down
53 changes: 35 additions & 18 deletions docs/signals/attributes/warehouse-config/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ To sync existing, pre-calculated attributes from your warehouse to Signals, conf
If your source data comes from Snowplow events, consider using a stream attribute group with the backfill option enabled instead. It gives you real-time updates from your event stream alongside historical backfill, without needing to maintain a separate warehouse table. Attributes that are fetched from a warehouse are best suited for pre-calculated values from non-Snowplow sources, or tables that already exist independently of your event pipeline.
:::

## Provide source table details
## Configure a warehouse source

<Tabs groupId="signals-impl" queryString>
<TabItem value="console" label="Console" default>
Expand All @@ -26,14 +26,36 @@ When creating an attribute group, select **Warehouse** as the data source.

![Create attribute group form showing where to select the Warehouse data source in Snowplow Console](../../images/warehouse.png)

Then provide the warehouse and table details, and select which fields you want to send to Signals.
### Provide the basic configuration

Select the **Attribute key** you have data for. The [attribute key](/docs/signals/attributes/attribute-keys/index.md) for the group must correspond to a column in the table that contains the key values.

Set the **TTL** to control how long the data persists in the Profiles Store.

Updates run hourly by default. Use the **Sync frequency** selector to sync less often (6h, 12h, or 24h).

You must also specify a **Backfill start date**: the earliest date from which Signals should read rows on the first sync.

![Create attribute group form showing where to add basic warehouse configuration in Snowplow Console](../../images/warehouse-config.png)

### Define which fields to sync

Select which **Table** to sync from.

Choose the **Modified date** column: the column that tells when a particular row was last written or updated in the table (for example, a `load_tstamp` or `updated_at` column populated by your ETL process on every write). The **Modified date** column must be in UTC. A column in local time causes rows to be silently matched against the wrong sync window, with no error raised. If multiple rows exist for the same attribute key within a sync period, the engine uses the row with the latest **Modified date** value.

:::warning[Use a last-modified timestamp]
Don't use a business or event timestamp that doesn't change when a row is reloaded or updated. The batch engine won't be able to tell the row has changed, and the data may be missed.
:::

Finally, select which **Table fields** you want to send to Signals.

![Warehouse source configuration showing warehouse table and field mapping options](../../images/attribute-group-warehouse-fields.png)

</TabItem>
<TabItem value="sdk" label="Python SDK">

Configure which table to sync by specifying a `BatchSource` object.
Start by configuring which table to sync by specifying a `BatchSource` object.

```python
from snowplow_signals import BatchSource
Expand All @@ -57,25 +79,18 @@ The table below lists all available arguments for a `BatchSource`:
| `database` | The database where the attributes are stored | `string` | ✅ |
| `schema` | The schema for the table of interest | `string` | ✅ |
| `table` | The table where the attributes are stored | `string` | ✅ |
| `timestamp_field` | Primary timestamp of the attribute value, indicating data freshness | `string` | |
| `timestamp_field` | Primary timestamp of the attribute value, indicating data freshness | `string` | |
| `owner` | The owner of the source | `string` | ❌ |

The batch engine only sends rows with a newer timestamp to the Profiles Store, based on the `timestamp_field`. If multiple rows exist for the same attribute key within a sync period, the engine uses the row with the greatest `timestamp_field` value.
The batch engine uses `timestamp_field` as the change-watermark for incremental syncs — it must reflect when a row was actually last written or updated in the table (for example, a `load_tstamp` or `updated_at` column populated by your ETL process on every write), not a business or event timestamp that stays fixed once set. `timestamp_field` must be in UTC. A column in local time causes rows to be silently matched against the wrong sync window, with no error raised.

</TabItem>
</Tabs>

## Define which fields to sync

<Tabs groupId="signals-impl" queryString>
<TabItem value="console" label="Console" default>

Select the fields (columns) from your warehouse table that you want to sync to the Profiles Store. You must also specify a backfill start date — the earliest date from which Signals should read rows on the first sync.
If multiple rows exist for the same attribute key within a sync period, the engine uses the row with the greatest `timestamp_field` value.

The [attribute key](/docs/signals/attributes/attribute-keys/index.md) for the group must correspond to a column in the table that contains the key values.
:::warning[Use a last-modified timestamp]
Don't use a business or event timestamp that doesn't change when a row is reloaded or updated. The batch engine won't be able to tell the row has changed, and the data may be missed.
:::

</TabItem>
<TabItem value="sdk" label="Python SDK">
### Define which fields to sync

Use `ExternalBatchAttributeGroup` to define which source table and fields to use. Instead of `attributes`, this class uses `fields` — abstractions over the warehouse columns.

Expand All @@ -84,7 +99,7 @@ You must set `backfill_since_tstamp` to tell the batch engine the earliest times
The attribute key must correspond to a column in the source table. The example below uses the built-in `domain_userid` key, so the table must have a `domain_userid` column. To key on a different column, [define a custom attribute key](/docs/signals/attributes/attribute-keys/index.md#custom-attribute-keys) with `external_column` set to the column name.

```python
from datetime import datetime, timezone
from datetime import datetime, timedelta, timezone
from snowplow_signals import ExternalBatchAttributeGroup, domain_userid, Field

attribute_group = ExternalBatchAttributeGroup(
Expand All @@ -94,6 +109,7 @@ attribute_group = ExternalBatchAttributeGroup(
owner="user@company.com",
batch_source=data_source,
backfill_since_tstamp=datetime(2026, 6, 1, tzinfo=timezone.utc),
refresh_rate=timedelta(hours=6),
fields=[
Field(name="TOTAL_TRANSACTIONS", type="int32"),
Field(name="TOTAL_REVENUE", type="int32"),
Expand All @@ -111,6 +127,7 @@ The table below lists all available arguments for `ExternalBatchAttributeGroup`:
| `attribute_key` | The key used to identify profiles. Its name must match a column in the source table | `AttributeKey` | ✅ |
| `batch_source` | The `BatchSource` defining the warehouse table to sync from | `BatchSource` | ✅ |
| `backfill_since_tstamp` | The earliest timestamp from which to read rows on the first sync. Accepts tz-aware or naive UTC `datetime`; tz-aware is recommended. | `datetime` | ✅ |
| `refresh_rate` | How frequently the attribute group should be synced (e.g. `timedelta(hours=1)`, `timedelta(hours=6)`, `timedelta(days=1)`). Defaults to hourly. | `timedelta` | ❌ |
| `fields` | The list of `Field` objects defining which columns to sync | `list` | ✅ |
| `description` | A description of the attribute group | `string` | ❌ |
| `owner` | The owner of the attribute group | `string` | ❌ |
Expand Down
6 changes: 5 additions & 1 deletion docs/signals/concepts/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,11 @@ Real-time stream flow:

Stream attribute groups by default only calculate attributes from the moment they are published. If you would like to pre-populate your attributes with values from your warehouse, you can optionally enable backfill when creating the group. Backfilling stream attributes requires a warehouse connection.

When backfill is enabled, you select a start date. On publish, Signals uses your Snowplow `atomic` events table to calculate attribute values for all events from that date up to the publish timestamp. The streaming engine starts immediately and processes all new events in real time. Backfill operates within this defined period and can take some time depending on data volume. Once backfill completes, only the streaming engine remains active.
When backfill is enabled, you select a start date. On publish, Signals uses your Snowplow `atomic` events table to calculate attribute values for all events from that date up to the publish timestamp, based on the `load_tstamp` column. The streaming engine starts immediately and processes all new events in real time. Backfill operates within this defined period and can take some time depending on data volume. Once backfill completes, only the streaming engine remains active.

:::info[Backfill query performance]
Signals filters the events table on the `load_tstamp` column, the time an event was loaded into your warehouse. This column is typically the partition or cluster key on Snowplow-loaded event tables, so filtering on it lets queries prune partitions efficiently.
:::

You can monitor progress in Snowplow Console on the attribute group's details page. While historical values are syncing, a status bar labeled `Backfill in progress` is shown. This status remains visible until the backfill period has completed.

Expand Down
2 changes: 1 addition & 1 deletion docs/signals/get-started/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ If you don't have a Snowplow account yet, sign up for a [Snowplow free trial](ht
You will need a [Snowplow pipeline](/docs/get-started/index.md) to use Signals. Your Signals infrastructure is deployed into the same cloud as your pipeline.

:::note[Warehouse support]
Only Snowflake and BigQuery are supported currently. However, you can also use Signals without the warehouse functionality.
Snowflake, BigQuery, and Databricks are supported. However, you can also use Signals without the warehouse functionality.
:::

Signals consists of several new infrastructure components. When running Signals, your Snowplow pipeline will continue to process events as usual.
Expand Down
Binary file added docs/signals/images/warehouse-config.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/signals/setup/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Log in to Console and navigate to the **Signals** section. Click **Enable** to s
![Snowplow Console Signals section showing Enable button to set up connection](../images/console-no-connection.png)

:::note[Warehouse connection]
Signals can also be deployed without connecting to a warehouse. Only Snowflake and BigQuery are supported currently.
Signals can also be deployed without connecting to a warehouse. Snowflake, BigQuery, and Databricks are supported.
:::

You'll need to:
Expand Down