Add automatic data-attribute-to-Postgres sync#308
Open
longquanzheng wants to merge 3 commits into
Open
Conversation
Mirror a workflow data attribute to a cell of a user-owned Postgres table: the iWF data attribute stays the source of truth; the DB cell is a mirror. - DataAttributeDef.create(type, key, DbAttributeSync) declares the mapping (DataSource + table + pk column + a (workflowId, persistence) -> CellLocation locator) - On start, Client swaps the start state to a worker-internal load state that SELECTs each mapped column, sets the data attribute, then transitions to the real starting state with the original input passed through. - When a state's execute mutates a mapped data attribute, the worker reroutes the decision through a worker-internal sync state (carrying DbSyncStatePayload) that UPDATEs the column, then replays the original decision verbatim. iWF persists first, so a DB failure retries only the sync step, not user code. - New: CellLocation, DbAttributeSync, PostgresDataAttributeSyncer (parameterized values + allowlisted/quoted identifiers, try-with-resources), DbSyncStatePayload. - Neither system state is registered; both are handled specially in WorkerService. - Integration test (DbSyncTest) against a real Postgres via docker-compose + init script (schema/table/seed); CI wired with a Postgres readiness wait. - Docs: user guide + concise overview, linked from README. Scope (v1): fixed-key data attributes only; sync on execute mutations only; load fires for starts via the registered Client; column stores the attribute JSON. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rver readiness The pinned Elasticsearch 7.16.2 crashes on startup on cgroup-v2 hosts (current GitHub runners) with a bundled-JDK NPE in DefaultSystemMemoryInfo/CgroupInfo, so ES never starts, Temporal's frontend never becomes ready, and every integration test fails on connectivity. ES 7.17.28 bundles a cgroup-v2-safe JDK and is compatible with Temporal 1.25 (verified locally: ES cluster health green, iWF server ready). Also replace the fixed `sleep 30` before the build with an active wait that polls the iWF server until it serves, so the build only starts once the full Temporal + Elasticsearch stack is actually ready. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in, SDK-side feature to automatically mirror a workflow data attribute to a single cell (row + column) of a user-owned Postgres table:
execute, the new value is written back to the column.The iWF data attribute stays the source of truth; the DB cell is a mirror. The SDK is driver-agnostic (depends only on
javax.sql.DataSource); the user owns the schema, credentials, and pooling.API
How it works
Clientswaps the start state to a worker-internal load state thatSELECTs each mapped column, sets the data attribute, then transitions to the real starting state (original input passed through).execute, the worker reroutes the decision through a worker-internal sync state (carrying the captured original decision asDbSyncStatePayload) thatUPDATEs the column, then replays the original decision verbatim — so any decision shape (single/multi-next, graceful/force complete/fail, conditionalClose) is preserved.executecode.Registry; both are handled specially inWorkerServiceby reserved stateId.Security
DataSource).PreparedStatement; identifiers validated against an allowlist and double-quoted.Tests / CI
DbSyncTestruns end-to-end against a real Postgres (script/docker-compose.ymliwf-integ-postgres+script/postgres-init/iwf-integ-init.sql) and a live iWF server; verified locally (load + sync).Scope (v1)
createByPrefix).executemutations only (notwaitUntil).Client.text/jsonb.Docs