Event replay poc

[zoldar]

Changes

This PR implements support for replaying events via existing ingestion pipeline. Although the current implementation is entirely contained in analytics repository, the eventual solution will be split among analytics and persistor.

The plan for release is following:

• ClickHouse database migration - merge, release and run; the only thing it does is renaming already existing columns in our cluster, which were added when we were recovering lost events from persistor some time ago
• Persistor logic changes - extend the ClickHouse schemas for event and session with replay_session_id field and use a different cache key for replayed events as well in CacheStore; the change will be merged and released as well
• Finally, analytics pipeline changes - a preview app will be created first to test recovery using an already setup staging instance of ingestvcr service; merging the PR will be held off until headers sanitisation will be implemented in our infra to prevent sending replay headers from outside our system

Implementation details

Recording

A simplified diagram of how recording is integrated into the event ingestion pipeline:

When the event payload reach Ingress proxy, the haproxy dispatches its body and headers to the agent process listening on a tcp socket. The agent deserialises the payload, assigns a random ID to the event and asynchronously persists the event in the store while immediately returning the newly assigned ID to the proxy. The proxy appends it as X-Replay-Event-ID header. As of now, it's not used and is ignored down the pipeline.

The analytics ingest pipeline processes the event normally without any additional processing and so does persistor, after which the event ends up batched and inserted into ClickHouse DB.

Replaying

When replaying, each payload will go through the same ingress proxy but it won't be sent to the recording agent on the basis of replay header being already present. The replay utility appends 3 following headers when sending the payloads:

• X-Replay-Event-ID (ignored for now, like in the case of recording)
• X-Replay-Session-ID
• X-Replay-Time

Once the payload reaches ingestion pipeline, it's decoded in the request building phase. There, for replayed event, the event timestamp is replaced with the one provided in X-Replay-Time and X-Replay-Session-ID is put under request.replay_session_id. The replay session ID is then carried over to event attributes.

The put_salts step is crucial in ensuring that replayed events are processed in complete isolation from other events even if some user sessions are still valid from the time of recording. Without that, the existing sessions will get corrupted. Processing replayed events in isolation is a trade-off we have to make - there's no feasible way to retroactively adjust existing sessions, especially somewhere in the middle.

The isolation is achieved by - instead of using the salt from the database - generating a pseudo-random hash on the basis of replay session ID using Plug.Crypto.KeyGenerator and mixing replay session ID with secret key base in a safe way. This brings another challenge, because the hash is generated using PBKDF2 with default number of 1000 iterations, which takes around 1-2 ms when tested locally. Fetching salt from ETS cache takes only a couple microseconds in comparison. Even reducing number of iterations to 1 (which raises safety concerns over risk of exposing secret key base too easily) the timing is still strongly in favor of ETS. To work around that, we use KeyGenerator's built-in ability to cache computations using an ETS table. A separate process manages the table and purges it every 2 hours, just to be safe (though cache table blowing up is unlikely, given session IDs can't be provided externally).

The last step is session management in persistor. Replayed sessions are stored and fetched using distinct shape of cache key, including replay session ID. Although risk of user ID collisions is low, it's still better to be on the safe side - that's one more level of ensuring isolation.

Lastly, when a new replayed session is provisioned, its replay session ID attribute is populated accordingly, like in the case of event.

Recording replay session ID allows relatively easy "rollback" of a botched replay and starting over again. Also, two different replays (with different session IDs), even if overlapping, are easy to tell apart and clean up properly as well. An interrupted session can also be safely resumed (the replay utility allows to provide an existing session ID; otherwise it generates a unique one on every run).

Tests

• Automated tests have been added