Feature/annotation cache by FineAndDandy · Pull Request #3957 · NationalSecurityAgency/datawave

FineAndDandy · 2026-09-23T02:00:52Z

Adds new annotation-cache microservice service

builds on hazelcast
manages a persistent cache for annotations
responsible for write though persistence of annotations to rabbitmq
cache validation and repair after hazelcast events
reading from federated rabbitmq to hydrate the cache with writes from a remote instance
creates an api package with common classes for both the annotation-cache service and other microservices
see README.md for details

apmoriarty · 2026-09-23T10:44:38Z

+        Message<AnnotationMessage> message = MessageBuilder.withPayload(annotationMessage).setHeader(AMQP_CORRELATION_DATA_HEADER, correlationData)
+                        .setHeader(AMQP_PUBLISH_CONFIRM_CORRELATION_HEADER, correlationData).build();
+
+        log.info("Sending message synchronously, ID: {}", correlationId);


I see a lot of logging at INFO level, confirming this is intended

apmoriarty · 2026-09-23T10:44:52Z

+                throw new AnnotationStorageException("Broker rejected message (NACK). Reason: " + confirm.getReason());
+            }
+        } catch (MessagingException | AmqpException e) {
+            log.info("caught messaging exception", e);


log at ERROR?

apmoriarty · 2026-09-23T11:09:23Z

+        processor.process(missingEntry);
+        assertNull(missingEntry.getValue());
+
+        Map.Entry<String,Set<String>> differentIdEntry = new AbstractMap.SimpleEntry<>(CACHE_KEY, Set.of("other"));


Map.entry(k, v)

apmoriarty · 2026-09-23T11:14:12Z

+
+    @Test
+    void federationLockWaitRejectsInvalidDurations() {
+        java.time.Duration[] invalidValues = {null, java.time.Duration.ZERO, java.time.Duration.ofNanos(-1), java.time.Duration.ofNanos(999_999)};


no fully qualified imports

apmoriarty

A few non-blocking comments (preferences)

tharke13 · 2026-09-24T14:56:58Z

+        annotationMapConfig.addEntryListenerConfig(listenerConfig);
+
+        MapConfig documentIndexConfig = config.getMapConfig(DOC_ANNOTATIONS_MAP);
+        documentIndexConfig.setTimeToLiveSeconds(0);


I think providing a merge-policy here that unions the set of ids may help prevent need for later reconciliation, although I think the reconciler will do the job.

Suggested change

documentIndexConfig.setTimeToLiveSeconds(0);

documentIndexConfig.setTimeToLiveSeconds(0);

documentIndexConfig.setMergePolicyConfig(

new MergePolicyConfig(

UnionAnnotationIdsMergePolicy.class.getName(),

MergePolicyConfig.DEFAULT_BATCH_SIZE));

tharke13 · 2026-09-24T15:04:28Z

Example union merge policy config:
UnionAnnotationIdsMergePolicy.java

tharke13

Looks good, just the one suggestion about adding a merge policy.

tharke13

Approving since I think the custom merge policy is optional and could be added later.

drewfarris · 2026-09-25T17:02:06Z

+
+## Purpose
+
+This service provides the regional Hazelcast cache used by Sonicweb. Annotation entries are immutable and are keyed by annotation ID within a document-specific map:


Suggested change

This service provides the regional Hazelcast cache used by Sonicweb. Annotation entries are immutable and are keyed by annotation ID within a document-specific map:

This service provides the multi-site Hazelcast cache used by Sonicweb. Annotation entries are immutable and are keyed by annotation ID within a document-specific map named `doc-annotations`:

drewfarris · 2026-09-25T17:04:10Z

+
+A local Sonicweb write is written to Hazelcast and published by `AnnotationMapStore` to RabbitMQ. RabbitMQ delivery is acknowledged before the write is considered accepted by this service; Accumulo persistence and regional federation are asynchronous consumers.
+
+Federated messages are consumed locally only when their `region.id` differs from the configured `region.name`. The message must also contain `id.type`, because `Annotation` contains the document ID but not the Sonicweb identifier type.


I've often wondered whether Annotation should contain the identifier type. I'm not quite happy that it doesn't.

drewfarris · 2026-09-25T17:08:20Z

+
+## Reconciliation
+
+`AnnotationCacheReconciler` periodically treats the `annotations:*` maps as authoritative and unions missing annotation IDs into `doc-annotations`. Reconciliation is intentionally add-only: stale IDs are measured and logged, while normal expiration/removal listeners remain responsible for deleting them. Repairing an index also invalidates that document's fetch records.


I think I'm missing fundamental information about what the annotations:* maps contain and why it might be necessary to "union missing annotation ID's into doc-annotations. Perhaps you could provide additional background or explanation earlier on in this document? Could you explain why reconciliation is needed (e.g, X falls out of sync with Y because Z)

drewfarris · 2026-09-25T17:19:20Z

+        }
+
+        ids.add(annotationId);
+        entry.setValue(ids);


Is it necessary to set the value every time, or could you move the setValue inside the null check or utilize computeIfAbsent here instead? I'm thinking this must be a hazelcastism, because I see similar behavior in RemoveAnnotationIdProcessor. If it's a hazelcastism, it might be worth explicitly calling this out briefly in a comment or javadoc.

drewfarris · 2026-09-25T17:28:41Z

+    public static final String ANNOTATIONS_MAP = "annotations:";
+    public static final String DOC_ANNOTATIONS_MAP = "doc-annotations";
+    public static final String FETCH_MAP = "doc-fetch-record:";


It might be cool to add minimal javadoc for each of these here, but perhaps if each of these is explained in the README in detail, it won't be necessary.

drewfarris · 2026-09-25T17:45:20Z

+            }
+
+            @Override
+            protected Object convertToInternal(Object payload, org.springframework.messaging.MessageHeaders headers, Object conversionHint) {


Looks like a fully qualified class name snuck in here.

drewfarris · 2026-09-25T17:49:36Z

+    }
+
+    public boolean send(Message<AnnotationMessage> message) {
+        return streamBridge.send("persisted-out-0", message);


Should persisted-out-0 be configurable or a constant instead of a raw string?

drewfarris · 2026-09-25T18:13:14Z

+import datawave.microservice.annotationCache.config.AnnotationCacheProperties;
+
+/**
+ * Periodically repairs missing entries in the derived document annotation index.


I'm still not quite sure what the purpose of the derived document annotation index is based on the available documentation :)

drewfarris · 2026-09-25T18:19:04Z

+                            result.mapsScanned, result.missingIdsRepaired, result.staleIdsObserved, result.failures,
+                            result.cycleComplete ? "" : "; more maps remain");
+        } catch (HazelcastInstanceNotActiveException e) {
+            log.debug("Hazelcast stopped during annotation cache reconciliation", e);


It seems off to have a no-op debug message in an exception handler. Maybe this should be at least info?

Suggested change

log.debug("Hazelcast stopped during annotation cache reconciliation", e);

log.info("Hazelcast stopped during annotation cache reconciliation", e);

drewfarris · 2026-09-25T18:26:54Z

+import datawave.microservice.annotationCache.api.RegionConfiguration;
+
+@Component
+public class AnnotationMapStore implements MapStore<String,Object>, HazelcastInstanceAware, MapLoaderLifecycleSupport {


A tiny bit of class Javadoc would be nice.

FineAndDandy · 2026-10-09T01:59:51Z

I wanted to document the motivations for the pom changes to annotation-cache here. This was inspired/patterned off of the authorization service

Aligned the aggregator and API with datawave-microservice-parent:4.0.14.
The API now inherits the shared parent directly rather than the annotation-cache aggregator, matching authorization’s structure and receiving the current shared dependency and build-policy baseline.
Updated spring-boot-starter-datawave from 4.0.9 to 4.0.13.
Aligns with authorization’s starter version and avoids selecting an older starter over newer transitive versions.
Moved annotation-cache API and annotation-messaging versions into .
Separates version policy from dependency declarations and makes overrides easier to maintain.
Removed unused aggregator version properties and standardized the Hazelcast service property name.
Keeps dependency configuration in the module that consumes it. The Hazelcast service dependency now uses released 4.0.7 rather than 4.0.8-SNAPSHOT.
Removed datawave-core-annotation and hazelcast-common from the API.
The API does not use core annotation classes or Hazelcast/common functionality in production. Removing these dependencies avoids exporting their substantial transitive graphs. Hazelcast remains explicitly declared with test scope for serialization and shared-map tests.
Declared the API’s Spring dependencies explicitly.
RegionConfiguration intentionally provides shared Spring configuration for annotation-cache and Sonicweb. Its spring-boot and spring-context dependencies are now explicit, with versions matching annotation-cache’s service baseline.
Declared directly used service and test libraries explicitly.
Added Hazelcast, Spring context/messaging, Spring AMQP/Rabbit, Spring Cloud commons/stream, Java annotations, and test dependencies rather than relying on unrelated modules to supply them transitively. Versions come from inherited management.
Added the Rabbit Stream binder as a direct dependency.
RabbitMQ is an intentional runtime transport requirement. The service should not depend on the DataWave starter continuing to include Spring Cloud Bus AMQP to obtain its binder. The inherited Cloud BOM manages its version.
Retained and documented the service’s core-annotation dependency and exclusions.
Service code directly uses its generated Annotation and AnnotationMessage classes. The exclusions prevent the broader core dependency graph from entering the service; annotation-messaging supplies the Protobuf runtime.
Added API/service project and SCM metadata and documented the existing JBoss annotation exclusion.
Completes the standalone module metadata and explains the use of the explicitly declared Java annotation API.

drewfarris · 2026-10-09T15:28:00Z

#3964 is now merged, so it might be useful to rebase on that to pick up the annotation-messaging module.

drewfarris · 2026-10-09T15:38:35Z

Consider integratingannotation-cache/api/README.md and annotation-cache/service/README.md into a single annotation-cache/README.md or providing a top level readme that references each?

drewfarris · 2026-10-09T15:47:56Z

+
+A fetch marker records a successful Datawave lookup for one authorization context. It does not guarantee that the annotation cache contains a complete snapshot. Its presence and TTL determine freshness; `FetchRecord` fields are diagnostic only.
+
+Sonicweb writes a marker after source retrieval and cache merge succeed. Empty and not-found responses also receive markers. Errors do not. Cache hits do not renew markers, and expired markers cause a source lookup on the next request, not a background refresh. Because markers are written after retrieval, source latency adds to the age of the source observation.


I'm a little confused by the statement "Sonicweb writes a marker after source retrieval and cache merge succeed" compared to the following statement: "Empty and not-found responses also receive markers".

Suggested change

Sonicweb writes a marker after source retrieval and cache merge succeed. Empty and not-found responses also receive markers. Errors do not. Cache hits do not renew markers, and expired markers cause a source lookup on the next request, not a background refresh. Because markers are written after retrieval, source latency adds to the age of the source observation.

Sonicweb writes fetch markers for source lookups that return annotations, no results, or “not found.” It writes the marker after retrieval and cache merging finish without error; failures do not create markers. Serving cached results does not extend a marker’s lifetime. Once a marker expires, the next request queries the source again; expiration does not trigger a background refresh. The marker’s lifetime starts after retrieval finishes, so the source data may be older than the marker by the time spent retrieving it.

drewfarris · 2026-10-09T19:38:33Z

+  topology-poll-interval-ms: 5000
+```
+
+Both TTLs must be positive whole seconds, and `max-fetch-age` must not exceed `max-cache-age`. Max-idle expiration is disabled. Federation lock wait and topology poll interval must be positive; topology settle delay may be zero.


If the configuration property was x-y-ttl, it would be fine to refer to them as TTLs in the documents, but it take a second more to parse this paragraph since it's not obvious for the attention starved developer to make the connection. Suggest a sligh modification

Suggested change

Both TTLs must be positive whole seconds, and `max-fetch-age` must not exceed `max-cache-age`. Max-idle expiration is disabled. Federation lock wait and topology poll interval must be positive; topology settle delay may be zero.

Both `max-cache-age` and `max-fetch-age` must be positive whole seconds. `max-fetch-age` must not exceed `max-cache-age`. Max-idle expiration is disabled, so a lack of activity does not cause entries to drop from the cache. `federation-lock-wait` wait and `topology-poll-interval-ms` must be positive; `topology-settle-delay` may be zero.

drewfarris · 2026-10-09T19:40:51Z

+
+Both TTLs must be positive whole seconds, and `max-fetch-age` must not exceed `max-cache-age`. Max-idle expiration is disabled. Federation lock wait and topology poll interval must be positive; topology settle delay may be zero.
+
+RabbitMQ bindings and connection settings are supplied separately through application configuration. Bind `persisted-out-0` and `loadCache-in-0` to `annotation`, use consumer group `cache`, and set the content type to `application/x-protobuf`. Publication requires correlated confirms and returns. The companion Helm configuration is `configuration/configMapFiles/annotation-cache.yml` in the `datawave-helm-charts` repository.


Take a look at the docker/config directory - there are sample configurations here for use in docker-compose that also serve as good exemplars for use elsewhere.

…lcast

…e.annotation

…er to user AnnotationMessage and process all varients of eviction. LoadCacheConsumer to use AnnotationMessage and avoid republishing to AnnotationMapStore

…interaction

Upgrade datawave dependency to 8.1.0 Remove now redundant classes MessageConfig and SpecificProtobufConverterConfig

…unique maps for each document

… patterns similar to authorization service

apmoriarty reviewed Sep 23, 2026

View reviewed changes

apmoriarty previously approved these changes Sep 23, 2026

View reviewed changes

tharke13 self-assigned this Sep 24, 2026

tharke13 reviewed Sep 24, 2026

View reviewed changes

tharke13 previously approved these changes Sep 24, 2026

View reviewed changes

drewfarris added the no-downstream-effects label Sep 24, 2026

drewfarris reviewed Sep 25, 2026

View reviewed changes

FineAndDandy dismissed stale reviews from tharke13 and apmoriarty via 9c4afc4 September 30, 2026 23:32

drewfarris assigned FineAndDandy and drewfarris Oct 9, 2026

drewfarris reviewed Oct 9, 2026

View reviewed changes

FineAndDandy added 8 commits October 10, 2026 02:35

add annotation cache services

213c54c

Added sync publisher confirms and manditory exchange/topic setup

706b724

Add consumer listening on annotation.cache topic

35a8bd8

demonstrate pulling an object off the queue and adding it to the haze…

9e9af5f

…lcast

Add annotation-cache support for AnnotationMessageProto

af35750

Migrate from internal protobuf AnnotationMessageProto to datawave.cor…

3b08a37

…e.annotation

formatting

8018eca

Migrate AnnotationCacheProperties from sonicweb. AnnotationSyncListen…

209b77b

…er to user AnnotationMessage and process all varients of eviction. LoadCacheConsumer to use AnnotationMessage and avoid republishing to AnnotationMapStore

FineAndDandy added 21 commits October 10, 2026 02:35

Add integration test for TTL handling

c81c479

Add integration tests that cover federation, removal, and multi-node …

c40b1cd

…interaction

Add PersistenceMode to annotation-cache api

1d25d66

Use AnnotationMessage Region and PersistenceMode

29d2e29

Add federated annotations directly to the document index

49fb05a

Added AnnotationCacheReconciler to monitor for and repair cache breaks

5f0dd23

Change lock to tryLock with a timeout in LoadCacheConsumer

aee83fe

Clarify unclear documentation in AnnotationCacheReconciler

7739055

Remove doc-annotations map and associated index usage

3b7335c

Add dependency on annotation.messaging

e3e5af9

Upgrade datawave dependency to 8.1.0 Remove now redundant classes MessageConfig and SpecificProtobufConverterConfig

Remove duplicate starter dependency in service/pom.xml

163726c

Add annotation-cache service to aggregate services profile

feb822d

Updated hazelcast versions to match integration

e27f48a

Remove fully qualified names embedded in code

6711a9a

Require id type and region on annotations

f07338c

Add annotation-cache api/service modules to microservice workflows

74a7281

Add repository to annotation-cache poms

9eb5d76

Add AnnotationMapStore documentation and update README

afe1917

Documentation updates

914dd02

Use a fixed two maps for annotations and doc-fetch-record instead of …

8cf8422

…unique maps for each document

Updated the annotation-cache POMs to follow the dependency-management…

bc90b9c

… patterns similar to authorization service

FineAndDandy force-pushed the feature/annotation-cache branch from 5f56a69 to bc90b9c Compare October 10, 2026 02:36

-        documentIndexConfig.setTimeToLiveSeconds(0);
+        documentIndexConfig.setTimeToLiveSeconds(0);
+        documentIndexConfig.setMergePolicyConfig(
+                new MergePolicyConfig(
+                        UnionAnnotationIdsMergePolicy.class.getName(),
+                        MergePolicyConfig.DEFAULT_BATCH_SIZE));


		## Purpose

		This service provides the regional Hazelcast cache used by Sonicweb. Annotation entries are immutable and are keyed by annotation ID within a document-specific map:

	This service provides the regional Hazelcast cache used by Sonicweb. Annotation entries are immutable and are keyed by annotation ID within a document-specific map:
	This service provides the multi-site Hazelcast cache used by Sonicweb. Annotation entries are immutable and are keyed by annotation ID within a document-specific map named `doc-annotations`:


		A local Sonicweb write is written to Hazelcast and published by `AnnotationMapStore` to RabbitMQ. RabbitMQ delivery is acknowledged before the write is considered accepted by this service; Accumulo persistence and regional federation are asynchronous consumers.

		Federated messages are consumed locally only when their `region.id` differs from the configured `region.name`. The message must also contain `id.type`, because `Annotation` contains the document ID but not the Sonicweb identifier type.


		## Reconciliation

		`AnnotationCacheReconciler` periodically treats the `annotations:*` maps as authoritative and unions missing annotation IDs into `doc-annotations`. Reconciliation is intentionally add-only: stale IDs are measured and logged, while normal expiration/removal listeners remain responsible for deleting them. Repairing an index also invalidates that document's fetch records.

	log.debug("Hazelcast stopped during annotation cache reconciliation", e);
	log.info("Hazelcast stopped during annotation cache reconciliation", e);


		A fetch marker records a successful Datawave lookup for one authorization context. It does not guarantee that the annotation cache contains a complete snapshot. Its presence and TTL determine freshness; `FetchRecord` fields are diagnostic only.

		Sonicweb writes a marker after source retrieval and cache merge succeed. Empty and not-found responses also receive markers. Errors do not. Cache hits do not renew markers, and expired markers cause a source lookup on the next request, not a background refresh. Because markers are written after retrieval, source latency adds to the age of the source observation.

	Both TTLs must be positive whole seconds, and `max-fetch-age` must not exceed `max-cache-age`. Max-idle expiration is disabled. Federation lock wait and topology poll interval must be positive; topology settle delay may be zero.
	Both `max-cache-age` and `max-fetch-age` must be positive whole seconds. `max-fetch-age` must not exceed `max-cache-age`. Max-idle expiration is disabled, so a lack of activity does not cause entries to drop from the cache. `federation-lock-wait` wait and `topology-poll-interval-ms` must be positive; `topology-settle-delay` may be zero.


		Both TTLs must be positive whole seconds, and `max-fetch-age` must not exceed `max-cache-age`. Max-idle expiration is disabled. Federation lock wait and topology poll interval must be positive; topology settle delay may be zero.

		RabbitMQ bindings and connection settings are supplied separately through application configuration. Bind `persisted-out-0` and `loadCache-in-0` to `annotation`, use consumer group `cache`, and set the content type to `application/x-protobuf`. Publication requires correlated confirms and returns. The companion Helm configuration is `configuration/configMapFiles/annotation-cache.yml` in the `datawave-helm-charts` repository.

Conversation

FineAndDandy commented Sep 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apmoriarty left a comment

Choose a reason for hiding this comment

Uh oh!

tharke13 Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tharke13 left a comment

Choose a reason for hiding this comment

Uh oh!

tharke13 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FineAndDandy commented Oct 9, 2026

Uh oh!

drewfarris commented Oct 9, 2026

Uh oh!

drewfarris Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tharke13 Sep 24, 2026 •

edited

Loading

drewfarris Oct 9, 2026 •

edited

Loading