perf(detector): share one vuls2 db session across a server's detect and enrich#2600
Merged
Merged
Conversation
7c11cb8 to
4b0412d
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes vuls2-backed detection/enrichment by introducing a lazily-opened vuls2 DB Session that can be shared across a single server’s detection and subsequent enrichment, reusing the warmed read cache instead of reopening the DB and rebuilding cache multiple times.
Changes:
- Introduces
vuls2.Sessionwith lazy open/close behavior and updates vuls2 detect/enrich entry points to use it. - Updates per-server detection flow (
detector.Detect) to create one session per server and share it across package detection, CPE detection, and enrichment. - Updates server-mode handler to share one session across request-scoped detection and enrichment.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| server/server.go | Creates a request-scoped vuls2 Session and reuses it for detection + enrichment in server mode. |
| detector/vuls2/vuls2.go | Adds the Session type, factors DB opening into openSession, and routes detect/enrich through the shared session. |
| detector/detector.go | Creates a per-server vuls2 Session and threads it through detection and enrichment steps. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
4b0412d to
292162d
Compare
292162d to
1753596
Compare
2f17531 to
b4ac0e6
Compare
b4ac0e6 to
72160a3
Compare
…nd enrich detect (DetectPkgs/DetectCPEs) and enrich (EnrichVulnInfos) each opened, used, and discarded their own vuls2 boltDB session — so within one report run a single server opened the db multiple times, each redoing the metadata read / schema-version check and, more importantly, throwing away the WithCache read cache on close. detect and enrich query largely the same CVEs, so enrich was rebuilding a cache detect had just warmed. Introduce a lazily-opened, shared vuls2.Session: the db is opened at most once (on the first path that queries it) and reused by every later path within one server's turn, then Closed by the owner. Detection warms the cache and the enrichment that immediately follows reuses it. The session opens exactly when and only when the unshared code opened one — no earlier, no more often: a family that skips vuls2 detection (FreeBSD, pseudo, trivy-scanned, ...) does not open it during detection, but enrichment still opens it when the result carries CVEs (e.g. FreeBSD's pkg-audit findings), and a server with nothing to detect and no CVEs to enrich never opens it at all. So no unnecessary db open/download is forced. DetectPkgs / DetectCPEs / EnrichVulnInfos / DetectPkgCves / DetectCpeURIsCves now take the db handle as a required *vuls2.Session, created with NewSession (which carries the Vuls2Conf/noProgress the db needs) and Closed by the caller. The Session is the single handle threaded through one server's detection and enrichment; detector.Detect (per-server loop) and the server-mode handler create one per server / request and share it across all three paths. A nil Session is rejected with an error rather than panicking. Scope is per-server: the session — and its unbounded read cache — is freed after each server, keeping peak memory at ~one server's working set rather than growing across the whole run, and leaving detection naturally safe to parallelize over servers later. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
72160a3 to
280ef04
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Within one report run,
detect(vuls2.DetectPkgs/DetectCPEs) andenrich(vuls2.EnrichVulnInfos) each opened, used, and discarded their own vuls2 boltDB session. So a single server opened the db multiple times, each time redoing the metadata read / schema-version check and — the costly part — throwing away theWithCacheread cache on close. Since detect and enrich query largely the same CVEs, enrich was rebuilding a cache detect had just warmed.This PR introduces a lazily-opened, shared
vuls2.Sessionand reuses one session per server across that server's detection and the enrichment that follows.How
vuls2.Session: a lazily-opened handle. The db is opened at most once — on the first path that actually queries it — and reused by every later path within one server's turn, then closed by the owner.pkg auditfindings); a server with nothing to detect and no CVEs to enrich never opens it at all. So no unnecessary db open/download is forced.*vuls2.Session, created withNewSession(which carries theVuls2Conf/noProgressthe db needs) andClosed by the caller.DetectPkgs/DetectCPEs/EnrichVulnInfos/DetectPkgCves/DetectCpeURIsCvestake it and share it across one server's detection + enrichment. A nilSessionis rejected with an error rather than panicking.detector.Detect(per-server loop) and the server-mode handler create oneSessionper server / request and share it across all three paths.Scope of sharing: per-server (deliberate)
A whole-run variant (one session for all servers) was prototyped and compared. The vuls2 read cache is an unbounded
sync.Mapkeyed by advisory/vulnerability ID, freed only onClose(). Whole-run keeps it warm across servers (faster on homogeneous fleets) but grows peak memory monotonically with the distinct advisories/vulns touched across the whole run.Per-server was chosen because:
Detectis ever parallelized over servers, movingNewSessioninto the goroutine is naturally safe; a run-wide shared session would need synchronization.Compatibility
DetectPkgs/DetectCPEs/EnrichVulnInfos) and the detector wrappers (DetectPkgCves/DetectCpeURIsCves) drop the oldvuls2Conf/noProgressparameters and take a required trailing*vuls2.Sessioninstead — that config now lives on theSession(viaNewSession). External library consumers must create aSessionwithNewSessionand pass it; a compile error flags every call site. Passingnilis not a supported fallback — it returns an error.Session.Test
go build ./...,go vet ./detector/... ./server/...,go test ./detector/... ./server/..., andgolangci-lint run ./...all pass.🤖 Generated with Claude Code