Skip to content

perf(detector): share one vuls2 db session across a server's detect and enrich#2600

Merged
MaineK00n merged 1 commit into
masterfrom
MaineK00n/share-vuls2-session-per-server
Jul 7, 2026
Merged

perf(detector): share one vuls2 db session across a server's detect and enrich#2600
MaineK00n merged 1 commit into
masterfrom
MaineK00n/share-vuls2-session-per-server

Conversation

@MaineK00n

@MaineK00n MaineK00n commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

What

Within one report run, detect (vuls2.DetectPkgs / DetectCPEs) and enrich (vuls2.EnrichVulnInfos) each opened, used, and discarded their own vuls2 boltDB session. So a single server opened the db multiple times, each time redoing the metadata read / schema-version check and — the costly part — throwing away the WithCache read cache on close. Since detect and enrich query largely the same CVEs, enrich was rebuilding a cache detect had just warmed.

This PR introduces a lazily-opened, shared vuls2.Session and reuses one session per server across that server's detection and the enrichment that follows.

How

  • New vuls2.Session: a lazily-opened handle. The db is opened at most once — on the first path that actually queries it — and reused by every later path within one server's turn, then closed by the owner.
    • Opens exactly when and only when the unshared code opened one: a family that skips vuls2 detection (FreeBSD, pseudo, trivy-scanned, …) does not open it during detection, but enrichment still opens it when the result carries CVEs (e.g. FreeBSD's pkg audit findings); a server with nothing to detect and no CVEs to enrich never opens it at all. So no unnecessary db open/download is forced.
  • The db handle is a required *vuls2.Session, created with NewSession (which carries the Vuls2Conf / noProgress the db needs) and Closed by the caller. DetectPkgs / DetectCPEs / EnrichVulnInfos / DetectPkgCves / DetectCpeURIsCves take it and share it across one server's detection + enrichment. A nil Session is rejected with an error rather than panicking.
  • detector.Detect (per-server loop) and the server-mode handler create one Session per server / request and share it across all three paths.

Scope of sharing: per-server (deliberate)

A whole-run variant (one session for all servers) was prototyped and compared. The vuls2 read cache is an unbounded sync.Map keyed by advisory/vulnerability ID, freed only on Close(). Whole-run keeps it warm across servers (faster on homogeneous fleets) but grows peak memory monotonically with the distinct advisories/vulns touched across the whole run.

Per-server was chosen because:

  • Memory: the cache is freed after each server, so peak stays at ~one server's working set instead of the union across the run — no unbounded growth on large/diverse fleets.
  • Future concurrency: if Detect is ever parallelized over servers, moving NewSession into the goroutine is naturally safe; a run-wide shared session would need synchronization.

Compatibility

  • The vuls2 detect/enrich entry points (DetectPkgs / DetectCPEs / EnrichVulnInfos) and the detector wrappers (DetectPkgCves / DetectCpeURIsCves) drop the old vuls2Conf / noProgress parameters and take a required trailing *vuls2.Session instead — that config now lives on the Session (via NewSession). External library consumers must create a Session with NewSession and pass it; a compile error flags every call site. Passing nil is not a supported fallback — it returns an error.
  • Server-mode detection (per request = one server) creates and shares a request-scoped Session.

Test

  • go build ./..., go vet ./detector/... ./server/..., go test ./detector/... ./server/..., and golangci-lint run ./... all pass.

🤖 Generated with Claude Code

@MaineK00n MaineK00n self-assigned this Jul 6, 2026
@MaineK00n MaineK00n force-pushed the MaineK00n/share-vuls2-session-per-server branch 6 times, most recently from 7c11cb8 to 4b0412d Compare July 7, 2026 03:34
@MaineK00n MaineK00n requested a review from Copilot July 7, 2026 05:17
@MaineK00n MaineK00n marked this pull request as ready for review July 7, 2026 05:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes vuls2-backed detection/enrichment by introducing a lazily-opened vuls2 DB Session that can be shared across a single server’s detection and subsequent enrichment, reusing the warmed read cache instead of reopening the DB and rebuilding cache multiple times.

Changes:

  • Introduces vuls2.Session with lazy open/close behavior and updates vuls2 detect/enrich entry points to use it.
  • Updates per-server detection flow (detector.Detect) to create one session per server and share it across package detection, CPE detection, and enrichment.
  • Updates server-mode handler to share one session across request-scoped detection and enrichment.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
server/server.go Creates a request-scoped vuls2 Session and reuses it for detection + enrichment in server mode.
detector/vuls2/vuls2.go Adds the Session type, factors DB opening into openSession, and routes detect/enrich through the shared session.
detector/detector.go Creates a per-server vuls2 Session and threads it through detection and enrichment steps.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread detector/vuls2/vuls2.go
Comment thread detector/vuls2/vuls2.go
@MaineK00n MaineK00n force-pushed the MaineK00n/share-vuls2-session-per-server branch from 4b0412d to 292162d Compare July 7, 2026 06:48
@MaineK00n MaineK00n requested a review from Copilot July 7, 2026 06:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread detector/vuls2/vuls2.go
Comment thread detector/vuls2/vuls2.go
Comment thread detector/vuls2/vuls2.go
@MaineK00n MaineK00n force-pushed the MaineK00n/share-vuls2-session-per-server branch from 292162d to 1753596 Compare July 7, 2026 07:11
@MaineK00n MaineK00n requested a review from Copilot July 7, 2026 07:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread detector/detector.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

@MaineK00n MaineK00n force-pushed the MaineK00n/share-vuls2-session-per-server branch 3 times, most recently from 2f17531 to b4ac0e6 Compare July 7, 2026 07:33
@MaineK00n MaineK00n requested a review from Copilot July 7, 2026 08:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread detector/vuls2/vuls2.go
@MaineK00n MaineK00n force-pushed the MaineK00n/share-vuls2-session-per-server branch from b4ac0e6 to 72160a3 Compare July 7, 2026 09:05
@MaineK00n MaineK00n requested a review from Copilot July 7, 2026 09:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread detector/detector.go Outdated
…nd enrich

detect (DetectPkgs/DetectCPEs) and enrich (EnrichVulnInfos) each opened,
used, and discarded their own vuls2 boltDB session — so within one report
run a single server opened the db multiple times, each redoing the metadata
read / schema-version check and, more importantly, throwing away the
WithCache read cache on close. detect and enrich query largely the same
CVEs, so enrich was rebuilding a cache detect had just warmed.

Introduce a lazily-opened, shared vuls2.Session: the db is opened at most
once (on the first path that queries it) and reused by every later path
within one server's turn, then Closed by the owner. Detection warms the
cache and the enrichment that immediately follows reuses it.

The session opens exactly when and only when the unshared code opened one —
no earlier, no more often: a family that skips vuls2 detection (FreeBSD,
pseudo, trivy-scanned, ...) does not open it during detection, but
enrichment still opens it when the result carries CVEs (e.g. FreeBSD's
pkg-audit findings), and a server with nothing to detect and no CVEs to
enrich never opens it at all. So no unnecessary db open/download is forced.

DetectPkgs / DetectCPEs / EnrichVulnInfos / DetectPkgCves / DetectCpeURIsCves
now take the db handle as a required *vuls2.Session, created with NewSession
(which carries the Vuls2Conf/noProgress the db needs) and Closed by the
caller. The Session is the single handle threaded through one server's
detection and enrichment; detector.Detect (per-server loop) and the
server-mode handler create one per server / request and share it across all
three paths. A nil Session is rejected with an error rather than panicking.

Scope is per-server: the session — and its unbounded read cache — is freed
after each server, keeping peak memory at ~one server's working set rather
than growing across the whole run, and leaving detection naturally safe to
parallelize over servers later.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@MaineK00n MaineK00n force-pushed the MaineK00n/share-vuls2-session-per-server branch from 72160a3 to 280ef04 Compare July 7, 2026 09:11
@MaineK00n MaineK00n requested a review from Copilot July 7, 2026 09:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

@MaineK00n MaineK00n requested a review from shino July 7, 2026 09:16

@shino shino left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1️⃣

@MaineK00n MaineK00n merged commit cf9e99e into master Jul 7, 2026
8 checks passed
@MaineK00n MaineK00n deleted the MaineK00n/share-vuls2-session-per-server branch July 7, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants