Skip to content

perf(docker): reduce MongoDB healthcheck overhead - #3878

Open
eralpozcan wants to merge 4 commits into
bluewave-labs:developfrom
eralpozcan:perf/reduce-mongodb-healthcheck-overhead
Open

perf(docker): reduce MongoDB healthcheck overhead#3878
eralpozcan wants to merge 4 commits into
bluewave-labs:developfrom
eralpozcan:perf/reduce-mongodb-healthcheck-overhead

Conversation

@eralpozcan

@eralpozcan eralpozcan commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Describe your changes

Replace the MongoDB healthchecks that start a new mongosh process with a lightweight TCP connection check.

  • Keep the existing five-second healthcheck cadence and failure-detection settings.
  • Check whether MongoDB accepts a connection on 127.0.0.1:27017 using Bash's built-in /dev/tcp support.
  • Apply the same healthcheck to the production, development, and docs/hosting/checkmate-so Compose configurations.

Review-driven revision

The initial version reduced the expensive mongosh probe frequency from every 5 seconds to every 30 seconds. Following maintainer review, the implementation was revised to remove the expensive process startup instead of running it less often.

Area Initial proposal Revised implementation
Healthcheck command mongosh ping Bash TCP connection check
Steady-state interval 30 seconds Existing 5 seconds retained
Startup/failure timings Adjusted Existing values retained
Compose coverage Production and development Production, development, and both docs/hosting/checkmate-so MongoDB services

All benchmark results below were rerun against the revised TCP implementation.

Write your issue number after "Fixes "

Fixes #3877

Testing

Local Docker reproduction

  • Reproduced the issue with six active monitors; five generated test monitors ran at 10-second intervals.
  • With the original five-second mongosh healthcheck, MongoDB container CPU showed recurring peaks of 56.64%, 47.10%, and 27.48%. A later comparable sample observed a 46.93% peak.
  • During a 30-second observation window, application activity remained modest: 0.43 inserts/s, 6.87 queries/s, 2.33 updates/s, and 4.43 commands/s.
  • mongotop showed only a few milliseconds of activity per second on the checks, jobs, and monitors collections, separating normal database work from healthcheck process startup.
  • Verified that the TCP command exits successfully against MongoDB on port 27017 and fails against a closed port.
  • Started MongoDB with a new empty volume and confirmed that the container reached healthy using the TCP healthcheck.
  • With the TCP healthcheck still running every five seconds, the comparable local sample recorded a 7.44% maximum MongoDB container CPU.
  • Verified all three Compose files with docker compose config --quiet.

Self-hosted validation

Validated on a Dokploy-managed self-hosted Checkmate deployment running on a 2-vCPU, 2-GB RAM server. The MongoDB container was limited to 0.75 CPU and 512 MB RAM. Both measurements use the same five-second healthcheck cadence.

Metric Before (mongosh) After (TCP) Change
Healthcheck executions per minute 12 12 unchanged
MongoDB CPU median (20 samples) 4.56% 0.83% -81.8%
MongoDB CPU p95 (20 samples) 77.85% 6.23% -92.0%
MongoDB CPU maximum (20 samples) 78.44% 7.08% -91.0%

Before:

MongoDB CPU before the healthcheck change

After:

MongoDB CPU after the TCP healthcheck change

Please ensure all items are checked off before requesting a review. "Checked off" means you need to add an "x" character between brackets so they turn into checkmarks.

  • (Do not skip this or your PR will be closed) I deployed the application locally.
  • (Do not skip this or your PR will be closed) I have performed a self-reviewing and testing of my code.
  • I have included the issue # in the PR.
  • I have added i18n support to visible strings (not applicable; no user-facing strings changed).
  • I have not included any files that are not related to my pull request, including package-lock and package-json if dependencies have not changed.
  • I didn't use any hardcoded application values (port 27017 is the MongoDB service port used by these Compose configurations).
  • I made sure font sizes, color choices etc. are all referenced from the theme (not applicable; no UI changes).
  • My PR is granular and targeted to one specific feature.
  • I ran the relevant formatting and validation checks (no server or client source files changed; all affected Compose files pass docker compose config --quiet).
  • I took a screenshot or a video and attached to this PR if there is a UI change (not applicable; no UI changes).

Keep fast startup probes while reducing steady-state mongosh process launches.\n\nRefs bluewave-labs#3877

@ajhollid ajhollid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a better solution here is to just not use mongosh and just see if 27017 accepts a connection.

There's no replica set here, so if 27017 accepts a connection then the service can considered ready.

In essence, make the health check very cheap instead of carrying out the existing expensive check less often.

Please note there is another docker compose file in the docs/hosting dir that also needs to be updated.

Thanks!

@eralpozcan

eralpozcan commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

I had not considered approaching it this way. Making the health check itself cheap instead of reducing its frequency makes sense. I will revise the change to use a TCP connection check and update the Compose file under docs/hosting as well, then rerun the validation against the revised implementation.

Updated Version:

TCP Screenshot

@eralpozcan
eralpozcan requested a review from ajhollid August 26, 2026 19:55

@ajhollid ajhollid left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check looks good, but there's an extra shell in there that I think we can safely skip

Comment thread docker/docker-compose.yaml Outdated
@eralpozcan
eralpozcan requested a review from ajhollid August 31, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce MongoDB healthcheck overhead in Docker Compose

2 participants