Description of Request for Enhancement
Kubernetes can use "health" end points to test whether a container is ready. In the PCP container, we would benefit from more reliable integration into the k8s ecosystem if our REST API provided support for the three distinct probe types (mapping to three API endpoints):
- Startup Probe (/health/startup): Delays all other checks while a (possibly slow) container initializes.
- Liveness Probe (/health/liveness): Determines if the container needs a hard restart.
- Readiness Probe (/health/readiness): Controls whether the container should receive live traffic.
The PCP container contains all of pmproxy, pmcd, pmlogger and pmie (and different combinations thereof) so the definitions of these probes may be a little fuzzy.
One problem area I've been encountering is that the PCP container starts using the systemd service startup mechanisms - which works great, but our scripts are designed to start components in parallel. For example, while pmcd may be very quick to start, a PMDA .NeedInstall may run in the background and worse, a pmlogconf configuration may be probing for that PMDA in order to configure pmlogger.
What happens often is the container starts really quickly but the pmlogger process is started and running (and dynamically configured) before the PMDA is ready. In the Kubernetes world this often means the OpenMetrics PMDA is either not yet running, or not yet receiving data from its configured endpoints, etc - and pmlogger starts up thinking these metrics do not exist. This corrects itself in a minute or so, but that is a fair bit of missing data at the start of an archive for any relatively short-runtime benchmarks / experiments.
This might be made more resilient by addition of a mode in the startup scripts allowing sequential startup with no background tasks like PMDA Installation (i.e. just run those in the foreground).
I currently use a script that waits and verifies the presence of given metrics, then restarts pmlogger, as a workaround. It'd be much better to not need this.
Component
pmproxy pmlogger
Description of Request for Enhancement
Kubernetes can use "health" end points to test whether a container is ready. In the PCP container, we would benefit from more reliable integration into the k8s ecosystem if our REST API provided support for the three distinct probe types (mapping to three API endpoints):
The PCP container contains all of pmproxy, pmcd, pmlogger and pmie (and different combinations thereof) so the definitions of these probes may be a little fuzzy.
One problem area I've been encountering is that the PCP container starts using the systemd service startup mechanisms - which works great, but our scripts are designed to start components in parallel. For example, while pmcd may be very quick to start, a PMDA .NeedInstall may run in the background and worse, a pmlogconf configuration may be probing for that PMDA in order to configure pmlogger.
What happens often is the container starts really quickly but the pmlogger process is started and running (and dynamically configured) before the PMDA is ready. In the Kubernetes world this often means the OpenMetrics PMDA is either not yet running, or not yet receiving data from its configured endpoints, etc - and pmlogger starts up thinking these metrics do not exist. This corrects itself in a minute or so, but that is a fair bit of missing data at the start of an archive for any relatively short-runtime benchmarks / experiments.
This might be made more resilient by addition of a mode in the startup scripts allowing sequential startup with no background tasks like PMDA Installation (i.e. just run those in the foreground).
I currently use a script that waits and verifies the presence of given metrics, then restarts pmlogger, as a workaround. It'd be much better to not need this.
Component
pmproxy pmlogger