worker use 100% cpu #41

Activity

1. tokers commented on Dec 17, 2024

   

   tokers

   on Dec 17, 2024

   Owner

   More actions

   @dolgovas It would be nice to have a minimum reproducible case. BTW, could you capture a flame graph when Nginx workers occupy nearly 100% CPU time.
2. dolgovas commented on Dec 18, 2024

   

   dolgovas

   on Dec 18, 2024

   Author

   More actions

   @dolgovas It would be nice to have a minimum reproducible case. BTW, could you capture a flame graph when Nginx workers occupy nearly 100% CPU time.

   I'll try to make it ASAP
3. dolgovas commented on Dec 18, 2024

   

   dolgovas

   on Dec 18, 2024

   Last edited by dolgovas

   Author

   More actions

   made via this instruction.
   
4. hackoffme commented on Dec 25, 2024

   

   hackoffme

   on Dec 25, 2024

   More actions

   I have encountered the same issue described in this thread. After integrating the zstd-nginx-module into my Nginx server, the worker processes consume 100% CPU following the first request, leading to server unresponsiveness.
   nginx/1.26.2
   zstd-nginx-module 0.1.1
   cat /usr/share/nginx/modules/zstd_* load_module "/usr/lib64/nginx/modules/ngx_http_zstd_filter_module.so"; load_module "/usr/lib64/nginx/modules/ngx_http_zstd_static_module.so";

   Name:           zstd-nginx-module
   Version:        0.1.1
   Release:        1%{?dist}
   Summary:        Zstandard compression module for Nginx
   License:        BSD-2-Clause
   Source0:        https://github.com/tokers/zstd-nginx-module/archive/refs/tags/%{version}.tar.gz
   
   BuildRequires:  gcc
   BuildRequires:  pcre-devel
   BuildRequires:  zlib-devel
   BuildRequires:  nginx-mod-devel
   Requires:       nginx
   
   %description
   The zstd-nginx-module provides Zstandard compression for Nginx.
   
   %prep
   %autosetup -n %{name}-%{version}
   
   %build
   %nginx_modconfigure
   %nginx_modbuild
   
   %install
   pushd %{_vpath_builddir}
   install -d %{buildroot}%{nginx_moddir}
   install -m 0755 ngx_http_zstd_filter_module.so %{buildroot}%{nginx_moddir}/
   install -m 0755 ngx_http_zstd_static_module.so %{buildroot}%{nginx_moddir}/
   
   install -d %{buildroot}%{nginx_modconfdir}
   echo 'load_module "%{nginx_moddir}/ngx_http_zstd_filter_module.so";' > %{buildroot}%{nginx_modconfdir}/zstd_filter.conf
   echo 'load_module "%{nginx_moddir}/ngx_http_zstd_static_module.so";' > %{buildroot}%{nginx_modconfdir}/zstd_static.conf
   popd
   
   %files
   %{nginx_modconfdir}/*
   %{nginx_moddir}/*
   
5. kamermans commented on Jan 10, 2025

   

   kamermans

   on Jan 10, 2025

   Last edited by kamermans

   More actions

   I'm also seeing this in an Ubuntu 22.04 container running Nginx:

   nginx version: nginx/1.27.3
   built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
   built with OpenSSL 3.0.2 15 Mar 2022
   TLS SNI support enabled
   configure arguments: 
     --prefix=/etc/nginx
     --sbin-path=/usr/sbin/nginx
     --add-module=/opt/nginx/ngx_brotli
     --add-module=/opt/nginx/zstd-nginx-module
     --modules-path=/usr/lib/nginx/modules
     --conf-path=/etc/nginx/nginx.conf
     --error-log-path=/var/log/nginx/error.log
     --http-log-path=/var/log/nginx/access.log
     --pid-path=/var/run/nginx.pid
     --lock-path=/var/run/nginx.lock
     --http-client-body-temp-path=/var/cache/nginx/client_temp
     --http-proxy-temp-path=/var/cache/nginx/proxy_temp
     --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp
     --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp
     --http-scgi-temp-path=/var/cache/nginx/scgi_temp
     --user=nginx
     --group=nginx
     --with-compat
     --with-file-aio
     --with-threads
     --with-http_addition_module
     --with-http_auth_request_module
     --with-http_dav_module
     --with-http_flv_module
     --with-http_gunzip_module
     --with-http_gzip_static_module
     --with-http_mp4_module
     --with-http_random_index_module
     --with-http_realip_module
     --with-http_secure_link_module
     --with-http_slice_module
     --with-http_ssl_module
     --with-http_stub_status_module
     --with-http_sub_module
     --with-http_v2_module
     --with-http_v3_module
     --with-mail
     --with-mail_ssl_module
     --with-stream
     --with-stream_realip_module
     --with-stream_ssl_module
     --with-stream_ssl_preread_module
     --with-cc-opt='-g -O2 -ffile-prefix-map=/opt/nginx/nginx-1.27.3=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wno-missing-field-initializers -Wp,-D_FORTIFY_SOURCE=2 -fPIC' 
     --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
   

   I repackage the official deb source from nginx.org with the zstd and brotli added.

   I'm using the official Ubuntu libzstd1 package:

   # dpkg -s libzstd1
   Package: libzstd1
   Status: install ok installed
   Priority: required
   Section: libs
   Installed-Size: 846
   Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
   Architecture: amd64
   Multi-Arch: same
   Source: libzstd
   Version: 1.4.8+dfsg-3build1
   Depends: libc6 (>= 2.14)
   

   Here's a graph of the CPU usage from the instance start time to 30 minutes later when it started failing health checks. During this time period, this instance was getting about 140 client requests per second, although most of them would not have used ztsd, probably 5-10% of those requests were using zstd.
   
6. dolgovas commented on Jan 15, 2025

   

   dolgovas

   on Jan 15, 2025

   Author

   More actions

   @tokers may be you need any additional info?
7. bouk commented on Jan 22, 2025

   

   bouk

   on Jan 22, 2025

   More actions

   Seems related: #23

   👀React with 👀1Reacted by Steve Kamerman
8. madrzejewski commented on Jun 23, 2025

   

   madrzejewski

   on Jun 23, 2025

   More actions

   Hello,

   Same issue here with Openresty 1.27.1.2, after a few days I always have a few Nginx processes stuck at 100% CPU usage.

       PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                                                                                                                             
   1066324 nobody    20   0 4245900   2.2g 108800 R  98.8   0.9 577:10.60 nginx: worker process                                                                                                                                                                                                                                                                               
   1066589 nobody    20   0 4245004   2.2g 107520 R  98.8   0.9 577:10.05 nginx: worker process                                                                                                                                                                                                                                                                               
   1066791 nobody    20   0 4244876   2.2g 110080 R  98.8   0.9 577:22.89 nginx: worker process                                                                                                                                                                                                                                                                               
   1066885 nobody    20   0 4246412   2.2g 110080 R  98.8   0.9 576:58.82 nginx: worker process
   

   When I run GDB on one of the process to get a backstrace, I have this :

   #0  0x00007f7742781e5c in ZSTD_compressStream2 () from /lib64/libzstd.so.1
   #1  0x00007f774278240b in ZSTD_compressStream () from /lib64/libzstd.so.1
   #2  0x000000000057f6c0 in ngx_http_zstd_filter_compress (r=0x19b1a070, r=0x19b1a070, ctx=0x19b1e8d0) at /usr/src/openresty/zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:420
   #3  ngx_http_zstd_body_filter (r=0x19b1a070, in=<optimized out>) at /usr/src/openresty/zstd-nginx-module/filter/ngx_http_zstd_filter_module.c:331
   #4  0x00000000004a9ad1 in ngx_http_ssi_body_filter (r=<optimized out>, in=<optimized out>) at src/http/modules/ngx_http_ssi_filter_module.c:440
   #5  0x00000000004ae876 in ngx_http_charset_body_filter (r=0x19b1a070, in=0x7ffdcba108c0) at src/http/modules/ngx_http_charset_filter_module.c:647
   #6  0x00000000004af9b4 in ngx_http_gunzip_body_filter (r=0x19b1a070, in=0x7ffdcba108c0) at src/http/modules/ngx_http_gunzip_filter_module.c:185
   #7  0x00000000004b1236 in ngx_http_trailers_filter (r=0x19b1a070, in=0x7ffdcba108c0) at src/http/modules/ngx_http_headers_filter_module.c:274
   #8  0x00000000005202e4 in ngx_http_lua_capture_body_filter (r=0x19b1a070, in=0x7ffdcba108c0) at ../ngx_lua-0.10.28/src/ngx_http_lua_capturefilter.c:133
   #9  0x000000000042d312 in ngx_output_chain (ctx=ctx@entry=0x19b1ecf8, in=in@entry=0x7ffdcba108c0) at src/core/ngx_output_chain.c:70
   #10 0x00000000004b1c44 in ngx_http_copy_filter (r=0x19b1a070, in=0x7ffdcba108c0) at src/http/ngx_http_copy_filter_module.c:145
   #11 0x000000000047bc27 in ngx_http_output_filter (r=r@entry=0x19b1a070, in=in@entry=0x7ffdcba108c0) at src/http/ngx_http_core_module.c:1874
   #12 0x000000000047fce0 in ngx_http_send_special (r=r@entry=0x19b1a070, flags=flags@entry=2) at src/http/ngx_http_request.c:3647
   #13 0x00000000004929bd in ngx_http_upstream_finalize_request (r=0x19b1a070, u=0x19b1d4c8, rc=-1) at src/http/ngx_http_upstream.c:4675
   #14 0x000000000049396a in ngx_http_upstream_process_request (r=0x19b1a070, u=0x19b1d4c8) at src/http/ngx_http_upstream.c:4239
   #15 0x0000000000492d94 in ngx_http_upstream_handler (ev=0xa45ca40) at src/http/ngx_http_upstream.c:1306
   #16 0x0000000000455df3 in ngx_epoll_process_events (cycle=0x7cdf2130, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:901
   #17 0x000000000044ab75 in ngx_process_events_and_timers (cycle=cycle@entry=0x7cdf2130) at src/event/ngx_event.c:258
   #18 0x00000000004539c5 in ngx_worker_process_cycle (cycle=cycle@entry=0x7cdf2130, data=data@entry=0x1e) at src/os/unix/ngx_process_cycle.c:793
   #19 0x00000000004523d7 in ngx_spawn_process (cycle=cycle@entry=0x7cdf2130, proc=proc@entry=0x453980 <ngx_worker_process_cycle>, data=data@entry=0x1e, name=name@entry=0x5d571d "worker process", respawn=respawn@entry=-4) at src/os/unix/ngx_process.c:207
   #20 0x0000000000453fcc in ngx_start_worker_processes (cycle=cycle@entry=0x7cdf2130, n=45, type=type@entry=-4) at src/os/unix/ngx_process_cycle.c:382
   #21 0x0000000000454e5a in ngx_master_process_cycle (cycle=0x7cdf2130) at src/os/unix/ngx_process_cycle.c:241
   #22 0x000000000042806c in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:387
   
9. eilandert

   added a commit that references this issue on May 10, 2026

   fix: Apply PR #23 + PR #49 fixes for CPU 100% infinite loop bug

   ...

   2af5889
10. gtoxlili commented on Aug 7, 2026

    

    gtoxlili

    on Aug 7, 2026

    More actions

    Still reproduces on nginx 1.29.8 / zstd-nginx-module 0.1.1 / libzstd 1.5.7. filter/ngx_http_zstd_filter_module.c is byte-identical between the 0.1.1 tag and current master, so nothing has changed here since this was filed.

    One worker sat at 100% CPU for 12 days before we noticed (utime 12d, stime 134s, zero syscalls). I sampled RIP 30 times via ptrace: 21 samples landed in libzstd.so inside a ~600 byte window, 9 in the module's filter code. The upstream socket was still in CLOSE-WAIT with Recv-Q exactly equal to the response Content-Length, so the response body had never been read — the loop starts before any body byte arrives. That matches the size: 0 on both buffers in @dolgovas's debug log.

    The loop is an oscillation between COMPRESS and FLUSH when the filter is entered with a zero-length buffer that has flush set (line numbers are master):

    add_data picks up the empty flush buf, sets ctx->flush = 1 (L518-519), computes buffer_in.size = 0, returns NGX_AGAIN (L531). After that it can never take anything else off ctx->in: L500-503 short-circuits on ctx->flush before reaching the ctx->in == NULL → NGX_DECLINED exit at L511.

    Each pass of the inner for ( ;; ) at L309 then does:

    • action COMPRESS — ZSTD_compressStream() with in.size == 0 returns > 0, so L443-448 sets action to FLUSH and redo = 1. out_buf is empty, so L462 returns NGX_AGAIN.
    • action FLUSH — ZSTD_flushStream() returns 0 (nothing buffered), redo = 0 at L441. rc > 0 is false, and ctx->last is 0 because last_buf never arrived, so L450 cannot switch to END. L458-459 restores action to COMPRESS. out_buf still empty, NGX_AGAIN again.

    Neither buffer_in.pos nor buffer_out.pos ever moves, so L439-440 are no-ops. This is exactly the alternating redo:0 / redo:1 with pos:0 size:0 in the log above.

    ctx->flush is the flag that would break the cycle, but it is only cleared at L480, which sits below the ngx_buf_size(ctx->out_buf) == 0 → return NGX_AGAIN at L462. A flush that produces no output never reaches it.

    On our side the trigger was proxy_buffering off set at server level on a vhost that also served a small proxied HTML page (200, text/html, Content-Length: 2281, client Accept-Encoding: gzip, deflate, br, zstd). Scoping the directive down to the upload-only location removed the exposure. That page took ~150 requests over two months and only one of them wedged a worker, so some timing has to line up.

    One consequence worth flagging, since @dolgovas hit it too ("cannot finishing with HUP signal"): a worker in this state never returns to the event loop, so it also never handles SIGQUIT. nginx -s reload leaves it running as an orphan from the previous cycle, indefinitely — ours survived a reload and burned a core for another 7 days before anyone looked. worker_shutdown_timeout bounds that. Setting zstd off while keeping zstd_static on avoids the code path entirely, since the static module does not compress at runtime.

    👍React with 👍1Reacted by dolgovas
11. eilandert commented on Aug 7, 2026

    

    eilandert

    on Aug 7, 2026

    Contributor

    More actions

    @gtoxlili would you be so kind to test a fork with an outstanding PR into this repo?
    https://github.com/myguard-labs/nginx-zstd-module

    I fixed this problem and more

    ❤️React with ❤️1Reacted by Emerson Gomes