windows: cross-compile Fluent Bit for Windows with the MinGW-w64 (UCRT64) toolchain#12069
Draft
hjiawei wants to merge 20 commits into
Draft
windows: cross-compile Fluent Bit for Windows with the MinGW-w64 (UCRT64) toolchain#12069hjiawei wants to merge 20 commits into
hjiawei wants to merge 20 commits into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
We need to send patches into upstream if we wanted to patch bundled libraries. |
hjiawei
force-pushed
the
mingw-ucrt64-cross-build
branch
2 times, most recently
from
4d96c59 to
6dc3994
Compare
Author
Agreed. Each bundled-library change has been submitted upstream. They are listed in the PR description. I will rebase or update the vendored libs once they are merged or brought into the Fluent Bit codebase. |
hjiawei
force-pushed
the
mingw-ucrt64-cross-build
branch
from
6dc3994 to
34b9aba
Compare
Make the cmake tree usable for a Linux-hosted MinGW-w64 UCRT64 cross-build targeting Windows, building the standard Windows plugin profile from cmake/windows-setup.cmake: - Define _POSIX_THREAD_SAFE_FUNCTIONS so the MinGW headers declare the POSIX gmtime_r()/localtime_r() family, and keep the Winsock char-pointer signature mismatches (warnings under MSVC) from becoming hard errors under GCC 14 and later unless the strict option is set. - Derive the autoconf host triple from the cross compiler when CMAKE_CROSSCOMPILING and no GNU_HOST was given, so the external autotools sub-builds (libbacktrace, jemalloc) configure for the target instead of silently producing build-host objects. - Use plain make for external autotools sub-builds under the Ninja generator on all targets; the $(MAKE) escape only exists in the Make-family generators. - Give the executable an import-library suffix distinct from the shared library's on MinGW, mirroring the MSVC branch; Ninja rejects the duplicate libfluent-bit.dll.a output NMake never noticed. - Link libbacktrace on fluent-bit-static, where the stacktrace helpers are compiled, so GNU ld orders the archive correctly. - Guard the MSVC /W2 warning flag with if(MSVC) in the core and plugin cmake files, link the Windows system libraries by their lowercase names (GNU ld resolves them against import libraries, MSVC accepts them unchanged), and append to FLB_DEPS instead of overwriting it so onigmo-static and friends survive on Windows. - Keep the ELF hardening link flags off for all Windows targets, not just MSVC, since GNU ld does not support them for PE. - Define YAML_DECLARE_STATIC for any Windows toolchain when FLB_LIBYAML_DIR is given; that path carries a static libyaml for MSVC (vcpkg static triplets) and MinGW alike. - Link OpenSSL statically only on MSVC; the MinGW cross-build uses the toolchain's dynamic OpenSSL to avoid GNU ld link-order issues. - Keep FLB_HAVE_UNIX_SOCKET off for non-MSVC Windows builds: the Windows unix socket paths are untested, so keep feature parity with MSVC. - Skip the tools/xxd-c host tool when cross-compiling, since it cannot run on the build host. - Overwrite onigmo's generated config.h with the win32/config.h it ships when cross-compiling to Windows; Onigmo's configure-time probes otherwise inspect the build host and misdetect the target. This keeps the vendored onigmo tree unpatched. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
MinGW-w64 UCRT headers already provide strcasecmp(), strncasecmp(), timezone, tzname, usleep() and, with _POSIX_THREAD_SAFE_FUNCTIONS set by the build system, the POSIX gmtime_r()/localtime_r() family, plus a real pthread implementation (winpthreads); redefining them or pulling in monkey's Windows pthread shim breaks compilation. Guard the MSVC replacement shims and route MinGW to <pthread.h>, mapping only the BSD random()/srandom() pair the MinGW unistd.h lacks. Use GCC's __thread for the thread-local basename() buffer since MinGW GCC ignores __declspec(thread), add a PTHREAD_STACK_MIN fallback for the coroutine stack sizing, lowercase the wincrypt.h include for cross-builds from case-sensitive filesystems, and use the Windows gate instead of the MSVC one in flb_langinfo.h. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
The header declares flb_net_server_unix() with four parameters, but the fallback stub compiled when FLB_HAVE_UNIX_SOCKET is not set still has the old three-parameter signature, so any platform taking that branch fails to compile with a conflicting declaration error. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
The glob(3) fallback and _fullpath() usage apply to any Windows toolchain, not just MSVC, so gate them on _WIN32 instead of _MSC_VER, and lowercase the windows.h include so it resolves against the lowercase MinGW SDK headers when cross-compiling from a case-sensitive filesystem. No behavior change for MSVC builds. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
Make the bundled WAMR runtime build for a Windows target with a non-MSVC toolchain: - Select the WAMR platform port from the target system (CMAKE_SYSTEM_NAME) instead of the build host, so a Linux-hosted Windows cross-build uses the windows port rather than compiling the linux port against missing POSIX headers. - The MASM assembler (ASM_MASM) is MSVC-only, so enable it only for MSVC; MinGW and other Windows toolchains still build WASM, using the generic C native-call wrapper the ARM64 Windows build already uses. - Link pathcch and ntdll, which WAMR's Windows platform needs for PathCchCombine (win_file.c) and NtQueryTimerResolution (win_clock.c). MSVC pulls these in via #pragma comment(lib, ...), which GCC ignores; declare them on vmlib-static so CMake orders them after the archive on consumers' link lines. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
GCC on a case-sensitive filesystem cannot resolve the capitalised <ConsoleApi.h>; MSVC is unaffected. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
The blocks guard Windows-only functionality, not the MSVC compiler, so MinGW builds must take the same path. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
The block guards Windows-only functionality, not the MSVC compiler. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
The blocks guard Windows-only functionality, not the MSVC compiler. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
Capitalised headers do not resolve on a case-sensitive filesystem under GCC; MSVC is unaffected. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
…table The prototype used empty parens; under C23 that means (void) and conflicts with the definition that takes a size_t. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
The block guards Windows functionality, not MSVC; the capitalised header also fails to resolve under GCC. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
The block guards Windows functionality, not MSVC; the capitalised header also fails to resolve under GCC. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
Only test the socket type when the platform provides the macro. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
Widen the MSVC-only CMake gate to WIN32, link shlwapi by its lowercase name, and lowercase the windows.h includes in the win32 helpers. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
…nGW build Add the missing math.h and we_util.h/we_perflib.h includes and adjust the headers so the plugin compiles with GCC. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
Capitalised headers do not resolve on a case-sensitive filesystem under GCC; MSVC is unaffected. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
Make the vendored libraries build and run with the MinGW-w64 UCRT toolchain. Each change belongs in the library's own upstream and has been submitted there -- monkey, chunkio and cmetrics via the fluent org, flb_libco via edsiper/flb_libco, and WAMR via bytecodealliance -- so these hunks are carried here only until those merge and are re-vendored. Two other vendored libraries needed no source change and are intentionally not touched: nghttp2 (its inline byte-order fallback compiles as-is; the consumers that include it do not pull in winsock2) and snappy (its MSVC intrinsic path builds under MinGW). Onigmo's cross-compile config is handled in the fluent-bit build files, not by patching its tree. - monkey: select the POSIX threading model for MinGW (winpthreads is a real pthread implementation; the hand-rolled shim stays MSVC-only), guard the MSVC-only typedefs and shims MinGW already provides, widen MSVC-only gates covering Windows functionality to _WIN32, use the standard GCC container_of, lowercase Windows header includes, and for the vendored libevent record the winsock dependency on the static archives; for MinGW only, pin NTDDI_VERSION next to the existing _WIN32_WINNT override in evutil.c (its sdkddkver.h errors when the two disagree, where MSVC does not, so MSVC compiles the file unchanged) and define EVENT__HAVE_STRTOK_R (libevent's WIN32 function check takes the function's address and only sees the force-included winsock headers, so it mis-detects strtok_r from <string.h> and then defines a conflicting static copy). - chunkio: guard the ssize_t/mode_t typedefs MinGW provides, widen the fts.h gate to _WIN32, lowercase Windows header includes, and link shell32 by its lowercase name. - cmetrics: define YY_NO_UNISTD_H only for MSVC, since MinGW has a real unistd.h the generated lexer needs for isatty(). - flb_libco: select the fiber backend on Windows before the arch check, matching the MSVC branch that avoids amd64.c due to its SIGSEGV bug (its co_swap machine-code blob faults under DEP). Applied to both the top-level lib/flb_libco copy and monkey's vendored deps/flb_libco copy, since the HTTP server build compiles the latter. - wasm-micro-runtime: lowercase the pathcch.h include and reword the // comments in win_file.c that end in a backslash; GCC treats a backslash-newline as a line continuation even inside a // comment and swallows the following line of code, while MSVC does not. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
Add a compile sanity check that cross-builds fluent-bit.exe for Windows from Linux with the Fedora MinGW-w64 UCRT64 toolchain, so the MinGW support does not bit-rot. The job runs in a fedora container, pins the same cmake version as the other compile checks, cross-builds a static libyaml, and configures with the distribution's ucrt64 toolchain file and the standard Windows plugin defaults. It is a single configuration with no matrix to keep the added CI cost low. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
Document the Linux-hosted MinGW-w64 UCRT64 cross-compilation option in the Windows section of the developer guide, pointing at the pr-compile-mingw-cross workflow job as the reference recipe. Signed-off-by: Jiawei Huang <jiawei@tigera.io>
hjiawei
force-pushed
the
mingw-ucrt64-cross-build
branch
from
34b9aba to
b21b836
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a draft / WIP PR that makes Fluent Bit cross-compile for Windows
from a Linux host using the MinGW-w64 UCRT64 toolchain (GCC), as an
alternative to a native Windows/MSVC build environment. The result is a working
fluent-bit.exe(PE32+, x86-64) with the WASM filter enabled, built entirely onLinux.
The change is deliberately conservative: every Windows code path stays gated so
the existing MSVC build is untouched. Where a symbol differs only between MSVC
and MinGW (Winsock
char*vs POSIXvoid*buffers, lowercase SDK headers,_WIN32_WINNT/NTDDI_VERSIONagreement,PTHREAD_STACK_MIN, libco backendselection on x86-64,
//-comments ending in a backslash, etc.) the fix is#ifdef __MINGW32__/if(MINGW)scoped so MSVC compiles the original codebyte-for-byte.
The only bundled-library source changes here are ones that belong upstream, and
each has been submitted there. This PR should not be merged until they merge
and Fluent Bit re-vendors those libraries; the corresponding
lib/…hunks arecarried here only so the cross-build works meanwhile, and are expected to arrive
via a re-vendor instead:
lib/monkeylib/chunkiolib/cmetricslib/wasm-micro-runtime-WAMR-2.4.1lib/flb_libco(and monkey's vendored copy)No other bundled libraries are patched
Two vendored libraries that an earlier revision of this PR touched turned out to
need no source change and are intentionally left untouched:
_BitScanForwardintrinsic path compiles under MinGW(the
unsigned long*-vs-int*warning is benign on LLP64, where both are32-bit).
nghttp2_net.hdon't pull inwinsock2.h, so its inline byte-order fallback compiles as-is withoutcolliding with the Winsock declarations.
onigmo needs no source patch either: when cross-compiling to Windows its
config.his misdetected by the configure-time probes, so the Fluent Bit buildfiles copy the
win32/config.hthat Onigmo already ships over the generated one— handled in
CMakeLists.txt, leaving the vendored tree unpatched.Fluent Bit's own sources
The rest is the actual cross-build enablement in Fluent Bit itself: the CMake
toolchain/crosscompile handling, MinGW gates and lowercase system headers in
src/,config_format/, the network stub, thecompatWindows headers, theWAMR build wiring, and the c-shared Go plugins. A CI workflow
(
build-windows-mingw) is included that performs the cross-build onfedora:44so the path stays green.Enter
[N/A]in the box, if an item is not applicable to your change.Testing
This is a build/toolchain change (no runtime behavior change on existing
platforms), so the runtime-oriented items below are N/A. The added CI workflow
cross-builds
fluent-bit.exeon every PR as the test evidence.[N/A] Example configuration file for the change
[N/A] Debug log output from testing the change
[N/A] Attached Valgrind output that shows no leaks or memory corruption was found
Run local packaging test showing all targets (including any new ones) build.
Set
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.