Add TensorFlow easyblock options for Bazel startup flags and TF_SYSTEM_LIBS exclusions by bedroesb · Pull Request #4193 · easybuilders/easybuild-easyblocks

bedroesb · 2026-08-06T20:22:53Z

This PR moves certain patches from easybuilders/easybuild-easyconfigs#26602 into the TensorFlow easyblock.

Changes included here:

add bazel_startup_opts, so easyconfigs can pass Bazel startup options such as --batch without editing .bazelrc;
add tf_system_libs_exclude, so easyconfigs can opt out of specific TF_SYSTEM_LIBS entries without patching .tf_configure.bazelrc;
skip absl_py automatically when TensorFlow’s tf_http_archive entry has no system_build_file;
pass the temporary TensorFlow home directory to Bazel tests via --test_env=HOME=...;
adjust TensorBoard sanity checking so an external tensorboard dependency is accepted instead of requiring bin/tensorboard inside the TensorFlow installation prefix.

The absl_py handling is needed for TensorFlow 2.21.0 because absl_py is still listed in TensorFlow’s VALID_LIBS, but its tf_http_archive entry no longer has a system_build_file. Without this, Bazel fails with:

ERROR: /tmp/bert.droesbeke/easybuild-tensorflow-build/TensorFlow/2.21.0/foss-2025b/TensorFlow/tensorflow-2.21.0/WORKSPACE:100:14: fetching _tf_http_archive rule //external:absl_py: java.io.IOException: _tf_http_archive rule //external:absl_py must create a directory
ERROR: no such package '@@absl_py//absl/flags': _tf_http_archive rule //external:absl_py must create a directory
ERROR: /tmp/bert.droesbeke/easybuild-tensorflow-build/TensorFlow/2.21.0/foss-2025b/TensorFlow/tensorflow-2.21.0/tensorflow/python/tools/api/generator2/generator/BUILD:10:22: //tensorflow/python/tools/api/generator2/generator:generator depends on @@absl_py//absl/flags:flags in repository @@absl_py which failed to fetch. no such package '@@absl_py//absl/flags': _tf_http_archive rule //external:absl_py must create a directory

The tf_system_libs_exclude option is needed because some TensorFlow system-library stubs are not sufficient for all Bazel compile actions. For TensorFlow 2.21.0, enabling system boringssl made gRPC pick up OpenSSL headers via Bazel actions, which failed with:

ERROR: /tmp/bert.droesbeke/easybuild-tensorflow-build/TensorFlow/2.21.0/foss-2025b/TensorFlow/bazel-root/29cea7ecf482505435700a02a31d6d02/external/com_github_grpc_grpc/BUILD:4219:16: Compiling src/core/tsi/ssl/session_cache/ssl_session_boringssl.cc failed: (Exit 1): clang failed: error executing CppCompile command (from target @@com_github_grpc_grpc//:tsi_ssl_session_cache)
external/com_github_grpc_grpc/src/core/tsi/ssl/session_cache/ssl_session.h:24:10: error: module com_github_grpc_grpc//:tsi_ssl_session_cache does not depend on a module exporting 'openssl/ssl.h'
/mnt/modules/easybuild/software/OpenSSL/3/include/openssl/ssl.h:29:10: fatal error: 'openssl/e_ostime.h' file not found

While validating the cleanup, using system cURL showed the same general problem with exposing dependency headers globally through CPATH:

ERROR: /tmp/bert.droesbeke/easybuild-tensorflow-build/TensorFlow/2.21.0/foss-2025b/TensorFlow/bazel-root/29cea7ecf482505435700a02a31d6d02/external/xla/xla/tsl/platform/cloud/BUILD:219:11: Compiling xla/tsl/platform/cloud/curl_http_request.cc failed: absolute path inclusion(s) found in rule '@@xla//xla/tsl/platform/cloud:curl_http_request':
the source file 'xla/tsl/platform/cloud/curl_http_request.cc' includes the following non-builtin files with absolute paths:
  '/data/groups/technologies/data.core/bert.droesbeke/easybuild/software/cURL/8.14.1-GCCcore-14.3.0/include/curl/curl.h'

With these easyblock changes, the TensorFlow 2.21.0 easyconfig no longer needs the previous sed/CPATH/TensorBoard-symlink workarounds. The corresponding TensorFlow/2.21.0-foss-2025b build completed successfully.

Flamefire · 2026-08-07T08:07:14Z

Before I jump into the implementation (which looks ok) I'd like to discuss the changes

add bazel_startup_opts, so easyconfigs can pass Bazel startup options such as --batch without editing .bazelrc;

Good idea, I'd say as an easy way to avoid modifying the easyblock. I'd expect you'd want --batch at least starting at a given Bazel version. Does it has any downsides or shall we always use it? Then we should add it here instead of in each easyconfig.

runs Bazel in batch mode to avoid lock/server startup issues

How does this look like and how does --batch avoids it?

add tf_system_libs_exclude, so easyconfigs can opt out of specific TF_SYSTEM_LIBS entries without patching .tf_configure.bazelrc;

I'd say this is not a good idea: Only direct dependencies are considered. So instead of excluding it you can simply not add it, can't you? Otherwise you would be mixing EasyBuild dependencies and downloaded/vendored software which likely leads to issues.

skip absl_py automatically when TensorFlow’s tf_http_archive entry has no system_build_file;

Similar to above: If it doesn't work (and we can't patch it) remove the dependency from the easyconfig to avoid conflicts.

adjust TensorBoard sanity checking so an external tensorboard dependency is accepted instead of requiring bin/tensorboard inside the TensorFlow installation prefix.

Good idea but I'd do it differently: It looks like TensorFlow did install tensorboard and since some version does it. So instead guard it on the version that introduced the change so you are not required to include tensorboard as a dependency of TensorFlow.

enabling system boringssl made gRPC pick up OpenSSL headers via Bazel actions, which failed with:

Not enabling it mixes the SSL installations leading to issues again. It looks rather like a problem in OpenSSL:

/mnt/modules/easybuild/software/OpenSSL/3/include/openssl/ssl.h:29:10: fatal error: 'openssl/e_ostime.h' file not found

--> OpenSSL header includes an OpenSSL header that doesn't exist?

the source file 'xla/tsl/platform/cloud/curl_http_request.cc' includes the following non-builtin files with absolute paths:
'/data/groups/technologies/data.core/bert.droesbeke/easybuild/software/cURL/8.14.1-GCCcore-14.3.0/include/curl/curl.h'

This looks like something we should fix too: We do want to use our cURL to avoid conflicts. Maybe this is caused by the filtering of CPATH you mentioned? Maybe there is some other option. ChatGPT mentions using build --copt=-isystem=/data/cURL/8.14.1-GCCcore-14.3.0/include etc to the bazelrc file instead of --action-env=CPATH=... that we currently do. Might be worth a try. Currently can't because our cluster is in maintenance

bedroesb · 2026-08-09T05:53:45Z

Thanks for the detailed feedback:

Based on the suggestions I could cleanup quite a bit:

removed the generic tf_system_libs_exclude option;
stopped auto-enabling boringssl merely because OpenSSL is present somewhere in the dependency tree;
boringssl is now only added when cURL is a direct TensorFlow dependency, matching the existing cURL-specific rationale;
made --batch automatic for Bazel >= 7.0.0, while keeping bazel_startup_opts only as an escape hatch for extra startup options;
changed TensorBoard sanity checking to be version-based for TensorFlow >= 2.21, so TensorFlow is no longer expected to install bin/tensorboard;
kept the absl_py check because absl-py is still a required Python dependency, but TensorFlow 2.21 cannot use it as a TF_SYSTEM_LIBS repository since its tf_http_archive entry has no system_build_file.

For absl_py, the exact failure was:

ERROR: .../WORKSPACE:100:14: fetching _tf_http_archive rule //external:absl_py:
java.io.IOException: _tf_http_archive rule //external:absl_py must create a directory
ERROR: no such package '@@absl_py//absl/flags': _tf_http_archive rule //external:absl_py must create a directory

I will go on holiday today, so feel free to change things without waiting on a response ;)

Flamefire · 2026-08-10T08:50:53Z

* stopped auto-enabling `boringssl` merely because OpenSSL is present somewhere in the dependency tree;

Why? If we have both and not using rpath the 2 will conflict, won't they?
The failure you see is because of use of system OpenSSL (via wrappers) and can be fixed by rebuilding the OpenSSL module. I'll add an update to the easyblock to avoid that in the future.

* kept the `absl_py` check because `absl-py` is still a required Python dependency, but TensorFlow 2.21 cannot use it as a `TF_SYSTEM_LIBS` repository since its `tf_http_archive` entry has no `system_build_file`.

Why is it a required Python dependency?
I do think the check is useful but more generic: The easyblock contains a version check for system deps, so we could state it as supported only until 2.21.
The check for system build file could then be done in the general "is systemlibs up to date, or update easyblock" function.

For absl_py, the exact failure was:

ERROR: .../WORKSPACE:100:14: fetching _tf_http_archive rule //external:absl_py:
java.io.IOException: _tf_http_archive rule //external:absl_py must create a directory
ERROR: no such package '@@absl_py//absl/flags': _tf_http_archive rule //external:absl_py must create a directory

Just reproduced that. I'm testing a patch as the removal seems accidental by upstream.

eylenth · 2026-08-10T10:13:32Z

I'm now also trying to build TensorFlow with the CUDA variant ( I will create a PR for the easyconfig once it is working).
I'm testing out this easyblock as well, but I'm getting the following error message in the build step of bazel:

ERROR: /tools/eb/build/TensorFlow/2.21.0/foss-2025b-CUDA-12.9.1/TensorFlow/bazel-root/c741a8d54995072d0506b6004cf6ee2e/external/curl/BUILD.bazel:26:11: Compiling lib/altsvc.c failed: absolute path inclusion(s) found in rule '@@curl//:curl':
the source file 'lib/altsvc.c' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
  '/cvmfs/biogrid-software.rnd.seeds.basf.net/tools/eb/x86_64/generic/software/OpenSSL/3/include/openssl/opensslv.h'
  '/cvmfs/biogrid-software.rnd.seeds.basf.net/tools/eb/x86_64/generic/software/OpenSSL/3/include/openssl/macros.h'
  '/cvmfs/biogrid-software.rnd.seeds.basf.net/tools/eb/x86_64/generic/software/OpenSSL/3/include/openssl/opensslconf.h'
  '/cvmfs/biogrid-software.rnd.seeds.basf.net/tools/eb/x86_64/generic/software/OpenSSL/3/include/openssl/configuration.h'
  '/cvmfs/biogrid-software.rnd.seeds.basf.net/tools/eb/x86_64/generic/software/OpenSSL/3/include/openssl/configuration-x86_64.h'

I tried to add the ('cURL', '8.14.1') as dependency in the easyconfig recipe of TensorFlow, But then I'm getting another error message again.
I'm not using rpath btw.

Maybe this could be useful by mentioning this.

Flamefire · 2026-08-10T15:27:43Z

I'm testing out this easyblock as well, but I'm getting the following error message in the build step of bazel:

That's (kind of) expected. The change here to not use OpenSSL/BoringSSL creates this "conflict". Adding cURL is correct anyway. Working on a solution for related fixes.

Flamefire · 2026-08-12T15:31:45Z

I made significant progress here after running into undeclared inclusions, (C++) module validation issues and more

A short test using system/our cURL can be done by passing the @xla//xla/tsl/platform/cloud:curl_http_request target instead of //tensorflow/tools/pip_package:wheel to bazel

This currently yields:

ERROR: /dev/shm/TensorFlow/2.21.0/foss-2025b/TensorFlow/bazel-root/38526f3dacc6677801c11293634cdb7f/external/xla/xla/tsl/platform/cloud/BUILD:219:11: Compiling xla/tsl/platform/cloud/curl_http_request.cc failed: absolute path inclusion(s) found in rule '@@xla//xla/tsl/platform/cloud:curl_http_request':
the source file 'xla/tsl/platform/cloud/curl_http_request.cc' includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain):
  '/software/cURL/8.14.1-GCCcore-14.3.0/include/curl/curl.h'
...
Target @@xla//xla/tsl/platform/cloud:curl_http_request failed to build

After lots of debugging and chatting with AI agents I stumbled upon a suspicious toolchain being used and git blamed it to: tensorflow/tensorflow@f7dfba3f742c (first released in 2.21)

It mentions the solution --config=clang_local

Another failure in the full build was:

In file included from external/com_github_grpc_grpc/src/core/tsi/ssl/session_cache/ssl_session_boringssl.cc:21:
external/com_github_grpc_grpc/src/core/tsi/ssl/session_cache/ssl_session.h:24:10: error: module com_github_grpc_grpc//:tsi_ssl_session_cache does not depend on a module exporting 'openssl/ssl.h'
   24 | #include <openssl/ssl.h>

This is that C++ module checker. I found that this can be disable by passing --features=-layering_check but still testing if we still need that after the above.

Currently that solution causes other issues: They assume their toolchain is used and pass flags accordingly causing e.g.

gcc: error: unrecognized command-line option '-fno-experimental-sanitize-metadata=all'

I'm not sure if we could use their toolchain, although they pass our flags like -march=native to that compiler I see --sysroot external/sysroot_linux_x86_64_glibc_2_27
I'm afraid that will bite us at least with EESSI. And of course it circumvents our rpath wrappers

Flamefire · 2026-08-13T16:49:12Z

        """
        dependency_mapping, python_mapping = get_system_libs_for_version(self.version)
+
+        # TensorFlow 2.21 still lists absl_py as a valid TF_SYSTEM_LIBS entry, but its


This can be fixed by a patch so does not need to be in the easyblock, see https://github.com/easybuilders/easybuild-easyconfigs/pull/26658/changes#diff-01cb94902867bc2e7ffd436faf712aca6604eb5ea903030bf45a6f5eb49f9792

Flamefire · 2026-08-13T17:15:19Z

I created #4205 which does what is required for TF 2.19 and 2.21. I think here the parts to be kept are the bazel startup options with the --batch default. Not critical though

apply fixes

ddd70ec

bedroesb mentioned this pull request Aug 6, 2026

{lib}[foss/2025b] TensorFlow v2.21.0 easybuilders/easybuild-easyconfigs#26602

Open

boegel added this to the release after release of Aug'26 milestone Aug 7, 2026

boegel added the enhancement label Aug 7, 2026

Implement feedback

46318e7

Flamefire reviewed Aug 13, 2026

View reviewed changes

Flamefire mentioned this pull request Aug 13, 2026

Update TensorFlow easyblock for TensorFlow 2.19+ #4205

Open

Flamefire mentioned this pull request Aug 13, 2026

{lib}[foss/2025b] TensorFlow v2.21.0 easybuilders/easybuild-easyconfigs#26658

Draft

Navigation Menu

Uh oh!

Add TensorFlow easyblock options for Bazel startup flags and TF_SYSTEM_LIBS exclusions - #4193

Add TensorFlow easyblock options for Bazel startup flags and TF_SYSTEM_LIBS exclusions#4193
bedroesb wants to merge 2 commits into
easybuilders:developfrom
bedroesb:TensorFlow

bedroesb commented Aug 6, 2026

Uh oh!

Flamefire commented Aug 7, 2026

Uh oh!

bedroesb commented Aug 9, 2026

Uh oh!

Flamefire commented Aug 10, 2026

Uh oh!

eylenth commented Aug 10, 2026 •

edited

Loading

Uh oh!

Flamefire commented Aug 10, 2026

Uh oh!

Flamefire commented Aug 12, 2026 •

edited

Loading

Uh oh!

Flamefire Aug 13, 2026 •

edited

Loading

Uh oh!

Flamefire commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

bedroesb commented Aug 6, 2026

Uh oh!

Flamefire commented Aug 7, 2026

Uh oh!

bedroesb commented Aug 9, 2026

Uh oh!

Flamefire commented Aug 10, 2026

Uh oh!

eylenth commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Flamefire commented Aug 10, 2026

Uh oh!

Flamefire commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Flamefire Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Flamefire commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eylenth commented Aug 10, 2026 •

edited

Loading

Flamefire commented Aug 12, 2026 •

edited

Loading

Flamefire Aug 13, 2026 •

edited

Loading