Add sentencepiece@0.2.1 - #9552
Merged
Merged
Conversation
BYVoid
force-pushed
the
sentencepiece-0.2.2pre1
branch
from
July 3, 2026 17:06
1582b63 to
6e85926
Compare
BYVoid
marked this pull request as ready for review
July 3, 2026 17:13
Member
|
Hello BCR maintainers, modules without existing maintainers (sentencepiece) have been updated in this PR. |
bazel-io
requested review from
a team,
cerisier and
fweikert
and removed request for
a team
July 3, 2026 17:13
Wyverald
added
the
presubmit-auto-run
BYVoid
force-pushed
the
sentencepiece-0.2.2pre1
branch
from
July 6, 2026 22:06
6e85926 to
812b18c
Compare
BYVoid
changed the title
BYVoid
force-pushed
the
sentencepiece-0.2.2pre1
branch
from
July 6, 2026 22:44
aa19625 to
68bbac0
Compare
Wyverald
approved these changes
Jul 7, 2026
bazel-io
pushed a commit
that referenced
this pull request
Jul 14, 2026
Adds [SentencePiece](https://github.com/google/sentencepiece) v0.2.2, following up on #9552 (sentencepiece@0.2.1). Same approach as 0.2.1, with the Bazel build definitions taken from the upstream PR google/sentencepiece#1280 (not yet merged), adapted for the 0.2.2 release tarball: - `overlay/BUILD.bazel` + `overlay/src/BUILD.bazel`: root aliases, `//src:sentencepiece` (runtime), `//src:sentencepiece_train` (trainer), and the `spm_{train,encode,decode,normalize,export_vocab}` CLIs. The protobuf code is regenerated from `src/*.proto` at build time (matching the `SPM_PROTOBUF_PROVIDER=package` CMake configuration). - `patches/use_bcr_dependency_includes.patch`: rewrites `#include "third_party/absl/..."` → `"absl/..."`, `"third_party/darts_clone/darts.h"` → `"darts.h"`, and `"third_party/esaxx/esa.hxx"` → `"esa.hxx"` so the sources resolve against the BCR modules (abseil-cpp, darts-clone@0.32h.bcr.1, esaxx@20250106.1.bcr.1) instead of the vendored copies. - The release tarball is a Python sdist, so `strip_prefix` is `sentencepiece-0.2.2/sentencepiece`. It does not ship the `data/` directory, so the upstream `spm_test` target is not included. Instead, the overlay adds a `bcr_tests/` test module (`bcr_test_module` in presubmit) with an in-memory train → encode → decode roundtrip `cc_test`; presubmit also verifies all library and CLI targets on Linux/macOS/Windows with C++17 flags. Differences from 0.2.1: 0.2.2 uses `init.cc` (renamed from `error.cc`), migrated to absl::Status/log (more abseil-cpp deps, no `_USE_EXTERNAL_ABSL` define needed), and its `MODULE.bazel` pins abseil-cpp 20260526.0 to match the `GIT_TAG` in `CMakeLists.txt`. Locally verified: all targets build (Bazel 9.1.1), the `bcr_tests` roundtrip test passes, and `bcr_validation.py --check sentencepiece@0.2.2` passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds SentencePiece v0.2.1 as a new module.
The Bazel build definitions are taken from the upstream PR google/sentencepiece#1280 (not yet merged), adapted for the 0.2.1 release tarball:
overlay/BUILD.bazel+overlay/src/BUILD.bazel: root aliases,//src:sentencepiece(runtime),//src:sentencepiece_train(trainer), and thespm_{train,encode,decode,normalize,export_vocab}CLIs. The protobuf code is regenerated fromsrc/*.protoat build time (matching theSPM_PROTOBUF_PROVIDER=packageCMake configuration).defines = ["_USE_EXTERNAL_ABSL"]selects the external-absl branch oferror.cc, which defines the--minloglevelflag (normally supplied by the vendored mini-absl shim'sflag.cc), matching theSPM_ABSL_PROVIDER=module|packageCMake configurations.patches/use_bcr_dependency_includes.patch: rewrites#include "third_party/absl/..."→"absl/...","third_party/darts_clone/darts.h"→"darts.h", and"third_party/esaxx/esa.hxx"→"esa.hxx"so the sources resolve against the BCR modules (abseil-cpp, darts-clone@0.32h.bcr.1, esaxx@20250106.1.bcr.1) instead of the vendored copies.strip_prefixissentencepiece-0.2.1/sentencepiece. It does not ship thedata/directory, so the upstreamspm_testtarget is not included. Instead, the overlay adds abcr_tests/test module (bcr_test_modulein presubmit) with an in-memory train → encode → decode roundtripcc_test; presubmit also verifies all library and CLI targets on Linux/macOS/Windows with C++17 flags.Locally verified: all targets build (Bazel 9.1.1), the
bcr_testsroundtrip test passes,spm_train/spm_encodework end to end, andbcr_validation.py --check sentencepiece@0.2.1passes.