cut: char-aware multibyte field, byte and delimiter handling by sylvestre · Pull Request #13927 · uutils/coreutils

sylvestre · 2026-08-13T20:08:31Z

Treat the C/POSIX locale as single-byte in the charmap, match field
delimiters on character boundaries (never inside a multibyte character),
and recognize Unicode whitespace for -w. paste tests that exercise
multibyte delimiters now declare a UTF-8 locale.

Should make test tests/cut/cut.pl pass

_{Stack created with GitHub Stacks CLI • Give Feedback 💬}

Copilot

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR makes cut and paste locale-aware for multibyte behavior: treating C/POSIX as single-byte, ensuring delimiters are matched on character boundaries (not inside multibyte characters), and expanding -w whitespace handling to recognize Unicode “blank” characters in UTF-8 locales.

Changes:

Update i18n charmap logic to treat C/POSIX as strictly single-byte and add Unicode “blank” detection for -w.
Update cut delimiter matching to avoid splitting multibyte characters and add corresponding regression tests.
Update paste delimiter parsing and tests to run under a UTF-8 locale when multibyte delimiters are involved.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/by-util/test_paste.rs	Sets a UTF-8 locale for multibyte-delimiter `paste` tests.
tests/by-util/test_cut.rs	Adds UTF-8-locale regression tests for `cut` byte/field behaviors with multibyte characters/delimiters and Unicode blanks.
src/uucore/src/lib/features/i18n/charmap.rs	Makes `C`/`POSIX` strictly single-byte and introduces `mb_blank_len` for Unicode blank detection in UTF-8 locales.
src/uu/paste/src/paste.rs	Uses `mb_char_len` directly for delimiter tokenization (character-stepped).
src/uu/cut/src/matcher.rs	Adds a multibyte-aware exact matcher and updates whitespace matching to use locale-aware blank detection.
src/uu/cut/src/cut.rs	Uses `MbExactMatcher` for non-ASCII delimiters and refactors field-cutting to share matcher/out-delim handling.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        let mut pos = 0;
+        while pos < haystack.len() {
+            if let Some(blank_len) = mb_blank_len(&haystack[pos..]) {
+                let start = pos;
+                pos += blank_len;
+                while pos < haystack.len() {
+                    match mb_blank_len(&haystack[pos..]) {
+                        Some(len) => pos += len,
+                        None => break,
+                    }
+                }
+                return Some((start, pos));
            }
+            pos += mb_char_len(&haystack[pos..]);
        }
-
-        Some((match_idx, skip))
+        None


+pub fn mb_blank_len(bytes: &[u8]) -> Option<usize> {
+    let len = mb_char_len(bytes);
+    if len == 1 {
+        return (bytes[0] == b' ' || bytes[0] == b'\t').then_some(1);
+    }
+    // `char_len` only looks at the shape of the sequence, so decode it for real
+    // here: an overlong or surrogate encoding is not a character at all, and
+    // must not pass for one of the blanks it decodes to.
+    if locale_encoding() == Encoding::Utf8
+        && std::str::from_utf8(&bytes[..len]).is_ok_and(|s| s.starts_with(is_unicode_blank))
+    {
+        return Some(len);
+    }
+    None
+}


Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

src/uu/cut/src/cut.rs:970

This validation now only checks “UTF-8 shape” via mb_char_len and may accept ill-formed UTF-8 byte sequences (e.g., surrogate-encoded triples) as a “single character” in a UTF-8 locale. In UTF-8 locales, delimiter validation should additionally require the bytes decode as UTF-8 and contain exactly one Unicode scalar value (while keeping the current byte-length logic for non-UTF-8 encodings).

                // The delimiter must be a single character in the *current locale's*
                // encoding: a multi-byte UTF-8 character in a UTF-8 locale, a 2-byte
                // GB18030 character, or any single byte like `b"\xAD"`. In a C/POSIX
                // (single-byte) locale even a valid UTF-8 multibyte sequence counts
                // as several characters, matching GNU.
                let bytes = os_str_as_bytes(os_string)?;
                let is_single_char = mb_char_len(bytes) == bytes.len();
                if !is_single_char {

src/uu/cut/src/matcher.rs:73

MbExactMatcher::next_match is O(n) over the full input slice and resolves locale encoding on every call; if the surrounding Searcher invokes next_match repeatedly while scanning, this can easily become O(n²) per line. Consider resolving Encoding once per line (or storing it in the matcher) and designing the search to advance incrementally over character boundaries (stateful searcher/iterator) rather than restarting from pos = 0 each call.

    fn next_match(&self, haystack: &[u8]) -> Option<(usize, usize)> {
        // Resolved once per line: character boundaries have to be walked from
        // the start, so there is no way to skip ahead with `memchr` here.
        let encoding = locale_encoding();
        let mut pos = 0;
        while pos < haystack.len() {
            let clen = encoding.char_len(&haystack[pos..]);
            if clen == self.needle.len() && &haystack[pos..pos + clen] == self.needle {
                return Some((pos, pos + clen));
            }
            pos += clen;
        }
        None
    }

tests/by-util/test_paste.rs:434

This sets LC_ALL=C.UTF-8 for all paste data examples, although the comment says only “some” require it. It would be more precise (and better at catching regressions in the C locale) to set the locale only for examples that actually contain multibyte delimiters (e.g., by tagging such cases in the test vector or checking the delimiter bytes).

        // Some examples use multibyte delimiters, which need a UTF-8 locale.
        ucmd.env("LC_ALL", "C.UTF-8")
            .args(example.args)

+        if self == Self::Utf8
+            && std::str::from_utf8(&bytes[..len]).is_ok_and(|s| s.starts_with(is_unicode_blank))
+        {
+            return Some(len);
        }
+        None


codspeed-hq · 2026-08-14T21:22:09Z

Merging this PR will degrade performance by 10.2%

❌ 2 regressed benchmarks
✅ 351 untouched benchmarks
⏩ 50 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`cut_characters_long_lines`	34.6 ms	39.4 ms	-12%
❌	Simulation	`cut_characters`	23.2 ms	25.4 ms	-8.36%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing cut-mb-non-utf8 (96dbf26) with main (be763fc)}

50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

github-actions · 2026-08-14T21:33:05Z

GNU testsuite comparison:

Skip an intermittent issue tests/pr/bounded-memory (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/date/date-locale-hour (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/cut/cut is no longer failing!

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/uu/cut/src/matcher.rs:92

The fast memchr2 path is currently limited to Encoding::SingleByte, but for the other non-UTF-8 multibyte encodings supported here (GB18030/EUC/Big5), blank_len() will only ever match ASCII space/tab anyway. You can likely extend the fast path to all encoding != Encoding::Utf8 to avoid per-character decoding on large inputs in those locales, while preserving correctness.

        // In a single-byte locale SPACE and TAB are the only blanks, so the run
        // can be found with a SIMD scan instead of decoding every character.
        let encoding = locale_encoding();
        if encoding == Encoding::SingleByte {
            let start = memchr2(b' ', b'\t', haystack)?;
            let mut end = start + 1;
            while end < haystack.len() && matches!(haystack[end], b' ' | b'\t') {
                end += 1;
            }
            return Some((start, end));
        }

src/uu/cut/src/cut.rs:860

With this PR, whitespace delimiter matching under -w recognizes Unicode blanks in a UTF-8 locale, but the --whitespace-delimited=trimmed code path (via cut_fields_whitespace_trimmed) still trims using only ASCII space/tab (per the current implementation in context). This creates inconsistent behavior where leading/trailing Unicode blanks won’t be trimmed even though they’re treated as delimiters. Consider updating the trimming logic to use the same blank definition (e.g., Encoding::blank_len / the whitespace matcher) so trimmed remains consistent with delimiter detection.

        Delimiter::Whitespace => {
            let out_delim = opts.out_delimiter.unwrap_or(b"\t");
            if field_opts.whitespace_trimmed {

src/uu/cut/src/matcher.rs:66

The comment says the encoding is “Resolved once per line”, but locale_encoding() is resolved per next_match() call. Consider either (a) adjusting the comment, or (b) storing Encoding inside MbExactMatcher (or passing it in) to align implementation with the comment and avoid repeated lookups.

impl Matcher for MbExactMatcher<'_> {
    fn next_match(&self, haystack: &[u8]) -> Option<(usize, usize)> {
        // Resolved once per line: character boundaries have to be walked from
        // the start, so there is no way to skip ahead with `memchr` here.
        let encoding = locale_encoding();
        let mut pos = 0;
        while pos < haystack.len() {
            let clen = encoding.char_len(&haystack[pos..]);

github-actions · 2026-08-14T22:58:47Z

Binary size comparison:

Individual binary size comparison VS main (threshold: >=5% AND >=4 KB).

Total size of compared binaries: 149.11 MB (-24 KB, -0.02%)

Significant per-binary changes:
  [           1.10 MB ->    1.18 MB  (+76 KB, +6.74%)
  mkfifo      1.09 MB ->    1.16 MB  (+76 KB, +6.81%)
  test        1.10 MB ->    1.18 MB  (+76 KB, +6.74%)
  install     1.25 MB ->    1.32 MB  (+72 KB, +5.62%)
  mkdir       1.10 MB ->    1.17 MB  (+72 KB, +6.38%)
  mknod       1.10 MB ->    1.17 MB  (+72 KB, +6.41%)
  chmod       1.15 MB ->    1.22 MB  (+68 KB, +5.76%)

Copilot

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

cakebaker · 2026-08-15T14:04:19Z

Is it intentional that this PR also contains changes of paste?

Treat the C/POSIX locale as single-byte in the charmap, match field delimiters on character boundaries (never inside a multibyte character), and recognize Unicode whitespace for -w. Should make test tests/cut/cut.pl pass

… in tests mb_char_len now guarantees the returned length never exceeds the slice size, making .min(remaining.len()) redundant. Set LC_ALL=C.UTF-8 in multibyte-delimiter tests and skip them under wasi_runner.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/uu/cut/src/matcher.rs:80

WhitespaceMatcher now treats additional Unicode “blank” characters as delimiters in UTF-8 locales, but the --whitespace-delimited=trimmed path in cut.rs trims only ASCII space/tab before splitting. This means leading/trailing Unicode blanks can survive “trimmed” preprocessing and yield empty/phantom fields even though they’re treated as delimiters later; trimming should use the same blank definition as matching (e.g., Encoding::blank_len).

// Matches any number of whitespace characters. ASCII space and tab are always
// recognized; in a UTF-8 locale Unicode space separators are too.
pub struct WhitespaceMatcher {}

impl Matcher for WhitespaceMatcher {

src/uu/cut/src/cut.rs:968

In get_delimiters, the --output-delimiter conversion later in this function uses os_str_as_bytes(os_string).unwrap(), which can panic on non-unix platforms if the OS string can’t be coerced to UTF-8 (see uucore::os_str_as_bytes). Since this function already returns UResult, it should propagate or map that error instead of unwrapping so invalid input is reported gracefully.

                // The delimiter must be a single character in the *current locale's*
                // encoding: a multi-byte UTF-8 character in a UTF-8 locale, a 2-byte
                // GB18030 character, or any single byte like `b"\xAD"`. In a C/POSIX
                // (single-byte) locale even a valid UTF-8 multibyte sequence counts
                // as several characters, matching GNU.

sylvestre · 2026-08-15T15:23:50Z

Is it intentional that this PR also contains changes of paste?

mb_char_len was introduced/fixed in this very PR to always return a length ≤ slice size, making .min(remaining.len()) redundant - so paste had to be updated to use the new guarantee.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Copilot AI lite review requested due to automatic review settings August 13, 2026 20:08

Copilot started reviewing on behalf of sylvestre August 13, 2026 20:24 View session

Copilot AI reviewed Aug 13, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 14, 2026 18:31

sylvestre force-pushed the cut-mb-non-utf8 branch from 6048734 to 3ab2ecd Compare August 14, 2026 18:31

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Comment thread src/uucore/src/lib/features/i18n/charmap.rs

Comment on lines +107 to +112

if self == Self::Utf8

&& std::str::from_utf8(&bytes[..len]).is_ok_and(|s| s.starts_with(is_unicode_blank))

{

return Some(len);

}

None

Copilot started reviewing on behalf of sylvestre August 14, 2026 18:49 View session

Copilot AI review requested due to automatic review settings August 14, 2026 21:21

sylvestre force-pushed the cut-mb-non-utf8 branch from 3ab2ecd to 69568dd Compare August 14, 2026 21:21

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Copilot started reviewing on behalf of sylvestre August 14, 2026 21:42 View session

Copilot AI review requested due to automatic review settings August 15, 2026 06:11

cakebaker force-pushed the cut-mb-non-utf8 branch from 69568dd to cd30e8e Compare August 15, 2026 06:11

Copilot AI reviewed Aug 15, 2026

Base automatically changed from cut-fopt to main August 15, 2026 13:22

cakebaker force-pushed the cut-mb-non-utf8 branch from cd30e8e to 8028d90 Compare August 15, 2026 13:22

Copilot AI review requested due to automatic review settings August 15, 2026 15:16

sylvestre force-pushed the cut-mb-non-utf8 branch from 8028d90 to 15eabab Compare August 15, 2026 15:16

Copilot started reviewing on behalf of sylvestre August 15, 2026 15:18 View session

sylvestre force-pushed the cut-mb-non-utf8 branch from 15eabab to 8028d90 Compare August 15, 2026 15:21

sylvestre added 2 commits August 15, 2026 17:23

cut: char-aware multibyte field, byte and delimiter handling

7253b1a

Treat the C/POSIX locale as single-byte in the charmap, match field delimiters on character boundaries (never inside a multibyte character), and recognize Unicode whitespace for -w. Should make test tests/cut/cut.pl pass

paste: remove redundant .min(remaining.len()) calls, set UTF-8 locale…

96dbf26

… in tests mb_char_len now guarantees the returned length never exceeds the slice size, making .min(remaining.len()) redundant. Set LC_ALL=C.UTF-8 in multibyte-delimiter tests and skip them under wasi_runner.

sylvestre force-pushed the cut-mb-non-utf8 branch from 8028d90 to 96dbf26 Compare August 15, 2026 15:23

Copilot AI reviewed Aug 15, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 15, 2026 15:23

Copilot started reviewing on behalf of sylvestre August 15, 2026 15:24 View session

Copilot AI reviewed Aug 15, 2026

View reviewed changes

sylvestre merged commit 49f2113 into main Aug 15, 2026
262 of 263 checks passed

sylvestre deleted the cut-mb-non-utf8 branch August 15, 2026 16:21

Navigation Menu

Uh oh!

Uh oh!

cut: char-aware multibyte field, byte and delimiter handling - #13927

cut: char-aware multibyte field, byte and delimiter handling#13927
sylvestre merged 2 commits into
mainfrom
cut-mb-non-utf8

sylvestre commented Aug 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

codspeed-hq Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

cakebaker commented Aug 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

sylvestre commented Aug 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

sylvestre commented Aug 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

codspeed-hq Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 10.2%

Performance Changes

Footnotes

Uh oh!

github-actions Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

cakebaker commented Aug 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

sylvestre commented Aug 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codspeed-hq Bot commented Aug 14, 2026 •

edited

Loading

github-actions Bot commented Aug 14, 2026 •

edited

Loading