Skip to content

Docker: a COPY --from carries an image reference in the grammar - #8604

Open
timtebeek wants to merge 3 commits into
mainfrom
tim/docker-copy-from-image-ref
Open

Docker: a COPY --from carries an image reference in the grammar#8604
timtebeek wants to merge 3 commits into
mainfrom
tim/docker-copy-from-image-ref

Conversation

@timtebeek

Copy link
Copy Markdown
Member

What changed

The lexer emits --name=value as a single FLAG token, so COPY --from=host:5000/img:1.2 was one flat literal and ImageReferences.split(List, Space) had to decide for itself which colon was a tag separator and which belonged to a registry port — the rule IMAGE_REF already knows. This is live: the docker-library corpus holds 72 --from=<image>:<tag>.

A FROM_FLAG token now pushes a FLAG_IMAGE_REF mode, and the value goes through the same imageReference rule a FROM does:

fromFlag
    : FROM_FLAG imageReference? FLAG_END?
    ;

Three things the mode needs:

  • A predicate can only disable the longer rule, not beat it. Lexing is maximal munch, so a rule that stops at : always loses to FLAG. The predicate has to sit at the very start of the longer rule, turning it off:
    FLAG      : {!atFromFlag()}? FLAG_TOKEN;
    FROM_FLAG : {atFromFlag()}?  '--from=' -> pushMode(FLAG_IMAGE_REF);
    atFromFlag() is a copyAddFlags member set at COPY/ADD, cleared at UNQUOTED_TEXT and NEWLINE, plus an _input.LA() peek for --from=. --chown=1:2, RUN echo --from=a:b and --mount=type=cache,from=builder therefore stay single FLAG tokens.
  • FLAG_IMAGE_REF differs from IMAGE_REF only in where it ends: at the whitespace before the paths rather than at AS or the end of the line.
  • Popping a mode does not bound a parser rule. The shared imageName : textElement+ happily consumed past the pop, making --link part of the image name of COPY --from=build --link /target/ /. The pop has to emit a visible token the parser rule names, hence FLAG_END : ( WS_CHAR+ | LINE_CONT ) -> popMode. Retyping the mode's tokens back to UNQUOTED_TEXT would erase the boundary the mode created.

ImageReferences.split(List, Space) is deleted. DockerCopyFrom reads the parts off the separators the mode left in the flag value's contents, the same convention --mount=type=bind already uses for its =; the value prints as before, and a colon in a registry port, a quoted name or a variable reference never becomes a content of its own. split(String, Space) stays for the reference a recipe supplies as text (withImageReference), and shrinks to a single loop now that it has one caller shape.

One fix came out of FLAG_IMAGE_REF being modelled on IMAGE_REF: ESCAPED_CHAR ('\\' .) swallowed the newline of a line continuation, so FROM nginx:1.25\ + newline parsed the tag as 1.25\␊. Both modes now use '\\' ~[\r\n], which also keeps COPY --from=nginx:1.25\ + newline from regressing against the old FLAG token.

Verification

:rewrite-docker:test is green (539 tests). Four LST-shape cases in CopyTest (separators split out, only --from splits, a following flag is not part of the value, a line continuation ends the value) and two in DockerCopyFromTest (a variable tag split from a variable image name, a recipe-written reference read back as its parts) — the corpus holds no flag value containing a $VAR, so it can only prove "no regression", never "the fix works".

The corpus diff: 713 Dockerfiles from 29 docker-library repos, LST parts + round-trip result + syntax errors (via ExecutionContext.getOnError()) dumped at origin/main and with this change. Every one of the 144 changed lines is a --from value splitting as intended; no other LST changed, no round-trip mismatch appeared or disappeared, and the syntax-error set is identical.

-  FLAG from prefix=' ' value=[L(eclipse-temurin:11-jre) ]
+  FLAG from prefix=' ' value=[L(eclipse-temurin) L(:) L(11-jre) ]

#8590 gave `FROM` an `IMAGE_REF` lexer mode, but `COPY --from=` never
reached it: the lexer emits `--name=value` as a single `FLAG` token, so
`COPY --from=host:5000/img:1.2` stayed one flat literal and
`ImageReferences.split(List, Space)` had to re-implement the registry-port
rule the lexer already knows.

A `FROM_FLAG` token now pushes a `FLAG_IMAGE_REF` mode, which is
`IMAGE_REF` ending at the whitespace before the paths rather than at `AS`
or the end of the line, and `fromFlag : FROM_FLAG imageReference? FLAG_END?`
runs the value through the same `imageReference` rule a `FROM` uses.

`ImageReferences.split(List, Space)` is deleted; `split(String, Space)`
stays for the reference a recipe supplies as text.
@timtebeek
timtebeek force-pushed the tim/docker-copy-from-image-ref branch from 01f91c9 to ebe9940 Compare August 22, 2026 00:26
A semantic predicate reachable without consuming anything sits in the ATN
start-state closure of its mode, and ANTLR refuses to cache a start state
whose closure carried a semantic context. `decisionToDFA[DEFAULT_MODE].s0`
therefore stayed null and every token re-ran a closure over all of the
default mode's rules: lexing a 12,001 line Dockerfile holding no `--from`
at all took 318ms rather than 3ms.

Both flag rules already begin with '--', so the predicate can sit after it.
Only a token starting with '-' pays an uncached edge, and lexing that same
file is back to 5ms.
`ImageReferences` was narrowed to text, which left the trait layer holding a
private copy of the same partition and a split-then-rejoin round trip through
a representation it did not want. It now owns both directions -- parts to
contents and contents back to parts -- so `DockerCopyFrom` is two thin calls,
`DockerParserVisitor` shares one `imageReference` walk between `FROM` and a
`COPY`'s `--from`, and a second consumer of a flag's reference has something
to call.

In the lexer, the reset of the flags scoped to a logical line is one method
rather than four inline copies that had already drifted, the two image
reference modes share the text fragment they had a copy of each of, and that
fragment reuses `ESCAPE_SEQUENCE`, which it had been a fourth spelling of.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant