OEL-4248: VRT Backstop by AaronGilMartinez · Pull Request #535 · openeuropa/oe_bootstrap_theme

AaronGilMartinez · 2026-01-16T08:40:07Z

Jira issue(s):

D8AGE-???

donquixote · 2026-04-22T20:44:23Z

+
+### Prerequisites
+
+- A running Drupal site reachable at `http://localhost:8080/build`. Override the target with the `VRT_BASE_URL` environment variable if needed.


I think we should simply point to the regular install steps.
The tests should pass on a fresh install following the steps above (docker-compose up, composer install, drupal:site-install).
There is no need to customize in the default case.
If you want to document these env vars, you could make a separate section about customization.

donquixote · 2026-04-22T20:50:23Z

We might want to check with QA if we can run this in gitlab.
But I am more than happy to first have it here in github, where we can control it all with code.

Let's only start this conversation once we are happy with what we have.

donquixote · 2026-04-22T20:53:31Z

+- `docker compose exec web npm run vrt:build` – scrapes the UI Patterns catalog and generates `tests/vrt/scenarios/patterns.json` plus `tests/vrt/backstop.generated.json`.
+- `docker compose exec web npm run vrt:reference` – captures fresh reference screenshots into `tests/vrt/bitmaps_reference`.
+- `docker compose exec web npm run vrt:test` – compares the current UI against the references, writing results to `tests/vrt/bitmaps_test` and `tests/vrt/html_report`.
+- `docker compose exec web npm run vrt:approve` – promotes the latest passing screenshots to the reference set.


When reading this, I would like to immediately understand in which category each of these commands fall, and when to run each of them:

I need to run this command as install / preparation before doing anything else.

I want to verify that the current appearance matches the captured snapshots.

I want to update the snapshots to align with changes in twig or CSS.

So here is what I understand:

I always need to run vrt:build before anything else, because the two json files are not tracked in git. I also need to run this any time after I created, deleted or renamed patterns.

I can run vrt:reference to update the snapshots after I changed the patterns.

I can run vrt:test to see if snapshots match current appearance. As a side effect this creates some tmp files.

I can run vrt:approve after vrt:test. Together, these achieve the same effect as vrt:reference. It is just a shortcut to avoid repeating the expensive calculation.

donquixote · 2026-04-22T20:56:56Z

+
+### Available commands
+
+- `docker compose exec web npm run vrt:build` – scrapes the UI Patterns catalog and generates `tests/vrt/scenarios/patterns.json` plus `tests/vrt/backstop.generated.json`.


When I run this, it tell me to run npx playwright install first.
I also see that the github action does this.
We could add this to the regular install commands, perhaps using runner.yml.dist.

donquixote · 2026-04-22T21:04:43Z

+      - name: Install dependencies
+        run: |
+          npm install
+          npx playwright install --with-deps


I notice that in README.md we run this npm stuff in web container, but here we run it in the root machine.

Nothing wrong, just an observation :)
(click "resolve" if this is as intended)

donquixote · 2026-04-22T21:10:55Z

Rebased on 1.x.
Interestingly, github action only runs if there are no merge conflicts.
Gitlab tests run either way.

donquixote · 2026-04-22T21:29:27Z

+      - name: Install dependencies
+        run: |
+          npm install
+          npx playwright install --with-deps


We might want to split this step into two steps, simply to make it easier to see the output from each step in isolation.

donquixote · 2026-04-22T22:27:09Z

+### Available commands
+
+- `docker compose exec web npm run vrt:build` – scrapes the UI Patterns catalog and generates `tests/vrt/scenarios/patterns.json` plus `tests/vrt/backstop.generated.json`.
+- `docker compose exec web npm run vrt:reference` – captures fresh reference screenshots into `tests/vrt/bitmaps_reference`.


I get a huge diff when I run this. But the images look the same.
Also, when I run it repeatedly, the files remain the same.
So maybe we just need to update once to make this PR green.

donquixote · 2026-04-22T22:27:55Z

+
+- `docker compose exec web npm run vrt:build` – scrapes the UI Patterns catalog and generates `tests/vrt/scenarios/patterns.json` plus `tests/vrt/backstop.generated.json`.
+- `docker compose exec web npm run vrt:reference` – captures fresh reference screenshots into `tests/vrt/bitmaps_reference`.
+- `docker compose exec web npm run vrt:test` – compares the current UI against the references, writing results to `tests/vrt/bitmaps_test` and `tests/vrt/html_report`.


I tried this.
How would I look at the output from this test, if there are failures? Which file do I look at?

donquixote · 2026-04-24T08:08:45Z

+- `docker compose exec web npm run vrt:build` – scrapes the UI Patterns catalog and generates `tests/vrt/scenarios/patterns.json` plus `tests/vrt/backstop.generated.json`.
+- `docker compose exec web npm run vrt:reference` – captures fresh reference screenshots into `tests/vrt/bitmaps_reference`.
+- `docker compose exec web npm run vrt:test` – compares the current UI against the references, writing results to `tests/vrt/bitmaps_test` and `tests/vrt/html_report`.
+- `docker compose exec web npm run vrt:approve` – promotes the latest passing screenshots to the reference set.


Can we run these commands targeting a specific snapshot or pattern?

donquixote · 2026-07-08T10:57:37Z

+*.jpg     -text diff
+*.otf     -text diff
+*.phar    -text diff
+*.png filter=lfs diff=lfs merge=lfs -text


I played a bit with LFS myself.
This is great! Let's go with it.

AaronGilMartinez changed the title ~~Oel 4248 backstop~~ OEL-4248: VRT Backstop Mar 17, 2026

AaronGilMartinez force-pushed the OEL-4248-backstop branch from deed894 to 09f114a Compare March 18, 2026 08:27

donquixote reviewed Apr 22, 2026

View reviewed changes

Comment thread README.md

donquixote reviewed Apr 22, 2026

View reviewed changes

donquixote force-pushed the OEL-4248-backstop branch from 164f022 to f2853bf Compare April 22, 2026 21:10

donquixote reviewed Apr 22, 2026

View reviewed changes

donquixote mentioned this pull request Apr 23, 2026

OEL-4643: Media Copyright Accessibility improvements #551

Merged

donquixote reviewed Apr 24, 2026

View reviewed changes

Aaron Gil Martinez added 8 commits June 2, 2026 12:08

OEL-3983: Add backstop with basic config and pattern scenario generator.

f6d0bf4

OEL-4248: Test CI on Github actions.

d389bc8

Trigger workflow

f4aee46

OEL-4248: Add documentation about workflow and allow manual run.

1e07320

OEL-4248: Install dependencies.

f0b6ef2

OEL-4248: Install site and upgrade node.

fcbd78b

OEL-4248: Remove patch for 11.3.

645e0e7

OEL-4248: Merge artifacts, remove unused variables.

5f8e190

AaronGilMartinez force-pushed the OEL-4248-backstop branch 2 times, most recently from bd9a536 to e296861 Compare June 3, 2026 07:58

AaronGilMartinez added 5 commits June 3, 2026 11:18

OEL-4248: Include .gitattributes.

e878b8d

OEL-4248: Add reference images to LFS.

0f87288

OEL-4248: Downgrade node version to align with web container.

1e73893

OEL-4248: Add flag for failed report generation.

01a66e6

OEL-4248: Support for LFS in actions.

8e5abfe

AaronGilMartinez force-pushed the OEL-4248-backstop branch from e957323 to 8e5abfe Compare June 3, 2026 09:21

AaronGilMartinez added 2 commits June 3, 2026 12:25

OEL-4248: Update readme documentation.

245f040

OEL-4248: Update reference images.

d10555b

donquixote reviewed Jul 8, 2026

View reviewed changes

Navigation Menu

Uh oh!

OEL-4248: VRT Backstop - #535

OEL-4248: VRT Backstop#535
AaronGilMartinez wants to merge 15 commits into
1.xfrom
OEL-4248-backstop

AaronGilMartinez commented Jan 16, 2026

Uh oh!

Uh oh!

donquixote Apr 22, 2026 •

edited

Loading

Uh oh!

donquixote Apr 22, 2026

Uh oh!

donquixote Apr 22, 2026

Uh oh!

donquixote Apr 22, 2026 •

edited

Loading

Uh oh!

donquixote Apr 22, 2026

Uh oh!

donquixote Apr 22, 2026

Uh oh!

donquixote commented Apr 22, 2026

Uh oh!

donquixote Apr 22, 2026

Uh oh!

donquixote Apr 22, 2026 •

edited

Loading

Uh oh!

donquixote Apr 22, 2026

Uh oh!

donquixote Apr 24, 2026

Uh oh!

donquixote Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		### Prerequisites

		- A running Drupal site reachable at `http://localhost:8080/build`. Override the target with the `VRT_BASE_URL` environment variable if needed.


		### Available commands

		- `docker compose exec web npm run vrt:build` – scrapes the UI Patterns catalog and generates `tests/vrt/scenarios/patterns.json` plus `tests/vrt/backstop.generated.json`.

Uh oh!

Conversation

AaronGilMartinez commented Jan 16, 2026

Uh oh!

Uh oh!

donquixote Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

donquixote Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

donquixote commented Apr 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

donquixote Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

donquixote Apr 22, 2026 •

edited

Loading

donquixote Apr 22, 2026 •

edited

Loading

donquixote Apr 22, 2026 •

edited

Loading