[Arch Linux] When building with a BaseTree that includes ALPM hooks, they will not be run

[Smujb]

mkosi commit the issue has been seen with

847d113

Used host distribution

Arch Linux

Used target distribution

Arch Linux

Linux kernel version used

Linux 7.2.4-arch1-2

CPU architectures issue was seen on

x86_64

Unexpected behaviour you saw

Rootful mkosi build with the /usr tree taken from a live sysupdate system (and package db copied across) can install further packages mostly correctly. However, pacman hooks that are provided by packages present on the base system are not run during the build, breaking packages like nautilus when some relevant gtk packages are already present. Specifically, hooks in /usr/share/libalpm/hooks which are provided by packages, as opposed to user-created hooks in /etc/pacman.d/hooks.

Additional context to the mkosi config: mkosi.base/var/lib/pacman contains a copy of the package db generated from the image and mkosi.base/usr has a full sysupdate /usr tree mounted using mount /dev/mapper/usr mkosi.base/usr -o ro

Used mkosi config

[Content]
BaseTrees=mkosi.base

[Output]
OutputDirectory=mkosi.output
ImageId=overlay-ext
Format=sysext
Overlay=yes
Output=%i_%v

The full contents of this mkosi config are here: https://github.com/Smujb/overlay-ext-skeleton

mkosi output

‣ Running configure script /tmp/overlay-ext-skeleton/mkosi.configure…
‣ Validating certificates and keys
‣ Syncing package manager metadata: no incremental images, no caches to preserve
‣ Populating pacman keyring
gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
gpg: no ultimately trusted keys found
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
==> Generating pacman master key. This may take some time.
gpg: Generating pacman keyring master key...
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/87BF6A3224B362AB630A9344D30476877D9D2E13.rev'
gpg: Done
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
==> Appending keys from archlinux.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signed 5 keys.
==> Importing owner trust values...
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: inserting ownertrust of 4
gpg: setting ownertrust to 4
==> Disabling revoked keys in keyring...
  -> Disabled 38 keys.
==> Updating trust database...
gpg: Note: third-party key signatures using the SHA1 algorithm are rejected
gpg: (use option "--allow-weak-key-signatures" to override)
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   5  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   5  signed:  89  trust: 0-, 0q, 0n, 5m, 0f, 0u
gpg: depth: 2  valid:  78  signed:  18  trust: 78-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2026-10-26
‣ Syncing package manager metadata
:: Synchronizing package databases...
 core                                                              127.5 KiB   911 KiB/s 00:00 [########################################################] 100%
 extra                                                               8.5 MiB  16.7 MiB/s 00:01 [########################################################] 100%
‣ Populating pacman keyring
==> Appending keys from archlinux.gpg...
==> Updating trust database...
gpg: next trustdb check due at 2026-10-26
‣ Building main image
‣ [main]  Mounting base trees…
‣ [main]   Populating pacman keyring
==> Appending keys from archlinux.gpg...
==> Updating trust database...
gpg: next trustdb check due at 2026-10-26
‣ [main]   Installing extra packages for Arch Linux
resolving dependencies...
looking for conflicting packages...

Packages (67) exempi-2.6.6-3  exiv2-0.28.9-1  ffmpeg-2:9.0.1-4  gexiv2-0.16.2-2  gexiv2-common-0.16.2-2  glslang-1:1.4.357.0-1  gnome-autoar-0.5.2-1
              gnome-desktop-4-1:44.5-1  gnome-desktop-common-1:44.5-1  gpgmepp-2.2.0-1  gsm-1.0.24-1  gupnp-dlna-0.12.0-5  l-smash-2.14.5-4  libao-1.2.2-7
              libass-0.17.5-1  libavc1394-0.5.4-7  libbs2b-3.1.0-10  libcue-2.3.0-2  libdovi-3.4.0-1  libdvdnav-7.0.0-1  libgsf-1.14.58-1  libgxps-0.3.2-5
              libiec61883-1.2.0-9  libiptcdata-1.0.5-5  libmodplug-0.8.9.0-7  libopenmpt-0.8.9-1  libosinfo-1.12.0-3  libplacebo-7.360.1-6
              libportal-0.11.0-1  libportal-gtk4-0.11.0-1  librashader-0.12.0-1  libraw1394-2.1.2-4  libretro-shaders-slang-20260620.024025.g8ec4c82cf61a-1
              libsoxr-0.1.3-5  libssh-0.12.2-1  libtheora-1.2.0-1  libunibreak-7.0-1  libvdpau-1.5-4  libvpl-2.17.0-1  libvpx-1.17.0-1  libzip-1.11.4-1
              localsearch-3.11.2-1  ocl-icd-2.3.5-1  openal-1.25.2-2  opencore-amr-0.1.6-2  osinfo-db-20260812-1  poppler-26.08.0-1  poppler-glib-26.08.0-1
              portaudio-1:19.7.0-4  rubberband-4.0.0-2  shaderc-2026.3-1  snappy-1.2.2-3  speex-1.2.1-2  speexdsp-1.2.1-2  srt-1.5.7-1
              totem-pl-parser-3.26.7-1  uchardet-0.0.8-4  v4l-utils-1.32.0-2  vapoursynth-79-1  vid.stab-1.1.2-1  x264-3:0.165.r3222.b35605a-2  x265-4.3-1
              xdg-user-dirs-gtk-0.16-1  xvidcore-1.3.7-4  zimg-3.0.6-2  ares-emu-148-2  nautilus-50.3.1-1

Total Installed Size:  283.14 MiB

:: Proceed with installation? [Y/n]
(67/67) checking keys in keyring                                                               [########################################################] 100%
(67/67) checking package integrity                                                             [########################################################] 100%
(67/67) loading package files                                                                  [########################################################] 100%
(67/67) checking for file conflicts                                                            [########################################################] 100%
:: Processing package changes...
( 1/67) installing libao                                                                       [########################################################] 100%
( 2/67) installing librashader                                                                 [########################################################] 100%
( 3/67) installing libretro-shaders-slang                                                      [########################################################] 100%
( 4/67) installing openal                                                                      [########################################################] 100%
Optional dependencies for openal
    qt6-base: alsoft-config GUI Configurator [installed]
    fluidsynth: MIDI rendering
    libmysofa: makemhr tool [installed]
( 5/67) installing ares-emu                                                                    [########################################################] 100%
( 6/67) installing exiv2                                                                       [########################################################] 100%
( 7/67) installing gexiv2-common                                                               [########################################################] 100%
( 8/67) installing gexiv2                                                                      [########################################################] 100%
( 9/67) installing gnome-autoar                                                                [########################################################] 100%
(10/67) installing gnome-desktop-common                                                        [########################################################] 100%
(11/67) installing gnome-desktop-4                                                             [########################################################] 100%
(12/67) installing libportal                                                                   [########################################################] 100%
Optional dependencies for libportal
    libportal-docs: API documentation
    libportal-gtk3: GTK 3 backend
    libportal-gtk4: GTK 4 backend [pending]
    libportal-qt5: Qt 5 backend
    libportal-qt6: Qt 6 backend
    xdg-desktop-portal: Portal service [installed]
(13/67) installing libportal-gtk4                                                              [########################################################] 100%
(14/67) installing exempi                                                                      [########################################################] 100%
(15/67) installing gsm                                                                         [########################################################] 100%
(16/67) installing libunibreak                                                                 [########################################################] 100%
(17/67) installing libass                                                                      [########################################################] 100%
(18/67) installing libraw1394                                                                  [########################################################] 100%
(19/67) installing libavc1394                                                                  [########################################################] 100%
(20/67) installing libbs2b                                                                     [########################################################] 100%
(21/67) installing libdvdnav                                                                   [########################################################] 100%
(22/67) installing libiec61883                                                                 [########################################################] 100%
(23/67) installing libmodplug                                                                  [########################################################] 100%
(24/67) installing portaudio                                                                   [########################################################] 100%
(25/67) installing libopenmpt                                                                  [########################################################] 100%
(26/67) installing glslang                                                                     [########################################################] 100%
(27/67) installing shaderc                                                                     [########################################################] 100%
(28/67) installing libdovi                                                                     [########################################################] 100%
(29/67) installing libplacebo                                                                  [########################################################] 100%
(30/67) installing libsoxr                                                                     [########################################################] 100%
(31/67) installing libssh                                                                      [########################################################] 100%
(32/67) installing libtheora                                                                   [########################################################] 100%
(33/67) installing libvdpau                                                                    [########################################################] 100%
Optional dependencies for libvdpau
    libvdpau-va-gl: driver using VAAPI
    nvidia-utils: driver for NVIDIA
(34/67) installing libvpx                                                                      [########################################################] 100%
(35/67) installing ocl-icd                                                                     [########################################################] 100%
Optional dependencies for ocl-icd
    opencl-driver: packaged opencl driver
(36/67) installing libvpl                                                                      [########################################################] 100%
Optional dependencies for libvpl
    intel-media-sdk: runtime implementation for legacy Intel GPUs
    vpl-gpu-rt: runtime implementation for Tiger Lake and newer GPUs
(37/67) installing opencore-amr                                                                [########################################################] 100%
(38/67) installing rubberband                                                                  [########################################################] 100%
(39/67) installing snappy                                                                      [########################################################] 100%
(40/67) installing speexdsp                                                                    [########################################################] 100%
(41/67) installing speex                                                                       [########################################################] 100%
(42/67) installing srt                                                                         [########################################################] 100%
(43/67) installing v4l-utils                                                                   [########################################################] 100%
Optional dependencies for v4l-utils
    qt6-base: for qv4l2 and qvidcap [installed]
    qt6-5compat: for qv4l2
    alsa-lib: for qv4l2 [installed]
(44/67) installing zimg                                                                        [########################################################] 100%
(45/67) installing vapoursynth                                                                 [########################################################] 100%
(46/67) installing vid.stab                                                                    [########################################################] 100%
(47/67) installing l-smash                                                                     [########################################################] 100%
(48/67) installing x264                                                                        [########################################################] 100%
(49/67) installing x265                                                                        [########################################################] 100%
(50/67) installing xvidcore                                                                    [########################################################] 100%
(51/67) installing ffmpeg                                                                      [########################################################] 100%
Optional dependencies for ffmpeg
    avisynthplus: AviSynthPlus support
    frei0r-plugins: Frei0r video effects support
    intel-media-sdk: Intel QuickSync support (legacy)
    ladspa: LADSPA filters
    nvidia-utils: Nvidia NVDEC/NVENC support
    onevpl-intel-gpu: Intel QuickSync support
(52/67) installing gupnp-dlna                                                                  [########################################################] 100%
(53/67) installing libcue                                                                      [########################################################] 100%
(54/67) installing libgsf                                                                      [########################################################] 100%
(55/67) installing libgxps                                                                     [########################################################] 100%
(56/67) installing libiptcdata                                                                 [########################################################] 100%
Optional dependencies for libiptcdata
    python: Python bindings [installed]
(57/67) installing osinfo-db                                                                   [########################################################] 100%
(58/67) installing libosinfo                                                                   [########################################################] 100%
(59/67) installing libzip                                                                      [########################################################] 100%
(60/67) installing gpgmepp                                                                     [########################################################] 100%
(61/67) installing poppler                                                                     [########################################################] 100%
Optional dependencies for poppler
    poppler-data: highly recommended encoding data to display PDF documents with certain encodings and characters
(62/67) installing poppler-glib                                                                [########################################################] 100%
(63/67) installing uchardet                                                                    [########################################################] 100%
(64/67) installing totem-pl-parser                                                             [########################################################] 100%
(65/67) installing localsearch                                                                 [########################################################] 100%
(66/67) installing xdg-user-dirs-gtk                                                           [########################################################] 100%
(67/67) installing nautilus                                                                    [########################################################] 100%
‣ [main]  Removing empty directories…
‣ [main]  Running finalize script /tmp/overlay-ext-skeleton/mkosi.finalize…
allexport       off
braceexpand     on
emacs           off
errexit         off
errtrace        off
functrace       off
hashall         on
histexpand      off
history         off
ignoreeof       off
interactive-comments    on
keyword         off
monitor         off
noclobber       off
noexec          off
noglob          off
nolog           off
notify          off
nounset         off
onecmd          off
physical        off
pipefail        off
posix           off
privileged      off
verbose         off
vi              off
xtrace          off
+ cat
++ cat base_image.id
‣ [main]  Building sysext extension image
Pre-populating erofs filesystem of partition 10-root.conf to calculate minimal partition size
Preparing to populate erofs filesystem.
Failed to open source file '/buildroot/opt', skipping: No such file or directory
Ready to populate erofs filesystem.
Formatting /var/tmp/.#repartf23c0a1b87cb82fd as erofs
mkfs.erofs 1.9.4
Build completed.
------
Filesystem UUID: 7a2b449c-4e4c-4393-a55c-50eb781b28d2
Filesystem total blocks: 72986 (of 4096-byte blocks)
Filesystem total inodes: 10997
Filesystem total metadata blocks: 4429
Filesystem total deduplicated bytes (of source files): 0
/var/tmp/.#repartf23c0a1b87cb82fd successfully formatted as erofs (uuid 7a2b449c-4e4c-4393-a55c-50eb781b28d2, no label)
Minimal partition size of erofs filesystem of partition 10-root.conf is 285.1M
Automatically determined minimal disk image size as 286.1M.
Sized '/work/var/tmp/mkosi-workspace-9ag7lyo5/staging/overlay-ext_20260914.0.raw' to 286.1M.
Applying changes to /work/var/tmp/mkosi-workspace-9ag7lyo5/staging/overlay-ext_20260914.0.raw.
Copying in '/var/tmp/.#repartf23c0a1b87cb82fd' (285.1M) on block level into future partition 0.
Copying in of '/var/tmp/.#repartf23c0a1b87cb82fd' on block level completed.
Syncing future partition 0 contents to disk.
Block level copying and synchronization of partition 0 complete in 70.011ms.
Adding new partition 0 to partition table.
Writing new partition table.
Partition table written.
‣ [main] Could not rename /var/tmp/mkosi-workspace-9ag7lyo5/staging/overlay-ext_20260914.0.raw to /tmp/overlay-ext-skeleton/mkosi.output/overlay-ext_20260914.0.raw as they are located on different devices, falling back to copying
‣ [main]  /tmp/overlay-ext-skeleton/mkosi.output/overlay-ext_20260914.0.raw size is 286.1M, consumes 285.1M.

[Smujb]

Taking a glance at the code, the issue seems to be that since the /usr tree isn't mounted at buildroot, nothing it provides can affect the transactions. While performing these transactions in a clean chroot is obviously the whole point, I wonder if an option could be provided when building from BaseTrees specifically? If BaseTrees are present during transactions solely as a read-only lowerdir then they will not be modified said transactions but can affect them.

I am aware that this is probably a pretty unorthodox usecase for mkosi. The main reason I am here and not simply making my own thing is that I think it makes more sense to work with upstream here, rather than building something completely from scratch. OCI tooling allows for this kind of thing by building FROM a base image and bootable OCI containers have projects like bluebuild which are designed specifically to make it straightforward to create derivative images. While obviously feature parity with OCI is not a priority or necessarily even desirable, I do think that this particular usecase has merit.

In fitting everything together, there is a section about "Developer Mode" where building and signing sysexts locally would be a possibility. When the package DB is a snapshot of some kind (e.g. the Arch Linux Archive), this kind of use of application of mkosi allows for exactly that - with a functional package manager to boot. The transactional nature of updates and secure/signed nature of such a system need not be compromised. And, when it is not, the next best thing in my opinion is building a second locally signed /usr tree which is a derivative of the one pulled from the remote. Both are faster and more efficient solutions than simply forking or locally checking out the build infrastructure and running everything from scratch.

[behrmann]

mkosi runs the package manager from outside of the image. I've not looked whether we hook up alpm hooks, but generally, package manager config for the build needs to go into a sandbox tree.

[Smujb]

Yep, that would make sense. I already tested adding /usr as a sandbox tree here but there are a few issues:

1. It is read-only (erofs, mounted), and mkosi expects it to be writeable. I would like for this to be better handled but it is hardly the end of the world - I just mount it with an upperdir which is simply discarded after the build.
2. The sandbox must copy the entire /usr tree rather than simply mounting it, which takes up quite a bit of space in /tmp and is far less efficient than simply mounting it itself
3. ... the post-install scripts still don't run? somehow? No error mind you.

[Smujb]

For what it's worth, here is the bash script I am using to automate this process alongside the repo I linked earlier:

#!/usr/bin/env bash

if [[ $EUID -ne 0 ]]; then
  echo "This script must be run as root"
  exit 1
fi


set -e

# Setup basic working directory
cd /tmp
git clone https://github.com/Smujb/overlay-ext-skeleton.git
cd overlay-ext-skeleton

if [[ ! -d "/etc/overay-ext/mkosi" ]]; then
  echo "No overlay configured."
  exit 1
fi

# Add base files needed
mkdir -p mkosi.base/{usr,var}
mkdir -p mkosi.sandbox/usr
mkdir -p workdirs/{upper,work}
# Booted partition
mount /dev/mapper/usr mkosi.base/usr -o ro
mount -t overlay overlay -o lowerdir=mkosi.base/usr,upperdir=workdirs/upper,workdir=workdirs/work mkosi.sandbox/usr
# mount /dev/nvme0n1p4 mkosi.base/usr -o ro
cp -R mkosi.base/usr/lib/sysimage/* mkosi.base/var

ln -s /etc/overay-ext/mkosi /tmp/overlay-ext-skeleton/mkosi.local

if command -v mkosi; then
  mkosi build
else
  /opt/mkosi/bin/mkosi build --debug-shell
fi

cp mkosi.output/*.raw /var/lib/extensions/
systemd-sysext refresh

umount mkosi.sandbox/usr
umount mkosi.base/usr
rm -rf /tmp/overlay-ext-skeleton

It is messy and bad but gives an idea of what I am trying to accomplish hopefully - after I have this working to a satisfactory degree I will rewrite it in a less hacky way, probably in Python.

[Smujb]

With /usr mounted in SandboxTrees and --debug-shell and mkosi patched to manually add /usr/share/libalpm/hooks as a hook dir:

--- a/mkosi/installer/pacman.py
+++ b/mkosi/installer/pacman.py
@@ -164,6 +164,7 @@ class Pacman(PackageManager):
             "--cachedir=/var/cache/pacman/mkosi",
             "--cachedir=/var/cache/pacman/pkg",
             "--hookdir=/buildroot/etc/pacman.d/hooks",
+            "--hookdir=/buildroot/usr/share/libalpm/hooks",
             "--arch", cls.architecture(context),
             "--color", "auto",
             "--noconfirm",

‣ Syncing package manager metadata
error: 'failed to resolve path '/buildroot/usr/share/libalpm/hooks' passed to '--hookdir': No such file or directory
‣ "pacman --root=/buildroot --logfile=/dev/null --dbpath=/var/lib/pacman --cachedir=/var/cache/pacman/mkosi --cachedir=/var/cache/pacman/pkg --hookdir=/buildroot/etc/pacman.d/hooks --hookdir=/buildroot/usr/share/libalpm/hooks --arch x86_64 --color auto --noconfirm --sync --refresh" returned non-zero exit code 1.
‣ (sandbox command omitted, rerun with --debug to see the full sandbox command.)
bash-5.3# ls
bin  buildroot  dev  etc  home  lib  lib64  media  mnt  opt  proc  proxy.cacert  repository  run  sandbox.py  sbin  srv  tmp  usr  var
bash-5.3# ls /usr/share/libalpm/hooks
10-glibc-locale-gen.hook                      25-systemd-hwdb.hook                  70-openssh-restart-sshd.hook     gio-remove-module-cache.hook
10-glibc-remove-locale-archive.hook           25-systemd-sysctl.hook                90-dracut-install.hook           glib-compile-schemas.hook
10-openssh-mark-sshd-for-restart.hook         30-systemd-daemon-reload-system.hook  90-update-appstream-cache.hook   glib-remove-compiled-schemas.hook
11-glibc-ldconfig.hook                        30-systemd-daemon-reload-user.hook    accounts-daemon-restart.hook     gtk-query-immodules-3.0.hook
11-glibc-remove-ldconfig-cache.hook           30-update-mime-database.hook          dbus-reload.hook                 gtk-update-icon-cache.hook
12-glibc-iconvconfig.hook                     35-systemd-enqueue-marked.hook        dconf-update.hook                gtk4-querymodules.hook
12-glibc-remove-iconvconfig-cache.hook        35-systemd-udev-reload.hook           detect-old-perl-modules.hook     gvfsd.hook
12-lib32-glibc-iconvconfig.hook               35-systemd-update.hook                fontconfig-32.hook               man-db-remove-cache.hook
12-lib32-glibc-remove-iconvconfig-cache.hook  36-systemd-modules-load.hook          fontconfig.hook                  shared-mime-info-remove-cache.hook
20-systemd-sysusers.hook                      40-fontconfig-config.hook             gdk-pixbuf-query-loaders.hook    update-desktop-database.hook
21-systemd-tmpfiles.hook                      40-update-ca-trust.hook               gio-querymodules-32.hook         vimdoc.hook
25-systemd-binfmt.hook                        60-depmod.hook                        gio-querymodules.hook
25-systemd-catalog.hook                       60-dracut-remove.hook                 gio-remove-module-cache-32.hook
bash-5.3# ls buildroot
etc  var
bash-5.3#

Earlier in the build log I see

‣ Copying in sandbox trees…
‣ Populating pacman keyring

But the second "Copying in sandbox trees" appears to be after the sync? As I don't see it at this stage.

That appears to be the actual issue here - pacman cannot find the libalpm hookdir at runtime and so it assumes that there are no existing hooks during the transaction.

[Smujb]

@linux-universe ran into this issue with AstrOS. It seems postinstall scripts are generally not run at all if they are present only on the base image, but a sub-image requires them.

This is a much broader problem which occurs even with much more normal system configurations and potentially not even just on Arch Linux.