Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
.git
.github
.agents
e2e
.codex
runtime-cli
102 changes: 102 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
name: CI

on:
pull_request:
push:
branches: [master]

permissions:
contents: read

concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true

jobs:
unit:
name: Unit tests (${{ matrix.os }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-24.04, macos-latest, windows-latest]
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true

- name: Build
run: go build ./...

- name: Test
run: go test -race -count=1 ./...

# Several service and capabilities tests skip themselves without root:
# the direct executor integration test, the namespace probes that need
# CAP_SYS_ADMIN, and the cgroup delegation checks. The unprivileged pass
# above covers everything else, so this pass is what actually exercises
# the privileged Linux paths.
- name: Test as root
if: runner.os == 'Linux'
run: go test -exec "sudo -n" -count=1 ./...

vet:
name: Vet and formatting
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true

- name: gofmt
run: |
unformatted="$(gofmt -l .)"
if [ -n "$unformatted" ]; then
echo "these files are not gofmt-formatted:" >&2
echo "$unformatted" >&2
exit 1
fi

- name: go vet
run: go vet ./...

- name: go mod tidy is up to date
run: |
go mod tidy
git diff --exit-code -- go.mod go.sum

crossbuild:
name: Build ${{ matrix.goos }}/${{ matrix.goarch }}
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
include:
- {goos: linux, goarch: amd64}
- {goos: linux, goarch: arm64}
- {goos: darwin, goarch: arm64}
- {goos: windows, goarch: amd64}
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true

# The runtime has three platform implementations behind build tags
# (linux, windows, and the unsupported fallback). Only a cross build
# keeps the two non-primary ones compiling.
- name: Build and vet
env:
GOOS: ${{ matrix.goos }}
GOARCH: ${{ matrix.goarch }}
run: |
go build ./...
go vet ./...
73 changes: 73 additions & 0 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: E2E

on:
pull_request:
push:
branches: [master]
workflow_dispatch:

permissions:
contents: read

concurrency:
group: e2e-${{ github.ref }}
cancel-in-progress: true

jobs:
# The runtime image is identical for every case, and building it once keeps
# the matrix legs to the part that actually differs: what the host grants the
# container.
image:
name: Build runtime image
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4

- name: Build
run: docker build -t mysterium-runtime:e2e .

- name: Export
run: docker save mysterium-runtime:e2e | gzip > runtime-image.tar.gz

- uses: actions/upload-artifact@v4
with:
name: runtime-image
path: runtime-image.tar.gz
retention-days: 1

isolation:
name: ${{ matrix.case }}
needs: image
# Pinned: the selected profile depends on the runner's kernel, cgroup
# layout, and AppArmor policy, so the image is part of the expectation.
runs-on: ubuntu-24.04
strategy:
fail-fast: false
matrix:
case:
- full
- limited-no-cgroups
- limited-no-netns
- unisolated
steps:
- uses: actions/checkout@v4

- uses: actions/download-artifact@v4
with:
name: runtime-image

- name: Load runtime image
run: docker load < runtime-image.tar.gz

- name: Report host facts
run: |
uname -a
stat -fc %T /sys/fs/cgroup
cat /sys/fs/cgroup/cgroup.controllers
sysctl kernel.apparmor_restrict_unprivileged_userns || true
docker version --format '{{.Server.Version}}'

- name: Run ${{ matrix.case }}
env:
E2E_REUSE_IMAGE: "1"
run: e2e/run.sh ${{ matrix.case }}
59 changes: 56 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,15 +243,22 @@ Build the runtime image:
docker build -t mysterium-runtime:dev .
```

Run the end-to-end demo on a native Linux Docker host. Docker Desktop is not a
supported host for this nested runtime because its VM blocks the subordinate
user namespace from mounting the workload procfs. The cgroup namespace and cgroup
Run the end-to-end demo on a Linux Docker host. The cgroup namespace and cgroup
filesystem options let the demo wrapper delegate `cpu`, `memory`, and `pids`
from the container cgroup to sibling workload cgroups managed by nested `runc`.
The capability list provides the operational privileges used by the full
profile; the outer Docker seccomp profile is disabled so `runc` can install the
stricter workload seccomp profile from the generated OCI spec.

Two of the options exist only so that the subordinate user namespace can be
established, and without them the demo reaches `limited` rather than `full`:

- `--cap-add=SETFCAP`: Linux 5.12 and later require `CAP_SETFCAP` to write a
`uid_map` that maps uid 0, which both the capability probe and `runc` do.
- `--security-opt systempaths=unconfined`: Docker's masked and read-only
`/proc` paths are locked over-mounts, and a nested user namespace may not
mount a fresh procfs while they hide part of the procfs it can already see.

```bash
docker run --rm --name mysterium-runtime-demo \
--cgroupns=host \
Expand All @@ -263,13 +270,15 @@ docker run --rm --name mysterium-runtime-demo \
--cap-add=KILL \
--cap-add=SETGID \
--cap-add=SETUID \
--cap-add=SETFCAP \
--cap-add=NET_ADMIN \
--cap-add=SYS_CHROOT \
--cap-add=SYS_PTRACE \
--cap-add=SYS_ADMIN \
--cap-add=MKNOD \
--security-opt seccomp=unconfined \
--security-opt apparmor=unconfined \
--security-opt systempaths=unconfined \
-e OCI_ARTIFACT="$OCI_ARTIFACT" \
-p 127.0.0.1:3000:3000 \
--entrypoint /usr/local/bin/runtime-demo \
Expand All @@ -293,6 +302,50 @@ only on loopback inside its isolated network namespace. It looks up the
manifest-defined service port and bridges a local listener through
`Backend.DialTCP`; it does not add a port override to the workload contract.

### Continuous integration

[`ci.yml`](.github/workflows/ci.yml) runs the unit tests on every pull request:
`go test -race` on Linux, macOS, and Windows, a second Linux pass under `sudo`
for the tests that skip themselves without root, `gofmt`, `go vet`, `go mod
tidy`, and a cross build of every platform implementation behind a build tag.

[`e2e.yml`](.github/workflows/e2e.yml) runs the demo workload end to end under
several isolation profiles. Each case is the same image and the same workload;
only what the host grants the runtime container changes, so the profile the
runtime selects is the thing under test:

| Case | What the host withholds | Expected |
| --- | --- | --- |
| `full` | nothing | `full-v1` |
| `limited-no-cgroups` | the delegated cgroup v2 tree | `best-effort-v1` without cgroups |
| `limited-no-netns` | `CAP_NET_ADMIN` and `CAP_SETFCAP` | `best-effort-v1` on a shared network namespace |
| `unisolated` | `runc` | `unisolated-v1` through the direct executor |

`limited-no-netns` withholds `CAP_SETFCAP`, and so user namespaces, along with
`CAP_NET_ADMIN`. That pairing is forced: the generated OCI spec always mounts
sysfs, and the kernel refuses a sysfs mount inside a user namespace that does
not own its network namespace, so a profile holding user namespaces without a
private network namespace cannot start a workload at all.

Each case asserts the reported runtime level, profile name, and the feature
vector the case is defined by; that the workload answers over the proxy; and
that stopping it leaves the service passive while its desired state survives.

The workload is served from a throwaway local registry, because the runtime
only accepts digest-pinned registry references. Docker pushes to it over
localhost, and the runtime pulls the same manifest through the docker bridge
gateway, which `go-containerregistry` treats as insecure because the address is
RFC1918.

A single case can be run on any Linux Docker host:

```bash
e2e/run.sh limited-no-cgroups
```

All four cases also pass against Docker Desktop, which needs
`E2E_ALLOW_NON_LINUX=1` to waive the platform guard.

---

## 3. Capability Detection Strategy
Expand Down
Loading
Loading