Skip to content

ci: fast CI #207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maxday merged 15 commits into from
Jun 18, 2026
Merged

ci: fast CI #207

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
2e6a4a4
ci: fast CI
maxday Jun 17, 2026
0abc519
fix: remove 3.9 runtime as deprecated
maxday Jun 17, 2026
0ac2256
fix: pr comments
maxday Jun 17, 2026
15e6911
fix: 3.14
maxday Jun 17, 2026
d380c4b
fix: put back comment
maxday Jun 17, 2026
816c214
fix: backoff delay if rate is exceeded
maxday Jun 17, 2026
0785e06
fix: try to remove codebuild
maxday Jun 17, 2026
2a9e4f2
fix: branch
maxday Jun 17, 2026
f07f335
fix: install curl for debian
maxday Jun 17, 2026
6a51f51
fix: pr comments
maxday Jun 18, 2026
f649f7b
fix: test
maxday Jun 18, 2026
35a7381
fix: tests
maxday Jun 18, 2026
e021617
Merge pull request #208 from aws/maxday/remove-codebuild
maxday Jun 18, 2026
f19187b
fix: remove mock
maxday Jun 18, 2026
dd73fd4
fix: tests
maxday Jun 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 70 additions & 10 deletions .github/workflows/test-on-push-and-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ on:
pull_request:
branches: [ '*' ]

permissions:
contents: read

jobs:
build:
runs-on: ubuntu-latest
Expand All @@ -17,32 +20,89 @@ jobs:

alpine:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
distro_version: ["3.19", "3.20"]
runtime_version: ["3.10", "3.11", "3.12", "3.13"]
include:
- distro_version: "3.21"
runtime_version: "3.14"

steps:
- uses: actions/checkout@v4
- name: Run alpine integration tests
run: DISTRO=alpine make test-integ
- name: Run integration test (alpine ${{ matrix.distro_version }} / python ${{ matrix.runtime_version }})
run: |
docker build -t codebuild-agent - < tests/integration/codebuild-local/Dockerfile.agent

@benrkia benrkia Jun 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is repeated across all jobs, having it in a common job would reduce time further

@maxday maxday Jun 18, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true, but this is going away with #208 so if that looks good to you let's merge both?

CODEBUILD_IMAGE_TAG=codebuild-agent tests/integration/codebuild-local/test_one.sh \
tests/integration/codebuild/buildspec.os.alpine.yml \
alpine "${{ matrix.distro_version }}" "${{ matrix.runtime_version }}"

amazonlinux:
debian:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
distro_version: ["bookworm", "bullseye"]
runtime_version: ["3.10", "3.11", "3.12", "3.13"]
include:
- distro_version: "bookworm"
runtime_version: "3.14"

steps:
- uses: actions/checkout@v4
- name: Run amazonlinux integration tests
run: DISTRO=amazonlinux make test-integ
- name: Run integration test (debian ${{ matrix.distro_version }} / python ${{ matrix.runtime_version }})
run: |
docker build -t codebuild-agent - < tests/integration/codebuild-local/Dockerfile.agent
CODEBUILD_IMAGE_TAG=codebuild-agent tests/integration/codebuild-local/test_one.sh \
tests/integration/codebuild/buildspec.os.debian.yml \
debian "${{ matrix.distro_version }}" "${{ matrix.runtime_version }}"

debian:
amazonlinux2:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
runtime_version: ["3.10", "3.11"]

steps:
- uses: actions/checkout@v4
- name: Run integration test (amazonlinux 2 / python ${{ matrix.runtime_version }})
run: |
docker build -t codebuild-agent - < tests/integration/codebuild-local/Dockerfile.agent
CODEBUILD_IMAGE_TAG=codebuild-agent tests/integration/codebuild-local/test_one.sh \
tests/integration/codebuild/buildspec.os.amazonlinux.2.yml \
amazonlinux2 "2" "${{ matrix.runtime_version }}"

amazonlinux2023:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
runtime_version: ["3.12", "3.13", "3.14"]

steps:
- uses: actions/checkout@v4
- name: Run debian integration tests
run: DISTRO=debian make test-integ
- name: Run integration test (amazonlinux 2023 / python ${{ matrix.runtime_version }})
run: |
docker build -t codebuild-agent - < tests/integration/codebuild-local/Dockerfile.agent
CODEBUILD_IMAGE_TAG=codebuild-agent tests/integration/codebuild-local/test_one.sh \
tests/integration/codebuild/buildspec.os.amazonlinux.2023.yml \
amazonlinux2023 "2023" "${{ matrix.runtime_version }}"

ubuntu:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
distro_version: ["22.04", "24.04"]
runtime_version: ["3.10", "3.11", "3.12", "3.13", "3.14"]

steps:
- uses: actions/checkout@v4
- name: Run ubuntu integration tests
run: DISTRO=ubuntu make test-integ
- name: Run integration test (ubuntu ${{ matrix.distro_version }} / python ${{ matrix.runtime_version }})
run: |
docker build -t codebuild-agent - < tests/integration/codebuild-local/Dockerfile.agent
CODEBUILD_IMAGE_TAG=codebuild-agent tests/integration/codebuild-local/test_one.sh \
tests/integration/codebuild/buildspec.os.ubuntu.yml \
ubuntu "${{ matrix.distro_version }}" "${{ matrix.runtime_version }}"
19 changes: 19 additions & 0 deletions tests/integration/codebuild-local/test_one.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,22 @@ function usage {
>&2 echo " env Additional environment variables file."
}

function pull_with_retry() {
local image="$1"
local max_retries=3
local wait=10
for attempt in $(seq 1 $max_retries); do
if docker pull "$image"; then
return 0
fi
>&2 echo "Docker pull attempt $attempt/$max_retries failed. Retrying in ${wait}s..."
sleep $wait
wait=$((wait * 2))
done
>&2 echo "Failed to pull $image after $max_retries attempts."
return 1
}

main() {
if (( $# != 3 && $# != 4)); then
>&2 echo "Invalid number of parameters."
Expand Down Expand Up @@ -49,6 +65,9 @@ main() {
ARTIFACTS_DIR="$CODEBUILD_TEMP_DIR/artifacts"
mkdir -p "$ARTIFACTS_DIR"

# Pre-pull the CodeBuild local agent image with retries to handle ECR rate limits.
pull_with_retry "public.ecr.aws/codebuild/local-builds:latest"

# Run CodeBuild local agent.
"$(dirname "$0")"/codebuild_build.sh \
-i "$CODEBUILD_IMAGE_TAG" \
Expand Down
30 changes: 19 additions & 11 deletions tests/test_concurrency.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Copyright 2025 Amazon.com, Inc. or its affiliates. All Rights Reserved.
"""

import multiprocessing
import threading
import unittest
from unittest.mock import patch, MagicMock

Expand All @@ -18,33 +18,41 @@ def setUp(self):
self.socket = "/tmp/sock"

def test_success_and_failure_isolation(self):

@benrkia benrkia Jun 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this test needs to be updated to reflect what are actually checking, using fork explicitly hides what's actually being used, so in the context of python 3.14 we force the test to use fork while we know the default now is to use forkserver. I'd use multiprocessing.Manager to track the shared data and this way we won't need to force usage of fork

@maxday maxday Jun 18, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, will change this

success_counter = multiprocessing.Value("i", 0)
fail_counter = multiprocessing.Value("i", 0)
success_counter = 0
fail_counter = 0
process_index = 0
lock = threading.Lock()

def fake_bootstrap_run(handler, lambda_runtime_client):
pid = multiprocessing.current_process().pid
if pid % 2 == 0:
nonlocal success_counter, fail_counter, process_index
with lock:
idx = process_index
process_index += 1
if idx % 2 == 0:
for _ in range(3):
with success_counter.get_lock():
success_counter.value += 1
with lock:
success_counter += 1
else:
with fail_counter.get_lock():
fail_counter.value += 1
with lock:
fail_counter += 1
raise RuntimeError("Simulated failure")

with patch(
"awslambdaric.lambda_multi_concurrent_utils.MultiConcurrentRunner._redirect_output"
), patch(
"awslambdaric.lambda_multi_concurrent_utils.bootstrap.run",
side_effect=fake_bootstrap_run,
), patch(
"awslambdaric.lambda_multi_concurrent_utils.multiprocessing.Process",
threading.Thread,

@benrkia benrkia Jun 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What i meant is that we should mock this at all, and rely on multiprocessing manager to track shared data, so here we're testing something different

@maxday maxday Jun 18, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I've changed back to no mock, but I still needed to add set_start_method as the default behaviour changed in python 3.14 (see: https://docs.python.org/3.14/whatsnew/3.14.html#concurrent-futures + python/cpython#84559)

):
# spawn 4 multi-concurrent processes
MultiConcurrentRunner.run_concurrent(
self.handler, self.addr, self.use_thread, self.socket, max_concurrency=4
)

self.assertEqual(success_counter.value, 6)
self.assertEqual(fail_counter.value, 2)
self.assertEqual(success_counter, 6)
self.assertEqual(fail_counter, 2)


if __name__ == "__main__":
Expand Down
Loading