Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,108 changes: 27 additions & 2,081 deletions .circleci/config.yml

Large diffs are not rendered by default.

2,080 changes: 2,080 additions & 0 deletions .circleci/continue_config.yml

Large diffs are not rendered by default.

14 changes: 14 additions & 0 deletions .circleci/continue_config_light.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
version: 2.1

# Lightweight continuation for non-code PRs (CLI-1625).
orbs:
prodsec: snyk/prodsec-orb@1

workflows:
light_test_and_release:
jobs:
- prodsec/secrets-scan:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Please make this pipeline run the same checks as we do for docs-only branches here.

name: secrets-scan
context: snyk-bot-slack
channel: cli-alerts
trusted-branch: main
38 changes: 38 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,44 @@ Your PR checks will run every time you push changes to your branch.
The [test pipeline](https://app.circleci.com/pipelines/github/snyk/cli?filter=mine) is on CircleCI. This is where your
changes are built and tested.

A short **setup** job runs first. It compares your branch to the **PR base branch** (or `main` for non-PR pushes) and continues one of two configs:

| Path | When | `test_and_release` runs |
| --------------- | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| **Full** | Any changed file is **not** on the [light-path list](#what-triggers-the-light-pipeline) | Builds, unit tests, acceptance tests, and the rest of the existing pipeline |
| **Lightweight** | Every changed file is on the light-path list | `secrets-scan` only |

Mixed PRs (for example `.md` and `.ts`) always take the **full** path.

**Branches named `docs/*`** use the same full vs lightweight rules. Md-only `docs/*` PRs are lightweight
(`secrets-scan` only), not a separate pipeline.

`docs-only-check` (full config, `docs/*` only) allows `.md`, `.svg`, and `.jpg`—see
[`scripts/ci/is-docs-only-changes.sh`](scripts/ci/is-docs-only-changes.sh) and
[`scripts/ci/docs-only-pathspecs.txt`](scripts/ci/docs-only-pathspecs.txt). It runs only when the full
config is selected, not on typical md-only PRs. Repo help assets: `help/ide.svg`, `help/monitor.svg`;
`help/snyk-cli-screenshot.png` is not in that allowlist.

#### What triggers the light pipeline

The setup job uses a **fail-closed** allowlist: the **full** pipeline runs unless **every** changed file (vs merge-base
with the PR base branch or `main`, excluding `node_modules/`) matches a path in
[`scripts/ci/light-pipeline-pathspecs.txt`](scripts/ci/light-pipeline-pathspecs.txt). Unknown or unlisted file types
default to **full** CI.

Light-path examples:

- Docs / assets: `*.md`, `*.svg`, `*.jpg`
- GitHub metadata: `.github/**`, `CODEOWNERS`
- CI config that only affects the light pipeline: `.circleci/continue_config_light.yml`
- Other YAML outside `.circleci/`: `*.yml`, `*.yaml`

Everything else — source files, Makefiles, lockfiles, `.nvmrc`, `.circleci/**` (except `continue_config_light.yml`),
`Dockerfile`, new languages, and so on — triggers the **full** pipeline.

[`scripts/ci/has-code-changes.sh`](scripts/ci/has-code-changes.sh) implements this check. The setup job resolves the
base branch via [`scripts/ci/resolve-base-branch.sh`](scripts/ci/resolve-base-branch.sh).

If any checks fail, fix them and force push your changes again. Make sure to review and tidy up your branch so that it
remains easy to follow.

Expand Down
34 changes: 34 additions & 0 deletions scripts/ci/continue-pipeline.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# POST a continuation config to the CircleCI dynamic configuration API.
set -euo pipefail

CONFIG_PATH="${1:-}"
if [ -z "$CONFIG_PATH" ] && [ -r /tmp/continuation-config-path.txt ]; then
CONFIG_PATH="$(cat /tmp/continuation-config-path.txt)"
fi

if [ -z "$CONFIG_PATH" ] || [ ! -r "$CONFIG_PATH" ]; then
echo "continue-pipeline: missing or unreadable config path" >&2
exit 2
fi

if [ -z "${CIRCLE_CONTINUATION_KEY:-}" ]; then
echo "continue-pipeline: CIRCLE_CONTINUATION_KEY is required" >&2
exit 1
fi

mkdir -p /tmp/circleci
jq -Rs '.' "$CONFIG_PATH" > /tmp/circleci/config-string.json

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: Isn't there a circleci cli function to do all of this? all the manually crafted API calls seems like quite some unnecessary maintenance burden?

jq -n \
--arg continuation "$CIRCLE_CONTINUATION_KEY" \
--slurpfile config /tmp/circleci/config-string.json \
'{"continuation-key": $continuation, "configuration": $config|join("\n"), "parameters": {}}' \
> /tmp/circleci/continue_post.json
code="$(curl -sS -o /tmp/circleci/continue_response.json -w '%{http_code}' \
-X POST \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
--data @/tmp/circleci/continue_post.json \
"https://${CIRCLECI_DOMAIN:-circleci.com}/api/v2/pipeline/continue")"
cat /tmp/circleci/continue_response.json
[ "$code" = "200" ]
5 changes: 5 additions & 0 deletions scripts/ci/docs-only-pathspecs.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Git pathspecs allowed on docs/* branches (strict subset of light-pipeline-pathspecs.txt).
# Used by: scripts/ci/is-docs-only-changes.sh
*.md
*.svg
*.jpg
43 changes: 43 additions & 0 deletions scripts/ci/has-code-changes.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
#!/usr/bin/env bash
# Exit 0 if the full CI pipeline is required; exit 1 if light pipeline is enough.
# Fail-closed: unknown paths default to full (exit 0).
set -euo pipefail

SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
PATHSPECS_FILE="${LIGHT_PIPELINE_PATHSPECS_FILE:-${CODE_CHANGE_PATHSPECS_FILE:-$SCRIPT_DIR/light-pipeline-pathspecs.txt}}"
BASE_REF="${BASE_REF:-origin/main}"

if [ ! -r "$PATHSPECS_FILE" ]; then
echo "has-code-changes: missing pathspecs file: $PATHSPECS_FILE" >&2
exit 2
fi

LIGHT_PATHSPECS=()
while IFS= read -r line; do
LIGHT_PATHSPECS+=("$line")
done < <("$SCRIPT_DIR/load-pathspecs.sh" "$PATHSPECS_FILE")
if [ "${#LIGHT_PATHSPECS[@]}" -eq 0 ]; then
echo "has-code-changes: no pathspecs in $PATHSPECS_FILE" >&2
exit 2
fi

ALL_CHANGED="$("$SCRIPT_DIR/list-changed-files.sh" | grep -Ev '(^|/)node_modules/' || true)"
if [ -z "$ALL_CHANGED" ]; then
exit 1
fi

MERGE_BASE="$(git merge-base HEAD "$BASE_REF")"
LIGHT_CHANGED="$(git diff --name-only "${MERGE_BASE}...HEAD" -- "${LIGHT_PATHSPECS[@]}" || true)"
while IFS= read -r file; do
[ -z "$file" ] && continue
case "$file" in
.circleci/*) ;;
*.yml|*.yaml) LIGHT_CHANGED="${LIGHT_CHANGED}"$'\n'"${file}" ;;
esac
done <<< "$ALL_CHANGED"

NON_LIGHT="$(comm -23 <(printf '%s\n' "$ALL_CHANGED" | sort) <(printf '%s\n' "$LIGHT_CHANGED" | sort -u))"
if [ -n "$NON_LIGHT" ]; then
exit 0
fi
exit 1
34 changes: 34 additions & 0 deletions scripts/ci/is-docs-only-changes.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# Exit 0 if all changed files are docs assets (.md, .svg, .jpg); else exit 1.
set -euo pipefail

SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
PATHSPECS_FILE="${DOCS_ONLY_PATHSPECS_FILE:-$SCRIPT_DIR/docs-only-pathspecs.txt}"
BASE_REF="${BASE_REF:-origin/main}"

DOCS_PATHSPECS=()
while IFS= read -r line; do
DOCS_PATHSPECS+=("$line")
done < <("$SCRIPT_DIR/load-pathspecs.sh" "$PATHSPECS_FILE")
if [ "${#DOCS_PATHSPECS[@]}" -eq 0 ]; then
echo "is-docs-only-changes: no pathspecs in $PATHSPECS_FILE" >&2
exit 2
fi

ALL_CHANGED="$("$SCRIPT_DIR/list-changed-files.sh" | grep -Ev '(^|/)node_modules/' || true)"
if [ -z "$ALL_CHANGED" ]; then
exit 0
fi

MERGE_BASE="$(git merge-base HEAD "$BASE_REF")"
DOCS_CHANGED="$(git diff --name-only "${MERGE_BASE}...HEAD" -- "${DOCS_PATHSPECS[@]}" || true)"
NON_DOCS="$(comm -23 <(printf '%s\n' "$ALL_CHANGED" | sort) <(printf '%s\n' "$DOCS_CHANGED" | sort -u))"
if [ -n "$NON_DOCS" ]; then
while IFS= read -r file; do
[ -z "$file" ] && continue
echo "Error: disallowed file type for docs-only branch: $file" >&2
done <<< "$NON_DOCS"
exit 1
fi

exit 0
11 changes: 11 additions & 0 deletions scripts/ci/light-pipeline-pathspecs.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Git pathspecs safe for the lightweight CI pipeline (one per line).
# Used by: scripts/ci/has-code-changes.sh
# Fail-closed: anything not listed here triggers the full pipeline.
# Docs-only pathspecs (docs/* branch gate) are a strict subset — see docs-only-pathspecs.txt.
# YAML outside .circleci/ is handled in has-code-changes.sh (pathspec exclude cannot re-include files).
*.md
*.svg
*.jpg
.github/
CODEOWNERS
.circleci/continue_config_light.yml
13 changes: 13 additions & 0 deletions scripts/ci/list-changed-files.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
# List files changed on HEAD since merge-base with a base ref (default origin/main).
set -euo pipefail

BASE_REF="${1:-${BASE_REF:-origin/main}}"

if ! git rev-parse --verify "$BASE_REF" >/dev/null 2>&1; then
echo "list-changed-files: unknown base ref: $BASE_REF" >&2
exit 2
fi

MERGE_BASE="$(git merge-base HEAD "$BASE_REF")"
git diff --name-only "${MERGE_BASE}...HEAD"
Comment thread
robertolopezlopez marked this conversation as resolved.
11 changes: 11 additions & 0 deletions scripts/ci/load-pathspecs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
# Print non-comment pathspec lines from a file (one per line).
set -euo pipefail

file="${1:?pathspec file required}"
if [ ! -r "$file" ]; then
echo "load-pathspecs: missing pathspecs file: $file" >&2
exit 2
fi

grep -v '^[[:space:]]*#' "$file" | grep -v '^[[:space:]]*$' || true
34 changes: 34 additions & 0 deletions scripts/ci/resolve-base-branch.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/usr/bin/env bash
# Print the branch to diff against for CI path routing (PR base, else main).
set -euo pipefail

if [ -n "${CIRCLE_PULL_REQUEST_BASE_BRANCH:-}" ]; then
echo "$CIRCLE_PULL_REQUEST_BASE_BRANCH"
exit 0
fi

pr_number="${CIRCLE_PR_NUMBER:-}"
repo_path=""
if [ -n "${CIRCLE_PULL_REQUEST:-}" ]; then
pr_number="${pr_number:-${CIRCLE_PULL_REQUEST##*/}}"
repo_path="${CIRCLE_PULL_REQUEST#*github.com/}"
repo_path="${repo_path%/pull/*}"
fi
if [ -z "$repo_path" ] && [ -n "${CIRCLE_PROJECT_USERNAME:-}" ] && [ -n "${CIRCLE_PROJECT_REPONAME:-}" ]; then
repo_path="${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}"
fi

token="${GITHUB_TOKEN:-${GITHUB_PRIVATE_TOKEN:-${HAMMERHEAD_GITHUB_PAT:-}}}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issue: Why do we send tokens here?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: Why do we generally need to interact with the github API.

if [ -n "$token" ] && [ -n "$pr_number" ] && [ -n "$repo_path" ]; then
base_ref="$(
curl -fsSL -H "Authorization: Bearer ${token}" -H 'Accept: application/vnd.github+json' \
"https://api.github.com/repos/${repo_path}/pulls/${pr_number}" \
| python3 -c 'import json,sys; print(json.load(sys.stdin)["base"]["ref"])'
)"
if [ -n "$base_ref" ]; then
echo "$base_ref"
exit 0
fi
fi

echo main
81 changes: 81 additions & 0 deletions scripts/ci/test_has_code_changes.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
HAS_CODE="$SCRIPT_DIR/has-code-changes.sh"

# ponytail: temp repo exercises real git diff + pathspecs (no hand-rolled pattern matcher).
with_repo() {
local dir
dir=$(mktemp -d)
trap 'rm -rf "$dir"' RETURN

cp "$SCRIPT_DIR/has-code-changes.sh" "$SCRIPT_DIR/list-changed-files.sh" "$dir/"
chmod +x "$dir"/*.sh

cd "$dir"
git init -q
git config user.email 'ci-test@example.com'
git config user.name 'ci-test'
echo base >README.md
git add . && git commit -qm base

export BASE_REF=HEAD
export LIGHT_PIPELINE_PATHSPECS_FILE="$SCRIPT_DIR/light-pipeline-pathspecs.txt"
"$@"
}

commit_paths() {
for f in "$@"; do
mkdir -p "$(dirname -- "$f")"
touch "$f"
git add -- "$f"
done
git commit -qm "add: $*"
}

assert_light() {
commit_paths "$@"
if BASE_REF=HEAD~1 "$HAS_CODE"; then
echo "FAIL: expected light pipeline for: $*" >&2
exit 1
fi
echo "ok (light): $*"
git reset --hard HEAD~1 >/dev/null
}

assert_full() {
commit_paths "$@"
if ! BASE_REF=HEAD~1 "$HAS_CODE"; then
echo "FAIL: expected full pipeline for: $*" >&2
exit 1
fi
echo "ok (full): $*"
git reset --hard HEAD~1 >/dev/null
}

with_repo assert_light CONTRIBUTING.md .github/CODEOWNERS
with_repo assert_light README.md docs/guide.md
with_repo assert_light test/fixtures/pnpm-app/node_modules/.pnpm/foo/index.mjs
with_repo assert_light .circleci/continue_config_light.yml
with_repo assert_light .github/workflows/ci.yml

with_repo assert_full src/cli/main.ts
with_repo assert_full cliv2/pkg/foo/bar.go
with_repo assert_full scripts/install-dev-dependencies.sh
with_repo assert_full Makefile
with_repo assert_full cliv2/Makefile
with_repo assert_full cliv2/go.mod
with_repo assert_full cliv2/go.sum
with_repo assert_full package.json
with_repo assert_full packages/foo/package-lock.json
with_repo assert_full .circleci/config.yml
with_repo assert_full .circleci/continue_config.yml
with_repo assert_full .nvmrc
with_repo assert_full dangerfile.js
with_repo assert_full CONTRIBUTING.md src/foo.ts
with_repo assert_full Dockerfile
with_repo assert_full foo.py
with_repo assert_full cliv2-private/go.mod

echo "test_has_code_changes: all passed"
64 changes: 64 additions & 0 deletions scripts/ci/test_is_docs_only_changes.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#!/usr/bin/env bash
set -euo pipefail

SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
IS_DOCS="$SCRIPT_DIR/is-docs-only-changes.sh"

with_repo() {
local dir
dir=$(mktemp -d)
trap 'rm -rf "$dir"' RETURN

cp "$SCRIPT_DIR/is-docs-only-changes.sh" \
"$SCRIPT_DIR/list-changed-files.sh" \
"$SCRIPT_DIR/load-pathspecs.sh" \
"$dir/"
chmod +x "$dir"/*.sh

cd "$dir"
git init -q
git config user.email 'ci-test@example.com'
git config user.name 'ci-test'
echo base >README.md
git add . && git commit -qm base

export DOCS_ONLY_PATHSPECS_FILE="$SCRIPT_DIR/docs-only-pathspecs.txt"
"$@"
}

commit_paths() {
for f in "$@"; do
mkdir -p "$(dirname -- "$f")"
touch "$f"
git add -- "$f"
done
git commit -qm "add: $*"
}

assert_docs() {
commit_paths "$@"
if ! BASE_REF=HEAD~1 "$IS_DOCS"; then
echo "FAIL: expected docs-only for: $*" >&2
exit 1
fi
echo "ok (docs): $*"
git reset --hard HEAD~1 >/dev/null
}

assert_not_docs() {
commit_paths "$@"
if BASE_REF=HEAD~1 "$IS_DOCS"; then
echo "FAIL: expected non-docs failure for: $*" >&2
exit 1
fi
echo "ok (not docs): $*"
git reset --hard HEAD~1 >/dev/null
}

with_repo assert_docs README.md docs/guide.md
with_repo assert_docs assets/logo.svg
with_repo assert_not_docs README.md src/foo.ts
with_repo assert_not_docs CODEOWNERS
with_repo assert_not_docs .github/workflows/ci.yml

echo "test_is_docs_only_changes: all passed"
Loading