Verify a release

Every release artifact is signed by the GitHub Actions workflow that built it, with cosign keyless signing:

• the workflow’s GitHub OIDC token gets a short-lived certificate from Sigstore’s Fulcio;
• the signature is recorded in the public Rekor transparency log.

There is no long-lived key to leak or rotate. You verify who built it: the workflow file in this repository, at a release tag.

You need cosign v2.4 or newer (the workflows sign with v3), plus docker buildx or crane to inspect images. Set these once:

OWNER=bobo-22lvd            # the GitHub owner the images are published under (lowercase)
REPO=jobwatch               # the repository name
ISSUER=https://token.actions.githubusercontent.com
VERSION=0.4.0

1. Agent image

Always verify, and then deploy, a digest, not a tag: a tag can be moved and a digest can’t.

DIGEST=$(docker buildx imagetools inspect ghcr.io/$OWNER/jobwatch-agent:$VERSION --format '{{json .Manifest}}' | jq -r .digest)

cosign verify "ghcr.io/$OWNER/jobwatch-agent@$DIGEST" \
  --certificate-oidc-issuer "$ISSUER" \
  --certificate-identity "https://github.com/$OWNER/$REPO/.github/workflows/release-agent.yml@refs/tags/agent-v$VERSION"

The command prints the verified claims, including the commit SHA (githubWorkflowSha) and the Rekor entry. Exit code 0 means verified. To accept any release tag instead of one version, use --certificate-identity-regexp "^https://github.com/$OWNER/$REPO/\.github/workflows/release-agent\.yml@refs/tags/agent-v".

Deploy that digest with the chart: --set image.digest=$DIGEST. Released charts already pin the digest they were released with.

2. Helm chart (OCI)

CHART_DIGEST=$(crane digest ghcr.io/$OWNER/charts/jobwatch-agent:$VERSION)   # or: helm pull … and read "Digest:"

cosign verify "ghcr.io/$OWNER/charts/jobwatch-agent@$CHART_DIGEST" \
  --certificate-oidc-issuer "$ISSUER" \
  --certificate-identity "https://github.com/$OWNER/$REPO/.github/workflows/release-agent.yml@refs/tags/agent-v$VERSION"

# install exactly what you verified
helm install jobwatch-agent "oci://ghcr.io/$OWNER/charts/jobwatch-agent@$CHART_DIGEST" -n jobwatch --create-namespace -f values.yaml

3. Host agent and binaries (verify-blob)

Download the file, its .sigstore.json bundle and SHA256SUMS (plus its bundle) from the GitHub release:

F=jobwatch-host-linux-amd64          # or jobwatch-host_0.4.0_amd64.deb, jobwatch-agent-linux-arm64, jobwatch-report-windows-amd64.exe, ...
BASE=https://github.com/$OWNER/$REPO/releases/download/agent-v$VERSION
curl -fsSLO "$BASE/$F" -O "$BASE/$F.sigstore.json" -O "$BASE/SHA256SUMS" -O "$BASE/SHA256SUMS.sigstore.json"

cosign verify-blob "$F" --bundle "$F.sigstore.json" \
  --certificate-oidc-issuer "$ISSUER" \
  --certificate-identity "https://github.com/$OWNER/$REPO/.github/workflows/release-agent.yml@refs/tags/agent-v$VERSION"

# checksums: verify the list's signature, then the file against it
cosign verify-blob SHA256SUMS --bundle SHA256SUMS.sigstore.json \
  --certificate-oidc-issuer "$ISSUER" \
  --certificate-identity "https://github.com/$OWNER/$REPO/.github/workflows/release-agent.yml@refs/tags/agent-v$VERSION"
sha256sum --ignore-missing -c SHA256SUMS

cosign verify-blob prints Verified OK. Each Go binary also ships a *.buildinfo.txt (go version -m: Go version, module versions, build flags).

4. Server image

DIGEST=$(docker buildx imagetools inspect ghcr.io/$OWNER/jobwatch-server:$VERSION --format '{{json .Manifest}}' | jq -r .digest)

cosign verify "ghcr.io/$OWNER/jobwatch-server@$DIGEST" \
  --certificate-oidc-issuer "$ISSUER" \
  --certificate-identity "https://github.com/$OWNER/$REPO/.github/workflows/release-server.yml@refs/tags/server-v$VERSION"

In docker-compose.yml (or Kubernetes), reference the image as ghcr.io/$OWNER/jobwatch-server@$DIGEST. The web app is served from this same image, so verifying the server image also verifies the JavaScript your browser runs to decrypt logs (see SECURITY.md, “Trusting the web app”).

5. SBOMs and provenance

BuildKit attaches an SPDX SBOM and SLSA provenance (mode=max) to every image, per platform:

# SPDX SBOM (packages in the image)
docker buildx imagetools inspect "ghcr.io/$OWNER/jobwatch-agent@$DIGEST" --format '{{ json (index .SBOM "linux/amd64").SPDX }}' \
  | jq -r '.packages[] | "\(.name) \(.versionInfo // "")"' | sort | head -50

# SLSA provenance: source repo, commit, workflow, build arguments
docker buildx imagetools inspect "ghcr.io/$OWNER/jobwatch-agent@$DIGEST" --format '{{ json (index .Provenance "linux/amd64").SLSA }}' \
  | jq '{builder: .runDetails.builder.id, source: .buildDefinition.externalParameters.configSource,
         args: .buildDefinition.externalParameters.request.args}'

The release also carries syft SBOMs as files: jobwatch-agent-image.spdx.json (the image), jobwatch-agent-source.spdx.json (the Go module) and jobwatch-server-image.spdx.json, each with a .sigstore.json bundle. Verify a file with cosign verify-blob as in section 3, then inspect it or scan it:

jq -r '.packages[] | select(.name | test("k8s.io|golang.org")) | "\(.name) \(.versionInfo)"' jobwatch-agent-source.spdx.json
grype sbom:jobwatch-server-image.spdx.json      # known vulnerabilities, if you use grype

Reproducing a build

The release workflows set SOURCE_DATE_EPOCH to the release commit time. Go binaries are built with -trimpath (the release binaries also embed the commit through Go’s VCS stamping). Images build with -buildvcs=false and an empty build id, and the .NET build uses ContinuousIntegrationBuild/Deterministic. Image layer timestamps are rewritten (rewrite-timestamp=true). To compare a release binary, rebuild it from the tag with the Go version named in its .buildinfo.txt:

git checkout agent-v$VERSION
make -C agent VERSION=$VERSION build-all build-host-all     # needs a clean checkout of the tag
(cd agent/bin && sha256sum jobwatch-agent-linux-* jobwatch-host-linux-* jobwatch-report-*)    # compare with SHA256SUMS

Images are reproducible only up to their base images. The Dockerfiles take the base images as build args (GO_IMAGE, BASE_IMAGE, NODE_IMAGE, SDK_IMAGE, RUNTIME_IMAGE). The defaults are tags, because the digests could not be resolved from the sandbox this was written in. Pin them by digest when you rebuild, and record the digests in the release notes:

docker buildx imagetools inspect gcr.io/distroless/static:nonroot --format '{{json .Manifest}}' | jq -r .digest
docker buildx build --build-arg BASE_IMAGE=gcr.io/distroless/static:nonroot@sha256:<digest> …