Quickstart

jobwatch tells you when scheduled work fails, misses its slot or runs slow. Kubernetes CronJobs and Jobs are found automatically: you add no ping lines to your jobs.

• Part 1 (about 5 minutes): the standalone agent in your cluster, alerting straight to Telegram or Slack. No server.
• Part 2: the jobwatch server with docker compose, which adds history, the web app, server-side checks and ping URLs.
• Part 3: connect the agent to the server.
• Part 4: report from jobs outside Kubernetes (API keys, jobwatch-report).

Before you install anything, verify what you are about to run (docs/verify.md). Everything below uses the image digests you verified.

You need:

• a Kubernetes cluster (k3s, k8s 1.27+) with kubectl and helm 3.8+;
• for part 2, Docker with the compose plugin;
• cosign to verify images (optional, but that’s the point).

Set these for the commands below:

OWNER=<github owner>                         # where the images are published: ghcr.io/$OWNER/...
VERSION=<agent version>                      # e.g. 0.4.0 (GitHub release agent-v0.4.0)

Part 1: the agent with Helm (Telegram or Slack)

1. Pick where alerts go

Telegram

1. Talk to @BotFather, run /newbot and keep the token (123456:ABC…).
2. Add the bot to your group and send the group a message.
3. Read the group’s chat id: open https://api.telegram.org/bot<token>/getUpdates and find "chat":{"id":-100…}.

Slack

1. Go to https://api.slack.com/apps, then Create New App → From scratch.
2. Under OAuth & Permissions → Bot Token Scopes, add chat:write and chat:write.public. The second one lets the bot post without being invited.
3. Click Install to Workspace and keep the Bot User OAuth Token (xoxb-…).
4. Get the channel id: open the channel’s details; it’s at the bottom (C0123ABC).

The Acknowledge/Snooze/Resolve buttons need the server (part 3). The standalone agent sends plain alerts.

2. Verify and install

CHART_DIGEST=$(crane digest ghcr.io/$OWNER/charts/jobwatch-agent:$VERSION)
cosign verify "ghcr.io/$OWNER/charts/jobwatch-agent@$CHART_DIGEST" \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity "https://github.com/$OWNER/jobwatch/.github/workflows/release-agent.yml@refs/tags/agent-v$VERSION"

The released chart pins the agent image by the digest the release signed (image.digest). Verify that image too (verify.md §1).

Create the namespace and a Secret with the token. Only the line for your platform is needed:

kubectl create namespace jobwatch
kubectl -n jobwatch create secret generic jobwatch-agent-secrets \
  --from-literal=JOBWATCH_TELEGRAM_TOKEN='123456:ABC…'
#  or: --from-literal=JOBWATCH_SLACK_TOKEN='xoxb-…'

Write values.yaml:

config:
  cluster: homelab                 # a name for this cluster; it shows in every alert
  channels:
    - name: ops
      type: telegram               # or: slack
      telegram: {chatId: "-1001234567890"}
      # slack: {channel: C0123ABC}
  routes:
    default: [ops]
secrets:
  existingSecret: jobwatch-agent-secrets

Then install:

helm install jobwatch-agent "oci://ghcr.io/$OWNER/charts/jobwatch-agent@$CHART_DIGEST" -n jobwatch -f values.yaml
kubectl -n jobwatch rollout status deploy/jobwatch-agent
kubectl -n jobwatch logs deploy/jobwatch-agent | head

The logs show starting jobwatch-agent (cluster, namespaces, channels), then initial sync processed with the number of jobs found, and informers synced. The agent’s RBAC is read-only: get, list and watch on CronJobs, Jobs and Pods, and get on pod logs.

3. See an alert

kubectl create namespace jobwatch-demo
kubectl apply -n jobwatch-demo -f - <<'YAML'
apiVersion: batch/v1
kind: CronJob
metadata:
  name: fails
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      backoffLimit: 0                  # fail the Job on the first error (the default retries for minutes)
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: app
              image: busybox:1.36
              command: ["sh", "-c", "echo working; echo PASSWORD=hunter2; exit 3"]
YAML

At the next full minute the Job runs and fails. A few seconds later you get FAILED · fails, with exit code 3, the last log lines and the image. Secrets in the log lines are masked before anything leaves the cluster. Each further failure edits the same message (Slack: chat.update), and the occurrence count goes up.

Don’t test with kubectl create cronjob … -- sh -c 'exit 3'. That gives restartPolicy: OnFailure and backoffLimit: 6, so Kubernetes restarts the container with back-off for several minutes before it marks the Job failed. The agent alerts when the Job fails, not while it retries.

Clean up with kubectl delete namespace jobwatch-demo. Deleting the CronJob makes the agent stop watching it and close the open incident: the message is edited to RESOLVED, with a threaded reply.

Useful annotations on a CronJob:

• jobwatch.dev/grace: 10m (missed-run grace);
• jobwatch.dev/max-duration: 30m (SLOW);
• jobwatch.dev/ignore: "true";
• jobwatch.dev/route: <tag> (routing).

All options are in agent/README.md and deploy/helm/jobwatch-agent/values.yaml.

From source (no registry, e.g. air-gapped): build the static binary and an image on a node, then install the chart from the repository:

make -C agent build-all                                   # bin/jobwatch-agent-linux-{amd64,arm64}
mkdir -p /tmp/jw-img && cp agent/bin/jobwatch-agent-linux-amd64 /tmp/jw-img/jobwatch-agent
printf 'FROM scratch\nCOPY jobwatch-agent /jobwatch-agent\nUSER 65532:65532\nENTRYPOINT ["/jobwatch-agent"]\n' > /tmp/jw-img/Dockerfile
docker build -t jobwatch-agent:local /tmp/jw-img           # k3s with the docker runtime sees it directly;
                                                           # containerd: docker save | k3s ctr images import -
helm install jobwatch-agent deploy/helm/jobwatch-agent -n jobwatch -f values.yaml \
  --set image.repository=jobwatch-agent --set image.tag=local --set image.pullPolicy=Never

Part 2: the server with docker compose

The server stores history, runs server-side detection (missed runs even when an agent dies, silent agents), serves the web app and ping URLs for jobs outside Kubernetes, and handles the Telegram/Slack buttons. It’s one image with Postgres 16 next to it.

git clone https://github.com/$OWNER/jobwatch && cd jobwatch/deploy/compose     # the whole repo: the server image is built from it
cp .env.example .env

Fill in .env:

• JOBWATCH_IMAGE (optional): leave it empty to build the server from this checkout (docker compose up -d --build, needs npm and NuGet access). To run the released image instead, set it to ghcr.io/$OWNER/jobwatch-server@sha256:…, the digest you verified with verify.md §4, then run docker compose pull server.
• The secrets, generated with the commands in the file:
  • POSTGRES_PASSWORD, JOBWATCH_API_DB_PASSWORD;
  • JOBWATCH_ACTION_KEY;
  • JOBWATCH_AGENT_TOKEN (jwa_…) and JOBWATCH_API_KEY (jwk_…).

docker compose up -d
docker compose logs -f server          # wait for "Now listening on: http://0.0.0.0:8080" (JSON log lines)
curl -s localhost:8080/readyz          # → ready

Try the API (the API key acts as Owner of the bootstrap org):

API=jwk_…                                   # JOBWATCH_API_KEY from .env
curl -s localhost:8080/v1/agents -H "Authorization: Bearer $API"
curl -s -X POST localhost:8080/v1/checks -H "Authorization: Bearer $API" -H 'Content-Type: application/json' \
  -d '{"name":"nightly-backup","schedule":"0 3 * * *","graceSeconds":1800}' | jq .pingUrls

The ping URLs contain a random 22-character ping key, which is the credential. The check id is not: /ping/<id> answers 404. Add curl -fsS <success URL> at the end of a host cron job (/start and /fail exist too). POST /v1/checks/{id}/rotate-ping-key replaces a leaked key.

Contact points (where the server sends alerts) are added in the web app under Notifications, or through the API:

# Slack with buttons: also add a Signing Secret (Slack app → Basic Information) and set the app's
# Interactivity Request URL to <PublicUrl>/slack/interactions/slack-ops
curl -s -X POST localhost:8080/v1/channels -H "Authorization: Bearer $API" -H 'Content-Type: application/json' -d '{
  "name": "slack-ops", "type": "slack",
  "config": {"channel": "C0123ABC", "actions": true},
  "secrets": {"botToken": "xoxb-…", "signingSecret": "…"}}'
curl -s -X PUT localhost:8080/v1/routes -H "Authorization: Bearer $API" -H 'Content-Type: application/json' \
  -d '{"rules": [], "default": ["slack-ops"]}'
curl -s -X POST localhost:8080/v1/channels/slack-ops/test -H "Authorization: Bearer $API"

Other types: telegram, discord (secret webhookUrl), email, webhook, apprise. The fields are in proto/api-ui.md and proto/phase4.md “Decisions (D)”.

Sign-in for the web app

The web app signs in with OIDC. Use your own provider: create a confidential web client with redirect URI <PublicUrl>/auth/callback and set JOBWATCH_OIDC_*. Or use the bundled Zitadel:

# in .env: ZITADEL_MASTERKEY (exactly 32 characters: openssl rand -hex 16), ZITADEL_DB_PASSWORD,
#          JOBWATCH_OIDC_AUTHORITY=http://auth.localhost:8090, JOBWATCH_OIDC_ALLOW_HTTP=true
# ZITADEL_DB_PASSWORD must be set before the *first* "up": init-db.sh creates Zitadel's database then.
chmod a+w zitadel                       # Zitadel (non-root) writes its setup token there
docker compose --profile auth up -d
# the OIDC app (redirect http://localhost:8080/auth/callback), printed into ./oidc.env:
ZITADEL_URL=http://auth.localhost:8090 ZITADEL_PAT_FILE=zitadel/admin.pat JOBWATCH_ENV_FILE=./oidc.env \
  ../../server/tools/zitadel-dev-setup.sh
cat oidc.env >> .env && docker compose up -d server

Open http://localhost:8080 and sign in as admin@jobwatch.local / Change-me-1! (you set a new password at first sign-in). Put that email in JOBWATCH_OWNER_EMAIL so this user becomes the org’s Owner. zitadel-dev-setup.sh also creates a demo user casino-ops; delete it in Zitadel if you don’t want it.

Production notes:

• serve both through HTTPS (JOBWATCH_PUBLIC_URL=https://…; cookies become Secure);
• keep .env and the database backups private: they hold the channel tokens (SECURITY.md);
• turn on end-to-end encrypted logs in Settings → Encryption.

Part 3: connect the agent

Add the server to the agent’s values and upgrade:

config:
  server:
    url: https://jobwatch.example.com        # how the cluster reaches the server (Part 2's JOBWATCH_PUBLIC_URL)
    publicUrl: https://jobwatch.example.com  # links in Slack/Discord alerts
    # allowPlainHTTP: true                   # only for an http:// URL on a trusted network
secrets:
  existingSecret: jobwatch-agent-secrets     # add JOBWATCH_SERVER_TOKEN=<JOBWATCH_AGENT_TOKEN> to it

kubectl -n jobwatch patch secret jobwatch-agent-secrets --type merge -p '{"stringData":{"JOBWATCH_SERVER_TOKEN":"jwa_…"}}'
helm upgrade jobwatch-agent "oci://ghcr.io/$OWNER/charts/jobwatch-agent@$CHART_DIGEST" -n jobwatch -f values.yaml
kubectl -n jobwatch rollout restart deploy/jobwatch-agent
curl -s localhost:8080/v1/agents -H "Authorization: Bearer $API" | jq '.[] | {cluster, name, lastSeenAt}'

The agent logs registered with server. It registers its own Ed25519 key with the token and signs every message from then on. Its jobs appear in the web app (Jobs) within seconds. If the agent stops reporting for 3 minutes, the server opens agent_silent.

To get Slack buttons on the agent’s own alerts:

1. Set actions: true on its Slack channel.
2. Add a server contact point of type slack with the same name, holding the signing secret.
3. Point the Slack app’s Interactivity URL at <PublicUrl>/slack/interactions/<name>.

People link their Slack account automatically when Slack’s verified email matches their jobwatch email, or with /jobwatch link <code> (profile menu → Link Slack). For the slash command, create /jobwatch in the Slack app with the same Request URL.

Part 4: jobs outside Kubernetes

Anything that is not a CronJob reports itself: Windows tasks, macOS launchd jobs, CI schedules, serverless functions, probes. In the web app go to Settings → API keys → New key, pick the role Reporter, limit it to a cluster name of your choice (for example ci) and copy the secret once. Then wrap the job:

export JOBWATCH_URL=http://localhost:8080          # your server
export JOBWATCH_API_KEY=jwk_...                    # the Reporter key
jobwatch-report run --check nightly-export --cluster ci --every 1d --grace 30m -- ./export.sh

The first run creates the check (Jobs → source report), failures alert within seconds, a missed run at due time plus grace, an over-long run when it exceeds --max-duration. Hosts you can install on can also run jobwatch-host (crontab, systemd timers, Proxmox vzdump), and a plain curl ping URL still works for the simplest cases. All options, curl, PowerShell, GitHub Actions and Lambda examples: docs/report-api.md.

Troubleshooting