- Shell 78.8%
- Dockerfile 21.2%
wg-quick runs under sudo but reads the conf path as root; current runner images deny that traversal on /home/runner/work/_temp and every deploy's finalize job died at 'Permission denied' (line 40). Install the peer config at the canonical /etc/wireguard/wg0.conf and bring the interface up by name. |
||
|---|---|---|
| .agents/skills | ||
| .claude/skills | ||
| .cursor/skills/cicd-onboard | ||
| .github | ||
| docker | ||
| examples | ||
| tools | ||
| README.md | ||
What the pipeline delivers
- Publishes a Docker image to
ghcr.io/herodotusdev/<app>-<env>:<version>when the version changes (skips the build if the tag already exists — version bumps drive deploys). - Images are multi-arch manifest lists (
linux/amd64+linux/arm64) by default — the home cluster schedules pods onto both amd64 (ms) and arm64 (RPi) nodes, and the kubelet pulls whichever arch matches the node. See "Multi-arch builds" below. - Tags the repo with that version.
- Opens a WireGuard tunnel to the home k3s cluster and applies Kubernetes manifests for each app as the
daloc-deployeridentity. - Syncs configuration from etcd (over the tunnel, via port-forward) and configmap dirs before deployment.
- Runs init jobs (apps with
init: true) ahead of the main rollout. - If configmap or secret changed, but version has not, deployment will be restarted to load changes.
- Waits for deployment rollout.
Branch → environment mapping
| Branch | namespace | image |
|---|---|---|
main |
<namespace>-prod (e.g. daloc-prod) |
ghcr.io/herodotusdev/<app>-prod |
develop |
<namespace>-stg |
ghcr.io/herodotusdev/<app>-stg |
Other branches are blocked. Environments are à la carte: wire only the branches you use; create the target namespace (as the tenant owner) before the first deploy.
Staging does not host its own etcd: on develop the deploy namespace is <namespace>-stg, but etcd is still read/forwarded from <namespace>-prod (keys are namespaced /stg/..., so prod and stg configs never collide). Apps that must not exist in staging set deploy_envs: [prod] in cicd-inputs.yaml (see the cicd-inputs reference).
Required secrets (org-level defaults, per-repo overrides)
| Secret | Content |
|---|---|
WG_CONFIG |
wg-quick config of a daloc-deployer WireGuard peer (tunnel to the cluster proxy only) |
DEPLOYER_KUBECONFIG |
kubeconfig for the capsule-users/daloc-deployer ServiceAccount, pointing at the capsule proxy |
Both are org secrets shared by consumer repos. WireGuard peers are one session per key — a second handshake with the same key kills the first tunnel. Hence the convention:
- Org secret
WG_CONFIGholds the default peer (used by the cicd smoke workflow). - Repos that deploy get their own peer + a repo-level
WG_CONFIGsecret (repo secrets shadow org secrets):daloc-deployer-monorepo,daloc-deployer-frontend-new,daloc-deployer-k8s, … Create the OPNsense peer, put its config in the repo secret. With per-repo peers, two repos deploying at the same time is harmless — different keys, independent tunnels. - Inside one repo, there is exactly one tunnel per job that needs it:
prepopens it only when an app pulls build-time env from etcd (the payload is then handed to the build jobs), andfinalizeopens it once for the whole deploy — all apps deploy in parallel through that single tunnel, so a repo can't clash with itself.
The pipeline logs in to ghcr with the caller's GITHUB_TOKEN — consumer workflows need packages: write. etcd credentials are read at deploy time from the in-namespace daloc-etcd secret; no etcd secret lives in GitHub.
Quick setup (consumer repo)
- Copy
examples/workflows/build-and-deploy-workflow.yamlinto.github/workflows/. Addpackages: writetopermissions. Add[ci skip]to a commit message to avoid triggering cicd. - Copy
examples/k8sinto your repo root. Edit manifests underk8s/<env>/to match your services (ingress TLS uses Traefikcertresolver=default— no cert-manager on the home cluster). - Ensure a Dockerfile exists for every app (default path
docker/Dockerfile.<app>unless you overridedockerfile). - Fill out
k8s/cicd-inputs.yaml. - Commit the version source files referenced in
version_file(for exampleCargo.toml,package.json). Bumping the version triggers the deploy. - Ensure org secrets
WG_CONFIG+DEPLOYER_KUBECONFIGexist and cover your repo. Push tomain(prod) ordevelop(stg).
AI-assisted setup (cicd-onboard skill)
This repo ships a cicd-onboard skill (.cursor/skills/cicd-onboard/) that lets an AI coding agent (Claude Code, Cursor, etc.) wire CI/CD into an app repo for you. Install it with the bundled script:
# global: symlink into ~/.claude + ~/.agents (default; tracks this checkout)
tools/install-skill.sh
# into a specific app repo's .claude/.agents/.cursor
tools/install-skill.sh --repo /path/to/app-repo
# copy instead of symlink (portable, but goes stale on cicd updates)
tools/install-skill.sh --copy
tools/install-skill.sh --repo /path/to/app-repo --copy
tools/install-skill.sh --help # full usage
Once installed, ask the agent to onboard the repo and it follows the skill.
Pipeline shape
Three jobs, four runner spawns total:
prep— branch guard, inputs, version resolution, app matrix, and the etcd build-time env pull (only when an app declaresetcd_build_env).build— one job per arch (ubuntu-latest/ nativeubuntu-24.04-arm); a prep script resolves apps (tag-exists skips, per-app build-env contexts) into a bake file, then onedocker buildx bakeinvocation (viadocker/bake-action) builds them all: BuildKit dedups the shared compile vertices (apps share the same Dockerfile DAG — per-appAPP_NAMEonly affects the runtime stage), so the workspace compile runs once and the thin runtime stages fan out concurrently.finalize— merges per-arch tags into manifest lists, creates the version tag, opens the cluster tunnel once, then deploys init pods + all apps in parallel (bounded bydeploy_concurrency, default 5) sharing that single tunnel + etcd port-forward.
Multi-arch builds
Images build per arch in parallel jobs — linux/amd64 on ubuntu-latest, linux/arm64 on the native ubuntu-24.04-arm runner — and push per-arch tags <image>:<version>-<arch>. finalize then assembles <image>:<version> + :latest as a manifest list, which is what the deploys apply; the kubelet resolves the platform per node, so Kubernetes never needs arch hints.
Caching: per-arch buildx GHA cache with scope resolution app.cache_scope → repo-level cache_scope (cicd-inputs top-level) → app name, always suffixed -{arch}. Builds must go through docker/bake-action (or docker/build-push-action): the gha backend needs the runner's cache-service token, which plain run: steps no longer receive — a raw docker buildx build --cache-to type=gha silently skips cache entirely. Repos with many apps sharing one Dockerfile should set the repo-level cache_scope — all apps then share one cache bucket per arch instead of duplicating the shared compile layers into a bucket per app (the GHA cache is 10 GB per repo, LRU-evicted after 7 days of no access). Dependency layers (cargo-chef cook, bun install) are cached per arch; a cold build of the daloc Rust workspace compiles in ~15 min per arch, warm builds faster. Per-app platforms in cicd-inputs excludes an arch from both build and merge.
Dockerfile requirements: base images (FROM ...) must be multi-arch manifests (all official rust/debian/node/bun images are) and must not hardcode x86_64 paths or tools. .github/workflows/multiarch-smoke.yaml exercises the per-arch build + merge path on every push touching the build actions.
Etcd configuration
Upload the configuration file to etcd before the workflow runs. The workflow reads /etcd_root/k8s_env/app_name/.env and writes it into <app_name>-secret.
etcd lives in the prod namespace (<namespace>-prod/daloc-etcd); CI reaches it over the WireGuard tunnel via kubectl port-forward (the cluster-access action sets that up, using etcd_namespace — on develop the deploy namespace is <namespace>-stg but etcd is still forwarded from <namespace>-prod, with keys namespaced /stg/...). The secret is mounted into the container and configuration is accessible as system envs.
ConfigMaps
If you place files in k8s/<env>/<app>-configmap/, they will be pulled into a config map object that can be mounted into the container as files.
Tips
- When deploying a new app — the namespace must exist first (created by the tenant owner through the capsule proxy).
- When deploying a new ingress — a Cloudflare DNS record (proxied) has to point the host at the home WAN.
- Pipeline can be triggered manually via GitHub UI — not only by a commit (main/develop only).
- Share a Dockerfile by assigning the same
dockerfilevalue to multiple entries inapp_names. - Each app can use a shared version source or a dedicated one, to be set in cicd_inputs.
- If the pipeline failed, check the logs before messaging ppl :)
- Mark init-only apps with
init: true; they will be built and deployed before standard apps. Those apps need to run to completion, else they will block the pipeline — so they have to be a job or pod, not a deployment. - Deploys need the home internet up (the tunnel is in the path); failures are retryable and apply is idempotent.