Docs/Hearth/Troubleshooting

Troubleshooting

hearth setup is the one-stop diagnostic. It re-runs every prerequisite check, regenerates an age key if missing, verifies your TLS certs, and confirms you can decrypt at least one secret.

shell
hearth setup

Most install issues resolve with hearth setup plus opening a fresh shell. If it surfaces something specific, the rest of this page covers the common cases.

Install issues

command not found: hearth

The shell rc block adds ~/.katforge/bin to PATH, but only takes effect in new shells.

shell
source ~/.bashrc       # or ~/.zshrc
# or just open a new terminal

If hearth still isn't found, check the rc block exists:

shell
grep "BEGIN katforge" ~/.bashrc ~/.zshrc 2>/dev/null

If missing, re-run ~/.katforge/hearth/install.sh — it's idempotent.

A binary is missing or stale

shell
hearth install --force         # reinstall everything
hearth install --only sops     # reinstall just one

hearth install is idempotent and only refetches when versions change.

Secrets

"no key could decrypt the data"

Your age key isn't on the recipient list for that file. Either:

  • Your public key isn't in .sops.yaml — ask a maintainer to run hearth team add age1... for you.
  • The file was encrypted before you were added — pull the latest, then retry.
  • SOPS_AGE_KEY_FILE isn't set or doesn't point at your age.txt — open a new shell, or echo $SOPS_AGE_KEY_FILE.
shell
hearth team whoami         # confirm your public key
hearth team list           # who can decrypt

"cannot decrypt" after a teammate's commit

shell
git -C ~/.katforge/hearth pull
hearth secret list dev

When somebody runs hearth team add or hearth team remove, every encrypted file is rewritten. Pull before retrying.

Lost age.txt

Your private key is gone — every secret it could decrypt is now uncovered to whoever has the file. Treat as a security incident:

  1. Ask a maintainer to hearth team remove age1<your-old-public-key>.
  2. Generate a new key by re-running ~/.katforge/hearth/install.sh (it regenerates if the file is missing).
  3. Run hearth team whoami and share the new key.
  4. Rotate any high-impact secrets the lost key could decrypt — see Secrets → Key rotation.

Compose stack

"required" env var error on hearth up

text
ERROR: APP_SECRET is required

A secret is missing from hearth/env/<stage>/api.enc.yaml (or its plaintext sibling). Either:

  • The key was added but never set in this stage — hearth secret edit <stage> api.
  • The compose file requires it but the env file dropped it — check hearth secret diff <stage> prod to see what's different.
  • You don't have permission to decrypt — see secrets above.

"network katforge not found"

shell
docker network create katforge

Or just re-run hearth up — it creates the network if missing.

TLS warnings in the browser

hearth setup should have imported the local CA. If browsers still complain:

  • Linux: check ~/.local/share/ca-certificates/ and update-ca-certificates.
  • WSL/Windows: the cert needs to be trusted on the Windows host, not WSL. Re-run hearth setup from inside WSL — it edits the Windows trust store.
  • Firefox: uses its own trust store. Import ~/.katforge/hearth/certs/rootCA.pem manually under Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import.

Apple OAuth fails locally

Apple requires HTTPS on :443 with no port suffix. The default :4443 setup won't work. See Networking → Apple OAuth caveat.

Mercure SSE replay returns stale events

Reset the bolt history volume:

shell
docker compose -f ~/.katforge/hearth/compose.yaml stop mercure
docker volume rm katforge-hearth_mercure_data
hearth up

Subscribers reconnect fresh.

Deploy

hearth ship fails on kubectl set image

Most often:

  • ~/.kube/katforge-config is missing or stale — hearth setup regenerates it.
  • The k3s cluster is unreachable — check kubectl --kubeconfig ~/.kube/katforge-config get nodes.
  • The image push to ghcr.io failed — check GITHUB_KATFORGE_TOKEN is set and has write:packages.

Rollout stuck on Progressing

The new pod is running but failing health checks. Tail logs to see why:

shell
hearth logs api

Common causes:

  • A required env var was added to the manifest but not to hearth/env/<stage>/.
  • A migration is failing — check the cron pod or run hearth run doctrine:migrations:status.
  • The image is missing a dependency from a recent package update.

You can force-revert at any time:

shell
hearth rollback api

"pull access denied" on ghcr.io

The cluster's registry-creds secret has the wrong token, or the token expired. Regenerate at https://github.com/settings/tokens with read:packages, then update the secret. (Talk to a maintainer if you don't have admin.)

Status / health

hearth status shows nothing

Wrong stage. The default is dev, which has no deployments — pass --stage prod or --stage qa.

shell
hearth status --stage prod

One pod is CrashLoopBackOff

shell
hearth logs <service>

Look at the last few seconds of output. If it's the API, missing env vars and DB connectivity are the top two suspects. If it's a frontend, a build-time env var (VITE_*) baked into the image was wrong — fix and re-ship.

Last resort

If everything is broken and you'd rather start clean:

shell
hearth down
docker volume rm katforge-hearth_db_data katforge-hearth_mercure_data
hearth setup
hearth up

This nukes local DB state and SSE history but keeps your repos, secrets, and age key intact.

← PreviousShipping