Avelia — Operations Runbook

Last updated: 2026-04-11 Scope: avelia-health.com website (Astro + Node SSR). For the Flutter app and backend, see their respective CLAUDE.md files.

1. Deploy flow

Push to main → Forgejo CI (.forgejo/workflows/build.yml) builds the Docker image for linux/arm64, tags it sha-<short> and latest, pushes to forgejo.anderscloud.de/avelia-heath/avelia-health.com-website.
CI dispatches an update-image workflow to cloudsourced-stacks, which bumps the image tag in apps/de-nc-hosting02/avelia-astro/docker-compose.yml and commits.
StackPilot on de-nc-hosting02 polls the stacks repo, pulls the new image, and docker compose up -d the service.

Manual deploy (bypass GitOps)

bash

cd /Volumes/exSSD/dev/cloudsourced/playbook
ANSIBLE_VAULT_PASSWORD_FILE=/Volumes/exSSD/dev/vault-pass.txt \
  ansible-playbook de-nc-hosting02.servers.cloudsourced.de.yaml --tags astro

Use this if the GitOps path is broken or you need to force-redeploy without a new image.

CI monitoring

bash

curl -s -H "Authorization: token $FORGEJO_TOKEN" \
  "https://forgejo.anderscloud.de/api/v1/repos/avelia-heath/avelia-health.com-website/actions/runs?limit=5"

2. Known gotchas (chronological, most recent first)

2026-04-11 — COCKPIT_API_KEY vs COCKPIT_API_TOKEN env var mismatch

Symptom: Every Cockpit API call in production returned HTTP 412 Precondition Failed:

Error: Cockpit /content/items/localities: 412
Error: Cockpit /content/items/tags: 412
Error: Cockpit /content/item/pages: 412

Root cause: The Astro code reads process.env.COCKPIT_API_KEY, but the sops-encrypted secret file (apps/de-nc-hosting02/avelia-astro/.env.sops) stored the value under the legacy name COCKPIT_API_TOKEN. The compose file was setting COCKPIT_API_TOKEN as an env var inside the container, leaving COCKPIT_API_KEY empty. An empty api-key header makes Cockpit return 412.

Fix: In cloudsourced-stacks/.../avelia-astro/docker-compose.yml:

yaml

# Before:
COCKPIT_API_TOKEN: ${COCKPIT_API_TOKEN}
# After:
COCKPIT_API_KEY: ${COCKPIT_API_TOKEN}

Proper long-term fix: Rename the variable inside the sops-encrypted file. Requires the age key:

bash

sops -d apps/de-nc-hosting02/avelia-astro/.env.sops > /tmp/env
sed -i '' 's/COCKPIT_API_TOKEN/COCKPIT_API_KEY/' /tmp/env
sops -e /tmp/env > apps/de-nc-hosting02/avelia-astro/.env.sops
rm /tmp/env

Lesson: The env var name mismatch is easy to miss because the value is encrypted. Always verify by running the app locally with the same env var names as production.

2026-04-10 — Middleware threw on immutable response headers

Symptom: TypeError: immutable stack trace on any request that returned a redirect Response:

at _Headers.set (node:internal/deps/undici/undici:9152:17)
at eval (src/middleware.ts:22:22)

Root cause: The middleware unconditionally calls response.headers.set(...) to inject security headers. But Response.redirect() (and some fetch-API responses) returns an object with headers that are immutable. Any .set() call throws.

Fix: Wrap the header-setting block in try/catch. Dropping security headers on a redirect is safe because redirects have no response body to protect (src/middleware.ts).

2026-04-10 — Satori requires TTF/OTF, not WOFF2

Symptom: /api/og endpoint returned 500, then fell back to the static SVG even when expected to render a custom image.

Root cause: Satori requires TTF or OTF font data. The project had Outfit fonts only as WOFF2 (which uses Brotli compression Satori can't decode).

Fix: Downloaded Outfit-Regular.ttf and Outfit-Bold.ttf from the Outfit Fonts GitHub repo into public/fonts/. Updated src/lib/og.ts to read the TTF files (with buffer.slice(byteOffset, ...) to preserve correct ArrayBuffer slicing).

2026-04-10 — `getStaticPaths` can't reference module-level consts

Symptom: Stage pillar pages (/en/stages/ivf etc.) returned 500: VALID_SLUGS is not defined.

Root cause: Astro runs getStaticPaths() in an isolated context that does not have access to module-level consts declared outside the function body.

Fix: Inline the array inside getStaticPaths(). Any consts needed by the body of the .astro file can be declared after the function.

2026-04-10 — Astro prerendered pages can't read `Astro.request.headers`

Symptom: Warning during build:

[WARN] Astro.request.headers was used when rendering the route ... Astro.request.headers is not available on prerendered pages.

Cause: Prerendered pages (the default or explicit export const prerender = true) run at build time and have no request context. Any code that reads request headers (including via Astro.locals populated by middleware from request.headers) must be on SSR pages (export const prerender = false).

When to force SSR: Pages that need jurisdiction, cookies, or per-request data. The Safe Mode page needs prerender = false because its resource filtering depends on the jurisdiction cookie.

When to keep static: Content-only pages without per-request data (zero-knowledge, security, glossary, stages) can stay prerendered for faster response times.

2026-04-10 — `alternateUrl()` dropped path segments for nested routes

Symptom: Language switcher on /en/privacy/zero-knowledge produced a 404 at /de/privacy (missing /zero-knowledge).

Root cause: The alternateUrl() helper in src/lib/i18n.ts only handled two-segment paths (/lang/section and /lang/section/slug). Nested paths beyond two segments dropped the remaining segments.

Fix: After computing the translated section, preserve all remaining path segments:

const rest = parts.slice(2);
if (rest.length) return `/${targetLang}/${targetSection}/${rest.join('/')}`;

2026-02 — Node 22.12+ required

Symptom: Local dev server fails to start on Node 20: Node.js v20.x is not supported by Astro! Please upgrade to >=22.12.0.

Fix: nvm install 22 && nvm use 22. The CI runner uses node:22-bookworm and the Docker runtime uses node:22-alpine — both compatible.

2026-02 — Node 25 Fetch spec: 204/304 responses must have null body

Symptom: new Response('', { status: 204 }) throws in Node 25.

Fix: Use new Response(null, { status: 204 }) for empty responses. Relevant for /api/a.ts and any API route returning 204 No Content.

2026-02 — Astro `security.checkOrigin` breaks `sendBeacon` analytics

Symptom: Analytics POST requests from navigator.sendBeacon() were blocked by Astro's CSRF origin check.

Fix: In astro.config.mjs:

security: { checkOrigin: false }

Caddy handles the actual origin check upstream. Astro's built-in check conflicts with beacon-style POSTs.

2026-02 — `process.env` vs `import.meta.env`

Rule: All runtime config uses process.env.X, never import.meta.env.X.

process.env.X is read at runtime on the Node.js server. The value is injected by Docker at container start. Safe for secrets.
import.meta.env.X is statically replaced by Vite at build time. If a secret is referenced this way, it gets BAKED INTO the compiled JavaScript — potentially into the client bundle. Only use import.meta.env.PUBLIC_* (which is explicitly public).

Verification: After a build, grep dist/client/ for secret names. Zero hits is the expected result.

3. Environment variables reference

Variable	Where	Purpose	Secret?
`COCKPIT_URL`	`src/lib/cms.ts`	Cockpit CMS base URL	No
`COCKPIT_API_KEY`	`src/lib/cms.ts`	Cockpit API authentication	Yes (sops-encrypted)
`LISTMONK_URL`	`src/pages/api/{subscribe,interest}.ts`	Listmonk base URL	No
`LISTMONK_API_USER`	same	Listmonk admin user	Yes (sops-encrypted)
`LISTMONK_API_TOKEN`	same	Listmonk API token	Yes (sops-encrypted)
`LISTMONK_LIST_ID`	`src/pages/api/subscribe.ts`	Waitlist mailing list ID	No
`LISTMONK_EXPERTS_LIST_ID`	`src/pages/api/interest.ts`	Experts list ID	No
`LISTMONK_PARTNERS_LIST_ID`	`src/pages/api/interest.ts`	Partners list ID	No
`NOTIFY_EMAIL`	`src/pages/api/interest.ts`	Internal notify recipient	No
`NOTIFY_TEMPLATE_ID`	`src/pages/api/interest.ts`	Listmonk template ID	No
`MATOMO_INTERNAL_URL`	`src/pages/api/a.ts`	Matomo proxy upstream	No
`MATOMO_AUTH_TOKEN`	`src/pages/api/a.ts`	Matomo token for real-IP forwarding	Yes (sops-encrypted)
`PUBLIC_MATOMO_SITE_ID`	`src/layouts/BaseLayout.astro`	Matomo site ID (exposed to browser as `<meta>`)	No (intentionally public)

Dev setup: Set via shell env vars when running pnpm dev:

bash

COCKPIT_URL=http://localhost:8055 \
COCKPIT_API_KEY=API-e0de96a5eebe53be07ad6193941d6d18b3894aba \
PUBLIC_MATOMO_SITE_ID=1 \
pnpm dev

Prod setup: Via sops-encrypted .env file in cloudsourced-stacks plus plain-text env in docker-compose.yml.

4. Build quirks

TypeScript `readonly` vs mutable

Any array literal from i18n files is readonly (because as const is implied by the type system). Components that accept these arrays as props must declare readonly T[] in their Props interface, not T[].

Example: FaqBlock.astro accepts items: readonly FaqItem[].

Asserting type-generic maps in i18n

When defining nested records like stages.details, use as Record<string, {...}> to avoid TypeScript's "readonly-deep" inference making downstream code painful.

Prefetch + viewport strategy

astro.config.mjs has prefetch: { prefetchAll: true, defaultStrategy: 'viewport' }. This means every internal link is prefetched on viewport-intersection. Keep this off for any pages that are expensive to render SSR (we have none currently).

5. Cockpit CMS operations

Applying model changes

bash

./cockpit/setup.sh                          # seeds + installs models
docker restart avelia-healthcom-cockpit-1   # REQUIRED to clear PHP memory cache

The PHP memory cache is per-worker; the cockpit cli only clears the CLI process cache. Without the Docker restart, the web workers still use the old schema.

Content link fields return single object if only one linked

const tags = asArray(item.tags); // always wrap before .map/.some/.filter

Cockpit returns content-link[multiple: true] as an object (not array) when only one item is linked. src/lib/cms.ts has an asArray() helper for this.

Locale mapping

en → Cockpit ?locale=default
de → Cockpit ?locale=de

Use this pattern: locale: lang === 'en' ? 'default' : lang.

6. Reverse proxy and CI/CD gotchas

Caddy handles origin check

astro.config.mjs disables Astro's checkOrigin. If you ever move off Caddy, re-enable it.

ARM64 throughout

Both the CI runner and prod server are ARM64. The Dockerfile uses --platform linux/arm64. No cross-arch, no emulation. If the runner pool changes, update the build args.

Forgejo registry authentication

CI uses REGISTRY_USER and REGISTRY_TOKEN secrets to push. If pushes start returning 403/unauthorized, the token likely expired or lost scope. Regenerate in Forgejo → User Settings → Applications → Generate New Token with write:package scope.

7. Recovery from deployment failures

Roll back to previous image

bash

cd /Volumes/exSSD/dev/anders-it/cloudsourced/cloudsourced-stacks
# Find previous working tag in git log:
git log --oneline apps/de-nc-hosting02/avelia-astro/
# Reset the compose file to the previous tag:
git checkout HEAD~1 -- apps/de-nc-hosting02/avelia-astro/docker-compose.yml
git commit -m "rollback avelia-astro to previous image"
git push

StackPilot will reconcile within ~1 minute.

Force-restart in production

SSH to the server, then:

bash

cd /opt/avelia-astro
docker compose restart astro
# Or, to force a fresh pull:
docker compose pull && docker compose up -d

Sanity test after deploy

bash

curl -sI https://avelia-health.com/en/ | head -5
curl -sI https://avelia-health.com/en/privacy/safe-mode | head -5  # SSR
curl -sI https://avelia-health.com/sitemap-index.xml | head -5
curl -sI https://avelia-health.com/llms.txt | head -5

All should return 200.

8. Cross-references

Content library: avelia-content-library.md
Product spec: avelia-product-spec.md
Website CLAUDE.md: avelia-health.com/CLAUDE.md (dev setup, stack, env vars)
Backend CLAUDE.md: avelia-backend/CLAUDE.md
Flutter CLAUDE.md: avelia-flutter-app/CLAUDE.md

Avelia — Operations Runbook ​

1. Deploy flow ​

Manual deploy (bypass GitOps) ​

CI monitoring ​

2. Known gotchas (chronological, most recent first) ​

2026-04-11 — COCKPIT_API_KEY vs COCKPIT_API_TOKEN env var mismatch ​

2026-04-10 — Middleware threw on immutable response headers ​

2026-04-10 — Satori requires TTF/OTF, not WOFF2 ​

2026-04-10 — getStaticPaths can't reference module-level consts ​

2026-04-10 — Astro prerendered pages can't read Astro.request.headers ​

2026-04-10 — alternateUrl() dropped path segments for nested routes ​

2026-02 — Node 22.12+ required ​

2026-02 — Node 25 Fetch spec: 204/304 responses must have null body ​

2026-02 — Astro security.checkOrigin breaks sendBeacon analytics ​

2026-02 — process.env vs import.meta.env ​

3. Environment variables reference ​

4. Build quirks ​

TypeScript readonly vs mutable ​

Asserting type-generic maps in i18n ​

Prefetch + viewport strategy ​

5. Cockpit CMS operations ​

Applying model changes ​

Content link fields return single object if only one linked ​

Locale mapping ​

6. Reverse proxy and CI/CD gotchas ​

Caddy handles origin check ​

ARM64 throughout ​

Forgejo registry authentication ​

7. Recovery from deployment failures ​

Roll back to previous image ​

Force-restart in production ​

Sanity test after deploy ​

8. Cross-references ​

Avelia — Operations Runbook

1. Deploy flow

Manual deploy (bypass GitOps)

CI monitoring

2. Known gotchas (chronological, most recent first)

2026-04-11 — COCKPIT_API_KEY vs COCKPIT_API_TOKEN env var mismatch

2026-04-10 — Middleware threw on immutable response headers

2026-04-10 — Satori requires TTF/OTF, not WOFF2

2026-04-10 — `getStaticPaths` can't reference module-level consts

2026-04-10 — Astro prerendered pages can't read `Astro.request.headers`

2026-04-10 — `alternateUrl()` dropped path segments for nested routes

2026-02 — Node 22.12+ required

2026-02 — Node 25 Fetch spec: 204/304 responses must have null body

2026-02 — Astro `security.checkOrigin` breaks `sendBeacon` analytics

2026-02 — `process.env` vs `import.meta.env`

3. Environment variables reference

4. Build quirks

TypeScript `readonly` vs mutable

Asserting type-generic maps in i18n

Prefetch + viewport strategy

5. Cockpit CMS operations

Applying model changes

Content link fields return single object if only one linked

Locale mapping

6. Reverse proxy and CI/CD gotchas

Caddy handles origin check

ARM64 throughout

Forgejo registry authentication

7. Recovery from deployment failures

Roll back to previous image

Force-restart in production

Sanity test after deploy

8. Cross-references