Add MVP plan for self-hosted Heroku clone on Raspberry Pi
Covers architecture, components, data model, security, monitoring, backup, milestones, and hardware recommendations. https://claude.ai/code/session_01FKCW3FDjNFj6jve4niMFXH
This commit is contained in:
Claude
commit
c46e3b4125
1 changed files with 301 additions and 0 deletions
301
plan.md
Normal file
301
plan.md
Normal file
|
|
@ -0,0 +1,301 @@
|
||||||
|
# HostItYourself — Heroku Clone MVP Plan
|
||||||
|
|
||||||
|
A self-hosted PaaS running on a Raspberry Pi that auto-deploys apps from GitHub pushes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
- Single operator, multiple apps (target: 5–15 small services)
|
||||||
|
- `git push` → live in under 2 minutes
|
||||||
|
- Subdomain routing with automatic HTTPS
|
||||||
|
- Central dashboard for logs, env vars, status
|
||||||
|
- Low idle resource footprint (Raspberry Pi 4, 4–8 GB RAM)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
Internet
|
||||||
|
│
|
||||||
|
Cloudflare DNS
|
||||||
|
│
|
||||||
|
┌──────▼──────┐
|
||||||
|
│ Caddy │ ← Reverse proxy + auto TLS
|
||||||
|
│ (port 443) │
|
||||||
|
└──────┬──────┘
|
||||||
|
│ routes by subdomain
|
||||||
|
┌──────────────┼──────────────┐
|
||||||
|
│ │ │
|
||||||
|
┌─────▼──┐ ┌─────▼──┐ ┌─────▼──┐
|
||||||
|
│ App A │ │ App B │ │Control │
|
||||||
|
│(Docker)│ │(Docker)│ │ Plane │
|
||||||
|
└────────┘ └────────┘ └───┬────┘
|
||||||
|
│
|
||||||
|
┌────────▼────────┐
|
||||||
|
│ Build Engine │
|
||||||
|
│ (clone→build→ │
|
||||||
|
│ run container) │
|
||||||
|
└────────┬────────┘
|
||||||
|
│
|
||||||
|
┌────────▼────────┐
|
||||||
|
│ GitHub Webhook │
|
||||||
|
│ Listener │
|
||||||
|
└─────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
### 1. Reverse Proxy — Caddy
|
||||||
|
|
||||||
|
- Automatic TLS via Let's Encrypt (ACME DNS challenge through Cloudflare)
|
||||||
|
- Dynamic config: each deployed app gets a `<appname>.yourdomain.com` subdomain
|
||||||
|
- Caddy reloads config via API on each deploy; no restart needed
|
||||||
|
- Runs as a systemd service, not inside Docker (avoids bootstrapping issues)
|
||||||
|
|
||||||
|
### 2. Control Plane — `hiy-server`
|
||||||
|
|
||||||
|
A small Go or Node.js HTTP API + web UI. Responsibilities:
|
||||||
|
|
||||||
|
- CRUD for apps (name, GitHub repo URL, branch, env vars, port)
|
||||||
|
- Trigger manual deploys
|
||||||
|
- Proxy log streams from running containers
|
||||||
|
- Store state in a local SQLite database (`~/.hiy/db.sqlite`)
|
||||||
|
- Exposes REST API consumed by the dashboard and CLI
|
||||||
|
|
||||||
|
Dashboard pages:
|
||||||
|
- App list with status badges (running / building / stopped / crashed)
|
||||||
|
- Per-app: logs, env vars editor, deploy history, resource sparklines
|
||||||
|
- System overview: CPU, RAM, disk on the Pi
|
||||||
|
|
||||||
|
### 3. GitHub Webhook Listener
|
||||||
|
|
||||||
|
- Receives `push` events from GitHub (configured per-repo in GitHub settings)
|
||||||
|
- Validates HMAC signature (`X-Hub-Signature-256`)
|
||||||
|
- Filters on configured branch (default: `main`)
|
||||||
|
- Enqueues a deploy job; responds 200 immediately
|
||||||
|
- Can also be triggered via the dashboard or CLI
|
||||||
|
|
||||||
|
### 4. Build Engine — `hiy-builder`
|
||||||
|
|
||||||
|
Sequential steps run in a build container:
|
||||||
|
|
||||||
|
```
|
||||||
|
1. git clone --depth=1 <repo> /build/<appname>/<sha>
|
||||||
|
2. Detect build strategy:
|
||||||
|
a. Dockerfile present → docker build
|
||||||
|
b. Procfile + recognised language → buildpack (Cloud Native Buildpacks via `pack`)
|
||||||
|
c. static/ directory → Caddy file server image
|
||||||
|
3. docker build -t hiy/<appname>:<sha> .
|
||||||
|
4. docker stop hiy-<appname> (if running)
|
||||||
|
5. docker run -d --name hiy-<appname> \
|
||||||
|
--env-file /etc/hiy/<appname>.env \
|
||||||
|
--restart unless-stopped \
|
||||||
|
--network hiy-net \
|
||||||
|
hiy/<appname>:<sha>
|
||||||
|
6. Update Caddy upstream to new container
|
||||||
|
7. Prune old images (keep last 2)
|
||||||
|
```
|
||||||
|
|
||||||
|
Build logs are streamed to the control plane and stored per-deploy.
|
||||||
|
|
||||||
|
### 5. App Runtime — Docker
|
||||||
|
|
||||||
|
- Each app runs in its own container on the `hiy-net` bridge network
|
||||||
|
- Containers are **not** port-exposed to host; only Caddy reaches them
|
||||||
|
- Resource limits set at start: default 512 MB RAM, 0.5 CPU (configurable per app)
|
||||||
|
- `--restart unless-stopped` handles crash recovery
|
||||||
|
- Persistent data: apps that need storage mount named volumes (`hiy-<appname>-data`)
|
||||||
|
|
||||||
|
### 6. CLI — `hiy`
|
||||||
|
|
||||||
|
Thin shell script or Go binary for operator convenience:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
hiy apps # list apps
|
||||||
|
hiy create myapp --repo <url> # register new app
|
||||||
|
hiy deploy myapp # trigger manual deploy
|
||||||
|
hiy logs myapp -f # tail logs
|
||||||
|
hiy env:set myapp KEY=value # set env var
|
||||||
|
hiy env:get myapp # list env vars
|
||||||
|
hiy restart myapp
|
||||||
|
hiy destroy myapp
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Data Model (SQLite)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
apps (
|
||||||
|
id TEXT PRIMARY KEY, -- slug e.g. "my-api"
|
||||||
|
repo_url TEXT NOT NULL,
|
||||||
|
branch TEXT DEFAULT 'main',
|
||||||
|
port INTEGER NOT NULL, -- internal container port
|
||||||
|
created_at DATETIME,
|
||||||
|
updated_at DATETIME
|
||||||
|
)
|
||||||
|
|
||||||
|
deploys (
|
||||||
|
id TEXT PRIMARY KEY, -- uuid
|
||||||
|
app_id TEXT REFERENCES apps(id),
|
||||||
|
sha TEXT, -- git commit sha
|
||||||
|
status TEXT, -- queued|building|success|failed
|
||||||
|
log TEXT, -- full build log
|
||||||
|
triggered_by TEXT, -- webhook|manual|cli
|
||||||
|
started_at DATETIME,
|
||||||
|
finished_at DATETIME
|
||||||
|
)
|
||||||
|
|
||||||
|
env_vars (
|
||||||
|
app_id TEXT REFERENCES apps(id),
|
||||||
|
key TEXT,
|
||||||
|
value TEXT, -- encrypted at rest (age or libsodium)
|
||||||
|
PRIMARY KEY (app_id, key)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Generic Infrastructure (Non-Functional Requirements)
|
||||||
|
|
||||||
|
### Security
|
||||||
|
|
||||||
|
| Layer | Mechanism |
|
||||||
|
|---|---|
|
||||||
|
| SSH access | Key-only, disable password auth, non-standard port |
|
||||||
|
| Firewall | `ufw`: allow only 22, 80, 443 inbound |
|
||||||
|
| Fail2ban | Bans IPs with repeated SSH/HTTP failures |
|
||||||
|
| Webhook auth | HMAC-SHA256 signature verification |
|
||||||
|
| Env var encryption | Encrypted at rest with `age`; decrypted into container env at deploy time |
|
||||||
|
| Container isolation | No `--privileged`, no host network, drop capabilities |
|
||||||
|
| Dashboard auth | HTTP Basic Auth behind Caddy (or simple JWT session) |
|
||||||
|
| Secrets never in logs | Build logs redact env var values |
|
||||||
|
| OS updates | `unattended-upgrades` for security patches |
|
||||||
|
|
||||||
|
### Monitoring
|
||||||
|
|
||||||
|
- **Metrics**: [Netdata](https://github.com/netdata/netdata) (single binary, ~50 MB RAM) — CPU, RAM, disk, network, per-container stats
|
||||||
|
- **Uptime checks**: [Gatus](https://github.com/TwiN/gatus) — HTTP health checks per app, alerts via email/Telegram
|
||||||
|
- **Alerting thresholds**: disk > 80%, RAM > 85%, any app down > 2 min
|
||||||
|
- **Dashboard link**: Netdata and Gatus UIs accessible via subdomains on the Pi
|
||||||
|
|
||||||
|
### Logging
|
||||||
|
|
||||||
|
- Container stdout/stderr captured by Docker logging driver
|
||||||
|
- `hiy logs <app>` tails `docker logs`
|
||||||
|
- Control plane stores last 10,000 lines per app in SQLite (ring buffer)
|
||||||
|
- Optional: ship to a free Loki/Grafana Cloud tier for persistence
|
||||||
|
|
||||||
|
### Backups
|
||||||
|
|
||||||
|
Daily cron job:
|
||||||
|
|
||||||
|
1. `sqlite3 ~/.hiy/db.sqlite .dump > backup.sql`
|
||||||
|
2. Tar all env var files from `/etc/hiy/`
|
||||||
|
3. Copy to an attached USB drive or remote (rclone to S3/Backblaze B2)
|
||||||
|
4. Retain 30 days of backups
|
||||||
|
|
||||||
|
### Deployment of the Platform Itself
|
||||||
|
|
||||||
|
The platform (Caddy, hiy-server, hiy-builder) is managed via a single `docker-compose.yml` at `~/hiy-platform/`. Upgrade process:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git pull origin main
|
||||||
|
docker compose up -d --build
|
||||||
|
```
|
||||||
|
|
||||||
|
Caddy runs as a systemd service outside Compose (avoids chicken-and-egg on port 443).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Repository Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
hostityourself/
|
||||||
|
├── plan.md ← this file
|
||||||
|
├── server/ ← control plane API + dashboard
|
||||||
|
│ ├── main.go (or index.ts)
|
||||||
|
│ ├── db/
|
||||||
|
│ ├── routes/
|
||||||
|
│ └── ui/ ← simple HTML/HTMX dashboard
|
||||||
|
├── builder/ ← build engine scripts
|
||||||
|
│ ├── build.sh
|
||||||
|
│ └── detect-strategy.sh
|
||||||
|
├── cli/ ← hiy CLI tool
|
||||||
|
│ └── hiy.sh
|
||||||
|
├── proxy/
|
||||||
|
│ └── Caddyfile.template
|
||||||
|
├── infra/
|
||||||
|
│ ├── docker-compose.yml ← platform compose file
|
||||||
|
│ ├── ufw-setup.sh
|
||||||
|
│ ├── fail2ban/
|
||||||
|
│ └── backup.sh
|
||||||
|
└── docs/
|
||||||
|
├── setup.md ← Pi OS bootstrap guide
|
||||||
|
└── add-app.md
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## MVP Milestones
|
||||||
|
|
||||||
|
### M1 — Foundation (Week 1–2)
|
||||||
|
- [ ] Pi OS setup: Raspberry Pi OS Lite, Docker, Caddy, ufw, fail2ban
|
||||||
|
- [ ] `hiy-server` skeleton: SQLite, REST endpoints for apps + deploys
|
||||||
|
- [ ] Manual deploy via CLI: clone → build → run container → Caddy config reload
|
||||||
|
|
||||||
|
### M2 — Auto-Deploy (Week 3)
|
||||||
|
- [ ] GitHub webhook listener integrated into server
|
||||||
|
- [ ] HMAC validation
|
||||||
|
- [ ] Build queue (single worker, sequential builds)
|
||||||
|
- [ ] Build log streaming to dashboard
|
||||||
|
|
||||||
|
### M3 — Dashboard (Week 4)
|
||||||
|
- [ ] App list + status
|
||||||
|
- [ ] Log viewer (last 500 lines, live tail via SSE)
|
||||||
|
- [ ] Env var editor
|
||||||
|
- [ ] Deploy history
|
||||||
|
|
||||||
|
### M4 — Hardening (Week 5)
|
||||||
|
- [ ] Env var encryption at rest
|
||||||
|
- [ ] Resource limits on containers
|
||||||
|
- [ ] Netdata + Gatus setup
|
||||||
|
- [ ] Backup cron job
|
||||||
|
- [ ] Dashboard auth
|
||||||
|
|
||||||
|
### M5 — Polish (Week 6)
|
||||||
|
- [ ] Buildpack detection (Dockerfile / Node / Python / static)
|
||||||
|
- [ ] `hiy` CLI binary
|
||||||
|
- [ ] `docs/setup.md` end-to-end bootstrap guide
|
||||||
|
- [ ] Smoke test: deploy a real app end-to-end from a GitHub push
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Hardware Recommendation
|
||||||
|
|
||||||
|
| Component | Recommendation |
|
||||||
|
|---|---|
|
||||||
|
| Pi model | Raspberry Pi 4 or 5, 4 GB RAM minimum, 8 GB preferred |
|
||||||
|
| Storage | 64 GB+ A2-rated microSD **or** USB SSD (much faster builds) |
|
||||||
|
| Network | Wired Ethernet; configure static IP or DHCP reservation |
|
||||||
|
| DNS | Cloudflare — free, supports ACME DNS challenge for wildcard TLS |
|
||||||
|
| Domain | Any registrar; point `*.yourdomain.com` → home IP + Cloudflare proxy |
|
||||||
|
| Power | Official Pi USB-C PSU + UPS hat (PiJuice or Geekworm X120) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key Design Decisions & Trade-offs
|
||||||
|
|
||||||
|
| Decision | Rationale |
|
||||||
|
|---|---|
|
||||||
|
| Docker over bare processes | Isolation, restart policy, easy image rollback |
|
||||||
|
| Caddy over Nginx/Traefik | Automatic HTTPS, simple API-driven config, single binary |
|
||||||
|
| SQLite over Postgres | Zero-ops, sufficient for one operator and dozens of apps |
|
||||||
|
| Sequential build queue | Avoids saturating the Pi CPU/RAM; builds rarely overlap for one person |
|
||||||
|
| Buildpacks as optional | Most personal projects have a Dockerfile; buildpacks add complexity for MVP |
|
||||||
|
| No Kubernetes | Massive overkill for a single Pi and one person |
|
||||||
|
| Age encryption for env vars | Simple, modern, no daemon needed vs. Vault |
|
||||||
Loading…
Add table
Reference in a new issue