301 lines
10 KiB
Markdown
301 lines
10 KiB
Markdown
# HostItYourself — Heroku Clone MVP Plan
|
||
|
||
A self-hosted PaaS running on a Raspberry Pi that auto-deploys apps from GitHub pushes.
|
||
|
||
---
|
||
|
||
## Goals
|
||
|
||
- Single operator, multiple apps (target: 5–15 small services)
|
||
- `git push` → live in under 2 minutes
|
||
- Subdomain routing with automatic HTTPS
|
||
- Central dashboard for logs, env vars, status
|
||
- Low idle resource footprint (Raspberry Pi 4, 4–8 GB RAM)
|
||
|
||
---
|
||
|
||
## Architecture Overview
|
||
|
||
```
|
||
Internet
|
||
│
|
||
Cloudflare DNS
|
||
│
|
||
┌──────▼──────┐
|
||
│ Caddy │ ← Reverse proxy + auto TLS
|
||
│ (port 443) │
|
||
└──────┬──────┘
|
||
│ routes by subdomain
|
||
┌──────────────┼──────────────┐
|
||
│ │ │
|
||
┌─────▼──┐ ┌─────▼──┐ ┌─────▼──┐
|
||
│ App A │ │ App B │ │Control │
|
||
│(Docker)│ │(Docker)│ │ Plane │
|
||
└────────┘ └────────┘ └───┬────┘
|
||
│
|
||
┌────────▼────────┐
|
||
│ Build Engine │
|
||
│ (clone→build→ │
|
||
│ run container) │
|
||
└────────┬────────┘
|
||
│
|
||
┌────────▼────────┐
|
||
│ GitHub Webhook │
|
||
│ Listener │
|
||
└─────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Components
|
||
|
||
### 1. Reverse Proxy — Caddy
|
||
|
||
- Automatic TLS via Let's Encrypt (ACME DNS challenge through Cloudflare)
|
||
- Dynamic config: each deployed app gets a `<appname>.yourdomain.com` subdomain
|
||
- Caddy reloads config via API on each deploy; no restart needed
|
||
- Runs as a systemd service, not inside Docker (avoids bootstrapping issues)
|
||
|
||
### 2. Control Plane — `hiy-server`
|
||
|
||
A small Go or Node.js HTTP API + web UI. Responsibilities:
|
||
|
||
- CRUD for apps (name, GitHub repo URL, branch, env vars, port)
|
||
- Trigger manual deploys
|
||
- Proxy log streams from running containers
|
||
- Store state in a local SQLite database (`~/.hiy/db.sqlite`)
|
||
- Exposes REST API consumed by the dashboard and CLI
|
||
|
||
Dashboard pages:
|
||
- App list with status badges (running / building / stopped / crashed)
|
||
- Per-app: logs, env vars editor, deploy history, resource sparklines
|
||
- System overview: CPU, RAM, disk on the Pi
|
||
|
||
### 3. GitHub Webhook Listener
|
||
|
||
- Receives `push` events from GitHub (configured per-repo in GitHub settings)
|
||
- Validates HMAC signature (`X-Hub-Signature-256`)
|
||
- Filters on configured branch (default: `main`)
|
||
- Enqueues a deploy job; responds 200 immediately
|
||
- Can also be triggered via the dashboard or CLI
|
||
|
||
### 4. Build Engine — `hiy-builder`
|
||
|
||
Sequential steps run in a build container:
|
||
|
||
```
|
||
1. git clone --depth=1 <repo> /build/<appname>/<sha>
|
||
2. Detect build strategy:
|
||
a. Dockerfile present → docker build
|
||
b. Procfile + recognised language → buildpack (Cloud Native Buildpacks via `pack`)
|
||
c. static/ directory → Caddy file server image
|
||
3. docker build -t hiy/<appname>:<sha> .
|
||
4. docker stop hiy-<appname> (if running)
|
||
5. docker run -d --name hiy-<appname> \
|
||
--env-file /etc/hiy/<appname>.env \
|
||
--restart unless-stopped \
|
||
--network hiy-net \
|
||
hiy/<appname>:<sha>
|
||
6. Update Caddy upstream to new container
|
||
7. Prune old images (keep last 2)
|
||
```
|
||
|
||
Build logs are streamed to the control plane and stored per-deploy.
|
||
|
||
### 5. App Runtime — Docker
|
||
|
||
- Each app runs in its own container on the `hiy-net` bridge network
|
||
- Containers are **not** port-exposed to host; only Caddy reaches them
|
||
- Resource limits set at start: default 512 MB RAM, 0.5 CPU (configurable per app)
|
||
- `--restart unless-stopped` handles crash recovery
|
||
- Persistent data: apps that need storage mount named volumes (`hiy-<appname>-data`)
|
||
|
||
### 6. CLI — `hiy`
|
||
|
||
Thin shell script or Go binary for operator convenience:
|
||
|
||
```bash
|
||
hiy apps # list apps
|
||
hiy create myapp --repo <url> # register new app
|
||
hiy deploy myapp # trigger manual deploy
|
||
hiy logs myapp -f # tail logs
|
||
hiy env:set myapp KEY=value # set env var
|
||
hiy env:get myapp # list env vars
|
||
hiy restart myapp
|
||
hiy destroy myapp
|
||
```
|
||
|
||
---
|
||
|
||
## Data Model (SQLite)
|
||
|
||
```sql
|
||
apps (
|
||
id TEXT PRIMARY KEY, -- slug e.g. "my-api"
|
||
repo_url TEXT NOT NULL,
|
||
branch TEXT DEFAULT 'main',
|
||
port INTEGER NOT NULL, -- internal container port
|
||
created_at DATETIME,
|
||
updated_at DATETIME
|
||
)
|
||
|
||
deploys (
|
||
id TEXT PRIMARY KEY, -- uuid
|
||
app_id TEXT REFERENCES apps(id),
|
||
sha TEXT, -- git commit sha
|
||
status TEXT, -- queued|building|success|failed
|
||
log TEXT, -- full build log
|
||
triggered_by TEXT, -- webhook|manual|cli
|
||
started_at DATETIME,
|
||
finished_at DATETIME
|
||
)
|
||
|
||
env_vars (
|
||
app_id TEXT REFERENCES apps(id),
|
||
key TEXT,
|
||
value TEXT, -- encrypted at rest (age or libsodium)
|
||
PRIMARY KEY (app_id, key)
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## Generic Infrastructure (Non-Functional Requirements)
|
||
|
||
### Security
|
||
|
||
| Layer | Mechanism |
|
||
|---|---|
|
||
| SSH access | Key-only, disable password auth, non-standard port |
|
||
| Firewall | `ufw`: allow only 22, 80, 443 inbound |
|
||
| Fail2ban | Bans IPs with repeated SSH/HTTP failures |
|
||
| Webhook auth | HMAC-SHA256 signature verification |
|
||
| Env var encryption | Encrypted at rest with `age`; decrypted into container env at deploy time |
|
||
| Container isolation | No `--privileged`, no host network, drop capabilities |
|
||
| Dashboard auth | HTTP Basic Auth behind Caddy (or simple JWT session) |
|
||
| Secrets never in logs | Build logs redact env var values |
|
||
| OS updates | `unattended-upgrades` for security patches |
|
||
|
||
### Monitoring
|
||
|
||
- **Metrics**: [Netdata](https://github.com/netdata/netdata) (single binary, ~50 MB RAM) — CPU, RAM, disk, network, per-container stats
|
||
- **Uptime checks**: [Gatus](https://github.com/TwiN/gatus) — HTTP health checks per app, alerts via email/Telegram
|
||
- **Alerting thresholds**: disk > 80%, RAM > 85%, any app down > 2 min
|
||
- **Dashboard link**: Netdata and Gatus UIs accessible via subdomains on the Pi
|
||
|
||
### Logging
|
||
|
||
- Container stdout/stderr captured by Docker logging driver
|
||
- `hiy logs <app>` tails `docker logs`
|
||
- Control plane stores last 10,000 lines per app in SQLite (ring buffer)
|
||
- Optional: ship to a free Loki/Grafana Cloud tier for persistence
|
||
|
||
### Backups
|
||
|
||
Daily cron job:
|
||
|
||
1. `sqlite3 ~/.hiy/db.sqlite .dump > backup.sql`
|
||
2. Tar all env var files from `/etc/hiy/`
|
||
3. Copy to an attached USB drive or remote (rclone to S3/Backblaze B2)
|
||
4. Retain 30 days of backups
|
||
|
||
### Deployment of the Platform Itself
|
||
|
||
The platform (Caddy, hiy-server, hiy-builder) is managed via a single `docker-compose.yml` at `~/hiy-platform/`. Upgrade process:
|
||
|
||
```bash
|
||
git pull origin main
|
||
docker compose up -d --build
|
||
```
|
||
|
||
Caddy runs as a systemd service outside Compose (avoids chicken-and-egg on port 443).
|
||
|
||
---
|
||
|
||
## Repository Layout
|
||
|
||
```
|
||
hostityourself/
|
||
├── plan.md ← this file
|
||
├── server/ ← control plane API + dashboard
|
||
│ ├── main.go (or index.ts)
|
||
│ ├── db/
|
||
│ ├── routes/
|
||
│ └── ui/ ← simple HTML/HTMX dashboard
|
||
├── builder/ ← build engine scripts
|
||
│ ├── build.sh
|
||
│ └── detect-strategy.sh
|
||
├── cli/ ← hiy CLI tool
|
||
│ └── hiy.sh
|
||
├── proxy/
|
||
│ └── Caddyfile.template
|
||
├── infra/
|
||
│ ├── docker-compose.yml ← platform compose file
|
||
│ ├── ufw-setup.sh
|
||
│ ├── fail2ban/
|
||
│ └── backup.sh
|
||
└── docs/
|
||
├── setup.md ← Pi OS bootstrap guide
|
||
└── add-app.md
|
||
```
|
||
|
||
---
|
||
|
||
## MVP Milestones
|
||
|
||
### M1 — Foundation (Week 1–2)
|
||
- [ ] Pi OS setup: Raspberry Pi OS Lite, Docker, Caddy, ufw, fail2ban
|
||
- [ ] `hiy-server` skeleton: SQLite, REST endpoints for apps + deploys
|
||
- [ ] Manual deploy via CLI: clone → build → run container → Caddy config reload
|
||
|
||
### M2 — Auto-Deploy (Week 3)
|
||
- [ ] GitHub webhook listener integrated into server
|
||
- [ ] HMAC validation
|
||
- [ ] Build queue (single worker, sequential builds)
|
||
- [ ] Build log streaming to dashboard
|
||
|
||
### M3 — Dashboard (Week 4)
|
||
- [ ] App list + status
|
||
- [ ] Log viewer (last 500 lines, live tail via SSE)
|
||
- [ ] Env var editor
|
||
- [ ] Deploy history
|
||
|
||
### M4 — Hardening (Week 5)
|
||
- [ ] Env var encryption at rest
|
||
- [ ] Resource limits on containers
|
||
- [ ] Netdata + Gatus setup
|
||
- [ ] Backup cron job
|
||
- [ ] Dashboard auth
|
||
|
||
### M5 — Polish (Week 6)
|
||
- [ ] Buildpack detection (Dockerfile / Node / Python / static)
|
||
- [ ] `hiy` CLI binary
|
||
- [ ] `docs/setup.md` end-to-end bootstrap guide
|
||
- [ ] Smoke test: deploy a real app end-to-end from a GitHub push
|
||
|
||
---
|
||
|
||
## Hardware Recommendation
|
||
|
||
| Component | Recommendation |
|
||
|---|---|
|
||
| Pi model | Raspberry Pi 4 or 5, 4 GB RAM minimum, 8 GB preferred |
|
||
| Storage | 64 GB+ A2-rated microSD **or** USB SSD (much faster builds) |
|
||
| Network | Wired Ethernet; configure static IP or DHCP reservation |
|
||
| DNS | Cloudflare — free, supports ACME DNS challenge for wildcard TLS |
|
||
| Domain | Any registrar; point `*.yourdomain.com` → home IP + Cloudflare proxy |
|
||
| Power | Official Pi USB-C PSU + UPS hat (PiJuice or Geekworm X120) |
|
||
|
||
---
|
||
|
||
## Key Design Decisions & Trade-offs
|
||
|
||
| Decision | Rationale |
|
||
|---|---|
|
||
| Docker over bare processes | Isolation, restart policy, easy image rollback |
|
||
| Caddy over Nginx/Traefik | Automatic HTTPS, simple API-driven config, single binary |
|
||
| SQLite over Postgres | Zero-ops, sufficient for one operator and dozens of apps |
|
||
| Sequential build queue | Avoids saturating the Pi CPU/RAM; builds rarely overlap for one person |
|
||
| Buildpacks as optional | Most personal projects have a Dockerfile; buildpacks add complexity for MVP |
|
||
| No Kubernetes | Massive overkill for a single Pi and one person |
|
||
| Age encryption for env vars | Simple, modern, no daemon needed vs. Vault |
|