Self-Hosting the Console

This guide runs Tuff Console for a team: one tuff process, one data folder, and a reverse proxy that authenticates the people who view it. The examples use the organisation acme and the address https://tuff.acme.dev.

The console is part of the tuff binary. tuff console serve needs tuff from a release that includes the console. Check with tuff console --help.

How the pieces fit

Piece	Setting
The process	`tuff console serve`, listening on loopback behind a proxy, or on `0.0.0.0` in a container
The data	One SQLite file, `console.sqlite`, in the folder given with `--data`
Publishers	GitHub Actions jobs through `--trust github:<owner>`, or any CI system through an API key
Viewers	Authenticated by the reverse proxy. The console does not authenticate viewers
The public address	`--public-url`, which is what publishers put in `TUFF_CONSOLE_URL`

The console checks credentials on POST /api/v1/reports only. Everything else is read only and open to whoever can reach the port, so the port must be reachable from the proxy and nowhere else.

Run with systemd

Install tuff on the server, create a user for the service, and write the unit. Installation lists the ways to install tuff. The unit below expects /usr/local/bin/tuff, where the curl installer puts it.

sudo useradd --system --home-dir /var/lib/tuff-console --shell /usr/sbin/nologin tuff

[Unit]
Description=Tuff Console
After=network-online.target
Wants=network-online.target

[Service]
User=tuff
Group=tuff
StateDirectory=tuff-console
StateDirectoryMode=0750
ExecStart=/usr/local/bin/tuff console serve \
  --addr 127.0.0.1:7474 \
  --data /var/lib/tuff-console \
  --trust github:acme \
  --public-url https://tuff.acme.dev
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

StateDirectory=tuff-console makes systemd create /var/lib/tuff-console, owned by tuff, and keep it writable under ProtectSystem=strict. The console creates console.sqlite inside it with mode 0600.

sudo systemctl daemon-reload
sudo systemctl enable --now tuff-console
journalctl -u tuff-console -n 5
curl http://127.0.0.1:7474/healthz

The journal shows the startup lines: the address, the data folder, and the OIDC audience. The audience must equal the public URL that publishers use.

The server stops on SIGTERM, which is what systemctl stop sends, and on SIGINT.

The unit binds 127.0.0.1, so --public-read is not needed. Bind a non-loopback address only when the proxy runs on another host. Then add --public-read, and make sure a firewall lets only the proxy connect.

A server with no --trust and no key accepts reports from anything that can reach it, including the proxy. The unit above has a trust. A console that publishes from other CI systems also needs a key, created as described in Keys on the host.

Run in a container

Each release from 0.15.0 publishes the image ghcr.io/kannandreams/tuff-console for linux/amd64 and linux/arm64. It holds the tuff binary from the same GitHub release and runs the console as an unprivileged user with /data as a volume. ca-certificates is in the image because the console fetches GitHub’s OIDC signing keys over HTTPS.

Tag, for example	Points to
`0.15.0`	That release. Pin this tag in production
`0.15`	The newest release with that minor version
`latest`	The newest release

The entry point is tuff console serve --addr 0.0.0.0:7474 --public-read, and the image sets TUFF_CONSOLE_DATA=/data, so the console and the tuff console key commands use the volume without --data. The console refuses to start on a non-loopback address until a publish credential exists, so create a key in the volume before the first start or pass --trust:

# Create a key in the volume. The key is printed once
docker run --rm -v tuff-console-data:/data --entrypoint tuff ghcr.io/kannandreams/tuff-console:<version> \
  console key create ci

# Start the console. Arguments after the image name are added to the entry point
docker run -d --name tuff-console --restart unless-stopped \
  -p 127.0.0.1:7474:7474 \
  -v tuff-console-data:/data \
  ghcr.io/kannandreams/tuff-console:<version> --trust github:acme --public-url https://tuff.acme.dev

Publishing 127.0.0.1:7474 keeps the port reachable from the host only, where the reverse proxy runs. GET /healthz answers for container health checks, such as a Kubernetes httpGet probe. The image has no curl, so run a check from outside the container.

To look at the console before setting it up, run it with generated sample data and open http://127.0.0.1:7474. The data lives in memory and is gone when the container stops. The demo needs no publish credential and refuses every publish:

docker run --rm -p 127.0.0.1:7474:7474 ghcr.io/kannandreams/tuff-console:<version> --demo

The 0.15.0 image passes --data /data in its entry point instead of setting TUFF_CONSOLE_DATA, and its console needs a credential for a demo. With that image, run the demo as docker run --rm -p 127.0.0.1:7474:7474 --entrypoint tuff ghcr.io/kannandreams/tuff-console:0.15.0 console serve --demo --addr 0.0.0.0:7474 --public-read --trust github:example.

Upgrade by pulling a newer tag and recreating the container with the same volume. The console migrates the database on first use.

Verify the image

The release workflow signs each image with cosign using GitHub’s OIDC identity, and attaches an SBOM and build provenance. To check that an image was built by the Tuff release workflow:

cosign verify ghcr.io/kannandreams/tuff-console:<version> \
  --certificate-identity-regexp '^https://github.com/kannandreams/tuff/.github/workflows/release.yml@refs/tags/v' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Build the image yourself

docker/Dockerfile packages the release binaries without compiling them. Its build context is a folder with amd64/tuff and arm64/tuff, taken from tuff-x86_64-unknown-linux-gnu.tar.gz and tuff-aarch64-unknown-linux-gnu.tar.gz on the GitHub release. Check the tarballs against checksums.txt before extracting them.

Put a reverse proxy in front

The console does not authenticate viewers, and viewer sign-in is planned for a later release. Until then the proxy decides who can open the pages and the read API.

Publishing needs different handling. Publishers send POST /api/v1/reports with their own Authorization: Bearer header, which the console verifies, so the proxy must leave that request to the console. Requiring a login there would reject every publish, and proxy basic auth would replace the Authorization header the console reads. Forward the header unchanged, and raise any request body limit to at least 16 MiB, which is the largest report the console accepts.

Caddy with basic auth

tuff.acme.dev {
  @publish {
    method POST
    path /api/v1/reports
  }

  handle @publish {
    reverse_proxy 127.0.0.1:7474
  }

  handle {
    basic_auth {
      # caddy hash-password
      alice $2a$14$replace-with-the-output-of-caddy-hash-password
    }
    reverse_proxy 127.0.0.1:7474
  }
}

Caddy obtains and renews the certificate for tuff.acme.dev by itself.

Caddy with an authentication service

To sign people in with the organisation’s identity provider, run a forward-auth service such as oauth2-proxy and point Caddy at it:

tuff.acme.dev {
  @publish {
    method POST
    path /api/v1/reports
  }

  handle @publish {
    reverse_proxy 127.0.0.1:7474
  }

  handle {
    forward_auth 127.0.0.1:4180 {
      uri /oauth2/auth
    }
    reverse_proxy 127.0.0.1:7474
  }
}

The sign-in redirect and the identity provider are configured in the authentication service. See the Caddy documentation for forward_auth.

nginx

map "$request_method $uri" $tuff_realm {
    default                 "Tuff Console";
    "POST /api/v1/reports"  off;
}

server {
    listen 443 ssl;
    server_name tuff.acme.dev;

    ssl_certificate     /etc/letsencrypt/live/tuff.acme.dev/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/tuff.acme.dev/privkey.pem;

    client_max_body_size 16m;

    location / {
        auth_basic           $tuff_realm;
        auth_basic_user_file /etc/nginx/tuff.htpasswd;
        proxy_pass           http://127.0.0.1:7474;
    }
}

Create the password file with htpasswd -c /etc/nginx/tuff.htpasswd alice. The map turns basic auth off for POST /api/v1/reports, which goes straight to the console. Every other request needs the password.

Checking the proxy

With a key in TUFF_CONSOLE_KEY:

# No password: the proxy refuses
curl -i https://tuff.acme.dev/api/v1/projects

# A publish reaches the console, which checks the key itself
TUFF_CONSOLE_URL=https://tuff.acme.dev tuff console publish --dry-run
TUFF_CONSOLE_URL=https://tuff.acme.dev tuff console publish

TUFF_CONSOLE_URL must equal --public-url, apart from a trailing slash, for GitHub Actions publishing to work. The OIDC token names that address as its audience.

Keys on the host

Keys are managed with tuff console key, run on the host against the data folder. The commands work while the server runs, and a revoked key stops working on the next request.

sudo -u tuff tuff console key create billing-ci --repository github.com/acme/agents --data /var/lib/tuff-console
sudo -u tuff tuff console key list --data /var/lib/tuff-console
sudo -u tuff tuff console key revoke billing-ci --data /var/lib/tuff-console

Run the commands as the service user. A key created as root leaves root-owned files in the data folder that the service cannot write.

create prints the key once. The database holds only its SHA-256, so a lost key is replaced with a new one. Copy the key straight into the CI system’s secret store. A key created with --repository publishes reports for that repository only. A key without it publishes for any repository, so give each repository or team its own.

In a container, run the same commands with docker run --rm -v tuff-console-data:/data --entrypoint tuff ghcr.io/kannandreams/tuff-console:<version> console key ..., or with docker exec tuff-console tuff console key list.

Back up

The database grows by one row for each stored report, and each row holds the report’s JSON. A publish that changes only the commit, the branch, or the dirty flag updates the latest row, so a project that publishes on every push adds a row only when its capabilities, their status, or its policy gaps change. The console has no pruning in this release.

All state is console.sqlite. The database runs in WAL mode, so the folder can also hold console.sqlite-wal and console.sqlite-shm while the server runs. Copying console.sqlite alone while the server is running can miss recent writes.

Take a consistent copy with the SQLite shell, which works while the server runs:

sudo -u tuff mkdir -p /var/backups/tuff-console
sudo -u tuff sqlite3 /var/lib/tuff-console/console.sqlite \
  ".backup '/var/backups/tuff-console/console-$(date +%F).sqlite'"

Alternatively, stop the service and copy the folder. A stopped server leaves only console.sqlite.

sudo systemctl stop tuff-console
sudo cp -a /var/lib/tuff-console /var/backups/tuff-console-$(date +%F)
sudo systemctl start tuff-console

To restore, stop the service, put the backup at /var/lib/tuff-console/console.sqlite, delete any console.sqlite-wal and console.sqlite-shm next to it, make tuff the owner with mode 0600, and start the service. The backup contains the SHA-256 of every API key and the full text of every report, so protect it as the database itself.

Upgrade

Replace the tuff binary and restart the service. The console migrates the database when it opens it. The schema version lives in SQLite’s PRAGMA user_version, and each migration runs once, in a transaction.

A database that a newer tuff has migrated is refused by an older one:

error: /var/lib/tuff-console/console.sqlite has schema version 99, and this tuff reads up to 2
hint: upgrade tuff, or point --data at another folder

To go back to an older tuff, restore a backup taken before the upgrade. Take one before each upgrade.

The console is excluded from the 1.0 stability promise, so read the changelog before upgrading.

Try it first

tuff console serve --demo serves generated sample projects from memory and keeps nothing on disk. It is the quickest way to show the pages to the people who will use them before the real deployment exists.