Install Checkend

Note

Checkend is not released yet. The install commands use placeholders.

Checkend is one binary and one data folder on one server. This page installs the server with the installer or with Docker. It also shows automatic TLS, a reverse proxy and an offline install.

Requirements

• Operating system: Linux on amd64 or arm64, with systemd and glibc 2.36 or newer. Debian 12 and Ubuntu 24.04 or newer work. An older glibc (for example Ubuntu 22.04) does not work. Docker works on every system.
• DNS: a name (A and AAAA records) that points at the server, for example stats.example.com. Automatic TLS and passkeys need it. Without a domain, see Without a domain.
• Ports: 80 and 443 open, for automatic TLS.
• Disk: about 1 GB for the program files, plus the data. See Disks, folders and memory.

First downloads

At the first start, the server downloads two sets of files into the cache folder:

File	Size	Where it goes
The DuckDB extensions ducklake and sqlite	About 85 MB	<cache>/duckdb-extensions, or the folder of --duckdb-extensions
The DB-IP City Lite file (IP location)	About 60 MB compressed, 127 MB on the disk	<cache>/geo

The cache folder is <data>/cache by default. See Disks, folders and memory.

The GeoIP download runs in the background. For a server without network access, see Offline install.

The installer

1. Run the installer as root. Give your domain for automatic TLS:

   curl -fsSL [INSTALL_URL] | sudo sh -s -- --domain stats.example.com

2. Read the setup URL at the end of the output. It looks like https://stats.example.com/setup#cesetup_....

3. Open the setup URL in a browser, and make the first admin. See First boot.

Caution

Keep the setup URL private until you use it. Anybody with the URL can make the first admin. A restart of the server makes a new URL, and the old URL does not work after the restart.

What the installer does

Step	What it does
Platform	Finds the OS and the CPU. Linux amd64 and arm64 get the server. macOS gets the ce client only.
Download	Gets the release archive and checksums.txt.
Signature	Checks the minisign signature of checksums.txt with the release key in the script. It uses minisign, or OpenSSL 3 when minisign is not installed. A missing or bad signature stops the install.
Checksum	Compares the SHA-256 of the archive with checksums.txt.
Binaries	Puts checkend and ce in /usr/local/bin.
User	Makes the system user checkend without a login shell.
Data	Makes /var/lib/checkend (mode 0750, owner checkend).
Flags	Writes /etc/checkend/checkend.env one time (mode 0640) with CHECKEND_FLAGS.
Service	Writes /etc/systemd/system/checkend.service, then enables and starts it.
Setup URL	Prints the setup URL.

The service runs as the user checkend with a hardened systemd unit. It can open ports 80 and 443 without root.

Installer flags

Flag	Meaning
--domain <name>	First install only: automatic TLS for this name (ports 80 and 443).
--addr <address>	First install only, without --domain: the listen address. The default is 0.0.0.0:8080.
--data-dir <folder>	The data folder on a mounted volume, for example /mnt/HC_Volume_1234/checkend. A later run keeps it. See Disks, folders and memory.
--version <v>	Install this version. The default is the newest release.
--ce-only	Install only the ce client.
--no-start	Install, but do not start or restart the server.
--uninstall	Remove the binaries and the service. Keep the data and the flags file.
--purge	With --uninstall: also delete the data folder, /var/cache/checkend, /etc/checkend and the user checkend.
--unsigned	Do not check the signature. The checksum check stays.

Danger

Do not use --unsigned for a release that you downloaded. The signature is the only proof that the files come from the Checkend project.

The installer reads these environment variables:

Variable	Meaning
CHECKEND_RELEASE_URL	The URL of the release folder.
CHECKEND_MINISIGN_PUBKEY	Another public key, for a release that you signed yourself.
CHECKEND_SIGNATURE_TOOL	minisign or openssl.
CHECKEND_SETUP_WAIT	The seconds to wait for the setup URL. The default is 60.

The signature check needs minisign or OpenSSL 3. Debian 12 and Ubuntu 24.04 have OpenSSL 3. The openssl of macOS is LibreSSL and does not work: install minisign with Homebrew.

Change the server flags

The installer writes the flags file one time and never changes it again.

1. Edit CHECKEND_FLAGS in /etc/checkend/checkend.env.

2. Restart the service:

   sudo systemctl restart checkend

Without a domain

Without --domain, the server listens on 0.0.0.0:8080 with plain HTTP. The setup URL then starts with http://localhost:8080. Passkeys need HTTPS on a domain name, or localhost.

1. Open an SSH tunnel from your computer:

   ssh -L 8080:localhost:8080 you@server

2. Open the setup URL on your computer. The passkey then belongs to localhost.

On a private network with an IP address only, the setup page can make the admin with an API token. See First boot.

Caution

Do not open plain HTTP to the internet. The session cookie and the tokens then go over the network without encryption. Use --domain, or a reverse proxy with TLS.

Disks, folders and memory

Checkend uses two folders. You select them one time. Checkend checks your choice, but it never moves data by itself.

Folder	Flag	What it holds	If you lose it
The data folder	--data	app.sqlite (sites, people, tokens, goals, findings), lake.sqlite (the lake catalog), the Parquet files of the lake (without a bucket), the event log, secret.key, the certificates.	You lose your data. Keep it safe, and make backups.
The cache folder	--cache-dir (default <data>/cache)	The DuckDB temporary files, the DuckDB extensions, the GeoIP file.	Nothing. Checkend makes or downloads the files again.

A local disk or a volume

Most cloud servers have a local disk. You can also add a volume (a network block disk) to them.

• Local disk only. This is the default: /var/lib/checkend on the root disk. It is fast. When you delete or rebuild the server, the disk and the data go too. Make backups to another place.
• A volume for the data. A volume stays when you delete or rebuild the server, and you can attach it to a new server. It can be slower than the local disk. Put --data on the volume, and put --cache-dir on the local disk.

Example with a Hetzner Cloud server and a Hetzner Volume (Hetzner docs: Volumes):

1. Create the volume in the Hetzner Cloud Console. Select the option to format and mount it automatically.

2. Find the mount folder of the volume with df -h, for example /mnt/HC_Volume_1234.

3. Run the installer with the data folder on the volume:

   curl -fsSL [INSTALL_URL] | sudo sh -s -- --domain stats.example.com \
     --data-dir /mnt/HC_Volume_1234/checkend

4. Open /etc/fstab and find the line of the volume. Make sure that it has the option nofail.

Caution

Add nofail to the /etc/fstab line of the volume. Without it, the server does not boot when the volume is missing. The installer does not change /etc/fstab.

With --data-dir, the installer sets up the service in a different way:

• The installer refuses a --data-dir that is not on a mounted volume.
• The service waits for the mount at boot (RequiresMountsFor=).
• The server runs with --require-mount. It starts only when the data folder is on its own mounted file system.
• The cache folder is /var/cache/checkend on the local disk.

Thus Checkend never writes new data into an empty folder on the root disk when the volume is missing.

Caution

Do not use a network file system (NFS, SMB) for the data folder. SQLite and the event log need a local disk or a block volume. checkend doctor warns about it.

What checkend doctor checks for disks

Run checkend doctor --data <folder>, and add --cache-dir <folder> when the server uses one. For each folder, it shows:

• the file system,
• whether the folder is on its own mount,
• the free space,
• a short write test: 16 MiB with fsync, and 16 small writes. --no-speed-test turns the test off.

It warns in these cases:

• The data folder is on the root file system, and the server has a separate volume. This is likely wrong.
• A mount point of /etc/fstab that holds the data folder is not mounted. This is a failure.
• The data folder is on a network file system, or on the root file system of a container.
• The cache folder is less than half as fast as the data folder.

The health report of the running server (ce health, Settings → Health) shows the same disk checks without the write test. It also shows the DuckDB settings.

Memory

DuckDB gets half of the memory of the server, and one thread for each CPU. Checkend reads the limits of its cgroup. Thus a Docker limit (--memory, --cpus) or a systemd MemoryMax= counts. The other half is for the Checkend process, the event log, SQLite, the GeoIP file and the file cache of the system. A query that needs more memory writes temporary files to <cache>/duckdb-tmp. These files use at most a quarter of the free space of the cache disk.

Flag	Meaning
--duckdb-memory	A size (1.5GiB) or a percent of the memory (40%, at most 90 %). The default is 50 %. The value is never less than 256 MiB.
--duckdb-threads	The count of DuckDB threads. The default is one for each CPU.

The server log line duckdb settings and ce health show the values at each start.

Docker

The image has checkend and ce. It runs as the user 65532 (nonroot), keeps the data in the volume /var/lib/checkend, and has a health check. By default, the container listens on port 8080.

Note

No image is published yet. The name ghcr.io/checkend/checkend is the planned name.

1. Start the container:

   docker run -d --name checkend -p 8080:8080 -v checkend:/var/lib/checkend ghcr.io/checkend/checkend

2. Find the setup URL in the log:

   docker logs checkend 2>&1 | grep setup

Put a reverse proxy with TLS in front of port 8080. See Behind a reverse proxy.

Docker Compose with automatic TLS

This example publishes ports 80 and 443 and gets a certificate for stats.example.com:

services:
  checkend:
    image: ghcr.io/checkend/checkend
    restart: unless-stopped
    command:
      - serve
      - --data
      - /var/lib/checkend
      - --duckdb-extensions
      - /var/lib/checkend/duckdb-extensions
      - --domain
      - stats.example.com
      # The health check of the image calls http://127.0.0.1:8080/health.
      - --addr
      - 127.0.0.1:8080
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - checkend:/var/lib/checkend
    stop_grace_period: 60s

volumes:
  checkend:

1. Point the DNS name at the server. Open ports 80 and 443.

2. Save the example as compose.yaml, and change stats.example.com to your name.

3. Start the service:

   docker compose up -d

4. Find the setup URL:

   docker compose logs checkend | grep setup

The certificates, the data, the DuckDB extensions and the GeoIP file stay in the volume.

Automatic TLS

The flag --domain stats.example.com turns on automatic TLS with ACME. The default ACME server is Let’s Encrypt. For more names, give a comma-separated list. The first name is the public URL.

With --domain, the server listens on two ports:

• Port 443 serves HTTPS and answers the TLS-ALPN-01 challenge.
• Port 80 answers the HTTP-01 challenge and sends all other requests to HTTPS.

The first HTTPS request for a name gets the certificate. The certificates and the ACME account key are in <data>/certs (mode 0700). The renewal starts 30 days before the end date.

Flag	Meaning
--domain	The names of the certificate. The first name is the public URL.
--https-addr	The HTTPS listen address. The default is :443.
--http-addr	The HTTP listen address. The default is :80. --http-addr - turns port 80 off: then only TLS-ALPN-01 works.
--acme-email	The contact address of the ACME account (optional).
--acme-directory	The ACME directory URL. The default is Let’s Encrypt.
--acme-ca-root	A PEM file of extra root certificates for the ACME server (a private CA).
--addr	With --domain: a plain listener that serves only /health, for a health check.

Caution

Test a new setup with the Let’s Encrypt staging directory: --acme-directory https://acme-staging-v02.api.letsencrypt.org/directory. Let’s Encrypt has rate limits for each name.

Each answer over TLS has the header Strict-Transport-Security: max-age=31536000 (one year).

Behind a reverse proxy

Run Checkend with plain HTTP on a local address, and let the proxy do TLS:

checkend serve --data /var/lib/checkend --addr 127.0.0.1:8080 \
  --public-url https://stats.example.com --trusted-proxies 127.0.0.1

• --public-url is the URL that browsers use. The script tag, the setup URL, the passkeys, the MCP metadata and the Secure flag of the session cookie come from it.
• --trusted-proxies lists the proxy addresses or CIDR ranges. Checkend reads X-Forwarded-For only from these addresses. It reads the header from the right to the left, and takes the first address that is not a trusted proxy.

Caution

Set --trusted-proxies behind a proxy. Without it, each visitor has the address of the proxy. Standard mode then counts one visitor, and the countries are wrong.

Caddy:

stats.example.com {
  reverse_proxy 127.0.0.1:8080
}

Nginx (with a certificate from your own tool):

server {
    listen 443 ssl;
    server_name stats.example.com;
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Behind a proxy with TLS, the proxy sets HSTS. Checkend does not, because its own answers are plain HTTP.

Offline install

Use these steps for a server without network access.

1. Copy the release archive, checksums.txt and checksums.txt.minisig to a folder on the server.
2. Serve the folder, for example with python3 -m http.server 8000.
3. Run the installer with CHECKEND_RELEASE_URL=http://127.0.0.1:8000.
4. Copy the DuckDB extensions ducklake and sqlite for the platform of the server from a computer with network access. They are in ~/.duckdb/extensions/<DuckDB version>/<platform>/. With --lake-bucket, also copy httpfs.
5. Put them in the folder of --duckdb-extensions in the service, under <DuckDB version>/<platform>/. This folder is /var/lib/checkend/duckdb-extensions, or /var/cache/checkend/duckdb-extensions with --data-dir.
6. Copy the DB-IP City Lite file to the server.
7. Add --geoip-file <file> to CHECKEND_FLAGS.

To run without location data, add --geoip-download=false in place of step 7. Events then have no location.

The flag --duckdb-extensions <folder> selects the folder of the DuckDB extensions. The installer sets it to /var/lib/checkend/duckdb-extensions. With --data-dir, it sets it to /var/cache/checkend/duckdb-extensions.

Check the install

Run checkend doctor on the server:

checkend doctor --data /var/lib/checkend --public-url https://stats.example.com

It checks the disk, the data folder and secret.key, the data and cache disks, the lake files, Litestream, the GeoIP file, the DuckDB extensions, the backups, the jobs and the certificates. Give it the same --cache-dir and --lake-bucket as the server. With --public-url, it also gets <public URL>/health as a browser does, and shows the end date of the certificate.

See Troubleshooting for the other checks.

Uninstall

Remove the binaries and the service. The data and the flags file stay:

curl -fsSL [INSTALL_URL] | sudo sh -s -- --uninstall

Add --purge to also delete the data folder, /var/cache/checkend, /etc/checkend and the user checkend.

Danger

Make a backup before you use --purge. It deletes all data and secret.key.

Next step

First boot: make the first admin.