Backups and upgrades

Checkend keeps all data in one data folder, for example /var/lib/checkend. This page shows how to back it up, restore it, limit its size and upgrade the server.

Danger

Do not lose <data>/secret.key. It opens the stored Search Console and Bing tokens and the webhook secrets. Without it, you must connect each search source and add each webhook again.

What a backup contains

• app.sqlite (the metadata) and lake.sqlite (the lake catalog), as consistent copies,
• the Parquet files of the lake (with the lake on a bucket, the backup reads them from the bucket),
• secret.key,
• manifest.json, with the size and the SHA-256 of each file.

The event log is not in a backup. The events that were not yet in the lake at the time of the backup are lost after a restore.

Caution

Keep a backup as safe as the data folder. It contains secret.key.

Make a backup by hand

A backup works while the server runs:

checkend backup create --data /var/lib/checkend --to /backups/checkend

With the lake on a bucket, add the same --lake-bucket as the server:

checkend backup create --data /var/lib/checkend --to /backups/checkend \
  --lake-bucket s3://<bucket>/checkend

The scheduled backup uses the --lake-bucket flag of the server.

List the backups in a folder:

checkend backup list --from /backups/checkend

Scheduled backups

Add these flags to CHECKEND_FLAGS in /etc/checkend/checkend.env:

--backup-dir /backups/checkend --backup-every 24h --backup-keep 7

Flag	Meaning	Default
--backup-dir	A folder, or s3://<bucket>/<prefix>. Without it, there is no scheduled backup.	None
--backup-every	The interval.	24 hours
--backup-keep	The count of backups to keep.	7

With the installer, the systemd service can write only to /var/lib/checkend. Add the backup folder to ReadWritePaths:

1. Open an override file:

   sudo systemctl edit checkend

2. Add these lines:

   [Service]
   ReadWritePaths=/backups/checkend

3. Restart the service:

   sudo systemctl restart checkend

The bucket

An S3-compatible bucket can hold the backups and the Parquet files of the lake. Examples are AWS S3, MinIO, Cloudflare R2 and Hetzner Object Storage. The bucket must exist. Make it in the console of your provider.

Set the bucket access in the environment of the server. For the service, put the variables in /etc/checkend/checkend.env:

Variable	Meaning	Default
CHECKEND_S3_ENDPOINT	The endpoint URL.	https://s3.<region>.amazonaws.com
CHECKEND_S3_REGION	The region.	us-east-1
CHECKEND_S3_ACCESS_KEY_ID	The access key ID.	None
CHECKEND_S3_SECRET_ACCESS_KEY	The secret access key.	None
CHECKEND_S3_URL_STYLE	path, or vhost for the bucket in the host name.	path

Checkend uses Signature Version 4. An example for Hetzner Object Storage in fsn1:

CHECKEND_S3_ENDPOINT=https://fsn1.your-objectstorage.com
CHECKEND_S3_REGION=fsn1
CHECKEND_S3_ACCESS_KEY_ID=...
CHECKEND_S3_SECRET_ACCESS_KEY=...
CHECKEND_S3_URL_STYLE=vhost

Hetzner documents the endpoint https://<location>.your-objectstorage.com and URLs with the bucket in the host name.

Note

Checkend was tested only with MinIO and path-style URLs. It was not tested with Hetzner or with other providers.

Backups in the bucket

Use s3://<bucket>/<prefix> in place of a folder, in --backup-dir, --to and --from:

checkend backup create --data /var/lib/checkend --to s3://<bucket>/backups

The lake on the bucket

The Parquet files of the lake can go to the bucket. The lake catalog (lake.sqlite) and the metadata (app.sqlite) stay in the data folder.

1. Add --lake-bucket s3://<bucket>/checkend to CHECKEND_FLAGS in /etc/checkend/checkend.env.

2. For a server that already has data, stop the server:

   sudo systemctl stop checkend

3. Copy the Parquet files to the bucket. A second run copies only the new files.

   sudo -u checkend env $(grep CHECKEND_S3 /etc/checkend/checkend.env | xargs) \
     checkend lake push --data /var/lib/checkend --to s3://<bucket>/checkend

4. Start the server:

   sudo systemctl start checkend

5. Run checkend doctor. The check lake_files must be OK.

   checkend doctor --data /var/lib/checkend --lake-bucket s3://<bucket>/checkend

6. When the server works on the bucket, delete /var/lib/checkend/objects.

With --lake-bucket, the first start also downloads the DuckDB extension httpfs. DuckDB keeps the bucket key in memory only. The writer sends new data to the bucket about every 5 seconds. The compaction job merges the small files each hour. The retention and the compaction also work on the bucket.

The server does not start when the catalog names data files and the lake store has none of them. The cause is a wrong --lake-bucket, or a missing checkend lake push. The message tells you what to do.

Caution

Do not give the same --lake-bucket prefix to two servers. Each server has its own catalog, and the cleanup of one server deletes the files of the other.

Litestream

Litestream copies app.sqlite and lake.sqlite to the bucket each second. It is optional, and Checkend does not start it. Checkend was tested with Litestream v0.5.17.

The Checkend source has an example configuration, deploy/litestream.yml. It takes its values from the environment. Copy it to the server, and give its path to -config:

CHECKEND_DATA=/var/lib/checkend CHECKEND_LITESTREAM_BUCKET=<bucket> \
CHECKEND_LITESTREAM_PREFIX=checkend/litestream \
CHECKEND_S3_ENDPOINT=... CHECKEND_S3_REGION=... CHECKEND_S3_ACCESS_KEY_ID=... CHECKEND_S3_SECRET_ACCESS_KEY=... \
  litestream replicate -config litestream.yml

Run Litestream as the user checkend, as its own systemd service. checkend doctor shows a litestream check. With the lake on a bucket, it warns when Litestream does not copy lake.sqlite.

Caution

Keep secret.key and a scheduled backup too. Litestream does not copy secret.key. Without it, the sealed tokens and webhook secrets do not open.

Restore from the bucket

Use these steps on a new server with the same bucket.

1. Install Checkend with --no-start.

2. Copy secret.key from a backup into the new data folder. Give it mode 0600 and the owner checkend.

3. Restore the two SQLite files into the data folder:

   • With Litestream:

     litestream restore -config litestream.yml -o /var/lib/checkend/app.sqlite /var/lib/checkend/app.sqlite
     litestream restore -config litestream.yml -o /var/lib/checkend/lake.sqlite /var/lib/checkend/lake.sqlite

   • Without Litestream, restore the newest backup. The restore writes a Parquet file to the bucket only when the bucket does not have it.

     checkend backup restore --from s3://<bucket>/backups --data /var/lib/checkend \
       --lake-bucket s3://<bucket>/checkend

4. Add the same --lake-bucket to CHECKEND_FLAGS.

5. Start the server.

6. Run checkend doctor. The checks lake_files and data_folder must be OK.

The events of the last seconds before the loss are lost. They were in the event log, and not yet in the lake.

Restore a backup

1. Stop the server:

   sudo systemctl stop checkend

2. Restore the newest backup:

   checkend backup restore --from /backups/checkend --data /var/lib/checkend

   To restore an older backup, add --id <id>. Get the ID from checkend backup list. With the lake on a bucket, add --lake-bucket. The restore then writes only the Parquet files that the bucket does not have.

3. Start the server:

   sudo systemctl start checkend

The restore checks every file before it writes. A data folder that already has data needs --force. The old data then moves to a folder next to it.

Caution

Make sure that the user checkend owns the restored files before you start the server. A restore as root can leave files that the server cannot write:

sudo chown -R checkend:checkend /var/lib/checkend

Retention

Checkend keeps the raw events for 24 months by default. A nightly job deletes the older months as whole months.

• The daily rollups stay permanently, so the numbers of old days stay.
• The search rows stay permanently.
• After a month is deleted, a visitor counts one time on each day of that month.

Show and change the retention (admin):

ce retention
ce retention set analytics --months 18 --dry-run
ce retention set analytics --months 18

The disk guard

The disk guard protects the server from a full disk. It runs every 10 minutes.

When the disk use goes above the limit, it deletes the oldest raw month first. It never deletes the current or the previous month. The rollups stay.

Flag	Meaning	Default
--disk-limit	The disk use in percent above which the guard deletes. 0 turns it off.	90
--disk-budget	The raw-data budget of a module, for example analytics=50GiB.	None
--disk-guard-every	The interval.	10 minutes

The disk guard tells you in three ways:

• an audit entry data.delete_raw for each deleted month,
• the webhook event disk.pressure,
• a failed disk check in ce health and checkend doctor.

Upgrades

With the installer

1. Make a backup.

2. Run the installer again:

   curl -fsSL [INSTALL_URL] | sudo sh

   It downloads the new version only when the version is different. It replaces the binaries and restarts the server.

3. Check the server:

   checkend doctor --data /var/lib/checkend --public-url https://stats.example.com

With Docker

1. Make a backup.
2. Pull the new image.
3. Start a new container on the same volume.

Migrations

The new binary runs the pending migrations at start. You never run a migration by hand. Migrations only go forward.

Before it migrates app.sqlite, Checkend copies the file to app.sqlite.before-migration-<n> (mode 0600). This is the rollback copy.

Caution

Delete app.sqlite.before-migration-<n> when the new version works. The copy can hold old secrets in clear, for example the webhook secrets of an older version.

Check the health

After a backup, a restore or an upgrade, run the checks in Troubleshooting.