Troubleshooting

Checkend has three health checks. Start with the check that fits your access.

Check	Where it runs	What it checks
checkend doctor	On the server	The files: disk, data folder, secret.key, data and cache disks, lake files, Litestream, GeoIP, DuckDB extensions, backups, jobs, certificates.
ce health	Anywhere, admin token	The running server: also the event log and the writer lag.
checkend healthcheck	On the server, or in a container	Only that /health answers.

checkend doctor

Run it on the server:

checkend doctor --data /var/lib/checkend

Give it the same --cache-dir and --lake-bucket as the server:

checkend doctor --data /var/lib/checkend --cache-dir /var/cache/checkend \
  --lake-bucket s3://<bucket>/checkend

It reads the files of the data folder directly. See What checkend doctor checks for disks for the disk warnings. It checks:

• the disk use,
• the data folder and secret.key (mode 0600),
• the data and cache disks: the file system, the own mount, the free space and a short write test (--no-speed-test turns the test off),
• the live lake files in the lake store (lake_files),
• Litestream: with the lake on a bucket, a warning when Litestream does not copy lake.sqlite,
• the GeoIP file,
• the DuckDB extensions,
• the backups (add --backup-dir <folder> to check a backup folder),
• the newest run and error of each job,
• the certificates of --domain: a warning under 14 days, a failure when a certificate has expired.

With --public-url, it also gets <public URL>/health as a browser does, and shows the end date of the certificate:

checkend doctor --data /var/lib/checkend --public-url https://stats.example.com

ce health

ce health

ce health needs an admin token. It shows the disk use and the disk guard, the data and cache disks without the write test, the DuckDB settings, the data folder and secret.key, the GeoIP file, the DuckDB extensions, the event log and the writer lag, the backups, and the newest run and error of each job. The web UI shows the same report in Settings → Health.

It exits with 0 when the status is ok or warn, and with 1 when the status is fail. A monitor can run it.

checkend healthcheck

checkend healthcheck --url http://127.0.0.1:8080/health

It exits with 0 when the server answers. The Docker image uses it. /health is a small public check that answers {"status": "ok"}.

Exit codes of ce

Code	Meaning	What to do
0	OK.	Read stdout.
1	Other error: a bad flag or value, a conflict, a server error. Also ce health with the status fail.	Read the error message.
2	Not authenticated: no token, a token that is not valid, or no server.	Run ce login <url>.
3	Not found.	Run ce sites list, or the next command of the error.
4	No data yet. stdout has the empty answer.	Run ce sites verify <site>.
5	Forbidden: the token is valid, but it has no scope or role for the action. Also a wrong or used setup token.	Ask an admin for a token with the scope.

On a pipe, or with --json, each error is one JSON object on stderr:

{"error": "not_authenticated", "message": "...", "next": "ce login https://stats.example.com"}

error	Exit code	When
usage	1	A bad flag, argument or value, an unknown command, or no --yes without a terminal.
invalid_input	1	The server refused a value (HTTP 400).
conflict	1	The action is not allowed now (HTTP 409).
rate_limited	1	HTTP 429.
server_error	1	HTTP 5xx.
connection_failed	1	No answer from the server.
health_fail	1	ce health: the status is fail.
credentials	1	The config file, the keychain or the credentials file failed.
error	1	All other errors, for example a local file that is not there.
not_authenticated	2	HTTP 401.
no_server	2	No context and no CHECKEND_URL.
not_found	3	HTTP 404.
no_data	4	The answer is empty.
forbidden	5	HTTP 403.

Common problems

ce says “no_server”

ce has no default server. Log in, or set the URL:

ce login https://stats.example.com

In CI, set CHECKEND_URL and CHECKEND_TOKEN.

ce refuses the credentials file

ce refuses to read the credentials file when other users can read it. Set the mode:

chmod 600 ~/.config/checkend/credentials

The passkey does not work

Passkeys need HTTPS on a domain name, or localhost.

• On 127.0.0.1, open the URL with localhost.
• On a remote server without a domain, use an SSH tunnel: ssh -L 8080:localhost:8080 you@server.
• On an IP address only, use the API token way of the setup page or the invite page.

The setup URL does not work

A restart of the server makes a new setup URL. Read the new URL in <data>/setup-url, in the server log, or with docker logs checkend 2>&1 | grep setup. After the first admin exists, the setup URL does not work again.

No events arrive

1. Check the install:

   ce sites verify example.com --send-test

2. If the test event arrives, but page views do not, check the script tag on the site. The data-site value must be the site domain.

3. Check that the page URL is on the site domain or a subdomain. Checkend refuses events for other hosts.

4. Check that the visitor is not on a hosting network. The data-center filter drops these events.

5. Check the Content-Security-Policy of the site. Add your Checkend URL to script-src and connect-src.

All visitors have one address, or the countries are wrong

The server is behind a proxy, but --trusted-proxies is not set. Add the proxy address to --trusted-proxies. With the first-party proxy route, also add the address of the site’s proxy. See Install Checkend.

An MCP client refuses the server metadata

Set --public-url to the URL that the MCP client uses. The issuer and the metadata URLs come from it.

Webhook deliveries fail

1. Read the delivery log:

   ce webhooks deliveries <id>

2. Check that the receiver answers with a 2xx status.

3. Check that the URL is not on a private address. Checkend refuses private targets unless the server runs with --webhook-allow-private.

4. After a restore with another secret.key, delete the webhook and add it again.

The Search Console sync shows an error

Run ce search status example.com. It shows the error and the next step. Check that the redirect URI of your OAuth client is <public URL>/v1/search/oauth/callback.

The disk is full

The disk guard deletes the oldest raw months above the disk limit (90 % by default). Check ce health, then lower the retention or add disk space. See Backups and upgrades.

The server does not start: the data folder is not on its own mount

The server runs with --require-mount, and the data folder is not on its own mounted file system. The installer sets this flag with --data-dir. Usually the volume is not mounted.

1. Run df -h, and look for the mount folder of the volume.
2. Mount the volume, or attach it to the server again.
3. Start the server: sudo systemctl start checkend.
4. Run checkend doctor. The check data_folder must be OK.

See Disks, folders and memory.

The server does not start: the lake store has no data files

The lake catalog names data files, but the lake store has none of them. The cause is a wrong --lake-bucket, or a missing checkend lake push. The message tells you what to do. See The lake on the bucket.

The first start is slow

The first start downloads the DuckDB extensions (about 85 MB) and the DB-IP file (about 60 MB compressed). For a server without network access, see Offline install.