# Search Console and Bing

> Connect Google Search Console with your own OAuth client, connect Bing Webmaster Tools, and read search data with visits and conversions.

Checkend joins Google Search Console and Bing Webmaster Tools data with your page views and conversions. The search rules use this data to make findings, for example a query on position 8 to 20.

## Connect Google Search Console

Checkend uses the OAuth client of your own Google Cloud project. Checkend never ships a shared OAuth client.

### Make the OAuth client

1. In the Google Cloud console, make a project.
2. Turn on the "Google Search Console API".
3. Set up the OAuth consent screen. Add the scope `https://www.googleapis.com/auth/webmasters.readonly`.
4. Make an OAuth client of the type "Web application".
5. Add the redirect URI `<public URL>/v1/search/oauth/callback`, for example `https://stats.example.com/v1/search/oauth/callback`.

`ce search status <site>` also shows the redirect URI.

### Give the client to the server

Start the server with the client ID as a flag and the client secret as an environment variable:

```sh
CHECKEND_GOOGLE_CLIENT_SECRET=<client secret> \
  checkend serve --google-client-id <client ID>.apps.googleusercontent.com
```

With the installer, put both in `/etc/checkend/checkend.env`, then restart the service.

> **Caution**
> Keep the client secret out of the command line. Use the environment variable, so that the secret is not in the process list.

### Connect a site

1. Run the connect command:

   ```sh
   ce search connect example.com --google
   ```

   Or select **Connect Search Console** on the Search tab, or use the MCP tool `connect_search_console`.

2. Open the consent page that Checkend prints. Use a Google account with access to the property.
3. Approve the access within 15 minutes.

The browser goes back to the Search tab.

The property is a domain property by default (`sc-domain:example.com`). For a URL-prefix property, use `--property`:

```sh
ce search connect example.com --google --property https://example.com/
```

The property must cover the site.

## Connect Bing Webmaster Tools

1. In Bing Webmaster Tools, open **Settings → API Access**, and copy the API key.
2. Give the key to Checkend on stdin:

   ```sh
   echo "$BING_KEY" | ce search connect example.com --bing --api-key-stdin
   ```

Checkend checks the key. The site must be a verified site of the key's user.

> **Caution**
> Keep the Bing API key safe. It gives access to all sites of its user.

## How Checkend keeps the tokens

Checkend seals the Google refresh token and the Bing API key with AES-256-GCM. The key is the file `<data>/secret.key`. A copy of the database without the key file does not give the tokens.

> **Danger**
> Back up `secret.key` with the data folder. Without it, you must connect each source again.

## The sync

A job syncs the search data each hour. Change the interval with `--search-every`.

- **Google:** the newest day is today minus 2 days (UTC). Each run gets the newest 4 days again, because Google changes recent data. The first run gets the newest 7 days. Then the backfill goes back 31 days in each run, until it reaches 16 months.
- **Bing:** Bing gives query rows and page rows, without a date range. Bing updates its data each week.

Checkend keeps the search rows permanently. The raw-event retention does not delete them. So you keep the history after the 16 months that Google gives.

After an error, the sync waits and tries again. The connection shows the error and the next step.

Run a sync now, or check the status:

```sh
ce search sync example.com
ce search status example.com
```

### Disconnect

```sh
ce search disconnect example.com google --yes
```

Disconnect deletes the stored token and revokes it at Google. The synced rows stay.

## Read the search data

```sh
ce search example.com --period 28d --by query
ce search example.com --by page
ce search example.com --by query --position 8-20
```

The answer has clicks, impressions, CTR and the average position, by query or by page. Page rows also have the Checkend page views, visitors, page views from Organic Search, the visitors who completed a goal, and the current title.

| Flag | Values |
|---|---|
| `--period` | `7d`, `28d` (default), `90d`, `365d`. The period ends at the newest search date. |
| `--by` | `query` or `page` |
| `--position` | A range, for example `8-20` for striking-distance rows |
| `--source` | `google`, `bing` or `all` |

The MCP tool is `search_performance`. The web UI has a Search tab.

### Limits of the search data

- Search Console data is 2 to 3 days late.
- Search Console gives at most 50,000 rows a day for each search type, and it leaves out rare queries. So the query totals are not equal to the page totals.
- Exit code 4 means that the site has no search data yet.

## The page crawler

A small crawler stores the title, meta description, H1, canonical URL and indexability of each page with traffic or search impressions. An agent needs this to propose a better title.

```sh
ce pages example.com --path /pricing
```

The MCP tool is `get_page`.

- The crawler runs each hour (`--crawl-every`). It crawls at most 100 pages of a site in each run, and a page again after 7 days.
- It obeys `robots.txt`. It reads the `CheckendBot` group, else `*`. When `robots.txt` gives a server error, it does not crawl.
- It waits 2 seconds between two requests to a site. It reads at most 2 MB of a page.
- Its user agent is `Mozilla/5.0 (compatible; CheckendBot/1.0; +<public URL>)`.
- It does not follow redirects. It stores a redirect as "redirects to <location>".
- It refuses loopback, private and special addresses. `--crawler-allow-private` turns this protection off.

A page is indexable when it has status 200, is an HTML page, has no `noindex`, and has a canonical URL that is the page itself.

## URL inspection

With a Search Console connection, Checkend also inspects the top pages with the URL Inspection API. It uses at most 50 inspections a day for each site. A failed inspection makes an `index_problem` finding. `ce pages` and `get_page` show the newest inspection.

## Search findings

The search data feeds these rules: `striking_distance_query`, `low_ctr_for_position`, `content_decay`, `cannibalization`, `impressions_without_visits`, `index_problem` and `slow_page_losing_position`. See [The closed loop](/docs/concepts/closed-loop.md#the-rules).

## Prompt for a coding agent

```text
Find the pages of example.com that are close to the first search page. Replace example.com with my site.
1. Run `ce search status example.com`. If Google is not connected, tell me to run `ce search connect example.com --google`, and stop.
2. Run `ce search example.com --period 28d --by query --position 8-20`.
3. List the 10 queries with the most impressions, with their page, position and clicks.
4. For each page, suggest a better title and meta description from the code of this repository.
Do not change files yet. Wait for my choice.
```