Notice Deduplication

Prevent database flooding during error loops by deduplicating identical notices within a configurable time window.

Overview

When your application encounters an error loop (e.g., a bug in a frequently-called endpoint), it can generate thousands of identical error reports in seconds. Notice deduplication protects your Checkend instance by:

  • Detecting identical errors within a configurable time window (default: 60 seconds)
  • Incrementing an occurrence counter instead of creating duplicate notice records
  • Still tracking total occurrences for accurate error frequency metrics
  • Skipping redundant notifications for duplicate errors

How It Works

Deduplication uses a composite key to identify identical errors:

  • App ID — Errors are deduplicated per app
  • Problem Fingerprint — Based on error class and location
  • Backtrace Fingerprint — Distinguishes different call paths to the same error

Deduplication Flow

1 Error arrives at ingestion endpoint
2 Calculate fingerprints (problem + backtrace)
3 Check cache for recent identical error
First occurrence: Create notice, send notifications
Duplicate in window: Increment counter, skip notice creation

Configuration

Deduplication is enabled by default. You can configure it using environment variables:

Variable Default Description
NOTICE_DEDUP_ENABLED true Enable or disable deduplication
NOTICE_DEDUP_WINDOW_SECONDS 60 Time window in seconds for deduplication
Example: Increase window to 5 minutes
export NOTICE_DEDUP_WINDOW_SECONDS=300

Disabling Deduplication

While you can disable deduplication with NOTICE_DEDUP_ENABLED=false, this is not recommended in production as error loops can quickly fill your database and degrade performance.

API Response

The errors endpoint response includes deduplication information:

First Occurrence (201 Created)

Response
{
  "id": 12345,
  "problem_id": 789,
  "deduplicated": false,
  "occurrence_count": 1
}

Duplicate Within Window (200 OK)

Note: No id field since no notice record was created.

Response
{
  "problem_id": 789,
  "deduplicated": true,
  "occurrence_count": 5
}

Tracking Occurrences

Even when notices are deduplicated, Checkend tracks the total number of error occurrences:

  • notices_count — Number of full notice records created
  • deduplicated_count — Number of deduplicated occurrences (no notice created)
  • total_occurrences — Sum of both counts

The last_noticed_at timestamp is updated for both regular and deduplicated notices, ensuring accurate "last seen" information.

Behavior Details

Scenario Behavior
Same error, different backtrace Not deduplicated (different call path)
Same error, different app Not deduplicated (separate apps)
Duplicate on resolved problem Problem is auto-unresolved
Notifications for duplicates Skipped (only first occurrence notifies)
After window expires New notice created (window resets)

Best Practices

The default 60-second window balances deduplication effectiveness with data granularity. If you're investigating time-sensitive issues, you may want to temporarily reduce the window. For high-volume applications, consider increasing it to 5 minutes (300 seconds).