Source verification methodology

The core rule

Every metric on this site has a citation, or it does not exist. A "metric" is anything a reader could act on or be misled by: pricing, context window, max output, modality, release/snapshot dates, knowledge cutoff, lifecycle, benchmark scores, latency, uptime, regions, features. The type system enforces this — metric fields are typed MaybeVerified<T> = VerifiedField<T> | null; a non-null value cannot exist without a SourceCitation attached to it.

What counts as a primary source

The allow-list is intentionally narrow:

• official-vendor-docs — the provider's own technical documentation hosted on a provider-controlled domain.
• official-vendor-pricing — the provider's own pricing reference, on a provider-controlled domain, in a stable URL.
• official-vendor-site — the provider's own marketing surface, used only when the relevant fact is structurally bound there (e.g. headquarters, status page URL).
• regulatory-filing — an authoritative regulator's record.
• research-paper — a peer-reviewed paper or a preprint on a recognised archive, used for factual claims about model architecture or training data when the provider has published them.
• public-dataset — a publicly-versioned dataset used as a benchmark reference.

Blog posts that are not on the provider's primary docs surface, social media, leaderboard sites, and AI-generated summaries are not primary sources. WebSearch tool output is also not a primary source — it is an AI-generated summary of pages we cannot verify directly, and the verification workflow forbids it.

The VerifiedField shape

Every verified value is wrapped via the verified() helper, which constructs:

interface VerifiedField<T> {
  value: T;
  citation: {
    url: string;            // absolute URL
    name: string;           // human-friendly source name
    type: SourceType;       // allow-listed
    retrievedAt: string;    // ISO-8601 datetime
    notes?: string;         // what was actually used from the page
  };
  confidenceLevel: 'high' | 'medium' | 'low' | 'unverified';
  notes?: string;
}

The constructor throws at build time if url, name, or retrievedAt is missing. There is no escape hatch — a metric without a citation cannot ship.

Verification states

• verified — every metric on the entity is wrapped in a VerifiedField with a current citation.
• partial — some metrics are verified, others are null. The partial-verification badge appears on entity pages with this state.
• unverified — entity identity is known but no metric has been confirmed. Common for entries blocked by 403/JS-only rendering.

Retrieval cadence

Pricing values: every 30 days, and immediately on any vendor pricing announcement. Lifecycle (deprecation, retirement): every 30 days plus on every published deprecation notice. Context window / max output / modality: every 90 days; these change less frequently. Benchmark scores: only when a new primary-source publication lands. Latency / uptime: continuously, once an independent monitor is wired; until then, null.

Each entity records a lastCheckedAt timestamp documenting the most recent full sweep. A field whose citation retrievedAt is older than the cadence above should either be re-verified or dropped back to null.

Freshness and the reverification queue

Sprint 21 added a source-freshness model on top of the retrieval cadence above. Every record carries a freshness state — fresh, review_due, stale, blocked, or unknown — computed deterministically against siteConfig.buildDate. Pricing rows use the shorter cadence (14d / 30d / 45d); general citations use the standard cadence (30d / 60d / 90d).

Records that age into review_due or stale appear on the /reverification queue (machine-readable at /api/reverification) with the source URL, the affected routes, and a suggested manual action. The catalogue never auto-fetches or auto-mutates verified values; the queue is an informational nudge for a human reviewer.

The workflow:

1. Pick the highest-priority queue item (critical > high > medium > low; pricing reviews rank higher than docs).
2. Open the source URL in a real browser. Confirm the value on the vendor page.
3. Update the on-disk record in data/. Stamp retrievedAt / lastCheckedAt with the date of the manual review.
4. Re-run npm run check:production. The queue refreshes on the next build.

A stale row is not asserted as wrong; it is asserted as unconfirmed since the last review. The renderer keeps showing the value (it was verified at the time) but pairs it with a chip so any reader can see how recently it was checked.

Blocked retrievals are recorded too

When an official documentation page cannot be retrieved (HTTP 403, JS-only rendering, redirect loop), the attempt itself is recorded in verification-attempts.ts with the URL, the date, the outcome category, and a free-text note. The audit log at /coverage surfaces every attempt — including the OpenAI 403s — so readers can see exactly which gaps exist and why.

JSON-LD exclusion policy

schema.org markup is generated per page from the verified fields only. The model JSON-LD helper at lib/model-jsonld.ts uses isVerified() to gate every metric and an integrity guard refuses to ship a build that emits unverified pricing, benchmark, latency, or uptime properties in JSON-LD. Search engines and AI surfaces never see an estimate from this site.

Allowed vs rejected source types

Manual verification workflow

For providers that block automated retrieval, the manual workflow is documented in VERIFICATION.md: open the source in a real browser, capture each URL and the retrieval timestamp, work through the field-by-field checklist, and encode each fact via the verified(value, citation, notes) call with the new citation added to data/citations.ts. Build validation runs the integrity guard suite end-to-end before deploy; nothing without a citation ships.