Releases and the CI check

1. Tag runs with the release

Every span carries the release of the code that produced it. Pass it to instrument(), or let the SDK read LANTERNFISH_RELEASE or your CI's commit variable (GITHUB_SHA and others; see Configuration).

from lanternfish import instrument

instrument(
    api_key="YOUR_API_KEY",
    endpoint="https://your-lanternfish-api",
    release="v1.8.2",
)

2. What is compared

The latest release is compared with the release before it, using runs from the last 30 days. Each side needs at least 30 runs. Quality means run duration, success rate and cost per run; behavior means how often each tool is called. A difference counts only when it is outside normal variation at 95% confidence.

The Releases page in the app shows the verdict, every finding with its numbers, and the runs behind it. When the latest release regresses, owners and admins of the organization get one email about it, and the same message is posted once to each channel in Settings → Notifications.

3. Check a release from the command line

The SDK installs a lanternfish command. It reads LANTERNFISH_API_KEY and LANTERNFISH_ENDPOINT.

lanternfish verdict --release v1.8.2

# also stop on a behavior change, and when there are too few runs to judge
lanternfish verdict --release v1.8.2 --fail-on changed --require-data

# compare only runs from one environment, e.g. your test suite in CI
lanternfish verdict --release v1.8.2 --environment ci

4. Check a release before you deploy it

Run a fixed set of at least 30 inputs through your agent in CI, tagged with the release and environment="ci". The verdict then compares this release's CI runs with the previous release's CI runs, so a regression stops the pipeline before any user sees it. Production runs are never mixed in.

# tests/agent_suite.py — runs in CI before deploy
import os
from lanternfish import instrument

instrument(release=os.environ["GITHUB_SHA"], environment="ci")

for case in load_cases("tests/agent_cases.jsonl"):  # your own inputs, 30 or more
    run_agent(case["input"])

# .github/workflows/deploy.yml (steps before deploy)
- name: Run the agent test suite
  run: python tests/agent_suite.py
  env:
    LANTERNFISH_API_KEY: ${{ secrets.LANTERNFISH_API_KEY }}
    LANTERNFISH_ENDPOINT: ${{ vars.LANTERNFISH_ENDPOINT }}
- name: Stop the deploy if the release regressed
  run: lanternfish verdict --release "$GITHUB_SHA" --environment ci
  env:
    LANTERNFISH_API_KEY: ${{ secrets.LANTERNFISH_API_KEY }}
    LANTERNFISH_ENDPOINT: ${{ vars.LANTERNFISH_ENDPOINT }}

The first release run this way has nothing to compare with yet, so its verdict is "not enough runs" and the gate stays open unless you pass --require-data. The Releases page and the regression email count runs from every environment.

5. Or check it after a canary

Without a test suite, run the check after the new release has served enough real traffic, for example in a canary stage, and before promoting it. A non-zero exit code fails the job.

- name: Check the release with Lanternfish
  run: |
    pip install lanternfish
    lanternfish verdict --release "$GITHUB_SHA" --environment production
  env:
    LANTERNFISH_API_KEY: ${{ secrets.LANTERNFISH_API_KEY }}
    LANTERNFISH_ENDPOINT: ${{ vars.LANTERNFISH_ENDPOINT }}

6. Get it in Slack or your own service

In Settings → Notifications, owners and admins add a Slack incoming webhook or any https address. A webhook receives a JSON POST (event, title, lines, link, data, sent_at) signed with the channel's secret in the X-Lanternfish-Signature header as t=<unix time>,v1=<hex>. Verify it before trusting the body:

import hashlib, hmac

def is_from_lanternfish(secret: str, header: str, body: bytes) -> bool:
    parts = dict(item.split("=", 1) for item in header.split(","))
    expected = hmac.new(secret.encode(), f"{parts['t']}.".encode() + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, parts["v1"])

Addresses that resolve to a private network are refused, and redirects are not followed. Each channel shows the result of its last message.