Verity docs

Verity is CI-native: every test runs in your pipeline and posts results to Verity's ingestion API. One platform covers UI, API, visual, agent (AI), performance, and security — each an additive engine you can turn on independently. We never host your browsers for paid usage, and never receive your secrets.

The fastest path: open the dashboard → a project → CI Setup, toggle the tests you want, and copy the generated config for your provider.

1 · Quickstart

Create a project in the dashboard and copy its ingest token into your CI secrets as VERITY_INGEST_TOKEN. Then add the config from CI Setup (it's generated for your exact selection). That's it — PRs get a fast Quick check, merges get the Full regression.

2 · Choosing which tests to run

You don't need all engines. In CI Setup, toggle any of: ui, api (flow), contract, visual, agent, perf, security. The engines input controls which steps run; the default ui keeps the classic behavior. Every engine maps its native output into one unified results schema, so the dashboard, PR comments, and predictions work the same regardless of what you enable.

3 · GitHub Actions

The first-party Action wraps everything. CI Setup generates this for your selection:

# .github/workflows/verity.yml
- uses: qashift/verity-action@v1
  with:
    tier: fast            # 'fast' on PRs, 'full' on merge
    token: ${{ secrets.VERITY_INGEST_TOKEN }}
    engines: ui,api,perf,security   # pick what you need

4 · GitLab CI

No GitHub Action needed — GitLab fetches the same self-contained runners via curl and reports to the same dashboard. Store VERITY_INGEST_TOKEN as a masked CI/CD variable. CI Setup generates the full .gitlab-ci.yml; the core is:

verity:fast:
  image: mcr.microsoft.com/playwright:v1.49.1-noble
  services: ["docker:dind"]
  script:
    - curl -Ls https://raw.githubusercontent.com/qashift/verity-action/v1/verity.mjs -o verity.mjs
    - curl -Ls https://raw.githubusercontent.com/qashift/verity-action/v1/verity-engines.mjs -o verity-engines.mjs
    - node verity.mjs run --tier fast --token "$VERITY_INGEST_TOKEN"

5 · Jenkins

Same approach via a Docker agent. Add VERITY_INGEST_TOKEN as a Jenkins credential (secret text); the agent needs docker run for the security/contract steps. CI Setup generates the full Jenkinsfile.

5b · CLI

Prefer not to use a CI integration? The CLI runs the same pipeline from any shell. Install it, then run the tier you want — it reruns failures to separate flaky from broken and posts results to Verity.

npm i -D @qashifthq/verity

# Quick check — diff-aware subset (fast, every PR)
npx verity run --tier fast --token $VERITY_INGEST_TOKEN

# Full regression — the entire suite (on merge)
npx verity run --tier full --token $VERITY_INGEST_TOKEN

There's also a local MCP server (@qashifthq/verity-mcp) so an AI IDE can propose, generate, and run tests in your repo. See the Local / MCP tab in your project.

6 · UI testing (Playwright)

AI-generated Playwright specs with grounded locators, run in your CI. Failures are rerun to separate flaky from broken; broken locators are self-healed (below). The fast tier is a diff-aware Quick check; only the Full regression tier can block a merge.

Verified generation: on the Tests page, Verify drafts runs newly generated specs in a real cloud browser and self-heals any failure before you trust them — each test is badged Verified ✓ (it actually passed) or Needs review. Verification is part of authoring: it consumes no cloud-run quota.

7 · API testing

Verity tests each endpoint two complementary ways and merges them so you see API health, not tooling:

• Example journeys — Playwright request tests for real flows (create → read → update), generated from your OpenAPI spec.
• Contract checks — Verity exercises every operation with automatically generated inputs and reports only the minimal reproducing failure.

The dashboard groups results by endpoint; a failure both methods catch is shown once and marked double-verified. Contract testing needs a reachable base URL — for internal APIs it runs only inside your CI.

8 · Visual

AI-generated Playwright visual specs in tests/verity/visual/ whose assertion is toHaveScreenshot. Each spec navigates to a stable UI state, masks volatile regions (timestamps, avatars, ads), and diffs a screenshot against the baseline committed in your repo. A diff over your maxDiffPixelRatio is reported as a changed snapshot — you approve the new baseline or fix the regression. Because baselines live in your repo and run in your CI, visual coverage is unlimited, and broken navigation locators self-heal exactly as UI specs do.

First run: a new visual test has no baseline yet, so CI captures one and reports it as a pass, not a regression. The GitHub Action uploads the captured images as a verity-visual-baselines-… artifact — download it and commit the files next to the spec, so later runs compare against them. Capture baselines on the same OS as your CI (Linux on GitHub-hosted runners), because fonts render differently per platform.

9 · Agent (AI)

Evals for your own AI features — chatbots, RAG assistants, and tool-using agents. Two formats, merged into one run: AI-generated Playwright API tests in tests/verity/agent/, and a scored eval suite (tests/verity/agent/verity.evals.yaml) that includes a built-in prompt-injection and PII pack. Both POST to your AI endpoint and assert a property of the reply, never an exact string (model output varies): grounding (no hallucination), prompt-injection resistance, correct tool-use, safety/PII, valid format, or clean refusal. Point Verity at the endpoint and store its token as a CI secret; evals run on every deploy so a prompt change or model swap can't quietly break behavior. Full regression asks each suite case 3 times, so inconsistent answers show up as flaky. Rubric (AI-graded) checks need a grader key secret (VERITY_GRADER_API_KEY, any OpenAI-compatible provider). Needs a reachable endpoint — for internal AI endpoints it runs only inside your CI.

10 · Performance

Verity's performance engine is a load test for your API, not a website crawler. It generates concurrent traffic against the endpoints/flows Verity builds from your OpenAPI spec and measures latency percentiles (p50/p95/p99) and error rate under load — one committed script (tests/verity/perf/verity.k6.js) runs six named profiles: smoke, load, stress, spike, soak and breakpoint. For page-load performance, enable Core Web Vitals (the front-end half).

Optimal config: the smoke profile (every PR) is 1 virtual user for 30s — it catches “this PR made the API slower,” never capacity. On merge, run load at your real peak concurrency; use stress or breakpoint to find the ceiling and soak for leaks. Healthy = error rate < 1% and a stable p95 as load rises. One CI runner tops out at a few hundred requests per second.

11 · Security

Dynamic analysis scans your running app at a URL — baseline on PRs, full active scan on merge. Static analysis scans your code (diff on PRs, full repo on schedule) in an isolated container. Findings are surfaced as severity-ranked issues, not pass/fail tests, and never block the build.

How to read them: work top-down — fix Critical/High before release, schedule Medium, batch Low/Info. SAST points at a file:line (precise — fix at source); DAST points at a URL (baseline can be noisy — confirm first). Follow the CWE link for the bug class and standard fix; a finding that disappears next scan is resolved.

12 · Self-healing

When a locator breaks, Verity sends the old selector + surrounding DOM context to the healing model, which proposes a corrected locator. It is never written into your source automatically — you review and apply the diff (or merge the PR). One delivery path for every discipline.

13 · Predictions

• Regression-prone module flagging (Live) — high-regression-rate files are flagged in PR comments.
• Flake vs. real-failure disambiguation (Live) — each failure shows its historical flake rate.
• Risk correlation & deploy confidence (Calibrating) — switch on after ~20 deploys of history.

14 · Ingestion API

The Action, CLI, and engine adapters all call this. The contract is stable; new disciplines use additive optional fields (findings, perf, per-result discipline/layer).

POST /api/ingest
Authorization: Bearer $VERITY_INGEST_TOKEN

{
  "tier": "fast",
  "commit_sha": "abc123",
  "branch": "feature/x",
  "results": [
    { "spec_path": "tests/checkout.spec.ts", "attempts": [false, true, true] }
  ]
  // security runs add "findings": [...]; performance adds "perf": {...}
}

attempts is the per-rerun pass/fail array: mixed = flaky, all-false = real failure, all-true = pass.

15 · Roles & team

Invite teammates from Team (they must have a Verity account first). Three roles:

• Owner — full control + billing. One per org.
• Admin — manage projects, members, and CI.
• Member — read-only access to projects, tests, runs, and all dashboards.

Access is enforced by row-level security: members only ever see their own org's data.

16 · Cloud trial

To get the feel without wiring CI, generation and a capped number of runs execute on our infra. It's intentionally limited — the product lives in your CI, where runs are unlimited.

17 · Licenses & open source

The core engine (@qashift/verity-core) is MIT-licensed. Every test engine Verity drives is best-in-class open source, invoked in isolation (never linked into our code), so you own the test files outright and there's no lock-in. The hosted dashboard, predictions, and team management are proprietary.