Qably centralizes QA tracking for engineering teams. The platform does not execute tests directly. Instead, external CI pipelines run the test suites and push results to Qably over HTTP. Qably stores the incoming data and provides a single view of test history, repository changes, and coverage status.

Two independent pipelines

Qably operates two decoupled data pipelines. Setting up one pipeline does not configure the other, which is the most frequent reason a screen appears empty.

Prerequisites

• A Qably account with at least one active organization.
• A repository hosted on GitHub or Bitbucket (the two currently supported providers).
• A CI pipeline capable of running tests and making HTTP requests upon completion.

Every project belongs to an organization. Create one by navigating to Projects > New project in the web app.

• Name (required, up to 80 characters)
• Description (optional, up to 500 characters)
• Technologies (optional, populated automatically when connecting a repository in the next step)

A connected repository is not required to create a project. Linking a repository, generating an API key, and configuring CI reporting operate independently and can be completed in any order.

This step activates the code change pipeline. Qably receives repository updates through webhooks sent by your Git host, completely independent of the project API key in step 3.

Repository selection

1. Sign in with GitHub or Bitbucket if you have not done so already. Qably uses this OAuth token to query accessible repositories across personal accounts and organizations.
2. In project settings under Integrations, select the target repository from the list, sorted by most recent push.
3. Selecting an unlinked repository creates an organization connection and generates a webhook secret. Selecting an already connected repository reuses its existing connection.

Registering the webhook with the provider

Qably does not register webhooks automatically on external hosts. Add the webhook manually in repository settings.

1. Retrieve the secret by calling POST /connections/:id/webhook-secret from connection settings. The response outputs the raw secret once upon creation or rotation, so copy it immediately.
2. In GitHub, navigate to Settings > Webhooks > Add webhook within the repository.
3. Payload URL: https://api.qably.dev/webhooks/scm/github
4. Content type: application/json
5. Secret: the value generated in the previous step
6. Events: select push at a minimum. Enable pull request events as well to capture complete branch activity.

Bitbucket connections operate identically, using Bitbucket-specific signature headers. GitHub and Bitbucket are the two supported repository providers.

This step configures the test results pipeline, supplying machine credentials so CI jobs can push test data without an interactive user session.

1. Under project API Keys settings, create a key with an identifying name like "CI/CD Pipeline". This requires owner or admin permissions within the organization.
2. The generated token follows the format qbly_<lookupId>_<secret> and appears only once in the confirmation dialog. Qably stores solely its SHA-256 hash and cannot display it again.
3. Store the value in your CI provider. In GitHub Actions, navigate to Settings > Secrets and variables > Actions > Secrets and create the QABLY_API_KEY repository secret. If using a custom URL, add the QABLY_API_BASE_URL variable under the Variables tab. Never commit plaintext tokens to source control.

Each API key is scoped strictly to a single project and only holds permissions to post test runs. It cannot read other projects, list suites, or modify organization configuration. The target project is determined directly from the key.

Revoking a key disables future requests immediately while preserving all historical runs linked to it. Projects can hold multiple active keys concurrently, allowing smooth key rotation without CI downtime.

To connect Qably to your continuous integration pipeline, create a workflow file in your repository (such as .github/workflows/ci.yml). The YAML syntax structures the pipeline through on trigger events, runner environments under jobs, and an ordered sequence of commands in steps to prepare the runner, execute tests, and send the report.

For Jest, add jest-junit and set JEST_JUNIT_ADD_FILE_ATTRIBUTE to "true" so the XML includes source file paths for each test case, enabling Qably to map tests to code. In Vitest, JUnit reports are generated natively by specifying the reporter flag on the command line.

          
            name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    name: Run tests and report to Qably
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install dependencies
        run: npm ci

      - name: Run Jest tests
        env:
          JEST_JUNIT_OUTPUT_DIR: ./reports
          JEST_JUNIT_OUTPUT_NAME: junit.xml
          JEST_JUNIT_ADD_FILE_ATTRIBUTE: 'true'
        run: npx jest --ci --reporters=default --reporters=jest-junit

      - name: Report results to Qably
        if: always()
        env:
          QABLY_API_KEY: ${{ secrets.QABLY_API_KEY }}
        run: |
          curl --fail --silent --request POST \
            "https://api.qably.dev/runs/ingest/junit?externalId=gha-${{ github.run_id }}-${{ github.job }}&source=github_actions" \
            --header "Authorization: Bearer $QABLY_API_KEY" \
            --header "Content-Type: application/xml" \
            --data-binary @./reports/junit.xml || true
          
        

          
            name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    name: Run tests and report to Qably
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install dependencies
        run: npm ci

      - name: Run Vitest tests
        run: npx vitest run --reporter=default --reporter=junit --outputFile=./reports/junit.xml

      - name: Report results to Qably
        if: always()
        env:
          QABLY_API_KEY: ${{ secrets.QABLY_API_KEY }}
        run: |
          curl --fail --silent --request POST \
            "https://api.qably.dev/runs/ingest/junit?externalId=gha-${{ github.run_id }}-${{ github.job }}&source=github_actions" \
            --header "Authorization: Bearer $QABLY_API_KEY" \
            --header "Content-Type: application/xml" \
            --data-binary @./reports/junit.xml || true
          
        

The POST /runs/ingest/junit endpoint receives raw XML and handles parsing server-side. Unrecognized suites or cases are provisioned automatically within that project.

Verify each pipeline separately, as one may function correctly while the other requires troubleshooting.

• Test results: open the project and locate the run submitted by CI. Overall run status is derived from individual test cases: any failing test fails the run; pending or executing tests keep it in progress; and it passes once at least one test passes or skips with no failures.
• Code changes: requires completing step 2. The Repository tab should display an ingestion batch matching the latest push or pull request.

Qably parses any report following the standard <testsuite>/<testcase> structure, deriving test status from <failure>, <error>, or <skipped> elements. Generate the XML file using your framework tooling and transmit it to Qably in your CI pipeline.

JavaScript and TypeScript

Jest and Vitest are covered in the previous step and share the same configuration.

          
            PLAYWRIGHT_JUNIT_OUTPUT_NAME=results.xml npx playwright test --reporter=junit
          
        

          
            pytest --junitxml=report.xml
          
        

          
            mvn test
# generates target/surefire-reports/TEST-*.xml
          
        

          
            ./gradlew test
# generates build/test-results/test/TEST-*.xml
          
        

          
            phpunit --log-junit junit.xml
          
        

          
            phpunit -c phpunit.xml
          
        

          
            dotnet test --logger:"junit;LogFilePath=test-result.xml"
          
        

          
            go test -v ./... 2>&1 | go-junit-report > report.xml
          
        

Playwright's built-in JUnit reporter writes to standard output unless a file is specified, either through an environment variable or the config file.

It can also be configured once in playwright.config.ts: reporter: [['junit', { outputFile: 'results.xml' }]].

mvn test writes one report per test class with no extra flag; the Surefire plugin does this by default.

PHPUnit 9 and earlier accept a direct CLI flag. PHPUnit 10 and later removed it, so the output file is configured in phpunit.xml instead.

The JunitXml.TestLogger NuGet package is added as a dependency, and the logger is passed on the command line.

Qably derives test and suite names directly from your test runner output and file paths. The labels defined in your test code appear across the platform, turning test names into clear technical documentation for the entire team.

These four conventions help produce clean, readable test suites:

1. Group tests by business feature rather than technical classes or file paths. Describe what the user does (such as "Shopping cart" instead of "CartServiceImpl").
2. Write test titles as an active verb combined with the expected condition: "rejects an empty token", "accepts a token up to 500 characters".
3. Isolate one behavior per test. A test title containing the word "and" usually indicates two separate assertions that should be split.
4. Name the test file after the feature under test, since this value determines the suite title in Qably.

Automatic name formatting

During ingestion, Qably converts technical identifiers into readable text by splitting camelCase or snake_case strings and stripping scaffolding prefixes like "test", "spec", or "case". A function name such as testTokenNull appears in the UI as "Token null".

When a test name is already written as a full phrase (for example, "should reject an empty token"), Qably preserves the phrasing and capitalizes only the first letter without modifying verbs or dropping words.

A before and after

Existing test suites do not require renaming before onboarding. Qably imports current titles as-is and preserves manual edits made in the UI across subsequent ingestion runs.

Ingests execution results for a test suite. Authenticate using Authorization: Bearer <project API key>. The target project and organization are resolved directly from the key.

          
            curl --fail --silent \
  --request POST \
  "https://api.qably.dev/runs/ingest" \
  --header "Authorization: Bearer $QABLY_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "externalId": "gh-run-482913",
    "source": "github_actions",
    "suiteId": "suite_123",
    "name": "Checkout regression - main",
    "startedAt": "2026-09-01T10:00:00Z",
    "finishedAt": "2026-09-01T10:04:12Z",
    "commitSha": "a1b2c3d",
    "commitMessage": "fix: checkout rounding",
    "commitAuthor": "Ada Lovelace",
    "cases": [
      { "name": "Adds an item to the cart", "status": "pass" },
      {
        "name": "Applies a discount code",
        "steps": ["open cart", "apply code SAVE10"],
        "expectedResult": "total is reduced by 10%",
        "status": "fail"
      }
    ]
  }'
          
        

          
            const response = await fetch('https://api.qably.dev/runs/ingest', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.QABLY_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    externalId: 'gh-run-482913',
    source: 'github_actions',
    suiteId: 'suite_123',
    name: 'Checkout regression - main',
    startedAt: '2026-09-01T10:00:00Z',
    finishedAt: '2026-09-01T10:04:12Z',
    commitSha: 'a1b2c3d',
    commitMessage: 'fix: checkout rounding',
    commitAuthor: 'Ada Lovelace',
    cases: [
      { name: 'Adds an item to the cart', status: 'pass' },
      {
        name: 'Applies a discount code',
        steps: ['open cart', 'apply code SAVE10'],
        expectedResult: 'total is reduced by 10%',
        status: 'fail',
      },
    ],
  }),
});
          
        

          
            import os
import requests

response = requests.post(
    "https://api.qably.dev/runs/ingest",
    headers={"Authorization": f"Bearer {os.environ['QABLY_API_KEY']}"},
    json={
        "externalId": "gh-run-482913",
        "source": "github_actions",
        "suiteId": "suite_123",
        "name": "Checkout regression - main",
        "startedAt": "2026-09-01T10:00:00Z",
        "finishedAt": "2026-09-01T10:04:12Z",
        "commitSha": "a1b2c3d",
        "commitMessage": "fix: checkout rounding",
        "commitAuthor": "Ada Lovelace",
        "cases": [
            {"name": "Adds an item to the cart", "status": "pass"},
            {
                "name": "Applies a discount code",
                "steps": ["open cart", "apply code SAVE10"],
                "expectedResult": "total is reduced by 10%",
                "status": "fail",
            },
        ],
    },
)
response.raise_for_status()
          
        

Run status is determined on the server and does not trust client-calculated summaries. A single failing case marks the entire run as failed. Unresolved pending or running cases maintain the run in an executing state. A run passes only when at least one case passes or is skipped with no accompanying failures. Runs where all cases are blocked count as failed because no verifications took place.

Submitting matching (project, source, externalId) tuples updates the existing run instead of duplicating records. The case list is replaced entirely, and optional metadata fields are updated when provided in subsequent payloads. The API responds with 200 OK on both initial creation and replays.

Allows submitting raw JUnit XML reports directly for server-side parsing. Shares identical authentication requirements with POST /runs/ingest.

The request body carries raw XML with Content-Type set to application/xml or text/xml (up to 10 MB). Additional run parameters are passed as query string arguments.

curl --fail --silent \
  --request POST \
  "https://api.qably.dev/runs/ingest/junit?externalId=ci-42" \
  --header "Authorization: Bearer $QABLY_API_KEY" \
  --header "Content-Type: application/xml" \
  --data-binary @junit.xml

Case titles are extracted from the <testcase name="..."> attribute; classname is consulted only as a fallback when name is omitted. Malformed XML or empty bodies return HTTP 400.

The :provider route parameter accepts github or bitbucket (case-insensitive). This route does not use API keys: incoming payloads are authenticated against the HMAC secret configured for the repository connection.

Rate limited to 60 requests per minute per instance. Accepted events queue and process asynchronously. The HTTP response acknowledges receipt of the payload rather than completed processing.

Rotating the secret

Each connection holds its own HMAC secret. If the secret configured on the Git provider falls out of sync with Qably, deliveries fail with HTTP 401. Rotating the secret generates a fresh token, persists its encrypted value, and displays the raw string once.

The 201 response returns { "webhookSecret": "<64 hexadecimal characters>" }. Copy this string into your repository webhook configuration before closing the dialog. The former secret is invalidated immediately. Calling this endpoint on a project without a linked repository returns 404.

API keys use the format qbly_<lookupId>_<secret>. They consist of a fixed prefix, a public 6-byte lookup ID, and a 32-byte secret token. Qably stores only the SHA-256 hash and performs constant-time comparisons. The plaintext token cannot be retrieved once issued.

Revoked keys are preserved in the database to maintain attribution for past test runs. Transmit the key via Authorization: Bearer qbly_<lookupId>_<secret>.

Environment variables configured in CI pipelines to integrate with Qably:

Qably dispatches notifications to Discord channels or Slack workspaces for failed or passed test runs, case regressions, ingestion failures, and credential updates. Webhooks are provisioned on Discord or Slack; Qably delivers outgoing notifications to the configured endpoint URL.

Discord

Slack

Get a webhook URL from Discord

1. In the destination Discord server, navigate to Server Settings > Integrations > Webhooks.
2. Click New Webhook, select the target channel, and copy the webhook URL.

Get a webhook URL from Slack

1. Create or select a Slack application at api.slack.com/apps and enable Incoming Webhooks.
2. Install the app to your workspace, select the channel, and copy the generated webhook URL.

Connect it to Qably

1. Under Settings > Integrations in Team notification channels, select Add channel.
2. Choose Discord or Slack, provide a channel name, paste the webhook URL, and select desired event triggers.
3. Use the send-test action to confirm message delivery before relying on the channel.

Why does my CI pipeline complete successfully, but no test runs appear in Qably?

Verify that the QABLY_API_KEY environment variable is defined as a repository secret for that CI job and that the reporting step executed. Using the if: always() condition ensures the upload runs even when tests fail.

Confirm that your test runner generated the XML report at the specified file path before the upload step. If the curl command returns an HTTP error code (such as 401 for a revoked key or 400 for malformed XML), inspect the runner logs to determine the exact cause.

How does Qably automatically create and organize test suites and cases?

When importing a JUnit XML report, Qably inspects the XML attributes to identify each suite and test case. If a suite with that name does not already exist in the project, the server creates it immediately and assigns the test cases to it.

You do not need to register tests in advance through the web interface or manage numerical identifiers. When posting JSON payloads directly to the API, pass the suiteName field so the system automatically matches or creates the suite.

What happens in Qably if I retry a failed test job in CI?

Qably handles retries idempotently when you supply an externalId query parameter. Deriving this value from runner execution identifiers (such as the workflow run ID and job name in GitHub Actions) instructs the server to update the existing run instead of creating duplicate records.

This ensures project pass rates and test metrics reflect the final status of the run without skewing historical analytics.

Can I record individual test steps and expected results when uploading JUnit XML?

The JUnit XML standard only records the final status of each test case (passed, failed, or skipped), its execution duration, and failure stack traces. For this reason, XML imports leave step definitions and expected outcomes empty.

To track structured test procedures with individual steps and expected results, send JSON payloads directly to the POST /runs/ingest endpoint, or maintain structured test definitions within the web dashboard.

Why are commits and branches missing from the project Repository section?

The Repository view relies exclusively on the SCM webhook configured for your repository provider (GitHub or Bitbucket). Submitting test reports from CI populates execution history, but does not synchronize commit activity or branch metadata.

To inspect code changes and link them with your test runs, configure the repository webhook using the payload URL and secret described in step 2.

Can a single API key submit test results to multiple projects?

No. Each API key is scoped strictly to a single project within your organization. The server infers the target project directly from the token and restricts its permissions solely to test ingestion.

For monorepos or multi-service architectures, generate a dedicated API key for each project and configure the corresponding secret in your CI workflows.