How to read a regression and its candidate commits

When Qably flags a regression, what the candidate commit range is, what it needs to show one, and what to do when it cannot.

A regression is a case that stopped passing. Qably flags it and, when it can, lists the commits that arrived between the two runs. Read that list as a set of leads, not as an answer.

What counts as a regression

Qably compares each finished run with the previous finished run of the same suite. A case is a regression when it passed in that run and fails in this one. The opposite change, from fail to pass, is a fix. A case that was skipped or blocked before, or that is new in this run, is neither.

You see it in three places:

Run list: each run carries three counts against the previous one: regressions, fixes and unchanged cases.
Run detail: the What changed block lists the cases under "Started failing" and "Got fixed".
Notification: Case regressed is sent only when someone records a failing result by hand on a manual run. Runs reported from CI send Run failed or Run completed.

What the candidate commits are, and what they are not

Under those lists, Candidate commits shows the commits Qably received in pushes between the previous run and this one, newest first: the five newest, each with short sha, first line of the message, author, branch and date, and a button for the older ones. It appears only when the run has at least one regression or fix.

Important: The list never says which commit caused the failure. It holds every commit received between the two runs, whether or not it touched the code behind the failing case.

When both runs report the same commit, the block reads Both runs ran the same commit: the code did not change, so look at the test and the environment, and at the Flaky signal described in how to read the signals of a case.

What you need for it to work

The list is only as good as what Qably recorded. Three things must be true:

The project has a GitHub repository connected, so Qably receives your pushes. See step 2 of the quickstart.
Both runs report commitSha, a field of POST /runs/ingest and a query parameter of POST /runs/ingest/junit. The workflow in step 4 of the quickstart does not send it: add it to the report step (example after this list).
The commits were pushed after the repository was connected. Qably rebuilds the list from the pushes its webhook recorded; it does not ask GitHub or fetch older history.

Query string of the report step

externalId=gha-${{ github.run_id }}-${{ github.job }}&source=github_actions&commitSha=${{ github.event.pull_request.head.sha || github.sha }}

The expression uses the head commit of a pull request and github.sha on a push, because on pull request builds github.sha is a temporary merge commit that no push contains.

When Qably says it does not know

When the recorded pushes cannot prove a chain from the earlier run to this one, Qably does not guess: the block shows one neutral line, information rather than an error.

Message	What it means	What to do
One of the runs did not report a commit.	One of the two runs has no `commitSha`.	Add it to the report. It applies from the next run.
This project has no repository connected.	No GitHub repository is linked, so there are no pushes to read.	Connect one (first condition).
A commit of these runs is not in the recorded pushes.	No recorded push mentions a commit that one of the runs reported.	Check the webhook log in GitHub for failed deliveries. Report a pushed commit, not a temporary one such as a pull request merge commit.
The runs come from different branches.	The earlier commit was recorded only on branches where this run's commit never was.	Nothing is broken: the previous run is on another line of history. Consecutive runs on one branch do get a list.
Some pushes between these runs were not recorded.	The chain is cut: a push never arrived, a push created the branch between the runs, or the history is past the 1000 most recent pushes Qably reads.	Check the webhook log for failed deliveries. Missing pushes are not recovered; read the history in GitHub.
History was rewritten (force push) between these runs.	A forced push replaced commits, so the recorded links cannot be trusted.	Nothing to fix. Read the history in GitHub.
The earlier run is on a newer commit than this one.	The previous run, by start time, ran a newer commit than this run.	Typical after re-running an old commit or when pipelines start out of order. Compare the commits in GitHub.
Too many commits between these runs to list.	More than 100 commits arrived between the runs.	Read the history in GitHub and report more often so runs sit closer together.

A four-step routine

Open the run and read What changed. Do the cases under "Started failing" share an area, or is it a single case?
Read Candidate commits, newest first, and open the ones close to the failing area. If the block says it does not know, find the reason in the table above.
Run the failing case on your machine at the commit of this run and at the commit of the previous one. With many candidates, git bisect narrows the range for you.
Fix the code or the test. In the next run the case shows under "Got fixed". If it keeps flipping, treat it as flaky.