How to read the signals of a case
What the five case signals in a Qably suite mean, when each one appears, and a short routine to triage them.
Qably checks every case in a suite against five fixed rules and shows a chip when one applies. A signal tells you which case to open next; it does not say your product is broken. The rules read only names, steps, automation keys and recent results, so the same data always gives the same signals.
Where the signals appear
Open a suite in your project. Above the case list, a strip shows each signal with the number of cases that trip it, and each case card repeats its own chips. Hover over or focus a chip to read its short explanation. An automated case with neither steps nor expected result shows Undocumented on its card, instead of a No steps chip.
The five signals
| Signal | When it appears | What it suggests | What to do |
|---|---|---|---|
| No steps | An automated case has no recorded steps. Manual cases are never flagged. | Results arrive, but nobody can tell what the test checks. | Write the steps and the expected result. See the guide on writing a test case. |
| Raw name | The name has no spaces, looks like a code identifier and either contains an underscore or mixes upper and lower case, as in test_login_1 or testTokenNull. A name equal to the case's automation key also counts. | Only the person who wrote the test can read it in a report. | Rename it as a phrase that says what the user does. See naming tests. |
| Flaky | The last 6 results of an automated case in finished runs, leaving out skipped and blocked ones, switch between pass and fail at least twice, as in pass, fail, pass. | The result may depend on timing, test order, shared data or the environment rather than on the code under test. | Compare a passing and a failing run of the case, find what differs and fix the test before you trust its status. |
| Duplicate key | Another case in the same project has the same automation key, the one Qably uses to link a result to a case. | The same test is registered as two cases, in different suites of the project, and one of them is probably stale. | Open both, keep the one that still receives results and delete the other. To see how a test becomes a case, read the JUnit reference. |
| Near-duplicate title | Another case in the same suite has the same title once capitals, punctuation and extra spaces are ignored. | Two cases read the same, so a failure in a report is ambiguous. | Rename them so each says what it checks, or delete the repeated one. The guide on what kinds of errors to look for helps you decide which cases earn a place. |
Flaky is a hint, not a verdict
A flaky case passes and fails without a change you can point at, so a red run may not be a regression. A case that changes result only once, for example from passing to failing, does not carry the signal: that is a regression, not flakiness. Neither does a case that fails every time.
Important: Flaky only marks the case. Qably does not isolate, skip or hide it: its results keep counting in every run exactly as your CI reported them.
A routine to triage a suite
- Open the suite and read the strip above the cases. Note which signal has the highest count.
- Fix duplicates first: Duplicate key and Near-duplicate title. Every other signal is easier to read when each test is one case.
- Look at Flaky cases next. Compare a passing and a failing run, and fix the test or the data it depends on.
- Then clear No steps and Raw name in small batches, for example whenever you touch the test file.
- If a case that always passed now fails, that is a regression. Continue with the guide on reading a regression and its candidate commits.