Skip to content

[Feature Request] In Tests tab, provide a normal pass/failure rate #808

@kg345

Description

@kg345

What product do you want to improve?
Tests tab

Is your feature request related to a problem? Please describe.
The failure rate and flake rate make little sense for someone who's trying to understand how often something is passing - just a vanilla num of passes / total attempts. Failure rate is also poorly named as it's not even a rate of failures - but, instead tied to flaky.

Tests that fail once get added to a flaky list and runs where they pass are not counted as passing. So our failure/flake rate will look like 100% for last 7 days for something that passed the last 20 runs (if it failed once on run proceeding aka last 30 times)

Describe the solution you'd like
A column in the Tests tab table that clearly indicates the percentage of runs where a test passed irrespective of a "flake list".

It seems pretty reasonable to expect a tests tab to give us more "normalized" data (how many times it passed, how many times it failed) without the opinionated/unique calculation that flake rate (and it's impact on failure rate).

Describe alternatives you've considered
The only way to seemingly get this info is to hover on the flake rate and see 1 passed, 0 failed (28 flaky) which isn't super helpful when you have 500 tests to get data

Additional context

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status

    Waiting for: Product Owner

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions