Ray Flaky Test Tracker (https://flaky-tests.ray.io/) is a static site refreshed every 30 minutes to report the health of Ray CI system. In particular, it gathers all the test results, and rank them in particular order to prioritize attention to failing test cases on master.

This document is a user guide to the site. Most items should be self explanatory.

Boxes Coloring

• Green means passing
• Red means failing
• Yellow means flaky (passed within three tries, but at least failed first time)
• Gray means missing data

Commit Status Bar

The bar should be read as left→right meaning newest commit→oldest commits. The site renders 100 commits by default. Currently we don’t store older data in publicly accessible dataset. Anyscale engineers can access internal data lake to diagnose failures older than 100 commits.

Stats and Teams

Ray engineering teams own their individual test suites and is (collectively) responsible for overall health of the CI system to reduce flakiness. Therefore we track key metrics.

Historically, windows build has been hard to debug and causing non-actionable test failures; thus tracking them separately.

Show All