CoverBot

Create a scheduled pipeline to backfill unit tests for recently modified files

Available on the Business and Enterprise plans.

Overview

Tusk CoverBot runs as a scheduled daily or weekly cron job and generates unit tests for recently modified files with low line coverage.

Think of this as Dependabot but for increasing unit test coverage across your repo. Go from 0 to 100% code coverage over time with minimal effort.

CoverBot is complementary to the Tusk CI workflow which is triggered per commit to a PR. This feature is best suited for engineering organizations that have set unit test coverage goals.

Prerequisites

You need the following to begin using CoverBot:

• Team Plan and up
• Enabled repo in Tusk
• Working Tusk test execution environment
• Using the following testing frameworks

How CoverBot Works

CoverBot is an async workflow so it is fully hands-off for your engineering team.

1. Navigate to CoverBot page

   Go to Repositories > Repo Settings > CoverBots to view and manage your existing CoverBots for the repo. If you haven’t yet received access to create your first CoverBot, reach out to Tusk support

   
2. Add new CoverBot

   Click Add CoverBot. Fill out the following fields:

   • Bot name
   • Test execution environment
   • Schedule for day(s) and time to run
   • Max no. of symbols to generate tests for
   • Default reviewers (optional)
   • File patterns for files/directories to focus on (optional)

   Make sure to toggle on the CoverBot after creating it.

   
3. Creates branch

   At your pre-determined schedule (daily or weekly at set time), CoverBot creates a branch off of your default branch.

   If you want to see CoverBot’s test generation output before your next scheduled run, click the ellipsis icon on your CoverBot card and click Run Now.

   
4. Analyzes code coverage

   CoverBot runs a coverage script and finds recently modified files that have low line coverage.

5. Backfills unit tests

   CoverBot generates unit tests for the files and iterates on them to make sure they are passing and fully executable.

6. Creates PR/MR

   CoverBot creates a PR/MR with the line/branch coverage gains and test case details in the PR/MR description.

   If no tests are generated, CoverBot deletes the branch from Step 1.
7. Assigns PR/MR reviewer

   CoverBot assigns the PR/MR to the engineer or group who is in charge of reviewing Tusk PRs. You can modify the default reviewer in the CoverBot settings panel.

8. Review and merge PR/MR

   Assigned engineer reviews the PR/MR, approves, and merges the tests if no changes required. Otherwise, they check out the branch and make modifications before merging.

CoverBot is triggered on a test execution environment level. For example, if you’ve enabled CoverBot for two separate environments (e.g., frontend unit tests, backend unit tests), the bot will raise 2 separate PRs.

FAQs

Who should review CoverBot PRs?

Some teams already have a dedicated engineer or group for reviewing PRs created by Tusk. If not, we recommend that the engineer who is the “repo owner” be the person reviewing the CoverBot PR.

Is there a limit to the number of files that Tusk will generate tests for?

The limit on number of files tested in the PR is 25. Let your Tusk representative know if you would like to change this default.

Why hasn't CoverBot created a PR at my pre-determined schedule?

This scenario most likely happened because you have not yet merged the most recent CoverBot PR that was raised. CoverBot will only raise a new PR if there are no other CoverBot PRs still open for the associated test execution environment. We do this to prevent duplicate tests across CoverBot PRs.

Can I specify directories that CoverBot should not be writing tests for?

Yes. You can do this in the CoverBot settings panel in your repo settings.

Does CoverBot guarantee 100% line coverage for the files that it is backfilling tests for?

It is not a guarantee. By design, CoverBot will iterate based on its collected coverage report multiple times to cover untested lines in chosen files up to 95-100% line coverage. However, certain files may not be backfilled up to that coverage level due to the “Max Symbols to Test” setting and the fact that some symbols aren’t worth testing.

Does Tusk add failing tests to the CoverBot PR?

No, it does not. CoverBot’s goal is to increase code coverage; we want to make it as easy as possible for your team to merge its PRs.

Will CoverBot uncover edge cases in the system under test (SUT)?

Yes, CoverBot will surface potential bugs that it detected in its PR body for you to investigate. However, it will not provide the failing test in “Files changed.”

We’ve optimized CoverBot for generating only passing tests for the SUT’s happy path. This is because CoverBot’s goal is to increase code coverage as opposed to finding bugs. You still enjoy the benefits of preventing future regressions by merging CoverBot’s tests.