Add code-coverage-tools package to compare code coverage on the PR build. (#22452)

## Description


[AB#14170](https://dev.azure.com/fluidframework/internal/_workitems/edit/14170)

Add code-coverage-tools package to compare code coverage on the PR
build.
1.) Before running coverage comparison, code coverage plugin identifies
the baseline build for the PR.
2.) Once the baseline build is identified, we download the build
artifacts corresponding to the `Code Coverage Report_<Build_Number>`
artifact name for this build
3.) We then collect the code coverage stats for the PR build and then
make the comparison with the baseline.
4.) If the code coverage diff (branch coverage) is more than a
percentage point change, then we fail the build for the PR. We also fail
the build in case the code coverage for the newly added package is less
than 50%.
5.) We post the comment on the PR specifying the code coverage change if
any for each package which is modified.

We needed this separate module as we need specifics to do the code
coverage comparison like the baseline with which we need to make the
comparison and then what we need to compare and then after comparison
what comment and in what format we want to report it.

Sample Comment:

<img width="945" alt="code coverage"
src="https://github.com/user-attachments/assets/d74e612e-0da3-43b6-87d5-7278c887518d">

---------

Co-authored-by: Jatin Garg <[email protected]>
Co-authored-by: Alex Villarreal <[email protected]>
Co-authored-by: Tyler Butler <[email protected]>

Author: 4 people

build-tools/packages/build-cli/.eslintrc.cjs

@@ -38,9 +38,13 @@ module.exports = {
 					// These are all excluded because they're "submodules" used for organization.
 					// AB#8118 tracks removing the barrel files and importing directly from the submodules.
 					"**/library/index.js",
+					"**/library/githubRest.js",
 					"**/handlers/index.js",
 					"**/machines/index.js",
 					"**/repoPolicyCheck/index.js",
+					"**/azureDevops/**",
+					"**/codeCoverage/**",
+					"azure-devops-node-api/**",
 				],
 			},
 		],

build-tools/packages/build-cli/README.md

@@ -47,6 +47,7 @@ USAGE
 * [`flub publish`](docs/publish.md) - Publish commands are used to publish packages to an npm registry.
 * [`flub release`](docs/release.md) - Release commands are used to manage the Fluid release process.
 * [`flub rename-types`](docs/rename-types.md) - Renames type declaration files from .d.ts to .d.mts.
+* [`flub report`](docs/report.md) - Report analysis about the codebase, like code coverage and bundle size measurements.
 * [`flub run`](docs/run.md) - Generate a report from input bundle stats collected through the collect bundleStats command.
 * [`flub transform`](docs/transform.md) - Transform commands are used to transform code, docs, etc. into alternative forms.
 * [`flub typetests`](docs/typetests.md) - Updates configuration for type tests in package.json files. If the previous version changes after running preparation, then npm install must be run before building.

build-tools/packages/build-cli/docs/codeCoverageDetails.md

@@ -0,0 +1,93 @@
+# Code coverage
+
+## Overview
+
+This module contains all the utilities required to analyze code coverage metrics from PRs. The coverage metrics generated in the PR build are compared against a baseline CI build for packages that have been updated in the PR. If the line or branch coverage for a package has been impacted in the PR, a comment is posted to the PR showing the diff of the code coverage between baseline and PR.
+
+## Code Coverage Metrics
+
+Code coverage metrics is generated when tests run in the CI pipeline. You can also generate the code coverage metrics for a package locally by running `npm run test:coverage` for the individual package or by running `npm run ci:test:mocha:coverage` from the root.
+
+## Pieces of the code coverage analysis
+
+Code coverage has several steps involving different commands. This section defines those pieces and how they fit together to enable overall code coverage reporting.
+
+### Cobertura coverage files
+
+Code coverage metrics is included in the cobertura-format coverage files we collect during CI builds. These files are currently published as artifacts from both our PR and internal build pipelines to ensure we can run comparisons on PRs against a baseline build.
+
+### Identifying the baseline build
+
+Before running coverage comparison, a baseline build needs to be determined for the PR. This is typically based on the target branch for the PR. For example, if a pull request was targeting `main`, we would consider the baseline to be the latest successful `main` branch build.
+
+### Downloading artifacts from baseline build
+
+Once the baseline build is identified, we download the build artifacts corresponding to the `Code Coverage Report_{Build_Number}` artifact name for this build. We unzip the files and extract the coverage metrics out of the code coverage artifact using the helper `getCoverageMetricsFromArtifact`. Currently, we track `lineCoverage` and `branchCoverage` as our metrics for reporting and
+use both `lineCoverage` and `branchCoverage` for success criteria. The metrics is in percentages from 0..100. The final structure of the extracted metrics looks like the following:
+
+```typescript
+/**
+ * The type for the coverage report, containing the line coverage and branch coverage(in percentage) for each package
+ */
+export interface CoverageMetric {
+	lineCoverage: number;
+	branchCoverage: number;
+}
+```
+
+### Generating the coverage metrics on PR build
+
+As mentioned earlier, the PR build also uploads coverage metrics as artifacts that can be used to run coverage analysis against a baseline build. To help with this, we use the `getCoverageMetricsFromArtifact` helper function to extract code coverage metrics of format `CoverageMetric` that contains code coverage metrics corresponding to the PR for each package.
+
+### Comparing code coverage reports
+
+Once we have the coverage metrics for the baseline and PR build, we use the `compareCodeCoverage` utility function that returns an array of coverage comparisons for the list of packages passed into it. The array returned contains objects of type `CodeCoverageComparison`. We ignore some packages for comparing code coverage such as definition packages and packages which are not inside `packages` folder in the repo. The logic is in the `compareCodeCoverage` function.
+
+```typescript
+/**
+ * Type for the code coverage report generated by comparing the baseline and pr code coverage
+ */
+export interface CodeCoverageComparison {
+	/**
+	 * Path of the package
+	 */
+	packagePath: string;
+	/**
+	 * Line coverage in baseline build (as a percent)
+	 */
+	lineCoverageInBaseline: number;
+	/**
+	 * Line coverage in pr build (as a percent)
+	 */
+	lineCoverageInPr: number;
+	/**
+	 * difference between line coverage in pr build and baseline build (percentage points)
+	 */
+	lineCoverageDiff: number;
+	/**
+	 * branch coverage in baseline build (as a percent)
+	 */
+	branchCoverageInBaseline: number;
+	/**
+	 * branch coverage in pr build (as a percent)
+	 */
+	branchCoverageInPr: number;
+	/**
+	 * difference between branch coverage in pr build and baseline build (percentage points)
+	 */
+	branchCoverageDiff: number;
+	/**
+	 * Flag to indicate if the package is new
+	 */
+	isNewPackage: boolean;
+}
+```
+
+### Success Criteria
+
+The code coverage PR checks will fail in the following cases:
+
+- If the **line code coverage** or **branch code coverage** decreased by > 1%.
+- If the code coverage(line and branch) for a newly added package is < 50%.
+
+The enforcement code is in the function `isCodeCoverageCriteriaPassed`.

build-tools/packages/build-cli/docs/report.md

@@ -0,0 +1,41 @@
+`flub report`
+=============
+
+Report analysis about the codebase, like code coverage and bundle size measurements.
+
+* [`flub report codeCoverage`](#flub-report-codecoverage)
+
+## `flub report codeCoverage`
+
+Run comparison of code coverage stats
+
+```
+USAGE
+  $ flub report codeCoverage --adoBuildId <value> --adoApiToken <value> --githubApiToken <value>
+    --adoCIBuildDefinitionIdBaseline <value> --adoCIBuildDefinitionIdPR <value>
+    --codeCoverageAnalysisArtifactNameBaseline <value> --codeCoverageAnalysisArtifactNamePR <value> --githubPRNumber
+    <value> --githubRepositoryName <value> --githubRepositoryOwner <value> --targetBranchName <value> [-v | --quiet]
+
+FLAGS
+  --adoApiToken=<value>                               (required) Token to get auth for accessing ADO builds.
+  --adoBuildId=<value>                                (required) Azure DevOps build ID.
+  --adoCIBuildDefinitionIdBaseline=<value>            (required) Build definition/pipeline number/id for the baseline
+                                                      build.
+  --adoCIBuildDefinitionIdPR=<value>                  (required) Build definition/pipeline number/id for the PR build.
+  --codeCoverageAnalysisArtifactNameBaseline=<value>  (required) Code coverage artifact name for the baseline build.
+  --codeCoverageAnalysisArtifactNamePR=<value>        (required) Code coverage artifact name for the PR build.
+  --githubApiToken=<value>                            (required) Token to get auth for accessing Github PR.
+  --githubPRNumber=<value>                            (required) Github PR number.
+  --githubRepositoryName=<value>                      (required) Github repository name.
+  --githubRepositoryOwner=<value>                     (required) Github repository owner.
+  --targetBranchName=<value>                          (required) Target branch name.
+
+LOGGING FLAGS
+  -v, --verbose  Enable verbose logging.
+      --quiet    Disable all logging.
+
+DESCRIPTION
+  Run comparison of code coverage stats
+```
+
+_See code: [src/commands/report/codeCoverage.ts](https://github.com/microsoft/FluidFramework/blob/main/build-tools/packages/build-cli/src/commands/report/codeCoverage.ts)_

build-tools/packages/build-cli/package.json

@@ -93,6 +93,7 @@
 		"@octokit/rest": "^21.0.2",
 		"@rushstack/node-core-library": "^3.59.5",
 		"async": "^3.2.4",
+		"azure-devops-node-api": "^11.2.0",
 		"chalk": "^5.3.0",
 		"change-case": "^3.1.0",
 		"cosmiconfig": "^8.3.6",
@@ -110,6 +111,7 @@
 		"issue-parser": "^7.0.1",
 		"json5": "^2.2.3",
 		"jssm": "5.98.2",
+		"jszip": "^3.10.1",
 		"latest-version": "^5.1.0",
 		"mdast": "^3.0.0",
 		"mdast-util-heading-range": "^4.0.0",
@@ -137,7 +139,8 @@
 		"table": "^6.8.1",
 		"ts-morph": "^22.0.0",
 		"type-fest": "^2.19.0",
-		"unist-util-visit": "^5.0.0"
+		"unist-util-visit": "^5.0.0",
+		"xml2js": "^0.5.0"
 	},
 	"devDependencies": {
 		"@biomejs/biome": "~1.8.3",
@@ -161,6 +164,7 @@
 		"@types/semver-utils": "^1.1.1",
 		"@types/sort-json": "^2.0.1",
 		"@types/unist": "^3.0.3",
+		"@types/xml2js": "^0.4.11",
 		"c8": "^7.14.0",
 		"chai": "^4.3.7",
 		"chai-arrays": "^2.2.0",
@@ -243,6 +247,9 @@
 			},
 			"transform": {
 				"description": "Transform commands are used to transform code, docs, etc. into alternative forms."
+			},
+			"report": {
+				"description": "Report analysis about the codebase, like code coverage and bundle size measurements."
 			}
 		}
 	}

build-tools/packages/build-cli/src/codeCoverage/codeCoveragePr.ts

@@ -0,0 +1,86 @@
+/*!
+ * Copyright (c) Microsoft Corporation and contributors. All rights reserved.
+ * Licensed under the MIT License.
+ */
+
+import { getAzureDevopsApi } from "@fluidframework/bundle-size-tools";
+import { type IAzureDevopsBuildCoverageConstants } from "../library/azureDevops/constants.js";
+import {
+	type IBuildMetrics,
+	getBaselineBuildMetrics,
+	getBuildArtifactForSpecificBuild,
+} from "../library/azureDevops/getBaselineBuildMetrics.js";
+import type { CommandLogger } from "../logging.js";
+import { type CodeCoverageComparison, compareCodeCoverage } from "./compareCodeCoverage.js";
+import { getCoverageMetricsFromArtifact } from "./getCoverageMetrics.js";
+
+/**
+ * Report of code coverage comparison.
+ */
+export interface CodeCoverageReport {
+	/**
+	 * Comparison data for each package.
+	 */
+	comparisonData: CodeCoverageComparison[];
+
+	/**
+	 * Baseline build metrics against which the PR build metrics are compared.
+	 */
+	baselineBuildMetrics: IBuildMetrics;
+}
+
+/**
+ * API to get the code coverage report for a PR.
+ * @param adoToken - ADO token that will be used to download artifacts from ADO pipeline runs.
+ * @param codeCoverageConstantsBaseline - The code coverage constants required for fetching the baseline build artifacts.
+ * @param codeCoverageConstantsPR - The code coverage constants required for fetching the PR build artifacts.
+ * @param changedFiles - The list of files changed in the PR.
+ * @param logger - The logger to log messages.
+ */
+export async function getCodeCoverageReport(
+	adoToken: string,
+	codeCoverageConstantsBaseline: IAzureDevopsBuildCoverageConstants,
+	codeCoverageConstantsPR: IAzureDevopsBuildCoverageConstants,
+	changedFiles: string[],
+	logger?: CommandLogger,
+): Promise<CodeCoverageReport> {
+	const adoConnection = getAzureDevopsApi(adoToken, codeCoverageConstantsBaseline.orgUrl);
+
+	const baselineBuildInfo = await getBaselineBuildMetrics(
+		codeCoverageConstantsBaseline,
+		adoConnection,
+		logger,
+	).catch((error) => {
+		logger?.errorLog(`Error getting baseline build metrics: ${error}`);
+		throw error;
+	});
+
+	const adoConnectionForPR = getAzureDevopsApi(adoToken, codeCoverageConstantsPR.orgUrl);
+
+	const prBuildInfo = await getBuildArtifactForSpecificBuild(
+		codeCoverageConstantsPR,
+		adoConnectionForPR,
+		logger,
+	).catch((error) => {
+		logger?.errorLog(`Error getting PR build metrics: ${error}`);
+		throw error;
+	});
+
+	// Extract the coverage metrics for the baseline and PR builds.
+	const [coverageMetricsForBaseline, coverageMetricsForPr] = await Promise.all([
+		getCoverageMetricsFromArtifact(baselineBuildInfo.artifactZip),
+		getCoverageMetricsFromArtifact(prBuildInfo.artifactZip),
+	]);
+
+	// Compare the code coverage metrics for the baseline and PR builds.
+	const comparisonData = compareCodeCoverage(
+		coverageMetricsForBaseline,
+		coverageMetricsForPr,
+		changedFiles,
+	);
+
+	return {
+		comparisonData,
+		baselineBuildMetrics: baselineBuildInfo,
+	};
+}