[Utils] Add a python tool for test coverage of patches#191721
Open
xgupta wants to merge 11 commits into
Open
Conversation
|
✅ With the latest revision this PR passed the Python code formatter. |
xgupta
added
the
skip-precommit-approval
xgupta
removed
the
skip-precommit-approval
xgupta
force-pushed
the
code-coverage
branch
5 times, most recently
from
April 14, 2026 18:02
6b339e3 to
1b88d3a
Compare
🐧 Linux x64 Test Results
✅ The build succeeded and all tests passed. |
xgupta
force-pushed
the
code-coverage
branch
3 times, most recently
from
April 14, 2026 20:26
9727fb7 to
be33086
Compare
Contributor
Author
|
Hello @hnrklssn, Created a draft pull request. Ping for your feedback. I have tested it for llvm changes mention in commit summary. |
Member
|
Awesome, thanks! Sorry about letting the old PR fizzle out. I'll try to find some time to review this one soon. Ping me again if I don't get back to you by the end of the week. |
|
@llvm/pr-subscribers-testing-tools Author: Shivam Gupta (xgupta) ChangesThis patch adds a python tool to show the test coverage of a patch. Complete details can be found on README.md doc. Example output for -
Assisted-by: AI Patch is 40.70 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/191721.diff 12 Files Affected:
diff --git a/llvm/utils/patch-coverage/README.md b/llvm/utils/patch-coverage/README.md
new file mode 100644
index 0000000000000..0faaa2ae6c5e9
--- /dev/null
+++ b/llvm/utils/patch-coverage/README.md
@@ -0,0 +1,185 @@
+# Patch-Based Code Coverage (LLVM Git Integration)
+
+This tool provides patch-level code coverage for LLVM development workflows.
+It helps developers and reviewers understand which lines introduced by a patch
+are covered by tests and which are not.
+
+## Overview
+
+Given recent commits, the tool:
+
+1. Generates a patch (`git diff`)
+2. Identifies modified source files and lines
+3. Detects modified lit tests and unit tests
+4. Builds LLVM with coverage instrumentation
+5. Runs only modified tests
+6. Collects coverage data using `llvm-profdata` and `llvm-cov`
+7. Reports:
+
+ * Covered lines introduced by the patch
+ * Uncovered lines introduced by the patch
+
+## Setup
+
+### 1. Build LLVM (standard workflow)
+
+If you have an existing LLVM build, point to it with `-b`. If not, skip this step —
+the tool will automatically configure and build LLVM in the specified build directory
+with a minimal configuration when first run.
+
+```bash
+git clone https://github.com/llvm/llvm-project.git
+cd llvm-project
+
+cmake -G Ninja -B build -S llvm \
+ -DCMAKE_BUILD_TYPE=Release \
+ -DLLVM_ENABLE_ASSERTIONS=ON \
+ -DLLVM_INCLUDE_TESTS=ON \
+ -DLLVM_BUILD_TESTS=ON
+
+ninja -C build
+```
+
+### 2. Install Python dependencies
+
+```bash
+pip install -r llvm/utils/patch-coverage/requirements.txt
+```
+or
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip install unidiff
+```
+
+### 3. Add tool to PATH
+
+```bash
+export PATH=$PATH:llvm/utils/patch-coverage/
+```
+
+This enables:
+
+```bash
+git patch-coverage ...
+```
+
+## Usage
+
+### Basic command
+
+```bash
+git patch-coverage -b <build-dir> <binary> <test-path>
+```
+
+### Example
+
+```bash
+git patch-coverage -b build -n 3 opt llvm/test
+```
+
+## Arguments
+
+### Required
+
+| Argument | Description |
+| ----------------- | ---------------------------------------------------- |
+| `<binary>` | LLVM binary to analyze (e.g., `opt`) |
+
+
+### Optional
+
+| Argument | Description |
+| ------------------------------ | ----------------------------------------------------------- |
+| `-n, --num-commits` | Number of recent commits to analyze (default: 1) |
+| `--projects` | LLVM projects to enable (e.g. clang;mlir (default: None) |
+| `-i, --instrumented-build-dir` | Use pre-built instrumented directory (Dafault: `build_inst`)|
+| `-b, --build-dir` | Path to LLVM build directory (default: `build`) |
+| `<test-path>` | One or more test suite paths (default: `build/test`) |
+
+
+
+## How It Works
+### 1. Patch Extraction
+The tool runs `git diff HEAD~N HEAD` to generate a patch from the last N commits.
+It parses the diff to extract:
+- **Source files** — modified `.cpp`, `.c`, or `.mm` files outside test directories
+- **Test files** — modified lit tests and unit tests within the patch
+- **Modified lines** — the exact line numbers added or changed in each source file
+
+### 2. Allowlist Generation
+To avoid instrumenting the entire codebase, the tool generates an allowlist
+containing only the source files present in the patch:
+source:/path/to/modified/file.cpp=allow
+default:skip
+
+This is passed to the compiler via `-fprofile-list`, so profiling overhead is
+scoped only to what changed.
+
+### 3. Instrumented Build
+The tool configures a second build directory (`build_inst`) with:
+- `LLVM_BUILD_INSTRUMENTED_COVERAGE=ON` — enables source-based coverage
+- `LLVM_INDIVIDUAL_TEST_COVERAGE=ON` — generates per-test `.profraw` files
+- The allowlist flags from step 2
+
+Rebuilds are skipped if the patch hash (SHA-256 of the diff) matches the hash
+from the last successful build, stored in `build_inst/.last_patch_hash`.
+
+### 4. Test Execution
+Only tests modified in the patch are run, against the instrumented binary.
+Each test writes its coverage data to a dedicated `.profraw` file under
+`build_inst/profiles/`, named by a hash of the test path to avoid collisions:
+
+- **Lit tests** — run via `llvm-lit` with `LLVM_PROFILE_FILE` set per test
+- **Unit tests** — run directly as binaries with `LLVM_PROFILE_FILE` set
+
+### 5. Coverage Processing
+For each `.profraw` file collected during test execution:
+1. `llvm-profdata merge` converts it to a `.profdata` file
+2. `llvm-cov show` produces a line-level coverage report for each patched
+ source file, using the instrumented binary as the symbol source
+
+### 6. Reporting
+The tool cross-references coverage results against the modified lines extracted
+in step 1. A line is considered:
+- **Covered** — if it was executed by at least one test
+- **Uncovered** — if it appears in the patch but was never executed
+- **Effectively covered** — if it was uncovered in some tests but covered in
+ others (union of all runs wins)
+
+Results are printed to the terminal in diff style with color coding, and the
+full details are written to `build_inst/patch_coverage.log`.
+
+## Output
+
+### `<build_inst>/patch_coverage.log`
+
+```
+...
+Modified File: llvm/lib/IR/Example.cpp
+ Covered Line: 42 : foo();
+ Uncovered Line: 45 : bar();
+```
+
+### Summary on Standard Output
+
+```
+[code-coverage] Coverage report for llvm/lib/IR/Example.cpp:
+ Line 42 : foo();
+ Line 45 : bar();
+
+```
+
+## Troubleshooting
+
+### No coverage generated
+
+* Ensure instrumented build succeeded and tests run.
+* Check `.profraw` files in `build_inst` directory.
+
+### Rebuild skipped unexpectedly
+
+* Delete `binary` to force rebuild
+
+### For any other doubt
+* Check `<build_inst>/patch_coverage.log`. It has full logging of tool.
diff --git a/llvm/utils/patch-coverage/__init__.py b/llvm/utils/patch-coverage/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/llvm/utils/patch-coverage/build.py b/llvm/utils/patch-coverage/build.py
new file mode 100644
index 0000000000000..7fee0b36acabf
--- /dev/null
+++ b/llvm/utils/patch-coverage/build.py
@@ -0,0 +1,153 @@
+import os
+import shutil
+import sys
+import subprocess
+
+from utils import log
+from utils import resolve_projects
+
+
+def is_configured_build(build_dir):
+ return os.path.exists(os.path.join(build_dir, "build.ninja"))
+
+
+def configure_llvm_build(build_dir, projects):
+ print(f"[patch-coverage] Configuring LLVM build in {build_dir}...")
+
+ cmake_cmd = [
+ "cmake",
+ "-G",
+ "Ninja",
+ "-S",
+ "llvm",
+ "-B",
+ build_dir,
+ "-DCMAKE_BUILD_TYPE=Release",
+ "-DLLVM_ENABLE_ASSERTIONS=ON",
+ "-DLLVM_INCLUDE_TESTS=ON",
+ "-DLLVM_BUILD_TESTS=ON",
+ ]
+
+ if projects:
+ cmake_cmd.append(f"-DLLVM_ENABLE_PROJECTS={projects}")
+
+ if projects and "lldb" in projects:
+ cmake_cmd.append(f"-DLLVM_ENABLE_RUNTIMES=libcxx")
+ cmake_cmd.append(f"-DPYTHON_EXECUTABLE={shutil.which('python3')}")
+
+ print("CMake cmd:", " ".join(cmake_cmd))
+ subprocess.check_call(cmake_cmd)
+
+
+def ensure_llvm_tools(build_dir, projects, binary):
+ print("Making sure we have required tools...")
+
+ if not is_configured_build(build_dir):
+ configure_llvm_build(build_dir, projects)
+
+ required_tools = [
+ "bin/llvm-lit",
+ "bin/FileCheck",
+ "bin/count",
+ "bin/not",
+ "bin/llvm-config",
+ ]
+
+ missing_tools = [
+ tool
+ for tool in required_tools
+ if not os.path.exists(os.path.join(build_dir, tool))
+ ]
+
+ if missing_tools:
+ print("Building missing core tools:", missing_tools)
+ subprocess.check_call(
+ ["ninja", "-C", build_dir, "FileCheck", "count", "not", "llvm-config"]
+ )
+ else:
+ print("Core tools already present.")
+
+ # Handle binary-specific requirements
+ extra_targets = []
+
+ if binary == "clang-tidy":
+ clang_path = os.path.join(build_dir, "bin", "clang")
+ if not os.path.exists(clang_path):
+ extra_targets.append("clang")
+
+ if binary == "lldb":
+ lldb_path = os.path.join(build_dir, "bin", "lldb")
+ if not os.path.exists(lldb_path):
+ extra_targets.append("lldb")
+
+ if extra_targets:
+ print("Building required targets to parse testsuite info:", extra_targets)
+ subprocess.check_call(["ninja", "-C", build_dir] + extra_targets)
+ else:
+ print("All targets already present to get testsuite info.")
+
+
+def build_llvm(inst_build_dir, build_dir, binary, projects, allowlist_path):
+ try:
+ if not is_configured_build(inst_build_dir):
+ print("Configuring the instrumented build for patch coverage.")
+
+ cmake_cmd = [
+ "cmake",
+ "-G",
+ "Ninja",
+ "-S",
+ "llvm",
+ "-B",
+ inst_build_dir,
+ "-DCMAKE_BUILD_TYPE=Release",
+ "-DLLVM_ENABLE_ASSERTIONS=ON",
+ "-DLLVM_INCLUDE_TESTS=ON",
+ "-DLLVM_BUILD_TESTS=ON",
+ "-DLLVM_BUILD_INSTRUMENTED_COVERAGE=ON",
+ "-DLLVM_INDIVIDUAL_TEST_COVERAGE=ON",
+ f"-DCMAKE_C_FLAGS=-fprofile-list={os.path.abspath(allowlist_path)}",
+ f"-DCMAKE_CXX_FLAGS=-fprofile-list={os.path.abspath(allowlist_path)}",
+ ]
+
+ if projects:
+ cmake_cmd.append(f"-DLLVM_ENABLE_PROJECTS={projects}")
+
+ if projects and "lldb" in projects:
+ cmake_cmd.append(f"-DLLVM_ENABLE_RUNTIMES=libcxx")
+ cmake_cmd.append(f"-DPYTHON_EXECUTABLE={shutil.which('python3')}")
+
+ print("CMake cmd:", " ".join(cmake_cmd))
+ subprocess.check_call(cmake_cmd)
+
+ target = [
+ "FileCheck",
+ "count",
+ "not",
+ "llvm-config",
+ "llvm-cov",
+ "llvm-profdata",
+ "llvm-dwarfdump",
+ "llvm-readobj",
+ binary,
+ ]
+
+ if binary == "clang-tidy":
+ target.append("clang")
+
+ if binary == "lldb":
+ target.extend(["clang", "lldb", "dsymutil"])
+
+ try:
+ print("Building the instrumented target")
+ subprocess.check_call(["ninja", "-C", inst_build_dir] + target)
+
+ except subprocess.CalledProcessError as ninja_error:
+ log(f"Error during Ninja build: {ninja_error}")
+ sys.exit(1)
+
+ log("LLVM build completed successfully.\n")
+
+ except subprocess.CalledProcessError as e:
+ log("Error during LLVM build:", e)
+ sys.exit(1)
diff --git a/llvm/utils/patch-coverage/git-patch-coverage b/llvm/utils/patch-coverage/git-patch-coverage
new file mode 100755
index 0000000000000..a63442cd3f782
--- /dev/null
+++ b/llvm/utils/patch-coverage/git-patch-coverage
@@ -0,0 +1,10 @@
+#!/bin/bash
+#
+# ===- git-patch-coverage - Patch-Based Code-Coverage Git Integration -----------===#
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ===----------------------------------------------------------------------------===#
+python3 "$(dirname "$0")/main.py" "$@"
diff --git a/llvm/utils/patch-coverage/lit.py b/llvm/utils/patch-coverage/lit.py
new file mode 100644
index 0000000000000..30b013bd8a64d
--- /dev/null
+++ b/llvm/utils/patch-coverage/lit.py
@@ -0,0 +1,54 @@
+import subprocess
+import os
+
+
+def parse_suite_info(s):
+ curr_suite = None
+ res = {}
+
+ for line in s.decode().splitlines():
+ leading_spaces = len(line) - len(line.lstrip(" "))
+
+ if leading_spaces == 2 and line.split():
+ curr_suite = line.split()[0]
+ elif curr_suite is not None and leading_spaces == 4 and "Source Root:" in line:
+ if curr_suite in res:
+ raise RuntimeError(f"Duplicate suite detected: {curr_suite}")
+ res[curr_suite] = line.split()[-1]
+
+ return res
+
+
+def find_lit_tests(lit_path, test_paths):
+ suites_cmd = [lit_path, "--show-suites"] + test_paths
+ output = subprocess.check_output(suites_cmd)
+
+ # `--show-suites` produce output in the following form -
+ # LLVM - 61914 tests
+ # Source Root: /Users/<username>/llvm-project/llvm/test
+ #
+ # Parse it to construct following format -
+ # {'LLVM': '/Users/<username>/llvm-project/llvm/test'}
+ test_suites = parse_suite_info(output)
+
+ tests_cmd = [lit_path, "--show-tests"] + test_paths
+ output = subprocess.check_output(tests_cmd)
+
+ lines = [line.decode() for line in output.splitlines()]
+
+ test_info = [line.split() for line in lines if "::" in line]
+
+ # Construct absolute path of each test case of test suite.
+ if test_info is not None and len(test_info) > 0:
+ if len(test_info[0]) == 3:
+ return [
+ os.path.join(test_suites[suite], test_case)
+ for (suite, sep, test_case) in test_info
+ ]
+ elif len(test_info[0]) == 4:
+ return [
+ os.path.join(test_suites[suite1], test_case)
+ for (suite1, suite2, sep, test_case) in test_info
+ ]
+ else:
+ return []
diff --git a/llvm/utils/patch-coverage/main.py b/llvm/utils/patch-coverage/main.py
new file mode 100644
index 0000000000000..3b9eb524e85fe
--- /dev/null
+++ b/llvm/utils/patch-coverage/main.py
@@ -0,0 +1,188 @@
+#!/usr/bin/env python3
+#
+# ===----------------------------------------------------------------------------===#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# ===----------------------------------------------------------------------------===#
+
+import argparse
+import logging
+import os
+import re
+import subprocess
+import sys
+
+from build import build_llvm
+from build import ensure_llvm_tools
+from lit import find_lit_tests
+from patch import extract_modified_source_lines_from_patch
+from patch import extract_source_files_from_patch
+from patch import create_patch_from_last_commits
+from patch import write_source_file_allowlist
+from print import report_covered_and_uncovered_lines
+from process import process_coverage_data
+from test import run_modified_lit_tests
+from test import run_modified_unit_tests
+from utils import configure_logging
+from utils import classify_tests
+from utils import delete_profraw
+from utils import log
+from utils import mark_build_success
+from utils import resolve_projects
+from utils import should_rebuild
+from utils import target_name
+
+sys.path.append(os.path.dirname(__file__))
+
+
+def parse_args():
+ parser = argparse.ArgumentParser(
+ prog="git patch-coverage",
+ description="Patch-based code coverage tool for LLVM",
+ formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+ )
+
+ parser.add_argument(
+ "-b",
+ "--build-dir",
+ help="path to build directory",
+ dest="build_dir",
+ default="build",
+ )
+
+ parser.add_argument(
+ "-i",
+ "--instrumented-build-dir",
+ help="path to directory in which the tool will build projects with intrumentation enabled",
+ dest="inst_build_dir",
+ default="build_inst",
+ )
+
+ parser.add_argument(
+ "-n",
+ "--num-commits",
+ help="number of commits to include in patch",
+ dest="num_commits",
+ default=1,
+ )
+
+ parser.add_argument(
+ "binary",
+ help="target binary to generate coverage for (e.g. opt, clang)",
+ )
+
+ parser.add_argument(
+ "test_path",
+ nargs="*",
+ help="path to test suite/s (default: <build-dir>/test)",
+ default=argparse.SUPPRESS,
+ )
+
+ parser.add_argument(
+ "--projects",
+ help="LLVM projects to enable (semicolon-separated, e.g. clang;mlir)",
+ )
+
+ args = parser.parse_args()
+
+ if not hasattr(args, "test_path") or not args.test_path:
+ args.test_path = [os.path.join(args.build_dir, "test")]
+
+ return (
+ args.build_dir,
+ args.inst_build_dir,
+ args.num_commits,
+ args.binary,
+ args.test_path,
+ args.projects,
+ )
+
+
+def main():
+ (
+ build_dir,
+ inst_build_dir,
+ num_commits,
+ binary,
+ test_paths,
+ projects,
+ ) = parse_args()
+
+ configure_logging(inst_build_dir)
+ projects = resolve_projects(projects, build_dir)
+
+ # Ensure we have required tools to parse test suite info.
+ ensure_llvm_tools(build_dir, projects, binary)
+
+ # Create a diff file from the commit/s.
+ patch_path = os.path.join(build_dir, "patch.diff")
+ create_patch_from_last_commits(patch_path, num_commits)
+ source_files = extract_source_files_from_patch(patch_path)
+
+ # Get all the modified lines of patch, from both source files and test files.
+ llvm_lit_path = os.path.join(build_dir, "bin/llvm-lit")
+ tests = frozenset(find_lit_tests(llvm_lit_path, test_paths))
+ modified_lines = extract_modified_source_lines_from_patch(patch_path, tests)
+
+ # Use allow list feature to generate ".profraw" data for only source files in the patch.
+ os.makedirs(inst_build_dir, exist_ok=True)
+ allowlist_path = os.path.join(inst_build_dir, "fun.list")
+ write_source_file_allowlist(source_files, allowlist_path)
+
+ # Print all the modified lines of the patch.
+ for file, lines in modified_lines.items():
+ log(f"File: {os.path.relpath(file)[2:]}")
+ for line_number, line_content in lines:
+ cleaned_line_content = line_content.rstrip("\n")
+ log(f"Line {line_number}: {cleaned_line_content}")
+ log("")
+
+ # Contruct the absolute path of binary that we want to check coverage for.
+ unit_tests, lit_tests = classify_tests(patch_path)
+ unit_binary = None
+ lit_binary = None
+ if lit_tests:
+ lit_binary = os.path.abspath(os.path.join(inst_build_dir, "bin", binary))
+ if unit_tests:
+ target = target_name(patch_path, inst_build_dir)
+ unit_binary = os.path.abspath(os.path.join(inst_build_dir, target))
+
+ # Build the LLVM in instrumented directory using LLVM_BUILD_INSTRUMENTED_COVERAGE.
+ rebuild = should_rebuild(inst_build_dir, patch_path, lit_binary or unit_binary)
+ if rebuild:
+ delete_profraw(inst_build_dir)
+ build_llvm(inst_build_dir, build_dir, binary, projects, allowlist_path)
+ mark_build_success(inst_build_dir, patch_path)
+ else:
+ print("\n[patch-coverage] Skipping patch coverage (no changes)")
+ sys.exit(0)
+
+ # Run all the test cases of patch with instrumented binary.
+ inst_lit_path = os.path.join(inst_build_dir, "bin/llvm-lit")
+ patch_path = os.path.abspath(patch_path)
+ run_modified_lit_tests(inst_lit_path, patch_path, tests, inst_build_dir)
+ run_modified_unit_tests(build_dir, inst_build_dir, patch_path)
+
+ # Report covered and uncovered lines of each source file.
+ coverage_files = process_coverage_data(
+ source_files,
+ inst_build_dir,
+ lit_binary,
+ unit_binary,
+ patch_path,
+ )
+ report_covered_and_uncovered_lines(coverage_files, modified_lines)
+
+ # Remove redundant "default.profraw" generated in source root.
+ curr_dir = os.path.dirname(os.getcwd())
+ default_profraw_path = os.path.join(curr_dir, "default.profraw")
+ try:
+ os.remove(default_profraw_path)
+ except FileNotFoundError:
+ pass
+
+
+if __name__ == "__main__":
+ main()
diff --git a/llvm/utils/patch-coverage/patch.py b/llvm/utils/patch-coverage/patch.py
ne...
[truncated]
|
Contributor
Author
|
Ping! |
Contributor
Author
|
I am not sure who else to ask for review, but added few people who might be care about the idea and would have some review comments. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This patch adds a python tool to show the test coverage of a patch. Complete details can be found on README.md doc.
Example output for -
$ git patch-coverage llccommit fe5d5b7 (containing lit test).
[SelectionDAG] Salvage debuginfo when combining load and z|s ext instrs. (#188544)
$ git patch-coverage SupportTestscommit 14da7e5 (containing unit test)
[llvm][Support] Avoid silent truncation of socket paths (#190869)
Assisted-by: AI