Skip to content

Commit aa3cc0f

Browse files
yash-atreyaclaudeactions-usergithub-actions
authored
feat: benchmark suite (#10804)
* Add Foundry multi-version benchmarking suite - Automated benchmarking across multiple Foundry versions using hyperfine - Supports stable, nightly, and specific version tags (e.g., v1.0.0) - Benchmarks 5 major Foundry projects: account, v4-core, solady, morpho-blue, spark-psm - Tests forge test, forge build (no cache), and forge build (with cache) - Generates comparison tables in markdown format - Uses foundryup for version management - Exports JSON data for detailed analysis 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * Fix benchmark script JSON data extraction and table formatting - Fix relative path issue causing JSON files to fail creation - Convert benchmark directories to absolute paths using SCRIPT_DIR - Improve markdown table formatting with proper column names and alignment - Use unified table generation with string concatenation for better formatting - Increase benchmark runs from 3 to 5 for more reliable results - Use --prepare instead of --cleanup for better cache management - Remove stderr suppression to catch hyperfine errors - Update table headers to show units (seconds) for clarity 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * parallel benchmarking * refac: mv to benches/ dir * feat: criterion benches * fix: install foundry versions at once * nit * - setup benchmark repos in parallel - run forge build in parallet for forge-test bench - switch foundry versions - README specifying prereqs * feat: shell script to run benches * feat: ci workflow, fix script * update readme * feat: enhance benchmarking suite with version flexibility - Add `get_benchmark_versions()` helper to read versions from env var - Update all benchmarks to use version helper for consistency - Add `--versions` and `--force-install` flags to shell script - Enable all three benchmarks (forge_test, build_no_cache, build_with_cache) - Improve error handling for corrupted forge installations - Remove complex workarounds in favor of clear error messages The benchmarks now support custom versions via: ./run_benchmarks.sh --versions stable,nightly,v1.2.0 🤖 Generated with Claude Code Co-Authored-By: Claude <[email protected]> * latest bench * rm notes * remove shell based bench suite * feat: benches using criterion (#10805) * feat: criterion benches * - setup benchmark repos in parallel - run forge build in parallet for forge-test bench - switch foundry versions - README specifying prereqs * feat: shell script to run benches * feat: ci workflow, fix script * update readme * feat: enhance benchmarking suite with version flexibility - Add `get_benchmark_versions()` helper to read versions from env var - Update all benchmarks to use version helper for consistency - Add `--versions` and `--force-install` flags to shell script - Enable all three benchmarks (forge_test, build_no_cache, build_with_cache) - Improve error handling for corrupted forge installations - Remove complex workarounds in favor of clear error messages The benchmarks now support custom versions via: ./run_benchmarks.sh --versions stable,nightly,v1.2.0 🤖 Generated with Claude Code Co-Authored-By: Claude <[email protected]> * latest bench * rm notes * remove shell based bench suite --------- Co-authored-by: Claude <[email protected]> * unified benchmarker - * main.rs * forge version is controlled by the bin * parses criterion json to collect results - writes to LATEST.md * parallel bench * refac * refac benchmark results table generation * cleanup main.rs * rm dep * cleanup main.rs * deser estimate * nit * cleanup CriterionResult type * feat: specify repos via flag * nits * update bench ci and README * bench fuzz tests * fmt * license * coverage bench * nits * clippy * clippy * separate benches into different jobs in CI * remove criterion * feat: hyperfine setup in foundry-bench * forge version details: hash and date * run benches again - run cov with --ir-min * del * bench in separate ci jobs * move combine bench results logic to scripts * setup foundryup in ci * setup foundryup fix * clippy * ci: run on foundry-runner * ci: don't use wget * ci: add build essential * ci: nodejs and npm * install hyperfine for each job * fix * install deps script * add benchmark-setup, using setup-node action, remove redundant files * fix * fix * checkout repo * nits * nit * fix * show forge test result in top comment * force foundry install * fix bench comment aggregation * nit * fix * feat: create PR for manual runs, else commit in the PR itself. * fix * fetch and pull * chore(`benches`): update benchmark results 🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions) Co-Authored-By: github-actions <[email protected]> * fix * chore(`benches`): update benchmark results 🤖 Generated with [Foundry Benchmarks](https://github.com/foundry-rs/foundry/actions) Co-Authored-By: github-actions <[email protected]> --------- Co-authored-by: Claude <[email protected]> Co-authored-by: GitHub Action <[email protected]> Co-authored-by: github-actions <[email protected]>
1 parent 387928b commit aa3cc0f

File tree

15 files changed

+1916
-1
lines changed

15 files changed

+1916
-1
lines changed

‎.github/actions/benchmark-setup/action.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
name: "Benchmark Setup"
2+
description: "Common setup steps for benchmark jobs"
3+
4+
runs:
5+
using: "composite"
6+
steps:
7+
- name: Setup Foundry
8+
shell: bash
9+
env:
10+
FOUNDRY_DIR: ${{ github.workspace }}/.foundry
11+
run: |
12+
./.github/scripts/setup-foundryup.sh
13+
echo "${{ github.workspace }}/.foundry/bin" >> $GITHUB_PATH
14+
15+
- name: Setup Node.js
16+
uses: actions/setup-node@v4
17+
with:
18+
node-version: "24"
19+
20+
- name: Install hyperfine
21+
shell: bash
22+
run: |
23+
curl -L https://github.com/sharkdp/hyperfine/releases/download/v1.19.0/hyperfine-v1.19.0-x86_64-unknown-linux-gnu.tar.gz | tar xz
24+
sudo mv hyperfine-v1.19.0-x86_64-unknown-linux-gnu/hyperfine /usr/local/bin/
25+
rm -rf hyperfine-v1.19.0-x86_64-unknown-linux-gnu
26+
27+
- name: Download benchmark binary
28+
uses: actions/download-artifact@v4
29+
with:
30+
name: foundry-bench
31+
32+
- name: Make binary executable
33+
shell: bash
34+
run: chmod +x foundry-bench

‎.github/scripts/combine-benchmarks.sh‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
#!/bin/bash
2+
set -euo pipefail
3+
4+
# Script to combine individual benchmark results into LATEST.md
5+
# Usage: ./combine-benchmarks.sh <output_dir>
6+
7+
OUTPUT_DIR="${1:-benches}"
8+
9+
# Create output directory if it doesn't exist
10+
mkdir -p "$OUTPUT_DIR"
11+
12+
# Define the benchmark files and their section names
13+
declare -A BENCHMARK_FILES=(
14+
["forge_test_bench.md"]="Forge Test"
15+
["forge_build_bench.md"]="Forge Build"
16+
["forge_coverage_bench.md"]="Forge Coverage"
17+
)
18+
19+
# Function to extract a specific section from a benchmark file
20+
extract_section() {
21+
local file=$1
22+
local section=$2
23+
local in_section=0
24+
25+
while IFS= read -r line; do
26+
if [[ "$line" =~ ^##[[:space:]]+"$section" ]]; then
27+
in_section=1
28+
echo "$line"
29+
elif [[ $in_section -eq 1 && "$line" =~ ^##[[:space:]] && ! "$line" =~ ^##[[:space:]]+"$section" ]]; then
30+
break
31+
elif [[ $in_section -eq 1 ]]; then
32+
echo "$line"
33+
fi
34+
done < "$file"
35+
}
36+
37+
# Function to extract summary info (repos and versions) from a file
38+
extract_summary_info() {
39+
local file=$1
40+
local in_summary=0
41+
local in_repos=0
42+
local in_versions=0
43+
44+
while IFS= read -r line; do
45+
# Check for Summary section
46+
if [[ "$line" =~ ^##[[:space:]]+Summary ]]; then
47+
in_summary=1
48+
continue
49+
fi
50+
51+
# Check for Repositories Tested subsection
52+
if [[ $in_summary -eq 1 && "$line" =~ ^###[[:space:]]+Repositories[[:space:]]+Tested ]]; then
53+
in_repos=1
54+
echo "### Repositories Tested"
55+
echo
56+
continue
57+
fi
58+
59+
# Check for Foundry Versions subsection
60+
if [[ $in_summary -eq 1 && "$line" =~ ^###[[:space:]]+Foundry[[:space:]]+Versions ]]; then
61+
in_repos=0
62+
in_versions=1
63+
echo "### Foundry Versions"
64+
echo
65+
continue
66+
fi
67+
68+
# End of summary section
69+
if [[ $in_summary -eq 1 && "$line" =~ ^##[[:space:]] && ! "$line" =~ ^##[[:space:]]+Summary ]]; then
70+
break
71+
fi
72+
73+
# Output repo or version lines
74+
if [[ ($in_repos -eq 1 || $in_versions -eq 1) && -n "$line" ]]; then
75+
echo "$line"
76+
fi
77+
done < "$file"
78+
}
79+
80+
# Function to extract benchmark table from a section
81+
extract_benchmark_table() {
82+
local file=$1
83+
local section=$2
84+
local in_section=0
85+
local found_table=0
86+
87+
while IFS= read -r line; do
88+
if [[ "$line" =~ ^##[[:space:]]+"$section" ]]; then
89+
in_section=1
90+
continue
91+
elif [[ $in_section -eq 1 && "$line" =~ ^##[[:space:]] && ! "$line" =~ ^##[[:space:]]+"$section" ]]; then
92+
break
93+
elif [[ $in_section -eq 1 ]]; then
94+
# Skip empty lines before table
95+
if [[ -z "$line" && $found_table -eq 0 ]]; then
96+
continue
97+
fi
98+
# Detect table start
99+
if [[ "$line" =~ ^\|[[:space:]]*Repository ]]; then
100+
found_table=1
101+
fi
102+
# Output table lines
103+
if [[ $found_table -eq 1 && -n "$line" ]]; then
104+
echo "$line"
105+
fi
106+
fi
107+
done < "$file"
108+
}
109+
110+
# Function to extract system information
111+
extract_system_info() {
112+
local file=$1
113+
# Extract from System Information to end of file (EOF)
114+
awk '/^## System Information/ { found=1; next } found { print }' "$file"
115+
}
116+
117+
# Start building LATEST.md
118+
cat > "$OUTPUT_DIR/LATEST.md" << EOF
119+
# 📊 Foundry Benchmark Results
120+
121+
**Generated at**: $(date -u '+%Y-%m-%d %H:%M:%S UTC')
122+
123+
EOF
124+
125+
# Process each benchmark file
126+
FIRST_FILE=1
127+
SYSTEM_INFO=""
128+
129+
for bench_file in "forge_test_bench.md" "forge_build_bench.md" "forge_coverage_bench.md"; do
130+
if [ -f "$OUTPUT_DIR/$bench_file" ]; then
131+
echo "Processing $bench_file..."
132+
133+
# Get the section name
134+
case "$bench_file" in
135+
"forge_test_bench.md")
136+
SECTION_NAME="Forge Test"
137+
;;
138+
"forge_build_bench.md")
139+
SECTION_NAME="Forge Build"
140+
;;
141+
"forge_coverage_bench.md")
142+
SECTION_NAME="Forge Coverage"
143+
;;
144+
esac
145+
146+
# Add section header
147+
echo "## $SECTION_NAME" >> "$OUTPUT_DIR/LATEST.md"
148+
echo >> "$OUTPUT_DIR/LATEST.md"
149+
150+
# Add summary info (repos and versions)
151+
extract_summary_info "$OUTPUT_DIR/$bench_file" >> "$OUTPUT_DIR/LATEST.md"
152+
echo >> "$OUTPUT_DIR/LATEST.md"
153+
154+
# Handle different benchmark types
155+
if [[ "$bench_file" == "forge_test_bench.md" ]]; then
156+
# Extract both Forge Test and Forge Fuzz Test tables
157+
extract_benchmark_table "$OUTPUT_DIR/$bench_file" "Forge Test" >> "$OUTPUT_DIR/LATEST.md"
158+
159+
# Check if Forge Fuzz Test section exists
160+
if grep -q "^## Forge Fuzz Test" "$OUTPUT_DIR/$bench_file"; then
161+
echo >> "$OUTPUT_DIR/LATEST.md"
162+
echo "## Forge Fuzz Test" >> "$OUTPUT_DIR/LATEST.md"
163+
echo >> "$OUTPUT_DIR/LATEST.md"
164+
extract_benchmark_table "$OUTPUT_DIR/$bench_file" "Forge Fuzz Test" >> "$OUTPUT_DIR/LATEST.md"
165+
fi
166+
elif [[ "$bench_file" == "forge_build_bench.md" ]]; then
167+
# Extract No Cache table
168+
echo "### No Cache" >> "$OUTPUT_DIR/LATEST.md"
169+
echo >> "$OUTPUT_DIR/LATEST.md"
170+
extract_benchmark_table "$OUTPUT_DIR/$bench_file" "Forge Build (No Cache)" >> "$OUTPUT_DIR/LATEST.md"
171+
echo >> "$OUTPUT_DIR/LATEST.md"
172+
173+
# Extract With Cache table
174+
echo "### With Cache" >> "$OUTPUT_DIR/LATEST.md"
175+
echo >> "$OUTPUT_DIR/LATEST.md"
176+
extract_benchmark_table "$OUTPUT_DIR/$bench_file" "Forge Build (With Cache)" >> "$OUTPUT_DIR/LATEST.md"
177+
else
178+
# Extract the benchmark table for other types
179+
extract_benchmark_table "$OUTPUT_DIR/$bench_file" "$SECTION_NAME" >> "$OUTPUT_DIR/LATEST.md"
180+
fi
181+
182+
echo >> "$OUTPUT_DIR/LATEST.md"
183+
184+
# Extract system info from first file only
185+
if [[ $FIRST_FILE -eq 1 ]]; then
186+
SYSTEM_INFO=$(extract_system_info "$OUTPUT_DIR/$bench_file")
187+
FIRST_FILE=0
188+
fi
189+
else
190+
echo "Warning: $bench_file not found, skipping..."
191+
fi
192+
done
193+
194+
# Add system information at the end
195+
if [[ -n "$SYSTEM_INFO" ]]; then
196+
echo "## System Information" >> "$OUTPUT_DIR/LATEST.md"
197+
echo >> "$OUTPUT_DIR/LATEST.md"
198+
echo "$SYSTEM_INFO" >> "$OUTPUT_DIR/LATEST.md"
199+
fi
200+
201+
echo "Successfully combined benchmark results into $OUTPUT_DIR/LATEST.md"

‎.github/scripts/commit-and-read-benchmarks.sh‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
#!/bin/bash
2+
set -euo pipefail
3+
4+
# Script to commit benchmark results and read them for GitHub Actions output
5+
# Usage: ./commit-and-read-benchmarks.sh <output_dir> <github_event_name> <github_repository>
6+
7+
OUTPUT_DIR="${1:-benches}"
8+
GITHUB_EVENT_NAME="${2:-pull_request}"
9+
GITHUB_REPOSITORY="${3:-}"
10+
11+
# Global variable for branch name
12+
BRANCH_NAME=""
13+
14+
# Function to commit benchmark results
15+
commit_results() {
16+
echo "Configuring git..."
17+
git config --local user.email "[email protected]"
18+
git config --local user.name "GitHub Action"
19+
20+
# For PR runs, fetch and checkout the PR branch to ensure we're up to date
21+
if [ "$GITHUB_EVENT_NAME" = "pull_request" ] && [ -n "${GITHUB_HEAD_REF:-}" ]; then
22+
echo "Fetching latest changes for PR branch: $GITHUB_HEAD_REF"
23+
git fetch origin "$GITHUB_HEAD_REF"
24+
git checkout -B "$GITHUB_HEAD_REF" "origin/$GITHUB_HEAD_REF"
25+
fi
26+
27+
echo "Adding benchmark file..."
28+
git add "$OUTPUT_DIR/LATEST.md"
29+
30+
if git diff --staged --quiet; then
31+
echo "No changes to commit"
32+
else
33+
echo "Committing benchmark results..."
34+
git commit -m "chore(\`benches\`): update benchmark results
35+
36+
🤖 Generated with [Foundry Benchmarks](https://github.com/${GITHUB_REPOSITORY}/actions)
37+
38+
Co-Authored-By: github-actions <[email protected]>"
39+
40+
echo "Pushing to repository..."
41+
if [ "$GITHUB_EVENT_NAME" = "workflow_dispatch" ]; then
42+
# For manual runs, we're on a new branch
43+
git push origin "$BRANCH_NAME"
44+
elif [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
45+
# For PR runs, push to the PR branch
46+
if [ -n "${GITHUB_HEAD_REF:-}" ]; then
47+
echo "Pushing to PR branch: $GITHUB_HEAD_REF"
48+
git push origin "$GITHUB_HEAD_REF"
49+
else
50+
echo "Error: GITHUB_HEAD_REF not set for pull_request event"
51+
exit 1
52+
fi
53+
else
54+
# This workflow should only run on workflow_dispatch or pull_request
55+
echo "Error: Unexpected event type: $GITHUB_EVENT_NAME"
56+
echo "This workflow only supports 'workflow_dispatch' and 'pull_request' events"
57+
exit 1
58+
fi
59+
echo "Successfully pushed benchmark results"
60+
fi
61+
}
62+
63+
# Function to read benchmark results and output for GitHub Actions
64+
read_results() {
65+
if [ -f "$OUTPUT_DIR/LATEST.md" ]; then
66+
echo "Reading benchmark results..."
67+
68+
# Output full results
69+
{
70+
echo 'results<<EOF'
71+
cat "$OUTPUT_DIR/LATEST.md"
72+
echo 'EOF'
73+
} >> "$GITHUB_OUTPUT"
74+
75+
# Format results for PR comment
76+
echo "Formatting results for PR comment..."
77+
FORMATTED_COMMENT=$("$(dirname "$0")/format-pr-comment.sh" "$OUTPUT_DIR/LATEST.md")
78+
79+
{
80+
echo 'pr_comment<<EOF'
81+
echo "$FORMATTED_COMMENT"
82+
echo 'EOF'
83+
} >> "$GITHUB_OUTPUT"
84+
85+
echo "Successfully read and formatted benchmark results"
86+
else
87+
echo 'results=No benchmark results found.' >> "$GITHUB_OUTPUT"
88+
echo 'pr_comment=No benchmark results found.' >> "$GITHUB_OUTPUT"
89+
echo "Warning: No benchmark results found at $OUTPUT_DIR/LATEST.md"
90+
fi
91+
}
92+
93+
# Main execution
94+
echo "Starting benchmark results processing..."
95+
96+
# Create new branch for manual runs
97+
if [ "$GITHUB_EVENT_NAME" = "workflow_dispatch" ]; then
98+
echo "Manual workflow run detected, creating new branch..."
99+
BRANCH_NAME="benchmarks/results-$(date +%Y%m%d-%H%M%S)"
100+
git checkout -b "$BRANCH_NAME"
101+
echo "Created branch: $BRANCH_NAME"
102+
103+
# Output branch name for later use
104+
echo "branch_name=$BRANCH_NAME" >> "$GITHUB_OUTPUT"
105+
fi
106+
107+
# Always commit benchmark results
108+
echo "Committing benchmark results..."
109+
commit_results
110+
111+
# Always read results for output
112+
read_results
113+
114+
echo "Benchmark results processing complete"

‎.github/scripts/format-pr-comment.sh‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/bin/bash
2+
set -euo pipefail
3+
4+
# Script to format benchmark results for PR comment
5+
# Usage: ./format-pr-comment.sh <benchmark_results_file>
6+
7+
RESULTS_FILE="${1:-}"
8+
9+
if [ -z "$RESULTS_FILE" ] || [ ! -f "$RESULTS_FILE" ]; then
10+
echo "Error: Benchmark results file not provided or does not exist"
11+
exit 1
12+
fi
13+
14+
# Read the file content
15+
CONTENT=$(cat "$RESULTS_FILE")
16+
17+
# Find where "## Forge Build" starts and split the content
18+
# Extract everything before "## Forge Build"
19+
BEFORE_FORGE_BUILD=$(echo "$CONTENT" | awk '/^## Forge Build$/ {exit} {print}')
20+
21+
# Extract everything from "## Forge Build" onwards
22+
FROM_FORGE_BUILD=$(echo "$CONTENT" | awk '/^## Forge Build$/ {found=1} found {print}')
23+
24+
# Output the formatted comment with dropdown
25+
cat << EOF
26+
${BEFORE_FORGE_BUILD}
27+
28+
<details>
29+
<summary>📈 View all benchmark results</summary>
30+
31+
${FROM_FORGE_BUILD}
32+
33+
</details>
34+
EOF

0 commit comments

Comments
 (0)