[ci] Add benchmark comparison

Updates the parser-benchmark script to use a benchmarking framework, and 

also brings in yargs to make it easier to handle parsing arguments. 

Non-CI usage: 

``` 

$ cd bench/parser-benchmark 

$ yarn bench --help 

Options: 

--help        Show help                                              [boolean] 

--ci                                                [boolean] [default: false] 

--sampleSize                                             [number] [default: 1] 

$ yarn bench --sampleSize=3 

[ISOBENCH ENDED] Parser Benchmark (3 times) 

OXC           - 32 op/s. 3 samples in 9518 ms. 5.288x (BEST) 

SWC           - 20 op/s. 3 samples in 9487 ms. 3.256x 

HermesParser  - 6 op/s. 3 samples in 9124 ms. 1.000x (WORST) 

Forget (Rust) - 15 op/s. 3 samples in 9749 ms. 2.499x 

``` 

In CI this outputs JSON instead of logging to stdout. The results are 

stored in github's action cache and used as a point of comparison when 

new commits are pushed to main. The comparison should be shown on 

workflow [summary 
pages](https://github.com/facebook/react-forget/actions/runs/5956421081), but 
nothing will be populated until we merge this PR. 

We intentionally only run this workflow on main as any run would 

override the last cached result, so a PR with multiple pushes would then 

start having comparisons with itself.
This commit is contained in:
Lauren Tan
2023-08-23 17:58:23 -04:00
parent aeae3ba0a7
commit 80efd75b8e
2 changed files with 54 additions and 18 deletions
+54
View File
@@ -0,0 +1,54 @@
# Runs the parser benchmark on every push to main, and compares it with the previous run
name: Parser Benchmark (Rust)
on:
push:
branches: ["main"]
paths:
- .github/workflows/**
- crates/**
- Cargo.*
- ./*.toml
env:
CARGO_TERM_COLOR: always
RUSTFLAGS: -Dwarnings
jobs:
bench:
name: Parser Benchmark
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: Swatinem/rust-cache@v2
- run: yarn install --frozen-lockfile
working-directory: crates/forget_napi
- run: mkdir dist -p
working-directory: crates/forget_napi
- run: yarn build
working-directory: crates/forget_napi
- run: yarn install --frozen-lockfile
working-directory: bench/parser-benchmark
- name: Run benchmark
run: yarn bench --ci --sampleSize=3 | tee results.json
working-directory: bench/parser-benchmark
# Download previous benchmark result from cache (if exists)
- name: Download previous benchmark data
uses: actions/cache@v3
with:
path: ./cache
key: ${{ runner.os }}-benchmark
- name: Store benchmark result
uses: benchmark-action/github-action-benchmark@v1
with:
name: "${{ runner.os }} Benchmark"
tool: customBiggerIsBetter
output-file-path: bench/parser-benchmark/results.json
# Where the previous data file is stored
external-data-json-path: ./cache/parser-benchmark-results.json
# Workflow will fail when an alert happens
fail-on-alert: true
# Enable Job Summary for PRs
summary-always: true
# GitHub API token to make a commit comment
github-token: ${{ secrets.GITHUB_TOKEN }}
-18
View File
@@ -66,21 +66,3 @@ jobs:
- uses: Swatinem/rust-cache@v2
- name: cargo build
run: cargo build --release
bench:
name: Parser Benchmark
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: Swatinem/rust-cache@v2
- run: yarn install --frozen-lockfile
working-directory: crates/forget_napi
- run: mkdir dist -p
working-directory: crates/forget_napi
- run: yarn build
working-directory: crates/forget_napi
- run: yarn install --frozen-lockfile
working-directory: bench/parser-benchmark
- name: Run benchmark
run: echo "$(node index.js --ci)" >> $GITHUB_STEP_SUMMARY
working-directory: bench/parser-benchmark