Motivation
* `check_benchmark_thresholds.sh` was updated to check for exit code 2
from `swift package benchmark thresholds check` to distinguish
threshold regression from build errors.
* It seems SwiftPM's `CommandPlugin` infrastructure always exits with code 1 when
`performCommand` throws, regardless of the error's raw value. So the
exit code is never 2, causing all regressions to fall through to the
`else` branch and be misreported as build errors, with no diff output.
Modifications
* Remove the `rc == 2` check and instead attempt `thresholds update`
for any non-zero `rc` from `thresholds check`.
* Use the result of `thresholds update` to distinguish regression
(success) from build error (failure).
* Remove `--exit-code` from `git diff` to prevent `set -uo pipefail`
from aborting the script before output is fully flushed.
Result
* Benchmark threshold regressions correctly output the `=== BEGIN DIFF
===` section again.
* Actual build errors are still correctly detected and reported.
### Motivation:
Currently if there is an error in the Benchmark targets, that prevents
the Benchmark to build correctly, we still get a CI success.
### Modifications:
- Check for the threshold changed error
### Result:
- More reliable CI
### Motivation:
The current method of parsing the ASCII table output is brittle and
because the text output and the stored thresholds in JSON have different
precisions doesn't always work.
### Modifications:
* In the event that a deviation is found re-run the benchmarks in update
mode and print the git diff to the logs.
* The threshold update script scrapes this from the logs and applies it
locally so that it can then be committed.
* Add the functionality in the update script to update thresholds based
on diff in logs
### Result:
More reliable threshold updates.
An example of this working:
https://github.com/apple/swift-nio/actions/runs/15302748893?pr=3257
---------
Co-authored-by: George Barnett <gbarnett@apple.com>
Rick Newton-Rogers
Fabian Fett
