Motivation:
Swift 5.9 is no longer supported, we should bump the tools version and
remove it from our CI.
Modifications:
* Bump the Swift tools version to Swift 5.10
* Remove Swift 5.9 jobs where appropriate in main.yml, pull_request.yml
Result:
Code reflects our support window.
# Motivation
We have quite a lot of shell scripts in our repo and want to make sure that they all pass `shellcheck`.
# Modification
This PR adds a GH action workflow to the soundness script for `shellcheck` and fixes up all errors and warnings.
# Result
No more shell/bash discussions
* Apply formatting
* Apply no block comments rule
* Apply OmitExplicitReturns
* Apple OnlyOneTrailingClosureArgument
* Apply NoAssignmentInExpressions
* Fix up DontRepeatTypeInStaticProperties lint errors
* Apply `OrderedImports`
* Apply `ReplaceForEachWithForLoop`
* format file
* Enable the formatting pipeline
* Adopt `AmbiguousTrailingClosureOverload`
* Fix license header
* Fix format check
* Fix `EndOfLineComment`
* Fix CI
* Adapt CI script to check if changes when running formatting
* Separate lint and format into to steps
* Fix format
* Adopt `UseEarlyExits`
* Revert "Adopt `UseEarlyExits`"
This reverts commit d1ac5bbe12.
Motivation:
Allocation counters are broken on macOS as described [here](https://github.com/apple/swift-nio/issues/2672). This PR fixes it.
Modifications:
`AtomicCounter` library is made `.dynamic`. This deduplicates the two (static) copies that used to exist. One embedded in the main binary, and the other in `HookedFunctions` dylib. This deplication fixes the issue, because there is just one copy of the counters in the process address space.
Result:
All the tests counting allocations and related statistics work on macOS.
* Update swift-tools-version to 5.7 in integration tests
* Fix flaky test
* Add missing AtomicCounter dependency in generated Package.swift
* Update swift-tools-version in integration test helper Swift projects
* Make test clearer
* Don't run MTELG integration test on Swift 5.11+
It hasn't been properly tested
Motivation
Fix build errors on Android
Modifications
- Fix previous Musl modifications that assumed Glibc wasn't imported on Android
- Add errors for all libc imports, so new platform ports error out early
Result
NIO builds natively on Android again, with all the same tests passing
* Extend the integration test harness to track FDs
Motivation
This patch extends the NIO integration test harness to track
file descriptors, in particular to search for leaks. This
change has been validated on Linux and Darwin, and in both cases
correctly diagnoses FD leaks.
The goal is to enable us to regression test for things like
Modifications
- Add support for hooking socket and close calls.
- Wire up this support into the test harness.
- Extend the test harness to handle the logging.
- Add new regression test for #2047.
Results
We can write regression tests for FD leaks.
* Disable FD checking in most builds.
I'm doing this for speed reasons
* Always print the leaked fds number
Motivation:
Our allocation counter tests still build for 5.0 by default. This isn't
great, but we've been getting away with it because building for 5.7
requires changing the way we express our dependencies. Doing that is a
big breaking pain in the neck that requires changes in multiple repos.
As a shorter-term goal, however, to enable testing in at least one more
repo (swift-nio-ssh), this PR lifts the version to 5.1, which is the
last release compatible with the Package.swift dependency structure we
use but that supports the Platforms we need to express.
Modifications:
- Move allocation counter to 5.1
- Add platforms from swift-nio-ssh
Result:
We can write allocation counter tests for swift-nio-ssh.
# Motivation
We landed the async bridge types a while back but never added allocation and performance tests. Since we expect these types to be used performance critical paths we really should cover those with tests.
# Modification
Extends the allocation counter scaffolding to support async tests. Furthermore, add allocations tests for both the writer and producer. Lastly, I a also added a performance test for the producer.
# Result
We now have baseline tests for the `NIOAsyncWriter` and `NIOAsyncSequenceProducer`
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
The allocation counter test framework only supports a single shared
file which is quite limiting, especially if you rely on generated code.
Modifications:
- The '-s' option can now be passed multiple times
- The symlink name for any of the passed files will be the basename of
the provided file rather than 'shared.swift'
Result:
Allocation counter test framework can use more than one shared file.
Motivation:
The alloc counters need to store the pointer to the original libc
implmentations of the hooked functions somewhere. Previously, we would
store them in thread locals. That works fine but creates quite some
overhead (and allocations) per thread (to do dlsym on every thread).
Modifications:
Instead, we now store the libc function pointers in (atomic) globals so
we need to only resolve each function once, no matter how many threads
we use.
Result:
Faster, and more accurate.
Motivation:
The Swift runtime is now using malloc_zone_*, we need to implement
replacements for these too. This is just a first pass, eventually, we
should implement _all_ replacements as `malloc_zone_memalign` which is
powerful enough to implement all others.
Modifications:
Provide new replacements.
Result:
Alloc tests work again on macOS.
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
The core problem is that we hook the system's allocator (malloc, free,
and friends) to get the most accurate allocation counts. This however
also means that all threads count into our limits which inherently means
we can't be 100% stable always.
Previously, we just waited a fixed 100ms and expected everything to be
as quiesced as it'll get. But we can do better.
Modifications:
- discard runs with negative remaining allocations (clearly bogus,
quiescing not long enough in a previous run)
- instead of waiting a fixed amount of time, wait until the results
appear quiesced (stable (for first run) or zero (for subsequent runs)
remaining allocation counts.
Result:
Allocation counter tests should be much more stable on busier machines.
Motivation:
Clang modules seem to mess with stuff like _GNU_SOURCE as soon as
they're included. We do need _GNU_SOURCE (but don't like the
`#define __USE_GNU` hack).
Modifications:
Include the modularised headers last.
Result:
Alloc counter tests work without `#define __USE_GNU`.
motivation: tsan is broken on ubuntu 18.04.use 16.04 so 5.2 build can pass
changes:
* change 5.2 docker setup to use 16.04 instead of 18.04
* add the ability to pipe arguments to the integration tests via the docker setup
* skip tests_02 integration tests suite since it broken on 5.2 (swift-pm issue?)
fake the new limits from #1362
Co-authored-by: tomer doron <tomer@apple.com>
Motivation:
It is useful/interesting to have an overview about total bytes
allocated, not only the number of malloc calls.
This is not used to fail tests though, more as an informational thing.
Modifications:
Introduce and hit new counter with allocated size whenever allocations
happen.
Result:
It is possible to inspect the total allocated bytes.
Motivation:
We should have better iOS compatibility.
Modifications:
- make all OS-conditional imports the same
- don't check TSI_S_ESTABLISHED (which is unavailable on iOS)
- update availability to reflect the OS versions we actually support
- make sure temporary UDS paths aren't too long
Result:
better iOS compatibility
Motivation:
The allocation tests shouldn't delete their working directory because
that's where you would navigate to debug them. Usually, they're run
through the integration tests which automatically clean up anyway.
Modifications:
Don't delete work dir
Result:
Easier to debug
Motivation:
The allocation counter tests should default to using the integration
tests' tmp dir.
Modifications:
Propagate the right tmp dir.
Result:
fewer orphaned tmp dirs lying around in case of error.
Motivation:
The new allocation counter suite hardcoded a few things:
- the NIO and NIOHTTP1 modules
- no extra dependencies apart from the repo you're in
Modifications:
remove these hardcoded things.
Result:
allocation counter script works for nio-http2
George Barnett
Rick Newton-Rogers
Franz Busch
Gregor Milos (Grzegorz Miłoś)
Gustavo Cairo
finagolfin
Cory Benfield
Johannes Weiss
Konrad `ktoso` Malawski