Motivation
* Other libraries often have byte buffer storage with similar concepts as
NIO's `ByteBuffer`, linear storage with reader and writer indices - it would
be nice to have efficient conversion to and from these objects.
* Efficienct conversion would likely mean taking ownership of the byte
storage allocated by the other object and exposing `ByteBuffer`s
internals so that they may be owned by another object, there is no API
for this currently.
Modifications
* Added new methods behind `CustomByteBufferAllocator` SPI.
* Added public `ByteBuffer` initializer for adopting externally-allocated memory
* Made `ByteBufferAllocator` initializer public with custom `malloc`/`realloc`/`free`/`memcpy` hooks
* Updated `realloc` signature to pass old capacity alongside new capacity to allow implementations to
copy over bytes
* Added `withVeryUnsafeMutableBytesWithStorageManagement` for mutable storage access
* Updated all existing tests using internal allocator API to use the new API
* Added tests for custom allocator functionality
Result
* Custom allocators can manage `ByteBuffer` memory with full size information
* External systems can transfer memory ownership to `ByteBuffer`
* System allocator maintains zero overhead due to inlining of the wrapper
* API (SPI) is extensible for future allocation system integrations
Apple platform simulator CI failures
### Motivation
* Several tests failed when testing against simulators for Apple
platforms (iOS, tvOS, watchOS, visionOS)
### Modifications
* Added `@available(macOS 13, iOS 16, tvOS 16, watchOS 9, *)` to each
`@Test` method in `NIOThreadPoolTest`, `EventLoopFutureTest`, and
`NIOTransportAccessibleChannelCoreTests` to match the availability of
the APIs they use (`timeLimit` and `withUnsafeTransportIfAvailable`).
* Removed `.timeLimit(.minutes(1))` from the `@Suite` declarations on
`NIOThreadPoolTest` and `EventLoopFutureTest` - the `@Suite` macro
cannot be used with `@available`, so `timeLimit` cannot be used.
* Skip `testHomeDirectoryFromPasswd` on simulator targets via
`#if targetEnvironment(simulator)` / `XCTSkip`, since simulators retu>
success but nil information when invoking `getpwuid_r`.
### Result
* Apple simulator platform builds and test runs no longer fail due to
these issues. Some infrastructure issues remain to be resolved
separately.
Adds capability to NIOFS to copy regular files and symlink, allowing to
overwrite the destination.
### Motivation:
Per https://github.com/apple/swift-nio/issues/3403 and
https://github.com/apple/swift-nio/pull/3470 we want to add
`replaceExisting: bool` to `FileSystem.copyItem`.
### Modifications:
1. Adds `replaceExisting: bool` parameter to
`FileSystemProtocol.copyItem`.
2. Adds `replaceExisting: bool` parameter to `FileSystem.copyItem` and
implementation for regular files and symbolic links.
3. Adds tests.
---------
Co-authored-by: George Barnett <gbarnett@apple.com>
## Motivation
NIO channels abstract over an underlying transport mechanisms (sockets,
pipes, etc.), which users typically need not interact with. However,
there are scenarios where users need direct access to the underlying
transport for low-level operations that work outside NIOs abstraction.
One example is performing out out-of-band operations on the underlying
file descriptor for a socket-based channel.
This PR adds a structured way to access the underlying transport of a
channel, for channels that choose to implement it.
## Modifications
- Add a new `public protocol
NIOTransportAccessibleChannelCore<Transport>` which provides a scoped
`withUnsafeTransport(_:)`, used for channel implementations to opt-in.
- Add conformance to
`NIOTransportAccessibleChannel<NIOBSDSocket.Handle>` for
`BaseSocketChannel`, to make this API available for all socket-based
channels, including channels returned from the socket-based bootstrap
public APIs.
- Add public API
`ChannelPipeline.SynchronousOperations.withUnsafeTransportIfAvailable(_:)`
Note that not all channels need to or should their transport, which is
why this was added as an additional protocol that refines `ChannelCore`,
vs. extending `Channel` or `ChannelCore` with a default implementation.
The protocol uses a primary associated type allowing channels to provide
typed access to the transport. E.g. this could be used in NIO Transport
Services to expose the underlying `NWConnection`, if desired.
The method itself is spelled with "unsafe", uses scoped access, and has
clear documentation that users must not violated any of NIOs assumptions
about the state of the underlying transport. It's very much not intended
for every day use. It being on `ChannelCore` should defer users of the
low-level API, since `Channel._channelCore` is marked as for NIO
internal use, and the public `withUnsafeTransportIfAvailable(_:)` will
take care of the runtime checks for channels that have opted into this
API.
## Result
- New opt-in API for channel implementations to expose their underlying
transport
- All socket-based channels from NIOPosix now expose the underlying
socket file descriptor
- New API
`ChannelPipeline.SynchronousOperations.withUnsafeTransportIfAvailable(_:)`
for users
---------
Co-authored-by: Agam Dua <agam_dua@apple.com>
### Motivation
Issue #2773 reports a problem where channels can receive
`channelInactive` before `channelActive`. This breaks expectations and
should not happen. Instead of sending them in the wrong order, this PR
suppresses both of them in those cases.
### Modifications
* Add a test to reproduce the problematic behavior.
* Update the state machine to split the active state into two. This
decouples succeeding the promise from sending the signal. Transiting to
closing only sends `channelInactive` when originating from the state
that sent `channelActive`.
### Result
Address #2773.
Edit: Updated modifications to reflect latests commit.
Adds three new system call wrappers for the `symlinkat`, `renameatx_np`,
and `unlinkat` system calls.
### Motivation:
Related to https://github.com/apple/swift-nio/issues/3403 and
https://github.com/apple/swift-nio/pull/3470. This PR adds syscall
wrappers needed to atomically overwrite existing files or symlinks at
the destination during copy operations. On Linux, atomic overwrites
require a "copy to temp file, then rename" strategy. We use the `*at`
family of syscalls (which operate relative to directory file
descriptors) to avoid TOCTOU race conditions.
### Modifications:
1. Adds three system call wrappers for the `symlinkat`, `renameatx_np`,
and `unlinkat` system calls.
2. Adds related tests
3. Updates the `FileSystemError` for `symlink` and `unlink` to take in
the system call name to allow for the `*at` names to be passed.
Ports NIOThreadPoolTest in NIOPosix from XCTest to Swift Testing.
### Motivation:
Tests should use the latest Swift Testing framework when possible.
### Modifications:
- Ported `NIOThreadPoolTest` from XCTest to Swift Testing
- Updated test implementation from Dispatch to Swift Concurrency as
needed to support test functionality
### Result:
All tests continue to pass, NIOThreadPoolTest now has more modern Swift
Testing syntax.
Port `EventLoopFutureTest` to Swift Testing. Includes some adjustments
to ensure tests are ran from the event loop, among other minor changes
### Motivation:
Tests should use the latest Swift Testing framework when possible.
### Modifications:
- Ported `EventLoopFutureTest ` from XCTest to Swift Testing
- Consolidated instantiations of `EmbeddedEventLoop()` into a private
function called `makeEventLoop` to allow for easier injection of a
different loop from a single location.
- Migrated some `.wait` usages to `.get` to allow tests to run async.
### Result:
All tests continue to pass, EventLoopFutureTest now has more modern
Swift Testing syntax.
We have a number of copies of `UnsafeTransfer` and two copies of
`UnsafeMutableTransferBox` in our code base. Before introducing more of
those, lets centralize to just one using a `package` access modifier.
### Motivation:
Channels based on `BaseSocketChannel` throw in both `getOption` and
`setOption` if the channel has been closed, since the `setsockopt` will
fail. However, the current behavior of `EmbeddedChannel` is options
remain writable and readable on closed channels.
There are situations where we'd like to be able to model the runtime
behaviour of the real channel in tests, e.g. to test this fix:
https://github.com/apple/swift-nio-extras/pull/304.
### Modifications:
- Add API to enable throwing in `EmbeddedChannel.getOption` and
`.setOption` if channel is closed.
- Add a test for this new behavior.
### Result:
- New API to enable throwing in `EmbeddedChannel.getOption` and
`.setOption` if channel is closed.
- No observable change for existing users of `EmbeddedChannel`.
### Motivation:
In reviewing a copy of this file in swift-log, @czechboy0 noticed that
the call to `pthread_mutexattr_destroy` is missing. Since
pthread_mutexattr_t is like allocated on the stack rather than heap on
most platforms, this is likely not a memory leak. But for correctness
the missing destroy call should be added.
This change is the start of a resolution
[requested](https://github.com/apple/swift-log/pull/398#discussion_r2689459360)
in a corresponding Lock rollout in swift-log
### Modifications:
- Added missing call to `pthread_mutexattr_destroy`
- Duplicated test for `NIOLock` to also test `Lock`, as a sanity test.
### Result:
The new sanity test passes. If there are other ways this change can be
realistically verified as safe and proper, please feel free to provide
feedback in the PR.
`NIOTypedHTTPClientUpgradeHandler.handlerAdded` will write upgrade
request if the channel is already active
### Motivation:
This has been added to make it easier to implement a websocket client
that supports proxies. If the upgrade handler is added after the proxy
connect has been processed previously nothing would happen.
### Modifications:
- Added new function
`NIOTypedHTTPClientUpgradeHandler.writeUpgradeRequest()` function. Call
this from `channelActive` and also from `handlerAdded` if the channel is
already active.
- Added test `testUpgradeHappensAfterHandlerAdded`
### Result:
NIOTypedHTTPClientUpgradeHandler now also works if it is added to a
channel after the channel is active
---------
Co-authored-by: Cory Benfield <lukasa@apple.com>
Add homeDirectory accessor to FileSystem
### Motivation:
Addresses #3381 by adding a `homeDirectory` property to FileSystem,
equivalent to `FileManager.default.homeDirectoryForCurrentUser`.
### Modifications:
- Added `homeDirectory` property to `FileSystemProtocol` and implemented
in `FileSystem`
- Implementation checks `HOME` environment variable first, falls back to
`USERPROFILE` on Windows, or uses `getpwuid_r(3)` on POSIX systems
- Added system call wrappers (`system_getuid`, `libc_getpwuid_r`) with
proper platform guards
- Added `FileSystemError.getpwuid_r()` error helper
### Result:
Users can now access the home directory via `FileSystem.homeDirectory`,
returning a `FilePath` asynchronously. Follows the same pattern as
`currentWorkingDirectory` and `temporaryDirectory`.
Co-authored-by: Cory Benfield <lukasa@apple.com>
### Motivation:
Currently, pending consumer closures remain in the
`{in}{out}boundBufferConsumer` queues of `EmbeddedChannelCore` even
after the channel closes. It is also possible to enqueue consumers
*after* the channel closes. In these cases, the consumer closures will
never be invoked and this can lead to unfavourable behaviour, as
observed in `NIOAsyncTestingChannel`'s `waitFor{In}{Out}boundWrite`
methods (the only places these queues are currently used).
`NIOAsyncTestingChannel`'s `waitFor{In}{Out}boundWrite` methods complete
a continuation *inside* the consumer closure. In the cases described
above, the continuation never completes and therefore
`waitFor{In}{Out}boundWrite` never returns.
### Modifications:
- Updated the element type in `EmbeddedChannelCore`'s
`{in}{out}boundBufferConsumer` from `(NIOAny) -> Void` to
`(Result<NIOAny, Error>) -> Void`.
- This is so that the `.failure` case can be used to notify the consumer
closure that the channel has closed.
- Changed the visibility of the `{in}{out}boundBufferConsumer`
properties from `internal` to `private` in order to prevent the queues
from being accessed and being appended to without the call site
considering whether the channel has been closed.
- Added new `internal` methods named
`enqueue{In}{Out}boundBufferConsumer(_:)` which take the consumer
closure as an argument and only append to the corresponding queue if the
channel isn't closed.
- If the channel is closed, the consumer closure is invoked immediately
with `.failure(ChannelError.ioOnClosedChannel)`.
- Updated `EmbeddedChannelCore`'s `close0` method to return a
`.failure(ChannelError.ioOnClosedChannel)` result to each closure in
`{in}{out}boundBufferConsumer` and empty both buffers.
- Updated `NIOAsyncTestingChannel`'s `waitFor{In}{Out}boundWrite` to
throw an error in the continuation upon receiving a `.failure` result.
- Added associated test cases.
### Result:
`EmbeddedChannelCore`'s `{in}{out}boundBufferConsumer` queues can be
used more safely: all pending closures will be invoked upon channel
close. As a result, `NIOAsyncTestingChannel`'s
`waitFor{In}{Out}boundWrite` no longer indefinitely blocks when the
channel closes.
Fix coreCount on Linux when using cgroup v2 with CFS throttling disabled
### Motivation:
When using `swift-nio` on Linux with cgroup v2 enabled, but with CFS
throttling disabled, it falls back to attempting to read the cpuset file
at the cgroup v1 path. This does not exist, which in turns falls back to
returning `_SC_NPROCESSORS_ONLN`, which will return the total number of
cores available (ignoring cgroup assignments).
This has unexpected effects, including the default behaviour of starting
the `MultiThreadedEventLoopGroup.singleton` with significantly more
event loops than cores available to the workload.
### Modifications:
- Adds `SystemCalls.statfs`, and associated constants, to determine the
cgroup version.
- Adds `Linux.cgroupVersion()` API to expose cgroup version.
- Adds `Linux.cgroupV2MountPoint` variable to determine the cgroup v2
mount point.
- Adds `Linux.cpuSetPathV1` & `Linux.cpuSetPathV2` (and
`Linux.cpuSetPath` convenience) variables to determine the correct cpu
set path.
- Alters `System.coreCount` to use the appropriate logic from above to
ensure that `cpuset.cpus` is parsed from the correct location.
### Result:
`Linux.coreCount` should correctly parse and return the core count on
Linux cgroup v2 enabled systems (when CFS throttling is disabled), while
maintaining correctness for other configurations.
---------
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Co-authored-by: Cory Benfield <lukasa@apple.com>
Add ByteBuffer.readableBytesUInt8Span
### Motivation:
Provide access to readableBytes as a Span<UInt8>
### Modifications:
Added computed property `ByteBuffer.readableBytesUInt8Span`. I also
attempted to add `ByteBuffer.mutableReadableBytesUInt8Span` but due to
an apparently faulty exclusivity issue I couldn't get this to work. See
https://github.com/swiftlang/swift/issues/81218.
### Result:
You can access the readable bytes of a ByteBuffer as a Span<UInt8>
Motivation:
This is a follow up to 5bf841dd to handle the yield task being cancelled
between dropping the lock and re-acquiring it a moment later in the
'withContinuation' block.
Modifications:
- Add an extra cancellation check when yielding with a continuation.
Result:
Fewer issues
### Motivation:
`NIOAsyncTestingChannel` stored its `localAddress` and `remoteAddress`
in a locked storage on itself for thread safety, however in doing so
left us open to bugs because a handler grabbing the addresses of the
context had no visibility of the values.
### Modifications:
Reach into `EmbeddedChannelCore` for the addresses instead of storing
them on the `NIOAsyncTestinghannel`. I also considered a delegate
approach where the `EmbeddedChannelCore` could offload the
responsibility for storing the values back to the
`NIOAsyncTestingChannel` but it was complicated and of questionable
value.
### Result:
* The correct address values are seen no matter how they are obtained.
* We probably take a performance hit locking the values in this way but
this is testing code so probably not the end of the world.
When having a state machine inside the `NIOLoopBoundBox` we currently
check the EL when reading and writing. This is unnecessary. Because of
this, this PR introduces a new `withValue` method, that allows users to read and
write the value inside the NIOLoopBoundBox while only paying the cost
for the EL check once.
Motivation:
NIO has existing GRO support. This GRO support is applied at Channel
scope, which makes it somewhat less than ideal, as it forces the
datagram sizes to be accessed using socket options. In real applications
this is less desirable than being able to do this on a per-superbuffer
context.
Linux supports this already by using recvmsg/recvmmsg control data. We
just need to wire this up in NIO. We can re-use the existing metadata
fields on AddressedEnvelope.
Modifications:
- Add a new ChannelOption to extra metadata
- Reject setting this option on non-Linux devices.
- On Linux, use this to load the control data.
- Added new tests that validate the behaviour is correct.
Result:
Enable per-message GRO.
Motivation:
NIO has existing GSO support. This GSO support is applied at Channel
scope, which makes it somewhat less than ideal, as it forces the
datagram sizes to be known statically ahead of time. In real
applications this is less common than being able to do this on a
per-superbuffer
context.
Linux supports this already by using sendmsg/sendmmsg control data. We
just need to wire this up in NIO.
An extension to this is that we can also do per-message GRO in Linux.
I'll tackle this in a later patch to keep the diffs small.
Modifications:
- Adding a new field on AddressedEnvelope Metadata.
- Reject setting this field on non-Linux devices.
- On Linux, use this to set the control data.
- Added new tests that validate the behaviour is correct.
Result:
Enable per-message GSO.
Motivation:
Users would like to be able to access the underlying memory of a
ByteBuffer, as evidenced by the plethora of `withUnsafe*` methods that
ByteBuffer has. As Swift 6.2 has introduced some initial APIs for safe
memory access to underlying storage, we should offer similar APIs on
ByteBuffer to enable users to get safer access to that storage.
For now, the obvious APIs to be able to supplement are:
- withUnsafeReadableBytes
- withUnsafeMutableReadableBytes
- writeWithUnsafeMutableWritableBytes
We can also offer some new APIs to allow initializing a buffer directly
from an OutputSpan.
Note that we can only do this because the Language Steering Group has
pinky promised that they will not break the "Lifetimes" experimental
feature: see
https://forums.swift.org/t/experimental-support-for-lifetime-dependencies-in-swift-6-2-and-beyond/78638
for more details. We are taking them at their word, and so we are
enabling that feature.
Modifications:
Many new methods and tests.
Result:
Safer access.
Motivation:
'copyItem' can be used to copy files/directories from a source to a
destination address and fails if the destination already exists. The
parallel version of it can wedge if a directory is being copied and the
directory can't be created at the destination path (for example, if a
file already exists at the destination).
The parallel copy works by feeding tasks (e.g. "copy this item from here
to there") into an async sequence and processing each task in a separate
child task within a task group. When copying directories a new directory
is created at the destination path and then each file within the source
directory is emitted as a separate item to process. Another item is sent
on the async sequence to indicate when the source directory is finished
with which may result in finishing the async sequence.
However, if creating the destination directory fails then that event
isn't sent. This results in the calling code never terminating the async
sequence and causes 'copyItem' to wedge.
Modifications:
- Use non-idempotent directory creation (copy item should fail if the
destination already exists, this was a regression introduced in
7124f096).
- Check whether to continue when a dir can't be created but always emit
the end of dir event
- If terminating when the task group hasn't reached its width limit then
check child tasks for errors
Result:
Parallel copy doesn't wedge
On filesystems without symlink support for Windows, the symlinked test
helper at is materialized as a plain text file containing the relative
path, which breaks compilation with errors like:
```
expressions are not allowed at the top level
cannot find 'assertNoThrowWithValue' in scope
```
This change inlines the contents, restoring the intended shared test
utilities for the NIOHTTP1 test target. This helps Windows compile the
Test target
~~Implement `AsyncSequence/split()` functions similar to
`String/split()` functions in std-lib.~~
Implement `AsyncSequence/splitLines()` functions similar to
`String/split(whereSeparator: \.isNewline)` in std-lib.
### Motivation:
~~Provide an easy way for users to split the data incoming from an async
sequence, using their preferred separator.~~
Provide an easy way for users to split the data incoming from an async
sequence, on new lines.
### Modifications:
Add `internal SplitMessageDecoder: NIOSingleStepByteToMessageDecoder`.
Add `public NIOSplitLinesMessageDecoder:
NIOSingleStepByteToMessageDecoder`.
Add `public
AsyncSequence/splitLines(omittingEmptySubsequences:maximumBufferSize) ->
AsyncSeq<ByteBuffer>`.
Add `public
AsyncSequence/splitUTF8Lines(omittingEmptySubsequences:maximumBufferSize)
-> AsyncSeq<String>`.
### Result:
Users can easily split the data.
### Motivation:
`ByteBuffer.lastIndex(where:)` is of suboptimal performance.
The default Collection implementations don't go through any "magic
underscored" functions like `_customIndexOfEquatableElement`.
### Modifications:
Manually implement `lastIndex(where:)`.
### Result:
Basically free performance boost. 2x+ boost even for not big buffers of
a few hundred bytes.
This function is currently used in
`ByteBufferView.trim(limitingElements:)`.
I have no immediate use case for this function, but it's still an issue
worth addressing.
The createDirectory function will succeed without error if the target
directory already exists.
### Motivation
This change addresses issue #3404. Currently,
`fileSystem.createDirectory` fails if the target directory already
exists, forcing users to write boilerplate try/catch blocks to handle
this common and expected case.
The goal is to make this function's behavior idempotent.
### Modifications
To achieve this, I've made the following changes:
**(Implementation)** A new private helper function,
`_handleCreateDirectoryFileExists`, was introduced. This function is
responsible for:
1. Performing a `stat` call on the path that failed.
2. Checking if the existing item is a directory (`S_IFDIR`).
3. Returning a success result if it's a directory, or re-throwing the
original `.fileExists` error if it's a file or another type of entity.
**(Logic)** The core `_createDirectory` function was updated to call
this new helper function whenever `Syscall.mkdir` fails with an `EEXIST`
(`.fileExists`) error. This check is applied in both internal loops to
correctly handle cases where either an intermediate directory or the
final target directory already exists.
### Result
With this change, users can now call the function
`fileSystem.createDirectory` and the operation will succeed even if the
directory is already present, leading to cleaner and more predictable
code.
### Motivation:
`ByteBuffer.firsIndex` is of suboptimal performance.
The default Collection implementations don't go through any "magic
underscored" functions like `_customIndexOfEquatableElement`.
### Modifications:
Manually implement `firstIndex(where:)`.
### Result:
Basically free performance boost. 2x+ boost even for not big buffers of
a few hundred bytes.
There are some usage of this function in `BufferedReader`. Those will
become much faster.
Also this function is used in `ByteBufferView.trim(limitingElements:)`.
Also makes #3411 stuff faster. See:
https://github.com/apple/swift-nio/pull/3411#discussion_r2436260691
Previous PR: #3405
Add an API on top of `AsyncSequence<ByteBuffer>` which can dynamically
decode values.
~~Add an API on top of the new `AsyncSequence<ByteBuffer>` APIs which
splits the file based on its content.~~
### Motivation:
Provides a nice API to decode files, instead of users having to go
though manually handling `BufferedReader.read(while:)`.
I struggled with this, as documented in
https://swift-open-source.slack.com/archives/C9MMT6VGB/p1760115481607159
### Modifications:
Add `NIODecodedAsyncSequence` + functions on `AsyncSequence<ByteBuffer>`
to create such a sequence.
~~Add `NIOSplitMessageDecoder` + stdlib-like functions on
`AsyncSequence<ByteBuffer>` to create such a sequence.~~
### Result:
Users can decode an async sequence of `ByteBuffer`s easier.
~~Users can easily split files based on their content.~~
### Checklist
See this comment for a checklist of the remaining things to do:
https://github.com/apple/swift-nio/pull/3407#issuecomment-3403382404
Motivation
In rare cases where the inbound side of a pipe channel is already closed
(either because it was never open or because it was closed during use),
we can get into trouble if the reader drops the read side of our write
pipe. In that context, the Linux kernel will deliver an EPOLLHUP, but
while we'll close the FD we'll let the channel hang out as a zombie.
This is, obviously, suboptimal.
Modifications
In writeEOF, we check whether the read side is already closed. If it is,
we trigger a close internally to shut the channel down. Added a test for
this as needed, and modified an existing test that would now trip over
this behaviour. Also added a new test for the reverse case to ensure we
don't end up with zombies there (we don't).
Results
No zombie channels.
Motivation:
This test occasionally fails in our ongoing continuous CI solution.
Flaky tests are bad, and we should aim not to have them, so let's fix
it.
My diagnosis of the most likely race here (which leads to
`alreadyClosed`) being thrown _somewhere_ from this code is that the
client channel got closed unexpectedly. This would happen if an unusual
series of events occurs:
1. The client connection succeeds and the channel starts up
2. The server close then goes through and the server channel is closed
_before_ it actually accepts the connection.
3. This causes the client connection to be reset.
4. The client channel observes this and handles that close.
5. We then close the client from the outside.
Modifications:
Add a ConditionLock we can use to wait for a connection to be
established.
Make the use of this ConditionLock resilient to weird test behaviour by
always using timeouts for blocking operations.
Result:
Less flaky tests.
Motivation
Availability guards on test functions don't work the way we'd want them
to: in particular, xctest still ends up calling these functions even on
platforms that are less available than the functions in question. We
need to move these to checks in in the function themselves.
Modifications
Replace @available with guard available
Result
No more segfaults in tests.
### Motivation:
Useful for parsing packets, for example `ipv4: InlineArray<4, UInt8>`
and `ipv6: InlineArray<16, UInt8>`.
### Modifications:
For now I've only added a `readInlineArray` function. I know some other
functions are missing, such as `writeInlineArray`.
I wanted to first open up a discussion and see if these changes are
acceptable. I can add those functions too if required, in this PR or
other PRs.
### Result:
Users can read `ByteBuffer` into stack-allocated memory, which can be
more performant than the other alternatives like `Array`, or more
convenient than reading as a tuple like `(UInt8, UInt8, UInt8, UInt8)`.
### Caveats:
Swift 6.2 is required so I've used `#if compiler(>=6.2)`.
Furthermore, `InlineArray` is marked as available on `macOS(9999)` since
the Swift team have yet to update that mark, although `InlineArray` is
planned for Swift 6.2 per the
[proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0453-vector.md).
Edit: to be clear things work fine on Linux and that's where I've been
using this same `readInlineArray` function that I've proposed.
---------
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
We accidentally removed the 'NIOFileSystem' module from the
'_NIOFileSystem' product in the last release.
Modifications:
- Rename 'NIOFileSystem' and 'NIOFileSystemFoundationCompat' to 'NIOFS'
and 'NIOFSFoundationCompat'
- Add back 'NIOFileSystem' which re-exports '_NIOFileSystem' (there was
no publicly available 'NIOFileSystemFoundationCompat' module to
remove, only '_NIOFileSystemFoundationCompat').
Result:
Fewer breaks
Motivation:
We changed the file path type in `_NIOFileSystem` to use `NIOFilePath`
instead of `FilePath`. Originally we planned to use API shims to avoid
breaking API, however in some cases this was inevitable (i.e. where a
file path is returned to the caller).
To roll over to the new API we will instead introduce the
`NIOFileSystem` module using `NIOFilePath` and have `_NIOFileSystem` use
`FilePath`. Users can then opt-in to the new stable API rather than
having it forced upon them.
The first wave of these changes turns `_NIOFileSystem` into
`NIOFileSystem` and uses `NIOFilePath` for its APIs.
Modifications:
- Remove disfavoured overloads (i.e. `FilePath` APIs)
- Update the `DirectoryEntry` API to use `NIOFilePath`.
- Update tests to use `NIOFilePath` since many were relying on the newly
removed shims
- Remove the underscore from the module names and products
- Remove deprecated methods
A follow up change will re-instate the `_NIOFileSystem` (and compat)
module and products using `FilePath`.
Result:
`NIOFileSystem` has a stable API.
### Motivation:
In #3297, I introduced a deadlock involving `SelectableEventLoop`'s
`debugDescription`. I was under the impression that it's not possible to
call this from outside the `NIO` module so I deemed it safe. That was a
mistake :).
### Modifications:
- Make it impossible to deadlock around
`SelectableEventLoop.debugDescription`.
### Result:
- Fewer deadlocks
Motivation:
NIOHTTPResponseHeadersValidator drops head/end response parts if they
contain invalid header fields. If a head part is dropped and the server
allows pipelining then the subsequent body/end part will reach the
pipelining handler and result in an assertion failure in debug builds.
In release builds, some parts may be incorrectly written out leading to
protocol violations.
Modifications:
- Drop all response parts after an invalid response part
Result:
- Fewer bugs
- Resolves#3326
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
Currently, file paths are represented by the `SystemPackage.FilePath`
type in `NIOFileSystem`. Following from #3322, we want to use
`NIOFilePath` instead.
Modifications:
This change provides `NIOFilePath` accepting/returning variants to all
existing public methods that accept/return `SystemPackage.FilePath`.
The bulk of the changes are in `FileSystemProtocol` and
`FileHandleProtocol`, and their conforming types. The protocols have
been modified to require `NIOFilePath` accepting/returning methods.
Default implementations are provided for `SystemPackage.FilePath`
accepting/returning methods---these call through to the `NIOFilePath`
variants.
Result:
`NIOFilePath` path representations can now be used with the methods for
interacting with the file system.
---------
Co-authored-by: George Barnett <gbarnett@apple.com>
Motivation:
testMetricsDelegateTickInfo fails occasionally because there are more
event loop ticks than it expects. This is because the expectation in the
test was incorrect.
Modifications:
- Update the test expectation and an explanation of how we got to that
number.
- Shutdown the ELG while we're at it.
Result:
Tests are less flaky
Support options on AsyncTestingChannel and EmbeddedChannel
Fixes#3305
### Motivation:
The channels are intended for testing purposes so while most options
have no practical use setting and getting them can be useful for
testing.
For example a traceroute implementation will set TTL to the current hop
number. Testing such an implementation requires the channels to pretend
to support the TTL option.
### Modifications:
Added option storage to AsyncTestingChannel and EmbedededChannel and
made `getOptionSync` and `setOptionSync` read and write from that
storage.
### Result:
EmbeddedChannel and AsyncTestingChannel support changing their options.
---------
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
In #3309 I fixed an issue where the channel initializer would not be
called in some PipeBootstrap paths. Unfortunately, that introduced a new
bug where the initializer was now called twice on some other paths.
Modifications:
Get the number of calls back to 1 on the non-async paths. Add some tests
Result:
Better pipe bootstrap behaviour.
Motivation:
Two separate types for representing file paths currently exist:
[`SystemPackage`'s
`FilePath`](https://swiftpackageindex.com/apple/swift-system/1.6.0/documentation/systempackage/filepath)
type and [`System`'s
`FilePath`](https://developer.apple.com/documentation/system/filepath)
type.
`NIOFileSystem` currently uses `SystemPackage`'s `FilePath`. However,
the lack of an API for converting between the two representations means
that users whose application also uses `System.FilePath` may find
interacting with `NIOFileSystem` difficult.
Modifications:
- Added `NIOFilePath`, and its subcomponents `NIOFilePath.Root`,
`NIOFilePath.Component`, and `NIOFilePath.ComponentView`.
- These types mirror the API of `SystemPackage`'s `FilePath` and are
internally backed by it.
- `NIOFilePath` can be initialized from either an instance of
`SystemPackage.FilePath` or `System.FilePath`.
Result:
Reduces the friction of interacting with `NIOFileSystem` by providing a
bridge between `SystemPackage.FilePath` and `System.FilePath`.
---------
Co-authored-by: George Barnett <gbarnett@apple.com>
Motivation:
For our async initializers on bootstraps, it's imortant that _both_
channel initializers are called, not only the one that comes as an
argument on the bind function.
Modifications:
Fix the takeOwnershipOfDescriptor async functions to call the
initializer from the bootstrap, and add regression tests.
Result:
Better behaved code.
### Motivation:
Detached threads are a bad idea, their lifetime becomes random and NIO
doesn't need this functionality (except for in some tests that are
easily refactored)
### Modifications:
- Refactor tests to not use detached threads
- Remove detached threads functionality
### Result:
- Cleaner, less complex and more correct code
Rick Newton-Rogers
Si Beaumont
Stepan UlyaninandGeorge Barnett
Raphael
scottmarchant
Fabian Fett
Adam FowlerandCory Benfield
Karan LokchandaniandCory Benfield
aryan-25
Zamderax
Mahdi Bahrami
Hamza Hassanain
finagolfin
Klaas Pieter AnnemaandCory Benfield
Max Desiatov