Adds capability to NIOFS to copy regular files and symlink, allowing to
overwrite the destination.
### Motivation:
Per https://github.com/apple/swift-nio/issues/3403 and
https://github.com/apple/swift-nio/pull/3470 we want to add
`replaceExisting: bool` to `FileSystem.copyItem`.
### Modifications:
1. Adds `replaceExisting: bool` parameter to
`FileSystemProtocol.copyItem`.
2. Adds `replaceExisting: bool` parameter to `FileSystem.copyItem` and
implementation for regular files and symbolic links.
3. Adds tests.
---------
Co-authored-by: George Barnett <gbarnett@apple.com>
## Motivation
NIO channels abstract over an underlying transport mechanisms (sockets,
pipes, etc.), which users typically need not interact with. However,
there are scenarios where users need direct access to the underlying
transport for low-level operations that work outside NIOs abstraction.
One example is performing out out-of-band operations on the underlying
file descriptor for a socket-based channel.
This PR adds a structured way to access the underlying transport of a
channel, for channels that choose to implement it.
## Modifications
- Add a new `public protocol
NIOTransportAccessibleChannelCore<Transport>` which provides a scoped
`withUnsafeTransport(_:)`, used for channel implementations to opt-in.
- Add conformance to
`NIOTransportAccessibleChannel<NIOBSDSocket.Handle>` for
`BaseSocketChannel`, to make this API available for all socket-based
channels, including channels returned from the socket-based bootstrap
public APIs.
- Add public API
`ChannelPipeline.SynchronousOperations.withUnsafeTransportIfAvailable(_:)`
Note that not all channels need to or should their transport, which is
why this was added as an additional protocol that refines `ChannelCore`,
vs. extending `Channel` or `ChannelCore` with a default implementation.
The protocol uses a primary associated type allowing channels to provide
typed access to the transport. E.g. this could be used in NIO Transport
Services to expose the underlying `NWConnection`, if desired.
The method itself is spelled with "unsafe", uses scoped access, and has
clear documentation that users must not violated any of NIOs assumptions
about the state of the underlying transport. It's very much not intended
for every day use. It being on `ChannelCore` should defer users of the
low-level API, since `Channel._channelCore` is marked as for NIO
internal use, and the public `withUnsafeTransportIfAvailable(_:)` will
take care of the runtime checks for channels that have opted into this
API.
## Result
- New opt-in API for channel implementations to expose their underlying
transport
- All socket-based channels from NIOPosix now expose the underlying
socket file descriptor
- New API
`ChannelPipeline.SynchronousOperations.withUnsafeTransportIfAvailable(_:)`
for users
---------
Co-authored-by: Agam Dua <agam_dua@apple.com>
### Motivation:
Currently if there is an error in the Benchmark targets, that prevents
the Benchmark to build correctly, we still get a CI success.
### Modifications:
- Check for the threshold changed error
### Result:
- More reliable CI
### Motivation
Issue #2773 reports a problem where channels can receive
`channelInactive` before `channelActive`. This breaks expectations and
should not happen. Instead of sending them in the wrong order, this PR
suppresses both of them in those cases.
### Modifications
* Add a test to reproduce the problematic behavior.
* Update the state machine to split the active state into two. This
decouples succeeding the promise from sending the signal. Transiting to
closing only sends `channelInactive` when originating from the state
that sent `channelActive`.
### Result
Address #2773.
Edit: Updated modifications to reflect latests commit.
Adds three new system call wrappers for the `symlinkat`, `renameatx_np`,
and `unlinkat` system calls.
### Motivation:
Related to https://github.com/apple/swift-nio/issues/3403 and
https://github.com/apple/swift-nio/pull/3470. This PR adds syscall
wrappers needed to atomically overwrite existing files or symlinks at
the destination during copy operations. On Linux, atomic overwrites
require a "copy to temp file, then rename" strategy. We use the `*at`
family of syscalls (which operate relative to directory file
descriptors) to avoid TOCTOU race conditions.
### Modifications:
1. Adds three system call wrappers for the `symlinkat`, `renameatx_np`,
and `unlinkat` system calls.
2. Adds related tests
3. Updates the `FileSystemError` for `symlink` and `unlink` to take in
the system call name to allow for the `*at` names to be passed.
### Motivation:
`NIOFileSystem` currently exposes the same API as `_NIOFileSystem`,
which is not API stable. `NIOFileSystem`
was created in error, and its lack of underscore incorrectly implies API
stability. Users who are currently importing `NIOFileSystem` should
ideally move to `_NIOFileSystem`. However this isn't made clear in the
docs:
1. The README talks about the non-underscored `NIOFileSystem`.
2. There are no hosted API docs `NIOFileSystem`.
3. The hosted API docs for `NIO` point to the docs for `NIOFileSystem`
-- results in 404.
### Modifications:
- Update the README to refer to `_NIOFileSystem`.
- Update the hosted API docs for `NIO` to point to docs for
`_NIOFileSystem`.
- Add hosted API docs for `NIOFileSystem` with a statement explaining
the situation and pointing people to the docs
for `_NIOFileSystem`.
### Result:
Clearer documentation on the state and relationship of the
`NIOFileSystem` and `_NIOFileSystem` modules.
We have a number of copies of `UnsafeTransfer` and two copies of
`UnsafeMutableTransferBox` in our code base. Before introducing more of
those, lets centralize to just one using a `package` access modifier.
### Motivation:
Channels based on `BaseSocketChannel` throw in both `getOption` and
`setOption` if the channel has been closed, since the `setsockopt` will
fail. However, the current behavior of `EmbeddedChannel` is options
remain writable and readable on closed channels.
There are situations where we'd like to be able to model the runtime
behaviour of the real channel in tests, e.g. to test this fix:
https://github.com/apple/swift-nio-extras/pull/304.
### Modifications:
- Add API to enable throwing in `EmbeddedChannel.getOption` and
`.setOption` if channel is closed.
- Add a test for this new behavior.
### Result:
- New API to enable throwing in `EmbeddedChannel.getOption` and
`.setOption` if channel is closed.
- No observable change for existing users of `EmbeddedChannel`.
### Motivation:
Packages that consume NIOPosix should be able to compile to WASI
platforms without special configurations. This change elides the
NIOPosix source from WASI platforms to simplify configuration.
Without this change:
```swift
dependencies: [
// Without this PR, downstream packages must maintain exhaustive platform list to exclude `.wasi`:
.product(
name: "NIOPosix",
package: "swift-nio",
condition: .when(platforms: [
.macOS,
.macCatalyst,
.iOS,
.tvOS,
.watchOS,
.visionOS,
.driverKit,
.linux,
.windows,
.android,
.openbsd,
// .wasi // <-- Need to exclude this, because there is no exclusion list api for SPM conditionals
])
),
]
```
With this change:
```swift
dependencies: [
// Without this PR, downstream packages can consume NIOPosix simply:
.product(name: "NIOPosix", package: "swift-nio"),
]
```
### Modifications:
- Add compiler directives (`#if !os(WASI)`) to all source files in
NIOPosix
### Result:
Downstream packages can compile to wasm without manually excluding the
NIOPosix dependency.
### Testing performed
- Verified `swift build --swift-sdk wasm32-unknown-wasip1 --target
NIOPosix` compiles, which demonstrates proper elision of source files in
NIOPosix that aren't wasm-ready.
- Confirmed GitHub [checks
pass](https://github.com/PassiveLogic/swift-nio/actions/runs/21151341738).
### Motivation:
Packages that consume NIOEmbedded should be able to compile to WASI
platforms without special configurations. This change elides the
NIOEmbedded source from WASI platforms to simplify configuration.
Without this change:
```swift
dependencies: [
// Without this PR, downstream packages must maintain exhaustive platform list to exclude `.wasi`:
.product(
name: "NIOEmbedded",
package: "swift-nio",
condition: .when(platforms: [
.macOS,
.macCatalyst,
.iOS,
.tvOS,
.watchOS,
.visionOS,
.driverKit,
.linux,
.windows,
.android,
.openbsd,
// .wasi // <-- Need to exclude this, because there is no exclusion list api for SPM conditionals
])
),
]
```
With this change:
```swift
dependencies: [
// Without this PR, downstream packages can consume NIOEmbedded simply:
.product(name: "NIOEmbedded", package: "swift-nio"),
]
```
### Modifications:
- Fix compiler directive using in AsyncTestingChannel.swift to include
an extension
### Result:
Packages can compile to wasm without manually excluding the NIOEmbedded
dependency.
### Testing performed
- Verified `swift build --swift-sdk wasm32-unknown-wasip1 --target
NIOEmbedded` compiles, which demonstrates proper elision of source files
in NIOEmbedded that aren't wasm-ready.
- Confirmed GitHub [checks
pass](https://github.com/PassiveLogic/swift-nio/actions/runs/21151287289).
### Motivation:
In reviewing a copy of this file in swift-log, @czechboy0
[noticed](https://github.com/apple/swift-log/pull/398#discussion_r2685605800)
that the `mutex` variable is essentially unused and unnecessarily
allocated for certain conditions. It is possible the compiler elided
usage. But given the unconditional variable reference in the `deinit`,
the compiler may not be able to.
This change ensures the `mutex` property is completely elided except in
the conditions where it is used.
### Modifications:
- Adjusted compiler directive pattern to completely elide the `mutex`
property for conditions where it would be unused.
- Adjusted `deinit` to avoid referencing `mutex` unless it is compiled
into the `Lock` class
### Result:
The `mutex` property is no longer allocated or even compiled into the
`Lock` class for unsupported configurations. All unit tests and checks
pass.
### Research:
The `deinit` used to have a blanket call to `mutex.deallocate()`. This
has been moved into the platform-specific checks within the `deinit`.
This allows complete elision of the `mutex` property altogether. The
`_runtime(_multithreaded)` condition [appears to be rooted to this
implementation](https://github.com/swiftlang/swift/blob/ffc51b914602765c5d680241796dbd3c3711fa6b/lib/Basic/LangOptions.cpp#L490).
Diving deeper, all [operating systems in this
list](https://github.com/swiftlang/swift/blob/ffc51b914602765c5d680241796dbd3c3711fa6b/lib/Basic/LangOptions.cpp#L554)
except for UnknownOS and WASI result in `_runtime(_multithreaded)`
returning true. That means that `OpenBSD` (and many other operating
systems besides windows) were almost certainly using the `#elseif
(compiler(<6.1) && !os(WASI)) || (compiler(>=6.1) &&
_runtime(_multithreaded))` condition in the `deinit`. In conclusion,
moving `mutex.deallocate()` up into the conditional clauses inside
`deinit` should be identical to the previous implementation for all
operating systems except `llvm::Triple::UnknownOS`. And
`llvm::Triple::UnknownOS` is likely not to have every compiled anyways
due to the [check on line
32](https://github.com/apple/swift-nio/blob/main/Sources/NIOConcurrencyHelpers/lock.swift#L32).
So in summary, it is expected that this clean up will compile identical
to the previous implementation for all platforms except WASI platforms
that don't have pthread support. For non-pthread WASI platforms, the
unused `mutex` is properly elided from compilation altogether.
### Testing Done:
- Confirmed that unit tests pass locally using Xcode
- Verified [PR checks
pass](https://github.com/PassiveLogic/swift-nio/actions/runs/21149729751)
As observed by @kkebo, this was missing from the implementation.
### Motivation:
Correct pthread API usage.
### Modifications:
Initialize the pthread_mutexattr_t with pthread_mutexattr_init before
passed to pthread_mutex_init.
### Result:
Even though this lock API appears to be deprecated, we'll correct the
usage anyway. The non-deprecated NIOLock class doesn't have this
problem.
### Motivation:
In reviewing a copy of this file in swift-log, @czechboy0 noticed that
the call to `pthread_mutexattr_destroy` is missing. Since
pthread_mutexattr_t is like allocated on the stack rather than heap on
most platforms, this is likely not a memory leak. But for correctness
the missing destroy call should be added.
This change is the start of a resolution
[requested](https://github.com/apple/swift-log/pull/398#discussion_r2689459360)
in a corresponding Lock rollout in swift-log
### Modifications:
- Added missing call to `pthread_mutexattr_destroy`
- Duplicated test for `NIOLock` to also test `Lock`, as a sanity test.
### Result:
The new sanity test passes. If there are other ways this change can be
realistically verified as safe and proper, please feel free to provide
feedback in the PR.
`NIOTypedHTTPClientUpgradeHandler.handlerAdded` will write upgrade
request if the channel is already active
### Motivation:
This has been added to make it easier to implement a websocket client
that supports proxies. If the upgrade handler is added after the proxy
connect has been processed previously nothing would happen.
### Modifications:
- Added new function
`NIOTypedHTTPClientUpgradeHandler.writeUpgradeRequest()` function. Call
this from `channelActive` and also from `handlerAdded` if the channel is
already active.
- Added test `testUpgradeHappensAfterHandlerAdded`
### Result:
NIOTypedHTTPClientUpgradeHandler now also works if it is added to a
channel after the channel is active
---------
Co-authored-by: Cory Benfield <lukasa@apple.com>
This re-enables documentation lost during refactoring work, under the
target _NIOFileSystem
resolves#3474
### Motivation:
During the refactoring of NIOFileSystem and recent Swift updates, the
mechanism to "shadow" symbols using `@_exported import` has stopped
working for documentation and imports for symbols, which means that
_NIOFileSystem is the target that needs to host the documentation for
this (for now)
### Modifications:
Moves DocC catalog into _NIOFileSystem target, and updates
disambiguation hashes on overloaded symbols in order to verify no
warnings are presented while generating documentation.
Updates .spi.yml to present _NIOFileSystem instead of NIOFileSystem
### Result:
Previous documentation should be available again, although at a slightly
different URI structure within Swift Package Index.
Co-authored-by: Cory Benfield <lukasa@apple.com>
Add homeDirectory accessor to FileSystem
### Motivation:
Addresses #3381 by adding a `homeDirectory` property to FileSystem,
equivalent to `FileManager.default.homeDirectoryForCurrentUser`.
### Modifications:
- Added `homeDirectory` property to `FileSystemProtocol` and implemented
in `FileSystem`
- Implementation checks `HOME` environment variable first, falls back to
`USERPROFILE` on Windows, or uses `getpwuid_r(3)` on POSIX systems
- Added system call wrappers (`system_getuid`, `libc_getpwuid_r`) with
proper platform guards
- Added `FileSystemError.getpwuid_r()` error helper
### Result:
Users can now access the home directory via `FileSystem.homeDirectory`,
returning a `FilePath` asynchronously. Follows the same pattern as
`currentWorkingDirectory` and `temporaryDirectory`.
Co-authored-by: Cory Benfield <lukasa@apple.com>
### Motivation:
The documentation for String(buffer:) initializer does not clearly
indicate
that this operation will always succeed, which may leave developers
uncertain
about error handling requirements.
fixes: https://github.com/apple/swift-nio/issues/3449
### Modifications:
Updated the documentation for String(buffer:) to explicitly mention that
the initialization will always succeed.
### Result:
Developers will have clearer understanding that String(buffer:) is a
safe
operation that does not require error handling.
Signed-off-by: Karan <karanlokchandani@protonmail.com>
### Motivation:
Currently, pending consumer closures remain in the
`{in}{out}boundBufferConsumer` queues of `EmbeddedChannelCore` even
after the channel closes. It is also possible to enqueue consumers
*after* the channel closes. In these cases, the consumer closures will
never be invoked and this can lead to unfavourable behaviour, as
observed in `NIOAsyncTestingChannel`'s `waitFor{In}{Out}boundWrite`
methods (the only places these queues are currently used).
`NIOAsyncTestingChannel`'s `waitFor{In}{Out}boundWrite` methods complete
a continuation *inside* the consumer closure. In the cases described
above, the continuation never completes and therefore
`waitFor{In}{Out}boundWrite` never returns.
### Modifications:
- Updated the element type in `EmbeddedChannelCore`'s
`{in}{out}boundBufferConsumer` from `(NIOAny) -> Void` to
`(Result<NIOAny, Error>) -> Void`.
- This is so that the `.failure` case can be used to notify the consumer
closure that the channel has closed.
- Changed the visibility of the `{in}{out}boundBufferConsumer`
properties from `internal` to `private` in order to prevent the queues
from being accessed and being appended to without the call site
considering whether the channel has been closed.
- Added new `internal` methods named
`enqueue{In}{Out}boundBufferConsumer(_:)` which take the consumer
closure as an argument and only append to the corresponding queue if the
channel isn't closed.
- If the channel is closed, the consumer closure is invoked immediately
with `.failure(ChannelError.ioOnClosedChannel)`.
- Updated `EmbeddedChannelCore`'s `close0` method to return a
`.failure(ChannelError.ioOnClosedChannel)` result to each closure in
`{in}{out}boundBufferConsumer` and empty both buffers.
- Updated `NIOAsyncTestingChannel`'s `waitFor{In}{Out}boundWrite` to
throw an error in the continuation upon receiving a `.failure` result.
- Added associated test cases.
### Result:
`EmbeddedChannelCore`'s `{in}{out}boundBufferConsumer` queues can be
used more safely: all pending closures will be invoked upon channel
close. As a result, `NIOAsyncTestingChannel`'s
`waitFor{In}{Out}boundWrite` no longer indefinitely blocks when the
channel closes.
Fix coreCount on Linux when using cgroup v2 with CFS throttling disabled
### Motivation:
When using `swift-nio` on Linux with cgroup v2 enabled, but with CFS
throttling disabled, it falls back to attempting to read the cpuset file
at the cgroup v1 path. This does not exist, which in turns falls back to
returning `_SC_NPROCESSORS_ONLN`, which will return the total number of
cores available (ignoring cgroup assignments).
This has unexpected effects, including the default behaviour of starting
the `MultiThreadedEventLoopGroup.singleton` with significantly more
event loops than cores available to the workload.
### Modifications:
- Adds `SystemCalls.statfs`, and associated constants, to determine the
cgroup version.
- Adds `Linux.cgroupVersion()` API to expose cgroup version.
- Adds `Linux.cgroupV2MountPoint` variable to determine the cgroup v2
mount point.
- Adds `Linux.cpuSetPathV1` & `Linux.cpuSetPathV2` (and
`Linux.cpuSetPath` convenience) variables to determine the correct cpu
set path.
- Alters `System.coreCount` to use the appropriate logic from above to
ensure that `cpuset.cpus` is parsed from the correct location.
### Result:
`Linux.coreCount` should correctly parse and return the core count on
Linux cgroup v2 enabled systems (when CFS throttling is disabled), while
maintaining correctness for other configurations.
---------
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
Co-authored-by: Cory Benfield <lukasa@apple.com>
Add ByteBuffer.readableBytesUInt8Span
### Motivation:
Provide access to readableBytes as a Span<UInt8>
### Modifications:
Added computed property `ByteBuffer.readableBytesUInt8Span`. I also
attempted to add `ByteBuffer.mutableReadableBytesUInt8Span` but due to
an apparently faulty exclusivity issue I couldn't get this to work. See
https://github.com/swiftlang/swift/issues/81218.
### Result:
You can access the readable bytes of a ByteBuffer as a Span<UInt8>
Motivation:
The various 'withMumbleContinuation' APIs are supposed to be invoked
synchronously with the caller. This assumption allows a lock to be
acquired before the call and released from the body of the
'withMumbleContinuation' after e.g. storing the continuation. However
this isn't the case and the job may be re-enqueued on the executor
meaning that this is pattern is vulnerable to deadlocks.
Modifications:
- Drop and reacquire the lock in TokenBucket.
- Switch to NIOLockedValueBox
Result:
Lower chance of deadlock
Added extra documentation to `channel.remoteAddress` field to capture a
scenario where this field might be `nil`
### Motivation:
It is somewhat common for users to have code like
`channel.remoteAddress!` in their implementation, as it is a reasonable
assumption to think a socket connection will have an associated remote
address. However, in at least one known situation this might not be the
case. When that happens, user's code might crash due to the force unwrap
of the optional field.
### Modifications:
Introduced more documentation to make it clear that users should be
prepared to handle the `nil` scenario.
### Result:
Less frequent mishandling of `channel.remoteAddress`.
### Motivation:
Additional partial platform support.
### Modifications:
* Create a new CNIOBSD module for OpenBSD, for general cleanliness, and
make use of it throughout. Some of the changes are borrowed directly
from CNIOLinux; some may technically be unnecessary; I am erring
somewhat on expediency to functionality.
* Since `malloc_size` is unavailable on OpenBSD, and some of the helpers
make use of ManagedBufferPointer which makes use of it, mark some of
this as unavailable as well.
* Usual pthread optional typing changes, since pthread types are
pointers on this platform.
* Add conditionals to exclude other functionality not available here,
like IP_RECVPKTINFO or IP_PKTINFO.
* d_ino is a backwards-compatibility macro valued token for dirent, so
instead, just expand with a conditional.
* Use kqueue on OpenBSD. This necessitates adding some conditionals for
type and feature compatibility.
* The vsock API is unavailable on OpenBSD.
### Result:
Tested the NIOTCPEchoClient and NIOTCPEchoServer appears to work and
that swift-nio-ssh (hopefully wlog) builds with a local repository with
these changes.
Because NIOFS/_NIOFileSystem depend on some non-portable components,
such as extended attributes, sendfile, non-portable linkat flags, and
renameat2, this is only just partial OpenBSD support, which means that
`swift build` on the full swift-nio project won't build cleanly, but at
least the portable parts can be used to build servers and clients. This
commit will mark the linked bug as fixed; I will open a new bug for file
system support.
Fixes#3383.
---------
Co-authored-by: Rick Newton-Rogers <rnro@apple.com>
Motivation:
The various 'withMumbleContinuation' APIs are supposed to be invoked
synchronously with the caller. This assumption allows a lock to be
acquired before the call and released from the body of the
'withMumbleContinuation' after e.g. storing the continuation. However
this isn't the case and the job may be re-enqueued on the executor
meaning that this is pattern is vulnerable to deadlocks.
Modifications:
- Drop and reacquire the lock in the NIOThrowingAsyncSequenceProducer.
- Merge 'next()' and 'next(for:)' methods in the state machine into a
single func; this reduces the amount of duplicated logic across the two
functions.
Result:
Lower chance of deadlock
Motivation:
This is a follow up to 5bf841dd to handle the yield task being cancelled
between dropping the lock and re-acquiring it a moment later in the
'withContinuation' block.
Modifications:
- Add an extra cancellation check when yielding with a continuation.
Result:
Fewer issues
Motivation:
The various 'withMumbleContinuation' APIs are supposed to be invoked
synchronously with the caller. This assumption allows a lock to be
acquired before the call and released from the body of the
'withMumbleContinuation' after e.g. storing the continuation. However
this isn't the case and the job may be re-enqueued on the executor
meaning that this is pattern is vulnerable to deadlocks.
Modifications:
- Drop and reacquire the lock in the NIOAsyncWriter
Result:
Lower chance of deadlock
### Motivation:
Unfortunately, even if no metics delegate has been configured,
significant work is being done _each EL tick_. Concretely:
- 2 `NIODealine.now()` calls which are expensive particularly on some
hypervisors (may cause a VMEXIT!)
- Some unnecessary calculations
### Modifications:
- Only perform the metrics delegate work if a delegate has actually been
set
### Result:
If no delegate is set, each EL tick will now be faster.
Co-authored-by: George Barnett <gbarnett@apple.com>
### Motivation:
`NIOAsyncTestingChannel` stored its `localAddress` and `remoteAddress`
in a locked storage on itself for thread safety, however in doing so
left us open to bugs because a handler grabbing the addresses of the
context had no visibility of the values.
### Modifications:
Reach into `EmbeddedChannelCore` for the addresses instead of storing
them on the `NIOAsyncTestinghannel`. I also considered a delegate
approach where the `EmbeddedChannelCore` could offload the
responsibility for storing the values back to the
`NIOAsyncTestingChannel` but it was complicated and of questionable
value.
### Result:
* The correct address values are seen no matter how they are obtained.
* We probably take a performance hit locking the values in this way but
this is testing code so probably not the end of the world.
When having a state machine inside the `NIOLoopBoundBox` we currently
check the EL when reading and writing. This is unnecessary. Because of
this, this PR introduces a new `withValue` method, that allows users to read and
write the value inside the NIOLoopBoundBox while only paying the cost
for the EL check once.
Motivation:
NIO has existing GRO support. This GRO support is applied at Channel
scope, which makes it somewhat less than ideal, as it forces the
datagram sizes to be accessed using socket options. In real applications
this is less desirable than being able to do this on a per-superbuffer
context.
Linux supports this already by using recvmsg/recvmmsg control data. We
just need to wire this up in NIO. We can re-use the existing metadata
fields on AddressedEnvelope.
Modifications:
- Add a new ChannelOption to extra metadata
- Reject setting this option on non-Linux devices.
- On Linux, use this to load the control data.
- Added new tests that validate the behaviour is correct.
Result:
Enable per-message GRO.
Motivation:
NIO has existing GSO support. This GSO support is applied at Channel
scope, which makes it somewhat less than ideal, as it forces the
datagram sizes to be known statically ahead of time. In real
applications this is less common than being able to do this on a
per-superbuffer
context.
Linux supports this already by using sendmsg/sendmmsg control data. We
just need to wire this up in NIO.
An extension to this is that we can also do per-message GRO in Linux.
I'll tackle this in a later patch to keep the diffs small.
Modifications:
- Adding a new field on AddressedEnvelope Metadata.
- Reject setting this field on non-Linux devices.
- On Linux, use this to set the control data.
- Added new tests that validate the behaviour is correct.
Result:
Enable per-message GSO.
Motivation:
Users would like to be able to access the underlying memory of a
ByteBuffer, as evidenced by the plethora of `withUnsafe*` methods that
ByteBuffer has. As Swift 6.2 has introduced some initial APIs for safe
memory access to underlying storage, we should offer similar APIs on
ByteBuffer to enable users to get safer access to that storage.
For now, the obvious APIs to be able to supplement are:
- withUnsafeReadableBytes
- withUnsafeMutableReadableBytes
- writeWithUnsafeMutableWritableBytes
We can also offer some new APIs to allow initializing a buffer directly
from an OutputSpan.
Note that we can only do this because the Language Steering Group has
pinky promised that they will not break the "Lifetimes" experimental
feature: see
https://forums.swift.org/t/experimental-support-for-lifetime-dependencies-in-swift-6-2-and-beyond/78638
for more details. We are taking them at their word, and so we are
enabling that feature.
Modifications:
Many new methods and tests.
Result:
Safer access.
Motivation:
Users who want to use typed throws would benefit from us propagating the
thrown error type through our APIs. Most of our surface doesn't allow
for that because it relies on standard library APIs that haven't been
updated, but we can do it for ByteBuffer which is entirely ours.
Modifications:
- Adopt typed rethrows in ByteBuffer
Result:
Users have an easier time working with ByteBuffer
Fix Swift for WebAssembly compilation in NIOFoundationCompat.
### Motivation:
NIO is a common dependency, and it is important to enable compiling NIO
using Swift for WebAssembly. This PR fixes compilation for
NIOFoundationCompat.
### Modifications:
- Fix NIOFoundationCompat compilation
### Result:
With these changes, the following build commands succeed: ✅
```
swift build --swift-sdk wasm32-unknown-wasip1-threads --target NIOCore
swift build --swift-sdk wasm32-unknown-wasip1-threads --target NIOFoundationCompat
```
### Context:
This PR is [part of a larger effort by
PassiveLogic](https://github.com/PassiveLogic/swift-web-examples/issues/1)
to improve Swift for WebAssembly support
Motivation:
'copyItem' can be used to copy files/directories from a source to a
destination address and fails if the destination already exists. The
parallel version of it can wedge if a directory is being copied and the
directory can't be created at the destination path (for example, if a
file already exists at the destination).
The parallel copy works by feeding tasks (e.g. "copy this item from here
to there") into an async sequence and processing each task in a separate
child task within a task group. When copying directories a new directory
is created at the destination path and then each file within the source
directory is emitted as a separate item to process. Another item is sent
on the async sequence to indicate when the source directory is finished
with which may result in finishing the async sequence.
However, if creating the destination directory fails then that event
isn't sent. This results in the calling code never terminating the async
sequence and causes 'copyItem' to wedge.
Modifications:
- Use non-idempotent directory creation (copy item should fail if the
destination already exists, this was a regression introduced in
7124f096).
- Check whether to continue when a dir can't be created but always emit
the end of dir event
- If terminating when the task group hasn't reached its width limit then
check child tasks for errors
Result:
Parallel copy doesn't wedge
Use fatalError in all .wait() calls for WASI builds only, and point
developers towards .get() instead.
### Motivation:
While working to adopt NIO in some wasm code, I've commonly ran into
issues any time code calls `.wait()` during wasm runtime. Typically the
executable either traps or crashes any time `.wait()` is called with an
ambiguous error: `Uncaught (in promise) RuntimeError: Atomics.wait
cannot be called in this context`. The error occurs because it is
forbidden to block the main thread for a wasm executable, and the
current implementation of `.wait()` blocks the calling thread.
The fix is straight forward, all calls to `.wait()` need refactor to
`.get()`, which sometimes involves some Swift Concurrency adoption (eg.
`async`) in the process. That change avoids blocking the main thread.
### Modifications:
Added `fatalError` with a descriptive error message to help developers
identify the issue easier.
### Result:
For WASI builds only, changes the trap error message `Uncaught (in
promise) RuntimeError: Atomics.wait cannot be called in this context`
into the following error message instead:
```
NIO's wait() function should not be called on WASI platforms. It will freeze or crash. Use get() instead.
```
### Alternatives considered
- We could instead conditionalize `.wait()` completely out of nio for
WASI builds, to turn runtime errors into compiler errors. That makes
detection of this issue easier, but forces a refactor and breaking
change, and somewhat precludes the possibility that WASI might support
this down the road with a future change.
- We could add a deprecation for WASI only. But long term WASI may be
able to support this blocking call, so deprecation and removal is
communicating the wrong message if the future ends up being reality.
### Context
This PR is [part of a larger effort by
PassiveLogic](https://github.com/PassiveLogic/swift-web-examples/issues/1)
to move Swift for WebAssembly support forward in a large number of
dependencies.
~~Implement `AsyncSequence/split()` functions similar to
`String/split()` functions in std-lib.~~
Implement `AsyncSequence/splitLines()` functions similar to
`String/split(whereSeparator: \.isNewline)` in std-lib.
### Motivation:
~~Provide an easy way for users to split the data incoming from an async
sequence, using their preferred separator.~~
Provide an easy way for users to split the data incoming from an async
sequence, on new lines.
### Modifications:
Add `internal SplitMessageDecoder: NIOSingleStepByteToMessageDecoder`.
Add `public NIOSplitLinesMessageDecoder:
NIOSingleStepByteToMessageDecoder`.
Add `public
AsyncSequence/splitLines(omittingEmptySubsequences:maximumBufferSize) ->
AsyncSeq<ByteBuffer>`.
Add `public
AsyncSequence/splitUTF8Lines(omittingEmptySubsequences:maximumBufferSize)
-> AsyncSeq<String>`.
### Result:
Users can easily split the data.
### Motivation:
`ByteBuffer.lastIndex(where:)` is of suboptimal performance.
The default Collection implementations don't go through any "magic
underscored" functions like `_customIndexOfEquatableElement`.
### Modifications:
Manually implement `lastIndex(where:)`.
### Result:
Basically free performance boost. 2x+ boost even for not big buffers of
a few hundred bytes.
This function is currently used in
`ByteBufferView.trim(limitingElements:)`.
I have no immediate use case for this function, but it's still an issue
worth addressing.
The createDirectory function will succeed without error if the target
directory already exists.
### Motivation
This change addresses issue #3404. Currently,
`fileSystem.createDirectory` fails if the target directory already
exists, forcing users to write boilerplate try/catch blocks to handle
this common and expected case.
The goal is to make this function's behavior idempotent.
### Modifications
To achieve this, I've made the following changes:
**(Implementation)** A new private helper function,
`_handleCreateDirectoryFileExists`, was introduced. This function is
responsible for:
1. Performing a `stat` call on the path that failed.
2. Checking if the existing item is a directory (`S_IFDIR`).
3. Returning a success result if it's a directory, or re-throwing the
original `.fileExists` error if it's a file or another type of entity.
**(Logic)** The core `_createDirectory` function was updated to call
this new helper function whenever `Syscall.mkdir` fails with an `EEXIST`
(`.fileExists`) error. This check is applied in both internal loops to
correctly handle cases where either an intermediate directory or the
final target directory already exists.
### Result
With this change, users can now call the function
`fileSystem.createDirectory` and the operation will succeed even if the
directory is already present, leading to cleaner and more predictable
code.
### Motivation:
`ByteBuffer.firsIndex` is of suboptimal performance.
The default Collection implementations don't go through any "magic
underscored" functions like `_customIndexOfEquatableElement`.
### Modifications:
Manually implement `firstIndex(where:)`.
### Result:
Basically free performance boost. 2x+ boost even for not big buffers of
a few hundred bytes.
There are some usage of this function in `BufferedReader`. Those will
become much faster.
Also this function is used in `ByteBufferView.trim(limitingElements:)`.
Also makes #3411 stuff faster. See:
https://github.com/apple/swift-nio/pull/3411#discussion_r2436260691
Previous PR: #3405
Add an API on top of `AsyncSequence<ByteBuffer>` which can dynamically
decode values.
~~Add an API on top of the new `AsyncSequence<ByteBuffer>` APIs which
splits the file based on its content.~~
### Motivation:
Provides a nice API to decode files, instead of users having to go
though manually handling `BufferedReader.read(while:)`.
I struggled with this, as documented in
https://swift-open-source.slack.com/archives/C9MMT6VGB/p1760115481607159
### Modifications:
Add `NIODecodedAsyncSequence` + functions on `AsyncSequence<ByteBuffer>`
to create such a sequence.
~~Add `NIOSplitMessageDecoder` + stdlib-like functions on
`AsyncSequence<ByteBuffer>` to create such a sequence.~~
### Result:
Users can decode an async sequence of `ByteBuffer`s easier.
~~Users can easily split files based on their content.~~
### Checklist
See this comment for a checklist of the remaining things to do:
https://github.com/apple/swift-nio/pull/3407#issuecomment-3403382404
Motivation
In rare cases where the inbound side of a pipe channel is already closed
(either because it was never open or because it was closed during use),
we can get into trouble if the reader drops the read side of our write
pipe. In that context, the Linux kernel will deliver an EPOLLHUP, but
while we'll close the FD we'll let the channel hang out as a zombie.
This is, obviously, suboptimal.
Modifications
In writeEOF, we check whether the read side is already closed. If it is,
we trigger a close internally to shut the channel down. Added a test for
this as needed, and modified an existing test that would now trip over
this behaviour. Also added a new test for the reverse case to ensure we
don't end up with zombies there (we don't).
Results
No zombie channels.
### Motivation:
Useful for parsing packets, for example `ipv4: InlineArray<4, UInt8>`
and `ipv6: InlineArray<16, UInt8>`.
### Modifications:
For now I've only added a `readInlineArray` function. I know some other
functions are missing, such as `writeInlineArray`.
I wanted to first open up a discussion and see if these changes are
acceptable. I can add those functions too if required, in this PR or
other PRs.
### Result:
Users can read `ByteBuffer` into stack-allocated memory, which can be
more performant than the other alternatives like `Array`, or more
convenient than reading as a tuple like `(UInt8, UInt8, UInt8, UInt8)`.
### Caveats:
Swift 6.2 is required so I've used `#if compiler(>=6.2)`.
Furthermore, `InlineArray` is marked as available on `macOS(9999)` since
the Swift team have yet to update that mark, although `InlineArray` is
planned for Swift 6.2 per the
[proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0453-vector.md).
Edit: to be clear things work fine on Linux and that's where I've been
using this same `readInlineArray` function that I've proposed.
---------
Co-authored-by: Cory Benfield <lukasa@apple.com>
Stepan UlyaninandGeorge Barnett
Si BeaumontandAgam Dua
Fabian Fett
Raphael
Kushal Pisavadia
scottmarchant
3405691582
Adam FowlerandCory Benfield
Joseph HeckandCory Benfield
Karan LokchandaniandCory Benfield
aryan-25
Rafael Cepeda
Max Desiatov
Marc Prud'hommeaux
Mahdi Bahrami
Hamza Hassanain