### Motivation:
To ensure `NIOHTTP1Server` concurrency safety.
### Modifications:
* Enable strict concurrency checking in the package manifest.
* User synchronous operations in pipeline configurations
### Result:
Builds will warn and CI will fail if regressions are introduced.
### Motivation:
Opening the `swift-nio` repository made me warning blind because there
were always so many trivially fixable warnings about things that were
correct but cannot be understood by the compiler.
### Modifications:
Fix all the sendable warnings that popped up, except for one test where
`NIOLockedValueBox<Thread?>` isn't sendable because `Foundation.Thread`
seemingly isn't `Sendable` which is odd. Guessing that'll be fixed on
their end.
### Result:
- Fewer warnings
- Less warning-blindness
- More checks
Motivation:
`IOData` is a legacy but alas also core type that needs to be
`Sendable`. Before this PR however it can't be `Sendable` because it
holds a `FileRegion` which holds a `NIOFileDescriptor`. So let's make
all of these `Sendable` but let's also start the deprecation journey for
the following types:
- `IOData`, now soft-deprecated (no warnings) because on its reliance on
`FileRegion`
- `FileRegion`, now soft-deprecated (no warnings) because on its
reliance on `NIOFileHandle`
- `NIOFileHandle`, now soft-deprecated (warnings on the
`NIOFileHandle(descriptor:)` constructor but with a
`NIOFileHandle(_deprecatedTakingOwnershipOfDescriptor:)` alternative
- `NonBlockingFileIO`, now soft-deprecated (warnings on the `openFile`
functions (but with `_deprecated` alternatives) because of their
reliance on `NIOFileHandle)
Modification:
- Make `NIOFileDescriptor`, `FileRegion` and `IOData` `Sendable` by
tracking the fd number and the usage state in an atomic
- Enforce singular access by making the `withFileDescriptor { fd ... }`
function atomically exchange the fd number for a "I'm busy" sentinel
value
- Start deprecating `IOData`, `NIOFileHandle`, `NonBlockingFileIO`,
`FileRegion`
Result:
- `NIOFileDescriptor`, `FileRegion` and `IOData` can be `Sendable`
### Motivation:
Teaching the compiler that NIO is safe is good.
### Modifications:
Use NIOLoopBound and friends to explain.
### Result:
Fewer warnings.
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
Since Swift 5.5 and [SE-0299](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0299-extend-generic-static-member-lookup.md) it is possible to add static members to protocols which are discoverable through the shorthand dot syntax.
This change reduces type repetition and improves call-site legibility.
Modifications:
Added extensions for ChannelOption with static members where Self is bound to a concrete type.
Result:
ChannelOption types can be used with the leading dot syntax. For eg:
```
//before
.channelOption(ChannelOptions.explicitCongestionNotification, value: true)
.channelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)
//after
.channelOption(.explicitCongestionNotification, value: true)
.channelOption(.socketOption(.so_reuseaddr), value: 1)
```
* Apply formatting
* Apply no block comments rule
* Apply OmitExplicitReturns
* Apple OnlyOneTrailingClosureArgument
* Apply NoAssignmentInExpressions
* Fix up DontRepeatTypeInStaticProperties lint errors
* Apply `OrderedImports`
* Apply `ReplaceForEachWithForLoop`
* format file
* Enable the formatting pipeline
* Adopt `AmbiguousTrailingClosureOverload`
* Fix license header
* Fix format check
* Fix `EndOfLineComment`
* Fix CI
* Adapt CI script to check if changes when running formatting
* Separate lint and format into to steps
* Fix format
* Adopt `UseEarlyExits`
* Revert "Adopt `UseEarlyExits`"
This reverts commit d1ac5bbe12.
### Motivation:
SwiftNIO allows and encourages to precisely manage all operating system resources like file descriptors & threads. That is a very important property of the system but in many places -- especially since Swift Concurrency arrived -- many simpler SwiftNIO programs only require a single, globally shared EventLoopGroup. Often even with just one thread.
Long story short: Many, probably most users would happily trade precise control over the threads for not having to pass around `EventLoopGroup`s. Today, many of those users resort to creating (and often leaking) threads because it's simpler. Adding a `.globalSingle` static var which _lazily_ provides singleton `EventLoopGroup`s and `NIOThreadPool`s is IMHO a much better answer.
Finally, this aligns SwiftNIO a little more with Apple's SDKs which have a lot of global singletons that hold onto system resources (`Dispatch`'s thread pool, `URLSession.shared`, ...). At least in `Dispatch`'s case the Apple SDKs actually make it impossible to manage the resources, there can only ever be one global pool of threads. That's fine for app development but wouldn't be good enough for certain server use cases, therefore I propose to add `NIOSingleton`s as an _option_ to the user. Confident programmers (especially in libraries) are still free and encouraged to manage all the resources deterministically and explicitly.
Companion PRs:
- https://github.com/apple/swift-nio-transport-services/pull/180
- https://github.com/swift-server/async-http-client/pull/697
### Modifications:
- Add the `NIOSingletons` type
- Add `MultiThreadedEventLoopGroup.singleton`
- Add `NIOThreadPool.singleton`
### Result:
- Easier use of NIO that requires fewer parameters for users who don't require full control.
- Helps with #2142
- Fixes#2472
- Partially addresses #2473
* Async support NIOPipeBootstrap
# Motivation
We want to support async bootstrapping with all our bootstraps.
# Modification
This PR adds support for the `NIOPipeBootstrap` and adds the three variants of methods to it (abstract output, `NIOAsyncChannel` based and protocol negotiation based)
# Result
We now support asynchronous interaction with the `NIOPipeBootstrap`
* Rename to takingOwnershipOfInputOutputDescriptor
* Rename to takingOwnershipOfDescriptor and takingOwnershipOfDescriptors
* _ prefix
* doc comment
Motivation:
The remaining NIO code really conceptually belongs in a module called
NIOPosix, and NIOCore should really be called NIO. We can't really do
that last step, but we can prepare by pushing the bulk of the remaining
code into a module called NIOPosix.
Modifications:
- Move NIO to NIOPosix
- Make NIO an umbrella module.
Result:
NIOPosix exists.
Motivation:
As we've largely completed our move to split out our core abstractions,
we now have an opportunity to clean up our dependencies and imports. We
should arrange for everything to only import NIO if it actually needs
it, and to correctly express dependencies on NIOCore and NIOEmbedded
where they exist.
We aren't yet splitting out tests that only test functionality in
NIOCore, that will follow in a separate patch.
Modifications:
- Fixed up imports
- Made sure our protocols only require NIOCore.
Result:
Better expression of dependencies.
Co-authored-by: George Barnett <gbarnett@apple.com>
Motivation:
The names of most of the BSD socket option values were brought over into
their NIO helper properties, with their namespace removed. This vastly
increases the risk of collision, particularly for things like TCP_INFO,
which just became .info.
Modifications:
- Added the prefixes back, e.g. `.info` is now `.tcp_info`.
- Renamed socket type `.dgram` to `.datagram`, as this is the nice clean
API and we should use nice clean names.
Result:
Better APIs
* NIO: Add new `BSDSocket` namespace
This starts the split of the POSIX/Linux/Darwin/BSD interfaces and the
BSD Socket interfaces in order to support Windows. The constants on
Windows are not part of the C standard library and need to be explicitly
prefixed. Use the import by name to avoid the `#if` conditions on the
wrapped versions. This will allow us to cleanly cleave the dependency
on the underlying C library.
This change adds a `BSDSocket.OptionLevel` and `BSDSocket.Option` types
which allow us to get proper enumerations for these values and pipe them
throughout the NIO codebase.
* Apply suggestions from code review
Co-Authored-By: Cory Benfield <lukasa@apple.com>
Motivation:
There are use-cases for SwiftNIO where there are no actual sockets but
rather two pipe file descriptors - one for input, one for output.
There's no real reason why SwiftNIO shouldn't work for those.
Modifications:
Add a PipeChannel.
Result:
More use-cases for SwiftNIO.
Motivation:
Previously the maths on NIODeadline/TimeAmount could sometimes crash and
weren't always right.
Modifications:
Fix the maths and add tests.
Result:
Fewer crashes, more happy times.
Motivation:
Networking software like SwiftNIO that always has explicit flushes
usually does not benefit from having TCP_NODELAY switched off. The
benefits of having it turned on are usually quite substantial and yet we
forced our users for the longest time to enable it manually.
Quite a bit of engineering time has been lost finding performance
problems and it turns out switching TCP_NODELAY on solves them
magically.
Netty has made the switch to TCP_NODELAY on by default, SwiftNIO should
follow.
Modifications:
Enable TCP_NODELAY by default.
Result:
If the user forgot to enable TCP_NODELAY, their software should now be
faster.
Motivation:
The file-io implementation in the example HTTP1 server just crashed on
empty file ;).
Modifications:
- fix the crash
- add integration test
Result:
fewer crashes
Motivation:
There is a UDP echo client which is very similar to the NIOEchoClient.
The styles of these two examples should match closely so ‘diff’ comparing will show the pertinent API differences.
Modifications:
Add self tags to member variables in the TCP echo client.
Add a self tag to a member variable in the HTTP1 server.
Remove an assert from the echo client.
Result:
The code in the TCP and UDP examples now matches and fits the SwiftNIO style.
Motivation:
`ctx` was always an abbreviation was 'context` and in Swift we don't
really use abbreviations, so let's fix it.
Modifications:
- rename all instances of `ctx` to `context`
Result:
- fixes#483
Motivation:
The thread pool implementation can be used for many things not just
blocking IO.
Modifications:
rename BlockingIOThreadPool to NIOThreadPool
Result:
fixes#186
Motivation:
- `ChannelPipeline.add(name:handler:...)` had a strange order of arguments
- `remove(handler:)` and `remove(ctx:)` both remove `ChannelHandler`s
but they read like they remove different things
So let's just fix the argument order and name them `addHandler` and
`removeHandler` making clear what they do.
Modifications:
- rename all `ChannelPipeline.add(name:handler:...)`s to `ChannelPipeline.addHandler(_:name:...)`
- rename all `ChannelPipeline.remove(...)`s to `ChannelPipeline.removeHandler(...)`
Result:
more readable and consistent code
Motivation:
ByteBuffer methods like `set(string:)` never felt very Swift-like and
also didn't look the same as their counterparts like `getString(...)`.
Modifications:
- rename all `ByteBuffer.set/write(<type>:,...)` methods to
`ByteBuffer.set/write<Type>(...)`
- polyfill the old spellings in `_NIO1APIShims`
Result:
code more Swift-like
Motivation:
While getting into NIO through the examples I stumbled
accross some chained dropFirst() calls, which can be
simplified by using dropFirst(_:) instead.
Modifications:
I did a quick project-wide search of chained dropFirst()
usage and modified every line to use dropFirst(_:) instead.
Result:
This will only effect the readability of the code,
not its functionality.
Motivation:
ELF's API should be as close as possible to the new Result's API.
Therefore, we should rename `then` to `flatMap`
Modifications:
- renamed `then` to `flatMap`
- renamed `thenIfError` to `flatMapError`
- renamed ELF's generic parameter from `T` to `Value`
Result:
- more like Result
- fixes#688
Motivation:
Addressing the https://github.com/apple/swift-nio/issues/713
Modifications:
Forward the statusCode from NIOHTTP1Server main handleJustWrite to ctx.writeAndFlush instead of always using "ok"
Result:
ChannelHandlerContext will write the correct status code even when it's not "ok"
Motivation:
Now that the stdlib has introduced the Result type, we can use it in the
implementation (and the whenComplete) function of EventLoopFuture
Modifications:
- replace EventLoopValue with Result
- make whenComplete provide the Result
Result:
use the new shiny stuff
* Use fixed default string to write in response and do not hardcode content length
Motivation:
Improve example code, resolve#714
Modifications:
Move hardcoded "Hello world" string to a constant property and removed hardcoding of content length header value
Result:
In example code content length will be written correctly when default response changes
Motivation:
Swift naming guidelines mandate that factory methods start with `make`,
like `makeSomething`. We had a few that were `newSomething`.
Modifications:
make all factories start with make
Result:
more compliant to Swift naming rules
Motivation:
making a promise of a type is more grammatical than making a promise for
a type.
Modifications:
s/newPromise(for/newPromise(of/g
Result:
more grammar
Motivation:
When creating new promises I always find it very frustrating to type the
`: EventLoopPromise<Type>` type annotation but it's necessary for the
compiler to know type the promise will be fulfilled with.
Modifications:
allow an optional `for: SomeType.self` parameter for `newPromise` as
let p = eventLoop.newPromise(for: Int.self)
is much easier to type than
let p: EventLoopPromise<Int> = eventLoop.newPromise()
Result:
easier to write code
Motivation:
_ = expression() is ugly and in many cases unimportant. In fact we train
ourselves to overread it which makes special cases where you'd actually
expect a result to be used just look normal.
Modifications:
remove _ = from a lot of places. In many cases we already had a better
(and sometimes cheaper way) to not return a value but in some cases
(mostly `remove` functions) I added `@discardableResult`
Result:
NIO source code looks nicer
Motivation:
The server attaches a dynamic handler to requests and will keep passing
data until the handlers are removed; which never happens.
Modifications:
`nil` out the `handler` when `completeResponse` is called.
Result:
When connections are reused we now get the correct dynamic handler for
it instead of the previous one.
Motivation:
We should be consistent with naming and also choose descriptive names.
Modifications:
- Deprecate old init method that uses numThreads
- Add new init with numberOfThreads param name
- Everywhere use the new init
Result:
More consistent and descriptive naming. Fixes https://github.com/apple/swift-nio/issues/432.
* Make NIOHTTP1 compile on 32-bit platforms
Motivation:
When building a datacenter out of Apple Watches w/ shattered
glasses, you'll need the ability to build on 32 bit platforms.
This fixes building NIOHTTP1 on such platforms.
Modifications:
This was resetting the parser context pointer using a very
long deadly beef. Had to shorted this for 32bit cows.
Result:
NIOHTTP1 compiles on 32 bit ARM platforms.
Motivation:
Opening a file, seeking it or querying its size (or other information)
is blocking on UNIX so `NonBlockingFileIO` should support that too.
Modifications:
Added a method to `NonBlockingFileIO` which lets the user open a file
without blocking the calling thread.
Result:
Less blocking is good :)
Motivation:
We had a number of problems:
1. We wanted to lazily process input EOFs and connection resets only
when the user actually calls `read()`. On Linux however you cannot
unsubscribe from `EPOLLHUP` so that's not possible.
2. Lazily processing input EOFs/connection resets wastes kernel
resources and that could potentially lead to a DOS
3. The very low-level `Selector` interpreted the eventing mechanism's
events quite a lot so the `EventLoop`/`Channel` only ever saw
`readable` or `writable` without further information what exactly
happened.
4. We completely ignored `EPOLLHUP` until now which on Unix Domain
Socket close leads to a 100% CPU spin (issue #277)
Modifications:
- made the `Selector` interface richer, it now sends the following
events: `readable`, `writable`, `readEOF` (input EOF), `reset`
(connection reset or some error)
- process input EOFs and connection resets/errors eagerly
- change all tests which relied on using unconnected and unbound sockets
to user connected/bound ones as `epoll_wait` otherwise would keep
sending us a stream of `EPOLLHUP`s which would now lead to an eager
close
Result:
- most importantly: fix issue #277
- waste less kernel resources (by dealing with input EOFs/connection
resets eagerly)
- bring kqueue/epoll more in line
Motivation:
`ab -k` behaves weirdly: it'll send an HTTP/1.0 request with keep-alive set
but stops doing anything at all if the server doesn't also set Connection: keep-alive
which our example HTTP1Server didn't do.
Modifications:
In the HTTP1Server example if we receive an HTTP/1.0 request with
Connection: keep-alive, we'll now set keep-alive too.
Result:
ab -k doesn't get stuck anymore.
Motivation:
Currently the HTTP decoders can throw errors, but they will be ignored
and lead to a simple EOF. That's not ideal: in most cases we should make
a best-effort attempt to send a 4XX error code before we shut the client
down.
Modifications:
Provided a new ChannelHandler that generates 400 errors when the HTTP
decoder fails.
Added a flag to automatically add that handler to the channel pipeline.
Added the handler to the HTTP sample server.
Enabled integration test 12.
Result:
Easier error handling for HTTP servers.
Motivation:
I wanted to make sure that I fixed the minor typos and that I addressed
the feedback that was provided to me in regards to force unwrapping.
I also wanted to make sure my error messages were accurate.
Modifications:
Fixed a few minor typos and updated the force unwrap error messages.
Result:
Fixed typos and guard statements with accurate error messages.
fix some typos, force unwraps and inconsistencies
Motivation:
Reviewing the code I found a few typos that hindered readability, some unnecessary force unwraps and inconsistencies in the codebase.
Modifications:
- The main file in NIOWebSocketServer now accepts command line arguments
and binds to a passed in host and port like the other server examples do.
- Minor typo fixes
- removed one force unwrap
Result:
- The result of my change will bring the NIOWebSocketServer more inline
with how the NIOHTTP1 and NIOChat servers examples are setup.
- less force unwrapping
- fewer typos
[message edited by @weissi]
Motivation:
Fix inconsistency of <Void> and <()> occurances in the codebase.
Modifications:
Changed EventLoop{Future,Promise}<()> occurances to EventLoop{Future,Promise}<Void>
Result:
Consistent code style for EventLoopPromise and EventLoopFuture.
Motivation:
Passing in a function directly makes the code a bit more concise.
Modifications:
Changing `flatMap { Int($0) }` to `flatMap(Int.init)`.
Result:
More concise code.
Motivation:
There are quite a few `switch`es that cover just two cases. Explicitly state this via an `if case` + `else` structure.
Modifications:
Changes something like:
switch a {
case .b:
return c
default:
return d
}
to
if case .b = a {
return c
} else {
return d
}
Result:
This should prevent misinterpretation and thus potential bugs in the future.
Motivation:
This changes two things:
- it removes the negation of non-used associated types (the removal of `(_)`).
- it prefers exhausting `switch`es over `default`ing.
This change does not impact bahavior; it is pure refactoring.
Modifications:
Omitting needless `(_)`'s and explicitly exhausting switches.
Result:
This change is pure refactoring.
Motivation:
The current HTTP server helpers are not modular, meaning that each
time we add a new first-party HTTP handler we need to add a brand new
method. Additionally, they do not currently compose, so if you want
assistance with HTTP pipelining *and* to support HTTP upgrade you are
out of luck.
Modifications:
Deprecate the two existing server helpers.
Add a new server helper that can be extended to support all of the
possible features that users may want in their HTTP pipelines.
Result:
Users will have a single place to go that can be used to configure
their HTTP server pipeline, and that understands how the different
handlers fit together in a complete pipeline.
Motivation:
In a few instances, we have code that ignores a `()` argument in a
closure with this syntax:
{ (_: Void) in
whilst that's a bit odd, it shouldn't be wrong and Swift 4.0.x is also
totally happy with it. The latest Swift 4.1 candidates however are not
and also there's a better way of spelling this:
{ () in
and in some cases even just
{
Modifications:
Removed instances of `{ (_: Void) in`
Result:
Compiles with `swift-4.1-DEVELOPMENT-SNAPSHOT-2018-03-12-a`
Motivation:
HTTP pipelining can be tricky to handle properly on the server side.
In particular, it's very easy to write out of order or inconsistently
mutate state. Users often need help to handle this appropriately.
Modifications:
Added a HTTPServerPipelineHandler that only lets one request through
at a time.
Result:
Better servers that are more able to handle HTTP pipelining
Motivation:
Currently our HTTP server almost always just assumes that keep-alive
is enabled. It really shouldn't, it should validate more carefully.
Additionally, it mishandles TCP half-close in a way that breaks with
netcat.
Modifications:
Added keep-alive tracking to all handlers and ensured that connections
will be closed after a response to a keep-alive request has been sent.
Also add support for handling TCP half-close.
Result:
Non-keep-alive connections will be closed by the server in all cases.
Half-closed connections will be appropriately managed.
Rick Newton-Rogers
Johannes Weiss
Ayush Garg
Franz Busch
Cory Benfield
Saleem Abdulrasool
Liam Flynn
Moritz Lang
JovanMilenkovic
Ilya Puchka
Esteban Torres
Norman Maurer
Helge Heß
Johannes Weiß
Cory Benfield
Matt Eaton
Tibor Bödecs
Bas Broek