- Refactored code from `TaskHandler` into new method `HTTPClient.Request.createRequestHead` that creates an `HTTPRequestHead` and matching `RequestFramingMetadata`
- Added required property `requestFramingMetadata` to `HTTPExecutingRequest`
- Added property `requestFramingMetadata` to `RequestBag`
Co-authored-by: Cory Benfield <lukasa@apple.com>
- Adding four new protocols:
- `HTTPExecutingRequest` a protocol representing an HTTP task that is executed on a ChannelHandler.
- `HTTPScheduledRequest` a protocol representing an HTTP task that is scheduled for execution, but in a waiting state (example: Waiting for an idle connection in the connection pool).
- `HTTPRequestExecutor` a protocol that can be used from the `HTTPExecutingRequest` abstracting away functionality that will normally be implemented by a `ChannelHandler`
- `HTTPRequestScheduler` a protocol that can be used from the `HTTPScheduledRequest` abstracting away functionality that will normally be implemented by a `ConnectionPool`
- An implementation of the `HTTPExecutingTask` and `HTTPScheduledRequest` called `RequestBag`. It implements our current API using the new protocols
Co-authored-by: George Barnett <gbarnett@apple.com>
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
Users of the HTTPClientResponseDelegate expect that the event loop
futures returned from didReceiveHead and didReceiveBodyPart can be used
to exert backpressure. To be fair to them, they somewhat can. However,
the TaskHandler has a bit of a misunderstanding about how NIO
backpressure works, and does not correctly manage the buffer of inbound
data.
The result of this misunderstanding is that multiple calls to
didReceiveBodyPart and didReceiveHead can be outstanding at once. This
would likely lead to severe bugs in most delegates, as they do not
expect it.
We should make things work the way delegate implementers believe it
works.
Modifications:
- Added a buffer to the TaskHandler to avoid delivering data that the
delegate is not ready for.
- Added a new "pending close" state that keeps track of a state where
the TaskHandler has received .end but not yet delivered it to the
delegate. This allows better error management.
- Added some more tests.
- Documented our backpressure commitments.
Result:
Better respect for backpressure.
Resolves#348
Motivation:
HTTPResponseAggregator attempts to build a single, complete response
object. This necessarily means it loads the entire response payload into
memory. It wants to provide this payload as a single contiguous buffer
of data, and it does so by aggregating the data into a single contiguous
buffer as it goes.
Because ByteBuffer does exponential reallocation, the cost of doing this
should be amortised constant-time, even though we do have to copy some
data sometimes. However, if this operation triggers a copy-on-write then
the operation will become quadratic. For large buffers this will rapidly
come to dominate the runtime.
Unfortunately in at least Swift 5.3 Swift cannot safely see that during
the body stanza the state variable is dead. Swift is not necessarily
wrong about this: there's a cross-module call to ByteBuffer.writeBuffer
in place and Swift cannot easily prove that that call will not lead to a
re-entrant access of the `HTTPResponseAggregator` object. For this
reason, during the call to `didReceiveBodyPart` there will be two copies
of the body buffer alive, and so the write will CoW.
This quadratic behaviour is a nasty performance trap that can become
highly apparent even at quite small body sizes.
Modifications:
While Swift can't prove that the `self.state` variable is dead, we can!
To that end, we temporarily set it to a different value that does not
store the buffer in question. This will force Swift to drop the ref on
the buffer, making it uniquely owned and avoiding the CoW.
Sadly, it's extremely difficult to test for "does not CoW", so this
patch does not currently come with any tests. I have experimentally
verified the behaviour.
Result:
No copy-on-write in the HTTPResponseAggregator during body aggregation.
Motivation:
When we stream request body, current implementation expects that body
will finish streaming _before_ we start to receive response body parts.
This is not correct, reponse body parts can start to arrive before we
finish sending the request.
Modifications:
- Simplifies state machine, we only case about request being fully sent
to prevent sending body parts after .end, but response state machine
is mostly ignored and correct flow will be handled by NIOHTTP1
pipeline
- Adds HTTPEchoHandler, that replies to each response body part
- Adds bi-directional streaming test
Result:
Closes#327
* fail if we get part when state is endOrError
* Prevent TaskHandler state change after `.endOrError`
Motivation:
Right now if task handler encounters an error, it changes state to
`.endOrError`. We gate on that state to make sure that we do not
process errors in the pipeline twice. Unfortunately, that state
can be reset when we upload body or receive response parts.
Modifications:
Adds state validation before state is updated to a new value
Adds a test
Result:
Fixes#297
Motivation:
TaskHandler unconditionally releases it's connection on error,
this can lead to double release. This issue actually indicates
a more general issue where handler continues to handle errors
even after its state is `.endOrError`. We need to fix this by
ignoring all subsequent errors.
Modifications:
1. Check state before calling out delegate and pool
2. Replace all error callouts with call to `errorCaught`
Result:
Fixes#294
* Added tests that ensure redirects to unix socket paths from a regular HTTP server are disallowed.
Motivation:
Currently, redirects to any supported URL scheme will always be allowed, despite code being in place to seemingly prevent it. See #230.
Modifications:
- Added a method to HTTPBin to redirect to the specified target.
- Added failing tests that perform redirects from a regular server to a socket-based server and vice versa.
Result:
Failing tests that show that the existing redirect checks were inadequate.
* Fixed an issue where redirects to socket path-based servers from any server was always allowed.
Motivation:
An arbitrary HTTP(S) server should not be able to trigger redirects, and thus activity, to a local socket-path based server, though the opposite may be a valid scenario. Currently, requests in either direction are allowed since the checks don't actually check the destination scheme.
Modifications:
- Refactored `hostSchemes`/`unixSchemes` to `hostRestrictedSchemes`/`allSupportedSchemes`, which better describes what they do.
- Refactored `Request.supports()` to `Request.supportsRedirects(to:)` since it is only used by Redirects now.
- Check the destination URL's scheme rather than the current URL's scheme when validating a redirect.
Result:
Closes#230
Co-authored-by: Artem Redkin <artem@redkin.me>
* Added additional tests for socketPath-based requests
Motivation:
While going through the existing tests, I identified a few more instances where we could add some testing.
Modifications:
Added one test that verifies Requests are being decoded correctly, and improved three others to check for path parsing, error throwing, and schema casing respectively.
Result:
Tests that continue to pass, but that will also catch any incompatible changes in the future.
* Added some convenience initializers to URL and methods to Request for making requests to socket paths
Motivation:
Creating URLs for connecting to servers bound to socket paths currently requires some additional code to get exactly right. It would be nice to have convenience methods on both URL and Request to assist here.
Modifications:
- Refactored the get/post/patch/put/delete methods so they all call into a one line execute() method.
- Added variations on the above methods so they can be called with socket paths (both over HTTP and HTTPS).
- Added public convenience initializers to URL to support the above, and so socket path URLs can be easily created in other situations.
- Added unit tests for creating socket path URLs, and testing the new suite of convenience execute methods (that, er, test `HTTPMETHOD`s). (patch, put, and delete are now also tested as a result of these tests)
- Updated the read me with basic usage instructions.
Result:
New methods that allow for easily creating requests to socket paths, and passing tests to go with them.
* Removed some of the new public methods added for creating a socket-path based request
Motivation:
I previously added too much new public API that will most likely not be necessary, and can be better accessed using a generic execute method.
Modifications:
Removed the get/post/patch/put/delete methods that were specific to socket paths.
Result:
Less new public API.
* Renamed execute(url:) methods such that the HTTP method is the first argument in the parameter list
Motivation:
If these are intended to be general methods for building simple requests, then it makes sense to have the method be the first parameter in the list.
Modifications:
Moved the `method: HTTPMethod` parameter to the front of the list for all `execute([...] url: [...])` methods, and made it default to .GET. I also changed the url parameter to be `urlPath` for the two socketPath based execute methods.
Result:
A cleaner public interface for users of the API.
* Fixed some minor issues introduces with logging
Motivation:
Some of the convenience request methods weren't properly adapted for logging.
Modifications:
- Removed a doc comment from patch() that incorrectly referenced a logger.
- Fixed an issue where patch() would call into post().
- Added a doc comment to delete() that references the logger.
- Tests for the above come in the next commit...
Result:
Correct documentation and functionality for the patch() and delete() methods.
* Updated logging tests to also check the new execute methods
Motivation:
The logging tests previously didn't check for socket path-based requests.
Modifications:
Updated the `testAllMethodsLog()` and `testAllMethodsLog()` tests to include checks for each of the new `execute()` methods.
Result:
Two more tests that pass.
Motivation:
AsyncHTTPClient is not a simple piece of software and nowadays also
quite stateful. To debug issues, the user may want logging.
Modification:
Support passing a logger to the request methods.
Result:
Debugging simplified.
* Added tests for http+unix and https+unix url schemes
Motivation:
Using a base URL as the socket path only works when the URL object is maintained as long as possible through the stack. Additionally, it doesn't currently provide a way to use TLS over UNIX sockets.
Modifications:
Added two tests to test out the to-be supported URL schemes, http+unix, and https+unix, which encode the socket path as a %-escaped hostname, as some existing services already do.
Result:
Better UNIX domain socket support.
Motivation:
Previously, we'd only use the server's connection header to determine if
we should close the connection or not. That's wrong because if we set
`connection: close` ourselves, we must not reuse again.
Modification:
Set `TaskHandler.closing = false` if we send a close header.
Result:
More HTTP correctness.
* Enable clients to call URLs that include %2F as an escaped backslash
Previously `percentEncodedPath` was using `path.addingPercentEncoding(withAllowedCharacters: .urlPathAllowed)`
which converts %2F to a literal '/'. This prevented users fetching URLs like https://api.travis-ci.org/repo/github/rails%2Frails
which use %2F as part of a path segment.
Migrating to `URLComponents(url: self, resolvingAgainstBaseURL: false)?.percentEncodedPath` has the desired behaviour
for the couple of test cases that exist.
Updated the test server to switch on the `percentEncodedPath` so it's easier
to understand the desired behaviour.
* Close idle pool connections
Motivation: Pooled connections should close at some point (see #168)
Changes:
- Add new poolingTimeout property to HTTPClient.Configuration, it's
default value is .seconds(60), it can be set to nil if one wishes to
disable this timeout.
- Add relevant unit test
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
motivation: Better performance thanks to connection reuse
changes:
- Added a connection pool for HTTP/1.1
- All requests automatically use the connection pool
- Up to 8 parallel connections per (scheme, host, port)
- Multiple additional unit tests
Previously, UNIX Domain Sockets would only work if the URL also had a
"base URL". If it didn't have a base URL, it would try to connect to the
empty string which would fail :).
Now, we support both cases:
- URLs with a baseURL (the path to the UDS) and a path (actual path)
- URLs that just have an actual path (path to the UDS) where we'll just
use "/" as the URL's path
As discussed in #128. We make the ResponseAccumulator public to give developers an easy time to create a Task. With the ResponseAccumulator the developer using this does not have to worry, about implementing the HTTPClientResponseDelegate all by himself.
Motivation:
More unit tests are good and now that SwiftNIO shipd
`NIOHTTP1TestServer`, writing integration tests for async-http-client is also
more straightforward.
Modification:
Demonstrate some tests using NIOHTTP1TestServer.
Result:
More tests.
motivation: the Swift Server Workgroup is not a legal entity and cannot hold copyrights. with this change, code authors continue and retain their copyrights under the apache license and previous copyrights note, but Apple steps up instead of the workgroup which has no legal status
changes:
* update header files to say "Apple Inc. and the AsyncHTTPClient project authors" instead of "Swift Server Workgroup and the AsyncHTTPClient project authors"
* update validation scripts to check for the correct header
* add CONTRIBUTING.md file to explain how to make contributions and include a legal notice about licensing the contribution to Apple and the project
* regenerate CONTRIBUTORS.md to reflect most recent contributions
Karl
David Nadoba
Fabian Fett
David Evans
Mads Odgaard
Cory Benfield
Artem Redkin
Tim Condon
Dimitri Bouniol
Johannes Weiss
Adam Fowler
Artem Redkin
Fabian Fett
Iain Smith
Trevör
Marcin Krzyzanowski
tomer doron