Motivation:
Users of the HTTPClientResponseDelegate expect that the event loop
futures returned from didReceiveHead and didReceiveBodyPart can be used
to exert backpressure. To be fair to them, they somewhat can. However,
the TaskHandler has a bit of a misunderstanding about how NIO
backpressure works, and does not correctly manage the buffer of inbound
data.
The result of this misunderstanding is that multiple calls to
didReceiveBodyPart and didReceiveHead can be outstanding at once. This
would likely lead to severe bugs in most delegates, as they do not
expect it.
We should make things work the way delegate implementers believe it
works.
Modifications:
- Added a buffer to the TaskHandler to avoid delivering data that the
delegate is not ready for.
- Added a new "pending close" state that keeps track of a state where
the TaskHandler has received .end but not yet delivered it to the
delegate. This allows better error management.
- Added some more tests.
- Documented our backpressure commitments.
Result:
Better respect for backpressure.
Resolves#348
Motivation:
AsyncHTTPClient attempts to avoid the problem of Happy Eyeballs making
it hard to know which Channel will be returned by only inserting the
TLSEventsHandler upon completion of the connect promise. Unfortunately,
as this may involve event loop hops, there are some awkward timing
windows in play where the connect may complete before this handler gets
added.
We should remove that timing window by ensuring that all channels always
have this handler in place, and instead of trying to wait until we know
which Channel will win, we can find the TLSEventsHandler that belongs to
the winning channel after the fact.
Modifications:
- TLSEventsHandler no longer removes itself from the pipeline or throws
away its promise.
- makeHTTP1Channel now searches for the TLSEventsHandler from the
pipeline that was created and is also responsible for removing it.
- Better sanity checking that the proxy TLS case does not overlap with
the connection-level TLS case.
Results:
Further shrinking windows for pipeline management issues.
Motivation:
When we stream request body, current implementation expects that body
will finish streaming _before_ we start to receive response body parts.
This is not correct, reponse body parts can start to arrive before we
finish sending the request.
Modifications:
- Simplifies state machine, we only case about request being fully sent
to prevent sending body parts after .end, but response state machine
is mostly ignored and correct flow will be handled by NIOHTTP1
pipeline
- Adds HTTPEchoHandler, that replies to each response body part
- Adds bi-directional streaming test
Result:
Closes#327
Motivation:
Flaky tests are bad.
This test is flaky because the server closes the connection immediately
upon channelActive. In practice this can mean that the handshake never
even gets a chance to start: by the time the SSLHandler ends up
in the pipeline the connection is already dead. Heck, by the time we
attempt to complete the connection the connection might be dead.
Modifications:
- Change the shutdown to be on first read.
- Remove the disabled autoRead.
- Change the expected NIOTS failure mode to connectTimeout,
which is how this manifests in NIOTS.
Result:
Test is no longer flaky.
Motivation:
Currently when either we or the server send Connection: close, we
correctly do not return that connection to the pool. However, we rely on
the server actually performing the connection closure: we never call
close() ourselves. This is unnecessarily optimistic: a server may
absolutely fail to close this connection. To protect our own file
descriptors, we should make sure that any connection we do not return
the pool is closed.
Modifications:
If we think a connection is closing when we release it, we now call
close() on it defensively.
Result:
We no longer leak connections when the server fails to close them.
Fixes#324.
Motivation:
Right now we only handle one type of SSL error: `.handshakeFailed`,
but in reality a multitude of errors can happen, for example, remote
party might just close connection that will in turn raise
`.uncleanShutdown` error, that will be dropped on the floor and
users will only get non-descriptive `NoResult` error.
Modifications:
Handle all types of SSL errors during handshake instead of just one.
Adds a test.
Result:
Closes#313
Co-authored-by: Cory Benfield <lukasa@apple.com>
allowed only using exposed API.
Motivation:
Having a setter for internal state of ConnectionsState led to a subset
of test testing invalid invariants, for example when we have at the same
time an available connecion and a waiter, which is an invalid state of
the system.
Modifications:
* test of ConnectionsState
* ConnectionsState
* ConnectionPool
are modified so the state under tests is achieved only by a sequence of
modifications invoked by state API.
During modification some tests are eliminated as they were testing
artificial state, which can not be achieved by exposed APIs.
ConnectionsState is pruned from "replace" as in no valid state we can
have a situation when we can "replace" a connection.
Invalid invariants and tests are removed.
Result:
We do not have a way to modify state of the ConnectionsState by direct
interaction with private state of the object. Getter on the state is
considered harmless and used for tests only.
* Introduce helper methods for test of ConnectionsState
Motivation:
Issue #234 highlights that we directly manipulate ConnectionsState and
this commit prepares tests to be refactored to manipulate the state
by exposed APIs instead.
Modifications:
* introduce helper methods in ConnectionPoolTestsSupport.swift
Result:
* no observable changes
* Move Connection tests out of ConnectionsState tests into separate file.
Motivation:
Clean up of code to address issue #234 - here we move away
connection tests to separate files outside of ConnectionsState tests
so we will be able to work on the ConnectionsState in focussed mode.
Modifications:
Connection tests moved to separate files.
Result:
No observable changes.
* Gather Connection code into Connection.swift
Motivation:
For tests we will need a simple version of Connection, so here I gather
Connection code in one place and will generify ConnectionsState on next
commit.
Modifications:
Code of Connection is moved from multiple files into single
Connections.swift.
Result:
All tests are passing, no observable behaviour change.
* Introduce generic type ConnectionType into ConnectionsState
Motivation:
To rework tests of ConnectionsState we want to have a "simpler" version
of Connection to be used, therefore here we convert ConnectionsState to
support generic type ConnectionType. We will substitute current
Connection with a test version in follow up commit.
Modifications:
ConnectionsState is altered to work on generic type ConnectionType
instead of solid type Connection.
Users of ConnectionsState are modified to provide type Connection into
ConnectionType in this commit.
Result:
Test are passing, no observable behaviour change.
* Add FileDownloadDelegate for simple file downloads
* Add testFileDownload to the allTests array
* Fix formatting
* Fix compatibility with Swift 5.0
* Add doc comments, update README.md
* Refine FileDownloadDelegate description in README
* Bump NIO version, remove weak self, cleanup test
* Fix formatting issues in a doc comment
* Create separate Progress struct, async open file
* Create an ad-hoc EventLoopGroup for opening a file
* Move file opening code to `didReceiveBodyPart`
* Fix linter error in FileDownloadDelegate.swift
* Fix wrong future assignment in FileDownloadDelegate
* Fix Swift 5.0 return statement compatibility
* Fix linter warning
* Fix Swift 5.0 return statement compatibility
* Remove redundant `write` function
* Add negative test case and separate testing endpoint
* Add missing testFileDownloadError to the manifest
* fail if we get part when state is endOrError
* Prevent TaskHandler state change after `.endOrError`
Motivation:
Right now if task handler encounters an error, it changes state to
`.endOrError`. We gate on that state to make sure that we do not
process errors in the pipeline twice. Unfortunately, that state
can be reset when we upload body or receive response parts.
Modifications:
Adds state validation before state is updated to a new value
Adds a test
Result:
Fixes#297
Motivation:
TaskHandler unconditionally releases it's connection on error,
this can lead to double release. This issue actually indicates
a more general issue where handler continues to handle errors
even after its state is `.endOrError`. We need to fix this by
ignoring all subsequent errors.
Modifications:
1. Check state before calling out delegate and pool
2. Replace all error callouts with call to `errorCaught`
Result:
Fixes#294
Motivation:
Build warnings fail our builds.
Modifications:
- Update minimum NIOSSL version
- Remove the use of the now-unnecessary `try` statement.
Result:
No warnings!
Motivation:
Streams length parameter is optional to allow cases were stream length is not known in advance, but we do not support this in request validation. This PR aims to address that.
Modifications:
Modifies request validation to default to chunked encoding if body length is zero or to passed in content-length header
Adds a test
Result:
Closes#218
* Added tests that ensure redirects to unix socket paths from a regular HTTP server are disallowed.
Motivation:
Currently, redirects to any supported URL scheme will always be allowed, despite code being in place to seemingly prevent it. See #230.
Modifications:
- Added a method to HTTPBin to redirect to the specified target.
- Added failing tests that perform redirects from a regular server to a socket-based server and vice versa.
Result:
Failing tests that show that the existing redirect checks were inadequate.
* Fixed an issue where redirects to socket path-based servers from any server was always allowed.
Motivation:
An arbitrary HTTP(S) server should not be able to trigger redirects, and thus activity, to a local socket-path based server, though the opposite may be a valid scenario. Currently, requests in either direction are allowed since the checks don't actually check the destination scheme.
Modifications:
- Refactored `hostSchemes`/`unixSchemes` to `hostRestrictedSchemes`/`allSupportedSchemes`, which better describes what they do.
- Refactored `Request.supports()` to `Request.supportsRedirects(to:)` since it is only used by Redirects now.
- Check the destination URL's scheme rather than the current URL's scheme when validating a redirect.
Result:
Closes#230
Co-authored-by: Artem Redkin <artem@redkin.me>
* Added additional tests for socketPath-based requests
Motivation:
While going through the existing tests, I identified a few more instances where we could add some testing.
Modifications:
Added one test that verifies Requests are being decoded correctly, and improved three others to check for path parsing, error throwing, and schema casing respectively.
Result:
Tests that continue to pass, but that will also catch any incompatible changes in the future.
* Added some convenience initializers to URL and methods to Request for making requests to socket paths
Motivation:
Creating URLs for connecting to servers bound to socket paths currently requires some additional code to get exactly right. It would be nice to have convenience methods on both URL and Request to assist here.
Modifications:
- Refactored the get/post/patch/put/delete methods so they all call into a one line execute() method.
- Added variations on the above methods so they can be called with socket paths (both over HTTP and HTTPS).
- Added public convenience initializers to URL to support the above, and so socket path URLs can be easily created in other situations.
- Added unit tests for creating socket path URLs, and testing the new suite of convenience execute methods (that, er, test `HTTPMETHOD`s). (patch, put, and delete are now also tested as a result of these tests)
- Updated the read me with basic usage instructions.
Result:
New methods that allow for easily creating requests to socket paths, and passing tests to go with them.
* Removed some of the new public methods added for creating a socket-path based request
Motivation:
I previously added too much new public API that will most likely not be necessary, and can be better accessed using a generic execute method.
Modifications:
Removed the get/post/patch/put/delete methods that were specific to socket paths.
Result:
Less new public API.
* Renamed execute(url:) methods such that the HTTP method is the first argument in the parameter list
Motivation:
If these are intended to be general methods for building simple requests, then it makes sense to have the method be the first parameter in the list.
Modifications:
Moved the `method: HTTPMethod` parameter to the front of the list for all `execute([...] url: [...])` methods, and made it default to .GET. I also changed the url parameter to be `urlPath` for the two socketPath based execute methods.
Result:
A cleaner public interface for users of the API.
* Fixed some minor issues introduces with logging
Motivation:
Some of the convenience request methods weren't properly adapted for logging.
Modifications:
- Removed a doc comment from patch() that incorrectly referenced a logger.
- Fixed an issue where patch() would call into post().
- Added a doc comment to delete() that references the logger.
- Tests for the above come in the next commit...
Result:
Correct documentation and functionality for the patch() and delete() methods.
* Updated logging tests to also check the new execute methods
Motivation:
The logging tests previously didn't check for socket path-based requests.
Modifications:
Updated the `testAllMethodsLog()` and `testAllMethodsLog()` tests to include checks for each of the new `execute()` methods.
Result:
Two more tests that pass.
Motivation:
AsyncHTTPClient is not a simple piece of software and nowadays also
quite stateful. To debug issues, the user may want logging.
Modification:
Support passing a logger to the request methods.
Result:
Debugging simplified.
* Added tests for http+unix and https+unix url schemes
Motivation:
Using a base URL as the socket path only works when the URL object is maintained as long as possible through the stack. Additionally, it doesn't currently provide a way to use TLS over UNIX sockets.
Modifications:
Added two tests to test out the to-be supported URL schemes, http+unix, and https+unix, which encode the socket path as a %-escaped hostname, as some existing services already do.
Result:
Better UNIX domain socket support.
Motivation:
Previously, we'd only use the server's connection header to determine if
we should close the connection or not. That's wrong because if we set
`connection: close` ourselves, we must not reuse again.
Modification:
Set `TaskHandler.closing = false` if we send a close header.
Result:
More HTTP correctness.
Motivation:
We didn't have a test that tested that streaming really streams, ie.
that request body part N arrives before N+1 is sent.
Modification:
Test it.
Result:
Better test coverage.
Motivation:
AHC's tests were split in HTTPClientTests (which only use the public
API) and HTTPClientInternalTests (which can use `internal` API by using
`@testable`). At some point, the public API tests had `@testable` added
to the AHC import which breaks this idea.
Modification:
Restore the intent by removing `@testable` and moving the 2 tests that
needed it over.
Result:
Cleaner test suite.
This is the continuation of the good work of @Yasumoto and @weissi in #135
The following code adds support for NIO Transport services. When the ConnectionPool asks for a connection bootstrap it is returned a NIOClientTCPBootstrap which wraps either a NIOTSConnectionBootstrap or a ClientBootstrap depending on whether the EventLoop we are running on is NIOTSEventLoop.
If you initialize an HTTPClient with eventLoopGroupProvider set to .createNew then if you are running on iOS, macOS 10.14 or later it will provide a NIOTSEventLoopGroup instead of a EventLoopGroup.
Currently a number of tests are failing. 4 of these are related to the NIOSSLUncleanShutdown error the others all seem related to various race conditions which are being dealt with on other PRs. I have tested this code with aws-sdk-swift and it is working on both macOS and iOS.
Things look into:
The aws-sdk-swift NIOTS HTTP client had issues with on Mojave. We should check if this is the case for async-http-client as well.
Co-authored-by: Joe Smith <yasumoto7@gmail.com>
Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
* Enable clients to call URLs that include %2F as an escaped backslash
Previously `percentEncodedPath` was using `path.addingPercentEncoding(withAllowedCharacters: .urlPathAllowed)`
which converts %2F to a literal '/'. This prevented users fetching URLs like https://api.travis-ci.org/repo/github/rails%2Frails
which use %2F as part of a path segment.
Migrating to `URLComponents(url: self, resolvingAgainstBaseURL: false)?.percentEncodedPath` has the desired behaviour
for the couple of test cases that exist.
Updated the test server to switch on the `percentEncodedPath` so it's easier
to understand the desired behaviour.
David EvansandGitHub
Cory BenfieldandGitHub
Artem RedkinandGitHub
Ilya TeterinandGitHub
Max DesiatovandGitHub
Adam FowlerandGitHub
Johannes WeissandGitHub
Artem RedkinandGitHub
Iain SmithandGitHub
Fabian FettandGitHub