Motivation:
The `CNIOSHA1` module from SwiftNIO is an implementation detail. The fact that we were able to import and build the project with it is a bug/feature of Swift Package Manager.
Should the Swift ecosystem standardize on an implementation, that will replace the one we have chosen later.
Modifications:
Use the SHA1 native Swift implementation from CryptoSwift by reproducing the necessary code in our project.
Updates to the appropriate files to provide attribution to CryptoSwift have been included.
Result:
A reliable and pure Swift implementation of the SHA1 digest algorithm in the project for the sake of Lua script cache hashing.
### General usage
Most users should only need to use the `evalScript` method. We take the approach recommended by the [redis EVAL docs](https://redis.io/commands/eval):
>>>
The client library implementation can always optimistically send EVALSHA under the hood even when the client actually calls EVAL, in the hope the script was already seen by the server. If the NOSCRIPT error is returned EVAL will be used instead.
>>>
To facilitate this, we introduce a RedisScript struct, which calculates the sha1 hash on the client and stores it alongside the script source. I took inspiration for this approach from the [radix golang library](https://github.com/mediocregopher/radix). The struct makes using scripts a bit easier/nicer, and guides the developer toward the highest performance approach, but I'm curious whether you think it's an unnecessary abstraction.
One disadvantage of the optimistic approach using `EVALSHA` is that the `evalScript` may not be atomic if the script is not cached yet, which will be a problem if we are using it during an explicit pipeline or MULTI/EXEC operation. The redis docs recommend using `SCRIPT LOAD` before a multi/exec operation to avoid this. I think we could leave this up to the application developer.
### Should we implement the basic commands?
The approach above forces the application developer to use the evalScript method instead of exposing either `EVAL` or `EVALSHA` directly. This is to avoid the behavior of only using `EVAL` without caching, which might otherwise seem like the natural choice. If you think this is too opinionated, we can also add basic `eval` and `evalSha` methods.
### SHA1 calculation
I've copied the approach used by the SwiftNIO developers - RediStack already includes the `CNIOSHA1` module required.
There's a comment in [their private implementation](https://github.com/apple/swift-nio/blob/master/Sources/NIOWebSocket/SHA1.swift#L17) which indicates they're not entirely happy with this, but consider it the best approach for server-side Swift for now. I think we could follow their lead and migrate to a standard library method if/when it becomes available.
The `sha1` method is added as an extension property on String. Because it can theoretically fail, the `sha1` property is optional and thus the RedisScript initializer is failable, which is a bit unpleasant. I don't think the sha1 calculation should never actually fail given a String input. A force unwrap in the sha1 property would remove several code smells [like this](https://gitlab.com/liamdon/swift-redi-stack/blob/scripting/Sources/RediStack/Commands/ScriptingCommands.swift#L92), but I left it out in case you have a strong opposition to force unwraps - let me know what you think.
### Not implemented
I have not implemented `SCRIPT DEBUG` or `SCRIPT KILL`, because these seem more like sysadmin tasks that should not be in a client library. But they can easily be added if we want full coverage of the scripting commands.
Motivation:
For ergonomics, users sometimes want to provide arguments as a variadic list rather than an array.
Modifications:
- Add variadic overloads for almost all methods that accept lists of homogenous types
Result:
Users should have more flexibility in the way arguments are passed to command methods
Motivation:
It is wrong to always assume that a GET operation is expecting a String response type, as users may be storing other types of data.
Modifications:
- Add `get` generic method with a constraint for types of `RESPValueConvertible` to convert values to the user desired type
- Change existing `get` method to specialize the generic overload
- Fix incorrect doc block regarding the ELF failure condition
Result:
Users should now be able to specialize the return type of a "GET" command
Motivation:
The assertions in `errorCaught(context:error:)` and `channelRead(context:data:)` are holdovers from when SwiftLog was not integrated into the package. The conditions where they were triggered are safely handled in other ways, so the asserts are not guarding against undefined behavior or bad app state.
Modifications:
Remove the two asserts regarding the `RedisCommandHandler.commandResponseQueue` to rely on Logging to bubble up errors to the user.
Result:
Debug builds using RediStack should not encounter unexpected assertions and should rely on other, better, error handling.
This contributes to issue #65
Motivation:
After enabling the thread sanitizer during testing, a data race with the `activeConnectionCount` in `RedisMetrics` was caught due to changing a primitive `Int` across threads.
Modifications:
Add a specialized class `ActiveConnectionGauge` to wrap the semantics of the `activeConnectionCount` with an `Atomic<Int>` property.
Result:
No data races should occur with the `RedisMetrics` subsystem.
Motivation:
While working to add more test coverage with `RESPTranslator`, it was made apparent that a `.bulkString(.none)` is impossible to create directly with the `RESPValue` initializers, even though it is a reasonable possibility.
Additionally, forcing all integer types to have to be stored in an `Int` is unnecessarily restrictive.
Modifications:
- Change `RESPValue.init(bulk:)` initializers to accept `Optional` instances
- Change `RESPValue.init(bulk:)` for `Int` initializer to be generic on `FixedWidthInteger`
Result:
Converting types to and from `RESPValue` should be more bi-directional and seamless.
Motivation:
Diagnostics for why `.bulkString` parses might fail were weak, and edge cases fell through gaps in coverage were found.
Modifications:
Added new cases to `RESPTranslator.ParsingError` for `.bulkString` parsing with additional test coverage.
Result:
Users should have better diagnostics for bogus data or failed parsing state.
Motivation:
After working with RedisKit with RediStackTestUtils as a dependency, it was realized how opinionated the module is in how RedisConnections can be created in test environments.
Modifications:
Require more information, with reasonable defaults for `RedisConnection.init()`. Provide subclass hooks for `RedisIntegrationTestCase` for implementors to make decisions for themselves at how to connect to Redis.
Result:
Users should have more freedom in how they connect to Redis in their units tests.
Motivation:
There is a reasonable way to compare if two `RedisErrors` are equal, which was seen as needed in the `Equatable` conformance for `RESPValue`.
Modifications:
Added `Equatable` conformance for `RedisError` by comparing the messages.
Result:
Two `RedisError` instances are now equatable.
Motivation:
While working on unit tests the need for conformance to `Equatable` for `RESPValue` has been needed a few times and it was decided to make it public.
Modifications:
Added conformance to `Equatable` for `RESPValue` with unit test.
Result:
Users should now be able to compare two `RESPValue` instances for equality.
Motivation:
During the refactor work (commit ea7c755) that was merged with MRs !71 and !53 - `sendCommandsImmediately` was accidentally lowered to `internal`
Modifications:
Properly mark `RedisConnection.sendCommandsImmediately` as `public`
Result:
Developers should now have proper access to the `sendCommandsImmediately` property
Motivation:
Issue #60 called for improving the type safety of the options available for the `zadd` command, and MR !70 made some great headway, but attempted to cram too much into a single enum.
Modifications:
- Break the `RedisSortedSetAddOption.returnChangedCount` value into an additional boolean param
Result:
Using `zadd` should now be more straight forward, while being type safe.
Motivation:
There are several cases where "command" could refer to a command keyword, or an entire message (keyword + args). This made working with `RedisCommand` and it's documentation ambiguous.
Modifications:
- Rename `RedisCommand.command` to `message`
- Rename initializer labels to `message` and `responsePromise`
Result:
When encountering a `RedisCommand` everyone should know that they are dealing with a message that should be sent to Redis as soon as possible.
Motivation:
To make it a little more generic, and to avoid turnover during renames (such as the planned rebranding in issue #61), `RedisClientError` more accurately reflects the source of the errors, as well as the responsibility of causing the bug.
Modifications:
- Rename `RedisNIOError` to `RedisClientError`
- Rename `RedisError` file to `RedisErrors`
- Add documentation of `RedisClientError`
- Remove no longer used `.unsupportedOperation(method:message:)` value
- Rename `.responseConversion(to:)` to `.failedRESPConversion(to:)`
Result:
Names of `RedisClientError` should be more descriptive, less prone to turnover, and more documented for users to understand the issues related to these thrown errors.
Motivation:
During proposal review, and while working within the codebase, several issues were identified with how `RedisConnection` was architectured.
Modifications:
- Change implementation of `RedisConnection` in some areas for new logic of internal `ConnectionState`
- Change behavior of logging in a few places
- The initializer for `RedisConnection` is now **internal**
- How users can override the default `ClientBootstrap` for a connection is by passing an instance to the `.connect` static method
- Change unit tests to inherit from a common XCTestCase class that handles creation and cleanup of `RedisConnection` in tests
- Remove Redis namespace enum
Result:
The API for `RedisConnection` should be much simpler, with the implementation being less buggy.
This resolves issues #49, #54, and #57.
Motivation:
While working through issue #59, it was noticed just how "stringly" the SortedSet command options for `zadd`, `zinterstore`, and `zunionstore` were, and Swift provides ways of having strong type safety for these options.
Modifications:
- Add `RedisSortedSetAddOption` and `RedisSortedSetAggregateMethod` to replace the String API in `zadd`, `zinterstore`, and `zunionstore`
- Fix an implication of how `overestimatedCountBeingAdded` documentation for `Array where Element == RESPValue` for `add(contentsOf:overestimatedCountBeingAdded:_:)`
Result:
Users should have a more discoverable and straightforward way that isn't error prone for calling `zadd`, `zinterstore`, and `zunionstore` with Redis supported options.
Motivation:
Johannes continues to provide great insight, and correctly pointed out that `RESPValueConvertible` was being used as an "existential" in all cases.
This can cause unexpected type-erasure and introduce unnecessary cost overhead with dynamic dispatch when in most cases we know the exact value we want for `RESPValue` to execute commands.
Modifications:
- Add new extensions to `Array where Element == RESPValue` for appending and adding elements into them
- Change `RedisClient.send(command:with:)` to require `[RESPValue]` instead of `[RESPValueConvertible]` as the `with` argument type
- Change all instances of `RESPValueConvertible` being an "existential" type for method arguments to instead be a generic constraint
Result:
The library should be safeguarded from a class of bugs, with the use of `send` being a bit more straight forward, with some new convenience methods for `[RESPValue]` types.
Motivation:
Johannes provided a fair code review of the project and summarized his findings in issue #48, and one of the prime offenders was all of the `unsafe*` APIs (pointers, buffers, bytes)
that were used with `RESPValue` and `RESPValueConvertible`.
He also provided great feedback and pointed out good points of confusion with the API design of `RESPValue` and `RESPValueConvertible`.
Modifications:
- Return to using `Array` instead of `ContiguousArray` for `RESPValue.array` storage
- Update all documentation to be more thorough in explaining how the types should be used and conformed to.
- Remove all uses of `unsafe*` APIs where possible
- Change implementations to be a lot more type and memory safe, double checking assumptions
- Remove conformance to `ExpressibleBy*Literal` as it is too easy for users to shoot themselves in the foot and saves only a few characters over `.init(bulk:)`
- Create new `RedisNIOTestUtils` target for common test extensions, making them public
- Move most almost all implementations of `RESPValue` computed properties into the `RESPValueConvertible` conformances
Result:
Users should be more safeguarded by the API against unknowingly getting incorrect `RESPValue` representations, the API design of `RESPValue` and `RESPValueConvertible` should be much clearer,
and memory safety should be at a higher bar from these changes.
This resolves issues #55 & #48, and contributes to issue #47.
Motivation:
During proposal review, it was noted that `RedisCommandContext` was a bit misleading, and the hidden reference semantics worrisome.
In addition, several of parts of the documentation around `RedisCommandHandler` were weak or also misleading.
Modifications:
- Rename `RedisCommandContext` to just `RedisCommand`
- Update documentation to be more explicit about the module who owns the types being referenced
- Update documentation to call out explicit usage semantics and behavior
- Change `RedisCommandHandler` to close the socket connection on error thrown
- Rename the "base" Redis Channel Handlers to be more explicitly named
Result:
Users should have clearer documentation on what happens when using `RedisCommandHandler` and `RedisCommand` without hidden semantics.
This contributes to issue #47.
Motivation:
During proposal review, it was appropriately pointed out that `RedisClient.expire` incorrectly mixes 'deadline' and 'timeout' terminology.
Modifications:
- Change references of 'deadline' to 'timeout' to follow Redis' established semantics for 'EXPIRE'
- Add additional unit test for `RedisClient.expire`
Result:
`RedisClient.expire` should now be more clear as to its semantics and not mix terminology incorrectly.
This contributes to issue #47
Motivation:
During proposal review, feedback was provided that the discoverability of the factory method for building a standard RESP `ChannelPipeline` was poor outside of documentation.
Modifications:
- Move `Redis.makeDefaultClientBootstrap` to `ClientBootstrap.makeRedisTCPClient`.
- Move the `channelInitializer` implementation into a new `Channel.addBaseRedisHandlers()` instance method.
Result:
Users should have an easier time discovering how to easily create baseline RESP `ChannelPipelines`.
This contributes to #47.
Motivation:
During proposal review, it was pointed out that the code with a position index was redundant and error prone over relying on `ByteBuffer`'s `readerIndex`.
Modifications:
Refactored `RESPTranslator` to rely on `ByteBuffer.readerIndex` for position of current parsing cursor,
and eliminated `ParsingResult` enum to instead return `RESPValue?`.
The implementation for writing out `RESPValue` has been expanded to `RESPValueConvertible` and moved to an extension of `ByteBuffer`.
Result:
Parsing `RESPValue` into and out of `ByteBuffer` should be less error-prone and more straight forward.
This contributes to issues #47 and #55
Motivation:
During SSWG review, feedback was provided on the API design of the `RESPValueConvertible.init` signature and how it should appropriately follow Swift Design Guidelines regarding labels.
Modifications:
`RESPValueConvertible.init(_:` is now `RESPValueConvertible.init(fromRESP:)`.
Result:
There should be more clarity at the call site when initializing a type from a `RESPValue`.
This contributes to #47
Motivation:
During SSWG review, feedback was provided that forced a re-evaluation of early design feedback interpretation on composability of RedisNIO.
First understanding was that the desire was to have "customization points" to gain benefits of implementation of RedisCommandHandler, while new understanding is that it just needs to be public in order for users to include it in their own custom ChannelPipeline schemes.
Modifications:
`RedisCommandHandler` is now a final class.
Result:
Users will no longer be able to subclass `RedisCommandHandler`, but gain a super slight performance increase.
This contributes to #47.
Motivation:
The default port for Redis is well published to be 6379, and it is common to want to pass this value around or use it as a static default value in methods and initializers.
Modifications:
Add `RedisConnection.defaultPort` static property for all users to use, and update references of the `6379` literals to use new the new property.
Result:
Users should have a reliable default defined to use everywhere to avoid bugs.
Motivation:
`Foundation.Data` was unexpectedly receiving `RESPValueConvertible` conformance through the `Collection` extension which lead to incorrect encoding.
Modifications:
Add explicit conformance to `RESPValueConvertible` for `Data`.
Result:
Users should see their `Data` encoding to RESP format as expected.
Motivation:
Reading output from `RESPValue` existentials is entirely verbose and does not provide a human-readable description of what is being represented.
Modifications:
- Add conformance to `CustomStringConvertible` for `RESPValue`
- Remove redundant logging of arguments in `RedisConnection.send`
Result:
String representations of `RESPValue` should now be more readable for human to understand.
Motivation:
The SSWG has identified a fast approaching reality of namespace clashes in SPM within the ecosystem and has proposed a rule on names that `NIORedis` no longer complies with.
Modifications:
All references to `NIORedis` have been switched to `RedisNIO` as this module name is unique (at least within GitHub's public repositories).
The goals for this name are as follows:
1. To indicate that this is a Redis client library that is built with SwiftNIO
2. That it is a lower level library, as it directly exposes SwiftNIO as an implementation detail
2a. The idea being that a higher level library (`Redis`) will be used, and to "go one level deeper" in the stack, you append the "deeper" `NIO` postfix
3. It follows a naming pattern adopted by Vapor who has expressed their desire to adopt this library as their Redis implementation
Result:
A repository, package name, and module name that are unique across GitHub's public repositories that achives the goals outlined above.
Motivation:
End users will want to capture some baseline performance metrics that can only be accurately gathered at this level of the stack.
Modifications:
Added new `RedisMetrics` struct that retains all SwiftMetrics objects used throughout the system lifecycle, and appropriate code to track desired metrics.
Result:
Users now have a way to peek into the system's performance to understand what NIORedis is being asked to do from their usage.
Motivation:
To be a comprehensive library, all commands should be implemented, even if they are highly discouraged. List's collection of commands were missing `brpop`, `blpop`, and `brpoplpush`.
Modifications:
`brpop`, `blpop` and `brpoplpush` are supported with defaults and overloads for an easier API.
Result:
Users now have access to `brpop`, `blpop` and `brpoplpush` commands.
Motivation:
To be a comprehensive library, all commands should be implemented, even if they are highly discouraged. Sorted Set's collection of commands were missing `bzpopmin` and `bzpopmax`.
Modifications:
`bzpopmin` and `bzpopmax` are supported with defaults and overloads for an easier API.
`RedisClient.channel` is now `internal` to have access during testing for bypassing normal guards for closing connections.
Result:
Users now have access to `bzpopmin` and `bzpopmax` commands.
Motivation:
Debugging and properly handling errors was difficult as the `ChannelHandler` names were the default generic names, in addition the `RedisCommandHandler` did not pipe errors further up the pipeline.
Modifications:
`RedisCommandHandler` now calls `context.fireErrorCaught(error)` when receiving errors, and the default Redis `ChannelPipeline` handlers are added with debugging names.
Result:
It should be easier for debugging purposes to work with any Redis `ChannelPipeline`.
Motivation:
The goal of the `Redis.makeConnection` factory method is to provide end users with a quick way to jump in and get started with Redis in Swift.
Right now, users have to provide an `EventLoopGroup` instance, when a reasonable default is available for us to define.
Modifications:
- Add: `MultiThreadedEventLoopGroup` for 1 thread as a default argument to the `using:` label in `Redis.makeConnection`
- Remove: The `with:` label for the password in `Redis.makeConnection`
- Change: The project README to reflect the current state of the project
Results:
Users should be able to create `RedisConnections` by just defining an IP Address & Port to connect to - and possibly a password.
In addition, the README should now properly direct users on how to use the latest version of the library.
Motivation:
During the discussion thread of the NIORedis SSWG proposal, there were concerns expressed over the implementation
of `RedisPipeline` that covered the following areas:
1. Clashes with SwiftNIO concepts such as "Pipeline"
2. Misleading users on library behavior with using a Pipeline vs. a single `RedisConnection`
3. The implementation was "too clever" in that it allowed users to easily find themselves in "Undefined Behavior"
due to lack of enough type safety in the API
However, the value in being able to control how frequently "flush" happens on a socket was discussed and considered
high - so something still needs to be in place to force flushes by users.
It was decided to leave the implementation of batching commands and their responses to library users, perhaps in
higher level frameworks while the library will provide said mechanism for controlling writing and flushing of commands.
Modifications:
- Add: `sendCommandsImmediately` bool property to `RedisConnection` to handle choice of flushing after each command
- Remove: `RedisPipeline`
Results:
Users should now have a more clear understanding on what type of control they have over the timing of when commands
are actually sent over the wire, with the greater emphasis placed on `RedisConnection`.
Motivation:
As it stands, the parsing / encoding implementations for RESP was directly tied to the NIO Channel Handlers.
Unit tests were sloppily spread across multiple files and it made it difficult to explicitly separate out
the Channel Handler behavior from the RESP specification implementation.
Modifications:
- Add: `RESPTranslator` enum helper object with static methods that only handles RESP parsing / encoding
- Rename: `RESPEncoder` to `RedisMessageEncoder`
- Rename: `RESPDecoder` to `RedisByteDecoder`
Results:
It should be easier to understand what type is intended to be used as part of a NIO channel pipeline while
still having a direct way of parsing / encoding between Swift types and the RESP specification.
Unit tests should be more maintainable as well.
Motivation:
The two provided factory methods for creating `RedisConnection` and `ClientBootstrap` were not readily discoverable or appropriately expressible.
Modifications:
- Add: `Redis` top-level namespace enum
- Move & Rename: `RedisConnection.connect` factory method to `Redis.makeConnection`
- Move & Rename: `ClientBootstrap.makeRedisDefault` factory method to `Redis.makeDefaultClientBootstrap`
- Rename: `EventLoopFuture` extension file to just `SwiftNIO` to have all framework extensions in a single file
Results:
Using NIORedis should be more discoverable and straight forward on how to create a connection with `Redis.makeConnection(...)` over `RedisConnection.connect(...)`
Nathan Harris
Liam Don
Nathan Harris
Ondrej Rafaj