Motivation:
The `CNIOSHA1` module from SwiftNIO is an implementation detail. The fact that we were able to import and build the project with it is a bug/feature of Swift Package Manager.
Should the Swift ecosystem standardize on an implementation, that will replace the one we have chosen later.
Modifications:
Use the SHA1 native Swift implementation from CryptoSwift by reproducing the necessary code in our project.
Updates to the appropriate files to provide attribution to CryptoSwift have been included.
Result:
A reliable and pure Swift implementation of the SHA1 digest algorithm in the project for the sake of Lua script cache hashing.
### General usage
Most users should only need to use the `evalScript` method. We take the approach recommended by the [redis EVAL docs](https://redis.io/commands/eval):
>>>
The client library implementation can always optimistically send EVALSHA under the hood even when the client actually calls EVAL, in the hope the script was already seen by the server. If the NOSCRIPT error is returned EVAL will be used instead.
>>>
To facilitate this, we introduce a RedisScript struct, which calculates the sha1 hash on the client and stores it alongside the script source. I took inspiration for this approach from the [radix golang library](https://github.com/mediocregopher/radix). The struct makes using scripts a bit easier/nicer, and guides the developer toward the highest performance approach, but I'm curious whether you think it's an unnecessary abstraction.
One disadvantage of the optimistic approach using `EVALSHA` is that the `evalScript` may not be atomic if the script is not cached yet, which will be a problem if we are using it during an explicit pipeline or MULTI/EXEC operation. The redis docs recommend using `SCRIPT LOAD` before a multi/exec operation to avoid this. I think we could leave this up to the application developer.
### Should we implement the basic commands?
The approach above forces the application developer to use the evalScript method instead of exposing either `EVAL` or `EVALSHA` directly. This is to avoid the behavior of only using `EVAL` without caching, which might otherwise seem like the natural choice. If you think this is too opinionated, we can also add basic `eval` and `evalSha` methods.
### SHA1 calculation
I've copied the approach used by the SwiftNIO developers - RediStack already includes the `CNIOSHA1` module required.
There's a comment in [their private implementation](https://github.com/apple/swift-nio/blob/master/Sources/NIOWebSocket/SHA1.swift#L17) which indicates they're not entirely happy with this, but consider it the best approach for server-side Swift for now. I think we could follow their lead and migrate to a standard library method if/when it becomes available.
The `sha1` method is added as an extension property on String. Because it can theoretically fail, the `sha1` property is optional and thus the RedisScript initializer is failable, which is a bit unpleasant. I don't think the sha1 calculation should never actually fail given a String input. A force unwrap in the sha1 property would remove several code smells [like this](https://gitlab.com/liamdon/swift-redi-stack/blob/scripting/Sources/RediStack/Commands/ScriptingCommands.swift#L92), but I left it out in case you have a strong opposition to force unwraps - let me know what you think.
### Not implemented
I have not implemented `SCRIPT DEBUG` or `SCRIPT KILL`, because these seem more like sysadmin tasks that should not be in a client library. But they can easily be added if we want full coverage of the scripting commands.
Motivation:
For ergonomics, users sometimes want to provide arguments as a variadic list rather than an array.
Modifications:
- Add variadic overloads for almost all methods that accept lists of homogenous types
Result:
Users should have more flexibility in the way arguments are passed to command methods
Motivation:
It is wrong to always assume that a GET operation is expecting a String response type, as users may be storing other types of data.
Modifications:
- Add `get` generic method with a constraint for types of `RESPValueConvertible` to convert values to the user desired type
- Change existing `get` method to specialize the generic overload
- Fix incorrect doc block regarding the ELF failure condition
Result:
Users should now be able to specialize the return type of a "GET" command
Motivation:
The assertions in `errorCaught(context:error:)` and `channelRead(context:data:)` are holdovers from when SwiftLog was not integrated into the package. The conditions where they were triggered are safely handled in other ways, so the asserts are not guarding against undefined behavior or bad app state.
Modifications:
Remove the two asserts regarding the `RedisCommandHandler.commandResponseQueue` to rely on Logging to bubble up errors to the user.
Result:
Debug builds using RediStack should not encounter unexpected assertions and should rely on other, better, error handling.
This contributes to issue #65
Motivation:
After enabling the thread sanitizer during testing, a data race with the `activeConnectionCount` in `RedisMetrics` was caught due to changing a primitive `Int` across threads.
Modifications:
Add a specialized class `ActiveConnectionGauge` to wrap the semantics of the `activeConnectionCount` with an `Atomic<Int>` property.
Result:
No data races should occur with the `RedisMetrics` subsystem.
Motivation:
Now that SwiftPM with 5.1 supports Linux thread sanitizers, we should run this as part of the normal test pass to catch threading issues.
Modifications:
Add `--sanitize=thread` argument to `swift test` commands for any job that runs Swift >=5.1.
Result:
More bugs should be caught as soon as possible now that thread races are being monitored.
Motivation:
With the release of Swift 5.1, the official docker containers are available for both 16.04 and 18.04 of Linux - so the test matrix should include them.
Modifications:
- Add new 5.1 jobs for 18.04 linux and change 16.04 5.1 jobs to use the official docker images.
- Update the README to capture the changes
Result:
The number of configurations the library is tested against should be as wide as possible for regular interval testing passes.
Motivation:
While working to add more test coverage with `RESPTranslator`, it was made apparent that a `.bulkString(.none)` is impossible to create directly with the `RESPValue` initializers, even though it is a reasonable possibility.
Additionally, forcing all integer types to have to be stored in an `Int` is unnecessarily restrictive.
Modifications:
- Change `RESPValue.init(bulk:)` initializers to accept `Optional` instances
- Change `RESPValue.init(bulk:)` for `Int` initializer to be generic on `FixedWidthInteger`
Result:
Converting types to and from `RESPValue` should be more bi-directional and seamless.
Motivation:
Diagnostics for why `.bulkString` parses might fail were weak, and edge cases fell through gaps in coverage were found.
Modifications:
Added new cases to `RESPTranslator.ParsingError` for `.bulkString` parsing with additional test coverage.
Result:
Users should have better diagnostics for bogus data or failed parsing state.
Motivation:
While working on issue #56, it was forgotten to add the new test cases to the linux manifest file.
Modifications:
Update the linux manifests to include all current unit tests
Result:
All written unit tests should be ran on Linux
Motivation:
MRs take unnecessarily too long to have completed pipelines with "redundant" builds happening in sequence.
The entire stage of building in release is an additional QA layer that can happen on merges into master, but for MRs are just added CPU cycles.
While it's useful to ensure that building in release is possible, we will get almost all build failures from the test stage, and the most important checks are the unit and integration tests.
Modifications:
Change the ordering of build vs. test, as well as only running "build in release" jobs on the master branch.
Result:
MRs should be able to be merged sooner as CI pipelines take less time to run.
Motivation:
After working with RedisKit with RediStackTestUtils as a dependency, it was realized how opinionated the module is in how RedisConnections can be created in test environments.
Modifications:
Require more information, with reasonable defaults for `RedisConnection.init()`. Provide subclass hooks for `RedisIntegrationTestCase` for implementors to make decisions for themselves at how to connect to Redis.
Result:
Users should have more freedom in how they connect to Redis in their units tests.
Motivation:
For users looking to contribute, and for those looking to validate the library, it was unclear what tests require an actual connection to a Redis instance in order to run.
Modifications:
Add a `RediStackIntegrationTests` that takes all tests that require a Redis instance in order to run.
Result:
Those looking to run just unit tests, or contribute new tests, can now directly point to a specific testTarget as defined in the Package manifest.
Motivation:
There is a reasonable way to compare if two `RedisErrors` are equal, which was seen as needed in the `Equatable` conformance for `RESPValue`.
Modifications:
Added `Equatable` conformance for `RedisError` by comparing the messages.
Result:
Two `RedisError` instances are now equatable.
Motivation:
While working on unit tests the need for conformance to `Equatable` for `RESPValue` has been needed a few times and it was decided to make it public.
Modifications:
Added conformance to `Equatable` for `RESPValue` with unit test.
Result:
Users should now be able to compare two `RESPValue` instances for equality.
Motivation:
During the refactor work (commit ea7c755) that was merged with MRs !71 and !53 - `sendCommandsImmediately` was accidentally lowered to `internal`
Modifications:
Properly mark `RedisConnection.sendCommandsImmediately` as `public`
Result:
Developers should now have proper access to the `sendCommandsImmediately` property
Motivation:
A handful of times, unit tests were forgotten to be added to the `allTests` extension, or were incorrectly copy/pasted.
Modifications:
Remove manual entries of `allTest` and use generated result from `swift test --generate-linuxmain`
Result:
There should be proper test parity between macOS and Linux.
Motivation:
As this project might take off and support be needed, users will want a direct line of communication to file support tickets.
Modifications:
Update docs to include references and links for contacting the RediStack maintainers for assistance.
Result:
Users should have more information on how to contact project maintainers.
Motivation:
Issue #60 called for improving the type safety of the options available for the `zadd` command, and MR !70 made some great headway, but attempted to cram too much into a single enum.
Modifications:
- Break the `RedisSortedSetAddOption.returnChangedCount` value into an additional boolean param
Result:
Using `zadd` should now be more straight forward, while being type safe.
Motivation:
There are several cases where "command" could refer to a command keyword, or an entire message (keyword + args). This made working with `RedisCommand` and it's documentation ambiguous.
Modifications:
- Rename `RedisCommand.command` to `message`
- Rename initializer labels to `message` and `responsePromise`
Result:
When encountering a `RedisCommand` everyone should know that they are dealing with a message that should be sent to Redis as soon as possible.
Motivation:
To give a better first impression, and providing more comprehensive information for first time viewers, the README file needed reworking.
Modifications:
- Add a CHANGELOG file that at least redirects readers to the releases page of GitLab
- Update the CONTRIBUTORS list
- Change project README to have more information, flow better, and to have a better "first impression"
Result:
The README should give a stronger indication into the project's health.
Motivation:
Up until now, Jazzy docs were generated on a private CI runner hosting macOS - which doesn't make CI portable as it requires that runner to always be available.
Modifications:
- Change: CI config to use a Linux Jazzy Docker image for portability
- Change: Doc theme templates to not reference GitHub
Result:
Jazzy docs should be able to be generated at any time, as the CI infrastructure is using GitLab's cloud solution with a Docker image
Motivation:
To make it a little more generic, and to avoid turnover during renames (such as the planned rebranding in issue #61), `RedisClientError` more accurately reflects the source of the errors, as well as the responsibility of causing the bug.
Modifications:
- Rename `RedisNIOError` to `RedisClientError`
- Rename `RedisError` file to `RedisErrors`
- Add documentation of `RedisClientError`
- Remove no longer used `.unsupportedOperation(method:message:)` value
- Rename `.responseConversion(to:)` to `.failedRESPConversion(to:)`
Result:
Names of `RedisClientError` should be more descriptive, less prone to turnover, and more documented for users to understand the issues related to these thrown errors.
Nathan Harris
Liam Don