Update docs/testing.md

Co-authored-by: Konrad `ktoso` Malawski <ktoso@apple.com>

.gitignore

@@ -1,2 +1,2 @@
-.DS_Store
-
+.DS_Store
+

README.md

@@ -1,87 +1,17 @@
-# Swift on Server Deployment Guide
+# Swift on Server Development Guide
 
 ## Introduction
 
-This guide is designed to help teams and individuals running Swift Server applications on Linux. It focuses on how to compile, test, deploy and debug such application and provides tips in those areas.
+This guide is designed to help teams and individuals running Swift Server applications on Linux and to provide orientation for those who want to start with such development. 
+It focuses on how to build, test, deploy and debug such application and provides tips in those areas.
 
-The guide is a community effort, and all are invited to share their tips and know-how.
+## Contents
+- [Setup and code editing](docs/setup-and-ide-alternatives.md)
+- [Building](docs/building.md)
+- [Testing](docs/testing.md)
+- [Debugging Memory leaks](docs/memory-leaks-and-usage.md)
+- [Performance troubleshooting and analysis](docs/performance.md)
+- [Debugging multithreading issues and memory checks](docs/llvm-sanitizers.md)
+- [Deployment](docs/deployment.md)
 
-## Building
-
-Triggering the build action in Xcode or running `swift build` from the terminal will trigger the build.
-
-Swift is architecture specific, so running the build command on macOS will create a macOS binary. Building on macOS is useful for development and for taking advantage of the great tooling that comes with Xcode. However, most server applications are designed to run on Linux.
-
-To build on Linux and create a Linux binary, use Docker. For example:
-
-`$ docker run -v $PWD:/code -w /code swift:5.1 swift build`
-
-The above command will run the build using the latest Swift 5.1 Docker image, utilizing bind mounts to the sources on your Mac. Apple publishes Docker images to Docker Hub.
-
-By default, SwiftPM will build a debug version of the application. Note that debug versions are not suitable for running in production as they are significantly slower. To build a release version of your app, run `swift build -c release`.
-
-Binary artifacts that could be deployed are be found under .build/x86_64-unknown-linux, or .build/x86_64-apple-macosx for macOS binaries. SwiftPM can show you the full binary path using `swift build --show-bin-path -c release`.
-
-### Building for production
-
-- Build production code in release mode by compiling with `swift build -c release`. Running code compiled in debug mode will hurt performance signficantly. 
-
-- For best performance in Swift 5.2 or later, pass `-Xswiftc -cross-module-optimization` (this won't work in Swift versions before 5.2)
-
-- Integrate [`swift-backtrace`](https://github.com/swift-server/swift-backtrace) into your application to make sure backtraces are printed on crash. Backtraces do not work out-of-the-box on Linux, and this library helps to fill the gap. Eventually this will become a language feature and not require a discrete library.
-
-## Testing 
-
-Triggering the test action in Xcode or running swift test from the terminal will trigger the unit tests.
-
-SwiftPM is integrated with XCTest, Apple’s unit test framework. Test results will be displayed in Xcode or printed out to the terminal.
-
-Like building on Linux, testing on Linux requires the use of Docker. For example:
-
-`$ docker run -v $PWD:/code -w /code swift:5.1 swift test`
-
-The above command will run the tests using the latest Swift 5.1 Docker image, utilizing bind mounts to the sources on your file system.
-
-Swift supports architecture-specific code. By default, Foundation imports architecture-specific libraries like Darwin or Glibc. While developing on macOS, you may end up using APIs that are not available on Linux. Since you are most likely to deploy a cloud service on Linux, it is critical to test on Linux.
-
-Another important detail about testing for Linux is the `Tests/LinuxMain.swift` file. For versions prior to Swift 5.1, This file provides SwiftPM an index of all the tests it needs to run on Linux. 
-
-- For version Swift 5.1 and later, this file is no longer required, and testing with `--enable-test-discovery` seamlessly discovers the tests. This will be the default behvior in future releases of Swift. Until this is the default, it can be useful 
-to keep `Tests/LinuxMain.swift` around but with with the following code to remind you to use the flag:
-
-   `fatalError("Please use `swift test --enable-test-discovery` to run the tests instead")`
-
-- For versions prior to Swift 5.1, it is critical to keep this file up-to-date as you add more unit tests. To regenerate this file, run `swift test --generate-linuxmain` after adding tests. It is also a good idea to include this command as part of your continuous integration setup.
-
-### Testing for production
-
-- For versions Swift 5.1 and later, always test with `--enable-test-discovery` to avoid forgetting tests on Linux.
-
-- Generally, whilst testing, you may want to build using `swift build --sanitize=thread`. The binary will run slower and is not suitable for production, but you'll see many many threading issues before you deploy your software. Often threading issues are really hard to debug and reproduce and also cause random problems. TSan helps catch them early.
-
-- Make use of the sanitizers. Before running code in production, do the following:
-    * Run your test suite with TSan (thread sanitizer): `swift test --sanitize=thread`
-    * Run your test suite with ASan (address sanitizer): `swift test --sanitize=address` and `swift test --sanitize=address -c release -Xswiftc -enable-testing`
-
-## Debugging / Development
-
-### Debugging Common Issues
-
-- [Memory leaks](memory-leaks.md)
-- [Performance issues](performance.md)
-
-### Deploying a Debuggable Configuration (Production on Linux)
-
-- If you have `--privileged`/`--security-opt seccomp=unconfined` containers or are running in VMs or even bare metal, you can run your binary with
-
-        lldb --batch -o "break set -n main --auto-continue 1 -C \"process handle SIGPIPE -s 0\"" -o run -k "image list" -k "register read" -k "bt all" -k "exit 134" ./my-program
-
-    instead of `./my-program` to get something something akin to a 'crash report' on crash.
-
-- If you don't have `--privileged` (or `--security-opt seccomp=unconfined`) containers (meaning you won't be able to use `lldb`) or you don't want to use lldb, consider using a library like [`swift-backtrace`](https://github.com/swift-server/swift-backtrace) to get stack traces on crash.
-
-## Deployment to Servers or Public Cloud
-
-Once an application is built for production, it needs to be packaged for deployment. There are several strategies for packaging Swift applications for deployment, see the [Packaging Guide](packaging.md) for more information.
-
-TODO: add guides for popular public clouds like AWS, GCP, Azure, Heroku, etc.
+_The guide is a community effort, and all are invited to share their tips and know-how. Please provide a PR if you have ideas for improving this guide!_

docs/building.md

@@ -0,0 +1,31 @@
+# Build system
+
+The recommended way of managing the builds for server applications is to use the [Swift Package Manager](https://swift.org/package-manager/), it provides a cross-platform foundation for building Swift code and works nicely for having one code base that can be edited as well as run on many Swift platforms
+
+## Building
+
+Running `swift build` from the terminal will trigger the build (or triggering the build action in Xcode).
+
+Swift is architecture specific, so running the build command on macOS will create a macOS binary. Building on macOS is useful for development and for taking advantage of the great tooling that comes with Xcode. However, most server applications are designed to run on Linux.
+
+To build on Linux and create a Linux binary, use Docker. For example:
+
+`$ docker run -v "$PWD:/code" -w /code swift:5.3 swift build`
+
+Note, if you want to run the Swift compiler for Intel CPUs on an Apple Silicon (M1) Mac, please add `--platform linux/amd64 -e QEMU_CPU=max` to the commandline. For example:
+
+`$ docker run -v "$PWD:/code" -w /code --platform linux/amd64 -e QEMU_CPU=max swift:5.3 swift build`
+
+The above commands will run the build using the latest Swift 5.3 Docker image, utilizing bind mounts to the sources on your Mac. Apple publishes Docker images to Docker Hub.
+
+By default, SwiftPM will build a debug version of the application. Note that debug versions are not suitable for running in production as they are significantly slower. To build a release version of your app, run `swift build -c release`.
+
+Binary artifacts that could be deployed are be found under .build/x86_64-unknown-linux, or .build/x86_64-apple-macosx for macOS binaries. SwiftPM can show you the full binary path using `swift build --show-bin-path -c release`.
+
+### Building for production
+
+- Build production code in release mode by compiling with `swift build -c release`. Running code compiled in debug mode will hurt performance signficantly. 
+
+- For best performance in Swift 5.2 or later, pass `-Xswiftc -cross-module-optimization` (this won't work in Swift versions before 5.2)
+
+- Integrate [`swift-backtrace`](https://github.com/swift-server/swift-backtrace) into your application to make sure backtraces are printed on crash. Backtraces do not work out-of-the-box on Linux, and this library helps to fill the gap. Eventually this will become a language feature and not require a discrete library.

docs/deployment.md

@@ -0,0 +1,18 @@
+
+## Deployment to Servers or Public Cloud
+
+Once an application is built for production, it needs to be packaged for deployment. There are several strategies for packaging Swift applications for deployment, see the [Packaging Guide](packaging.md) for more information.
+
+A separate guide for [Ubuntu on DigitalOcean](digital-ocean.md) is also available.
+
+### Deploying a Debuggable Configuration (Production on Linux)
+
+- If you have `--privileged`/`--security-opt seccomp=unconfined` containers or are running in VMs or even bare metal, you can run your binary with
+
+        lldb --batch -o "break set -n main --auto-continue 1 -C \"process handle SIGPIPE -s 0\"" -o run -k "image list" -k "register read" -k "bt all" -k "exit 134" ./my-program
+
+    instead of `./my-program` to get something something akin to a 'crash report' on crash.
+
+- If you don't have `--privileged` (or `--security-opt seccomp=unconfined`) containers (meaning you won't be able to use `lldb`) or you don't want to use lldb, consider using a library like [`swift-backtrace`](https://github.com/swift-server/swift-backtrace) to get stack traces on crash.
+
+_TODO: add guides for popular public clouds like AWS, GCP, Azure, Heroku, etc._

docs/digital-ocean.md

@@ -6,11 +6,11 @@ This guide will walk you through setting up an Ubuntu virtual machine on a Digit
 
 Use the create menu to create a new Droplet.
 
-![Create Droplet](images/digital-ocean-create-droplet.png)
+![Create Droplet](../images/digital-ocean-create-droplet.png)
 
 Under distributions, select Ubuntu 18.04 LTS.
 
-![Ubuntu Distro](images/digital-ocean-distributions-ubuntu-18.png)
+![Ubuntu Distro](../images/digital-ocean-distributions-ubuntu-18.png)
 
 > Note: You may select any version of Linux that Swift supports. You can check which operating systems are officially supported on the [Swift Releases](https://swift.org/download/#releases) page.
 
@@ -18,7 +18,7 @@ After selecting the distribution, choose any plan and datacenter region you pref
 
 Once the new server is ready, hover over the Droplet's IP address and click copy.
 
-![Droplet List](images/digital-ocean-droplet-list.png)
+![Droplet List](../images/digital-ocean-droplet-list.png)
 
 ## Initial Setup
 

docs/heroku.md

@@ -0,0 +1,114 @@
+# What is Heroku
+
+Heroku is a popular all in one hosting solution, you can find more at heroku.com
+
+## Signing Up
+
+You'll need a heroku account, if you don't have one, please sign up here: [https://signup.heroku.com/](https://signup.heroku.com/)
+
+## Installing CLI
+
+Make sure that you've installed the heroku cli tool.
+
+### HomeBrew
+
+```bash
+brew install heroku/brew/heroku
+```
+
+### Other Install Options
+
+See alternative install options here: [https://devcenter.heroku.com/articles/heroku-cli#download-and-install](https://devcenter.heroku.com/articles/heroku-cli#download-and-install).
+
+### Logging in
+
+Once you've installed the CLI, login with the following:
+
+```bash
+heroku login
+```
+
+### Create an application
+
+Visit dashboard.heroku.com to access your account, and create a new application from the drop down in the upper right hand corner. Heroku will ask a few questions such as region and application name, just follow their prompts.
+
+### Project
+
+Today we're going to be hosting Swift NIO's example http server, you can apply these concepts to your own project. Let's start by cloning NIO
+
+```bash
+
+git clone https://github.com/apple/swift-nio
+```
+
+Make sure to make our newly cloned directory the working directory
+
+
+```bash
+cd swift-nio
+```
+
+By default, Heroku deploys the **master** branch. Always make sure all changes are checked into this branch before pushing.
+
+#### Connect with Heroku
+
+Connect your app with Heroku (replace with your app's name).
+
+```bash
+$ heroku git:remote -a your-apps-name-here
+```
+
+### Set Stack
+
+As of 13 September 2018, Heroku’s default stack is Heroku 18, we need it to be 16 for swift projects.
+
+```bash
+heroku stack:set heroku-16 -a your-apps-name-here
+```
+
+### Set Buildpack
+
+Set the buildpack to teach Heroku how to deal with swift, the vapor-communnity buildpack is a good buildpack for *any swift project*. It doesn't install vapor, and it doesn't have any vapor specific setup.
+
+
+```bash
+heroku buildpacks:set vapor/vapor
+```
+
+### Swift version file
+
+The buildpack we added looks for a **.swift-version** file to know which version of swift to use.
+
+```bash
+echo "5.2" > .swift-version
+```
+
+This creates **.swift-version** with `5.2` as its contents.
+
+
+### Procfile
+
+Heroku uses the **Procfile** to know how to run your app. This includes the executable name and any arguments necessary. You'll see `$PORT` below, this allows heroku to assign a specific port when it launches the app. 
+
+```
+web: NIOHTTP1Server 0.0.0.0 $PORT
+```
+
+You can use this command in terminal to set the file
+
+
+```bash
+echo "web: NIOHTTP1Server 0.0.0.0 $PORT" > Procfile
+```
+
+### Commit changes
+
+We have now added the `.swift-version` file, and the `Procfile`, make sure these are committed into master or Heroku will not find them.
+
+### Deploying to Heroku
+
+You're ready to deploy, run this from the terminal. It may take a while to build, this is normal.
+
+```none
+git push heroku master
+```

docs/llvm-sanitizers.md

@@ -0,0 +1,32 @@
+# LLVM TSAN / ASAN
+
+For multithreaded and low-level unsafe interfacing server code, the ability to use LLVM's [ThreadSanitizer](https://clang.llvm.org/docs/ThreadSanitizer.html) and  
+[AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html) can help troubleshoot invalid thread usage and invalid usage/access of memory.
+
+There is a [blog post](https://swift.org/blog/tsan-support-on-linux/) outlining the usage of TSAN.
+
+The short story is to use the swiftc command line options `-sanitize=address` and `-santize=thread` to each respective tool.
+
+Also for Swift Package Manager projects you can use `--sanitize` at the command line, e.g.:
+
+```
+swift build --sanitize=address
+```
+
+or
+
+```
+swift build --sanitize=thread
+```
+
+and it can be used for the tests too:
+
+```
+swift test --sanitize=address
+```
+
+or
+
+```
+swift test --sanitize=thread
+```

docs/log-levels.md

@@ -0,0 +1,209 @@
+# Log Levels
+
+This guide serves as guidelines for library authors with regards to what [SwiftLog](https://github.com/apple/swift-log) log levels are appropriate for use in libraries, and in what situations to use what level.
+
+## Guidelines for Libraries
+
+SwiftLog defines the following 7 log levels via the [`Logger.Level` enum](https://apple.github.io/swift-log/docs/current/Logging/Structs/Logger/Level.html), ordered from least to most severe:
+
+* `trace`
+* `debug`
+* `info`
+* `notice`
+* `warning`
+* `error`
+* `critical`
+
+Out of those, only levels _less severe than_ info (exclusive) are generally okay to be used by libraries.
+In the following section we'll explore how to use them in practice.
+
+### Recommended log levels
+
+It is always fine for a library to log at `trace` and `debug` levels, and these two should be the primary levels any library is logging at.
+
+`trace` is the finest log level, and end-users of a library will not usually use it unless debugging very specific issues. You should consider it as a way for library developers to "log everything we could possibly need to diagnose a hard to reproduce bug." It is expected to take a toll on the performance of a system, and developers can assume trace level logging will not be used in production deployments, unless enabled specifically to locate some specific issue.
+
+This is in contrast with `debug` which some users _may_ choose to run enabled on their production systems.
+
+> Debug level logging should be not "too" noisy. Developers should assume some production deployments may need to (or want to) run with debug level logging enabled. 
+>
+> Debug level logging should not completely undermine the performance of a production system.
+
+As such, `debug` logging should not be overly noisy. It should provide a high value understanding of what is going on in the library for end users without diving deep into its internals. This is what `trace` is intended for.
+
+Use `warning` level sparingly. Whenever possible, try to rather return or throw `Error` to end users that are descriptive enough so they can inspect, log them and figure out the issue. Potentially they may then enable debug logging to find out more about the issue.
+
+It is okay to log a `warning` "once", for example on system startup. This may include some one off "more secure configuration is available, try upgrading to it!" log statement upon a server's startup. You may also log warnings from background processes, which otherwise have no other means of informing the end user about some issue.
+
+Logging on `error` level is similar to warnings: prefer to avoid doing so whenever possible. Instead, report errors via your library's API. For example, it is _not_ a good idea to log "connection failed" from an HTTP client. Perhaps the end-user intended to make this request to a known offline server to _confirm_ it is offline? From their perspective this connection error is not a "real" error, it is just what they expected -- as such the HTTP client should return or throw such an error, but _not_ log it.
+
+Do also note that in situations when you decide to log an error, be mindful of error rates. Will this error potentially be logged for every single operation while some network failure is happening? Some teams and companies have alerting systems set up based on the rate of errors logged in a system, and if it exceeds some threshold it may start calling and paging people in the middle of the night. When logging at error level, consider if the issue indeed is something that should be waking up people at night. You may also want to consider offering configuration in your library: "at what log level should this issue be reported?" This can come in handy in clustered systems which may log network failures themselves, or depend on external systems detecting and reporting this.
+
+Logging `critical` logs is allowed for libraries, however as the name implies - only in the most critical situations. Most often this implies that the library will *stop functioning* after such log has been issued. End users are thought to expect that a logged critical error is _very_ important, and they may have set up their systems to page people in the middle of the night to investigate the production system _right now_ when such log statements are detected. So please be careful about logging these kinds of errors.
+
+Some libraries and situations may not be entirely clear with regards to what log level is "best" for them. In such situations, it sometimes is worth it to allow the end-users of the library to be able to configure the levels of specific groups of messages. You can see this in action in the Soto library [here](https://github.com/soto-project/soto-core/pull/423/files#diff-4a8ca7e54da5b22287900dd8cf6b47ded38a94194c1f0b544119030c81a2f238R649) where an `Options` object allows end users to configure the level at which requests are logged (`options.requestLogLevel`) which is then used as `log.log(self.options.requestLogLevel)`.
+
+#### Examples
+
+`trace` level logging:
+
+- Could include various additional information about a request, such as various diagnostics about created data structures, the state of caches or similar, which are created in order to serve a request.
+- Could include "begin operation" and "end operation" logs.
+  - However, please consider using [swift-distributed-tracing](https://github.com/apple/swift-distributed-tracing) to instrument "begin" and "end" events, as tracing can be more efficient than logging. Logging may need to create string representations of the durations, log levels, and other fields.
+
+`debug` level logging:
+
+- May include a single log statement for opening a connection, accepting a request, and so on.
+- It can include a _high level_ overview of what is going on in the library. For example: "started work, processing step X, finished work X, result code 200".
+
+### Log levels to avoid
+
+All these rules are only _general_ guidelines, and as such may have exceptions. Consider the following examples and rationale for why logging at high log levels by a library may not be desirable:
+
+It is generally _not acceptable_ for a service client (for example, an http client) to log an `error` when a request has failed. End-users may be using the client to probe if an endpoint is even responsive or not, and a failure to respond may be _expected_ behavior. Logging errors would only confuse and pollute their logs.
+
+Instead, libraries should either `throw`, or return an `Error` value that users of the library will have enough knowledge about if they should log or ignore it.
+
+It is even less acceptable for a library to log any successful operations. This leads to flooding server side systems, especially if, for example, one were to log every successfully handled request. In a server side application, this can easily flood and overwhelm logging systems when deployed to production where many end users are connected to the same server. Such issues are rarely found in development time, because of only a single peer requesting things from the service-under-test.
+
+#### Examples (of things to avoid)
+
+Avoid using `info` or any higher log level for:
+
+- "Normal operation" of the library, that is there is no need to log on info level "accepted a request" as this is the normal operation of a web service.
+
+Avoid using `error` or `warning`:
+
+- To report errors which the end-user of the library has the means of logging themselves. For example, if a database driver fails to fetch all rows of a query, it should not log an error or warning, but instead return or throw an error on the stream of values (or function, async function, or even the async sequence) that was providing the returned values.
+  - Since the end-user is consuming these values, and has a mean of reporting (or swallowing) this error, the library should not log anything on their behalf.
+- Never report as warnings which is merely an information. For example. "weird header detected" may look like a good idea to log as a warning at first sight, however if the "weird header" is simply a misconfigured client (or just a "weird browser") you may be accidentally completely flooding an end-users logs with these "weird header" warnings (!)
+  - Only log warnings about actionable things which the end-user of your library can do something about. Using the "weird header detected" log statement as an example: it would not be a good candidate to log as a warning because the server developer has no way to fix the users of their service to stop sending weird headers, so the server should not be logging this information as a warning.
+- It may be tempting to implement a "log as warning only once" technique for per-request style situations which may be almost important enough to be a warning, but should not be logged repeatedly after all. Authors may think of smart techniques to log a warning only once per "weird header discovered" and later on log the same issue on a different level, such as trace... Such techniques result in confusing hard to debug logs, where developers of a system unaware of the stateful nature of the logging would be left confused when trying to reproduce the issue.
+  - For example, if a developer spots such a warning in a production system, they may attempt to reproduce it — thinking that it only happens in the production environment. However, if the logging system's log level choice is _stateful_ they may actually be successfully reproducing the issue but never seeing it manifest. For this, and related performance reasons (as implementing "only once per X" implies growing storage and per-request additional checking requirements), it is not recommended to apply this pattern.
+
+Exceptions to the "avoid logging warnings" rule:
+
+- "Background processes" such as tasks scheduled on a periodic timer, may not have any other means of communicating a failure or warning to the end user of the library other than through logging.
+  - Consider offering an API that would collect errors at runtime, and then you can avoid logging errors manually. This can often take the form of a customizable "on error" hook that the library accepts when constructing the scheduled job. If the handler is not customized, we can log the errors, but if it was, it again is up to the end-user of the library to decide what to do with them.
+- An exception to the "log a warning only once" rule is when things do not happen very frequently. For example, if a library is warning about an outdated license or something similar during _its initialization_ this isn't necessarily a bad idea. After all, we'd rather see this warning once during initialization rather during every request made to the library. Use your best judgement and consider the developers using your library when designing how often and where from to log such information.
+
+### Suggested logging style
+
+While libraries are free to use whichever logging message style they choose, here are some best practices to follow if you want users of your libraries to *love* the logs your library produces.
+
+Firstly, it is important to remember that both the message of a log statement as well as the metadata in [swift-log](https://github.com/apple/swift-log) are [autoclosures](https://docs.swift.org/swift-book/LanguageGuide/Closures.html#ID543), which are only invoked if the logger has a log level set such that it must emit a message for the message given. As such, messages logged at `trace` do not "materialize" their string and metadata representation unless they are actually needed:
+
+```swift
+    public func debug(_ message: @autoclosure () -> Logger.Message,
+                      metadata: @autoclosure () -> Logger.Metadata? = nil,
+                      source: @autoclosure () -> String? = nil,
+                      file: String = #file, function: String = #function, line: UInt = #line) {
+```
+
+And a minor yet important hint: avoid inserting newlines and other control characters into log statements (!). Many log aggregation systems assume that a single line in a logged output is specifically "one log statement" which can accidentally break if we log not sanitized, potentially multi-line, strings. This isn't a problem for _all_ log backends. For example, some will automatically sanitize and form a JSON payload with `{message: "..."}` before emitting it to a backend service collecting the logs, but plain old stream (or file) loggers usually assume that one line equals one log statement. It also makes grepping through logs more reliable.
+
+#### Structured Logging (Semantic Logging)
+
+Libraries may want to embrace the structured logging style.
+
+It is a fantastic pattern which makes consuming logs in automated systems, and through grepping and other means much easier and future proof.
+
+Consider the following not structured log statement:
+
+```swift
+// NOT structured logging style
+log.info("Accepted connection \(connection.id) from \(connection.peer), total: \(connections.count)")
+```
+
+It contains 4 pieces of information:
+
+- We accepted a connection.
+- This is its string representation.
+- It is from this peer.
+- We currently have `connections.count` active connections.
+
+While this log statement contains all useful information that we meant to relay to end users, it is hard to visually and mechanically parse the detailed information it contains. For example, if we know connections start failing around the time when we reach a total of 100 concurrent connections, it is not trivial to find the specific log statement at which we hit this number. We would have to `grep 'total: 100'` for example, however perhaps there are many other `"total: "` strings present in all of our log systems.
+
+Instead, we can express the same information using the structured logging pattern, as follows:
+
+```swift
+log.info("Accepted connection", metadata: [
+  "connection.id": "\(connection.id)",
+  "connection.peer": "\(connection.peer)",
+  "connections.total": "\(connections.count)"
+])
+
+// example output:
+// <date> info [connection.id:?,connection.peer:?, connections.total:?] Accepted connection
+```
+
+This structured log can be formatted, depending on the logging backend, slightly differently on various systems. Even in the simple string representation of such log, we'd be able to grep for `connections.total: 100` rather than having to guess the correct string.
+
+Also, since the message now does not contain all that much "human readable wording", it is less prone to randomly change from "Accepted" to "We have accepted" or vice versa. This kind of change could break alerting systems which are set up to parse and alert on specific log messages.
+
+Structured logs are very useful in combination with [swift-distributed-tracing](https://github.com/apple/swift-distributed-tracing)'s `LoggingContext`, which automatically populates the metadata with any present trace information. Thanks to this, all logs made in response to some specific request will automatically carry the same TraceID.
+
+You can see more examples of structured logging on the following pages, and example implementations thereof:
+
+- <https://tersesystems.com/blog/2020/05/26/why-i-wrote-a-logging-library/>
+- <https://cloud.google.com/logging/docs/structured-logging>
+- <https://stackify.com/what-is-structured-logging-and-why-developers-need-it/>
+- <https://kubernetes.io/blog/2020/09/04/kubernetes-1-19-introducing-structured-logs/>
+
+#### Logging with Correlation IDs / Trace IDs
+
+A very common pattern is to log messages with some "correlation id". The best approach in general here is to use a `LoggingContext` from [swift-distributed-tracing](https://github.com/apple/swift-distributed-tracing) as then your library will be able to be traced and used with correlation contexts regardless what tracing system the end-user is using (such as open telemetry, zipkin, xray, and other tracing systems) The concept though can be explained well with just a manually logged `requestID` which we'll explain below.
+
+Consider an HTTP client as an example of a library that has a lot of metadata about some request, perhaps something like this:
+
+```swift
+log.trace("Received response", metadata: [
+   "id": "...",
+   "peer.host": "...",
+   "payload.size": "...",
+   "headers": "...",
+   "responseCode": "...",
+   "responseCode.text": "...",
+])
+```
+
+The exact metadata does not matter, they're just some placeholder in this example. What matters is that there's "a lot of it".
+
+> Side note on metadata keys: while there is no single right way to structure metadata keys, we recommend thinking of them as-if JSON keys: camelCased and `.` separated identifiers. This allows many log analysis backends to treat them as such nested structure.
+
+Now, we would like to avoid logging _all_ this information in every single log statement. Instead, we are able to just repeatedly log the `"id"` metadata, like this:
+
+```swift
+// ... 
+log.trace("Something something...", metadata: ["id": "..."])
+log.trace("Finished streaming response", metadata: ["id": "..."]) // good, the same ID is propagated
+```
+
+Thanks to the correlation ID (or a tracing provided ID, in which case we'd log as `context.log.trace("...")` as the ID is propagated automatically), in each following log statement after the initial log statement we're able to correlate all those log statements. Then we know that this `"Finished streaming response"` message was about a response with a `responseCode` that we're able to look up from the `"Received response"` log message.
+
+This pattern is somewhat advanced and may not always be the right approach, but consider it in high performance code where logging the same information repeatedly can be too costly.
+
+##### Things to avoid with Correlation ID logging
+
+When logging with correlation contexts make sure to never "drop the ID". It is easiest to get this right when using distributed tracing's `LoggingContext` since propagating it ensures the carrying of identifiers, however the same applies to any kind of correlation identifier.
+
+Specifically, avoid situations like these:
+
+```swift
+debug: connection established [connection-id: 7]
+debug: connection closed unexpectedly [error: foobar] // BAD, the connection-id was dropped
+```
+
+On the second line, we don't know which connection had the error since the `connection-id` was dropped. Make sure to audit your logging code to ensure all relevant log statements carry necessary correlation identifiers.
+
+### Exceptions to the rules
+
+These are only general guidelines, and there always will be exceptions to these rules and other situations where these suggestions will be broken, for good reason. Please use your best judgement, and always consider the end-user of a system, and how they'll be interacting with your library and decide case-by-case depending on the library and situation at hand how to handle each situation.
+
+Here are a few examples of situations when logging a message on a relatively high level might still be tolerable for a library.
+
+It's permissible for a library to log at `critical` level right before a _hard crash_, as a last resort of informing the log collection systems or end-user about additional information detailing the reason for the crash. This should be _in addition to_ the message from a `fatalError` and can lead to an improved diagnosis/debugging experience for end users.
+
+Sometimes libraries may be able to detect a harmful misconfiguration of the library. For example, selecting deprecated protocol versions. In such situations it may be useful to inform users in production by issuing a `warning`. However you should ensure that the warning is not logged repeatedly! For example, it is not acceptable for an HTTP client to log a warning on every single http request using some misconfiguration of the client. It _may_ be acceptable however for the client to log such warning, for example, _once_ at configuration time, if the library has a good way to do this.
+
+Some libraries may implement a "log this warning only once", "log this warning only at startup", "log this error only once an hour", or similar tricks to keep the noise level low but still informative enough to not be missed. This is, however, usually a pattern reserved for stateful long running libraries, rather than clients of databases and related persistent stores.

docs/memory-leaks-and-usage.md

@@ -0,0 +1,240 @@
+# Debugging Memory Leaks and Usage
+
+There are many different tools for troubleshooting memory leaks both on Linux and macOS, each with different strengths and ease-of-use. One excellent tool is the Xcode's [Memory Graph Debugger](https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/debugging_with_xcode/chapters/special_debugging_workflows.html#//apple_ref/doc/uid/TP40015022-CH9-DontLinkElementID_1).
+[Instruments](https://help.apple.com/instruments/mac/10.0/#/dev022f987b) and `leaks` can also be very useful. If you cannot run or reproduce the problem on macOS, there are a number of server-side alternatives below. 
+
+## Example program
+
+The following program doesn't do anything useful but leaks memory so will serve as the example:
+
+```swift
+public class MemoryLeaker {
+    var closure: () -> Void = { () }
+
+    public init() {}
+
+    public func doNothing() {}
+
+    public func doSomethingThatLeaks() {
+        self.closure = {
+            // This will leak as it'll create a permanent reference cycle:
+            //
+            //     self -> self.closure -> self
+            self.doNothing()
+        }
+    }
+}
+
+@inline(never) // just to be sure to get this in a stack trace
+func myFunctionDoingTheAllocation() {
+    let thing = MemoryLeaker()
+    thing.doSomethingThatLeaks()
+}
+
+myFunctionDoingTheAllocation()
+```
+
+## Debugging leaks with `valgrind`
+
+If you run your program using
+
+    valgrind --leak-check=full ./test
+
+then `valgrind` will output
+
+```
+==1== Memcheck, a memory error detector
+==1== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
+==1== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
+==1== Command: ./test
+==1==
+==1==
+==1== HEAP SUMMARY:
+==1==     in use at exit: 824 bytes in 4 blocks
+==1==   total heap usage: 5 allocs, 1 frees, 73,528 bytes allocated
+==1==
+==1== 32 bytes in 1 blocks are definitely lost in loss record 1 of 4
+==1==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
+==1==    by 0x52076B1: swift_slowAlloc (in /usr/lib/swift/linux/libswiftCore.so)
+==1==    by 0x5207721: swift_allocObject (in /usr/lib/swift/linux/libswiftCore.so)
+==1==    by 0x108E58: $s4test12MemoryLeakerCACycfC (in /tmp/test)
+==1==    by 0x10900E: $s4test28myFunctionDoingTheAllocationyyF (in /tmp/test)
+==1==    by 0x108CA3: main (in /tmp/test)
+==1==
+==1== LEAK SUMMARY:
+==1==    definitely lost: 32 bytes in 1 blocks
+==1==    indirectly lost: 0 bytes in 0 blocks
+==1==      possibly lost: 0 bytes in 0 blocks
+==1==    still reachable: 792 bytes in 3 blocks
+==1==         suppressed: 0 bytes in 0 blocks
+==1== Reachable blocks (those to which a pointer was found) are not shown.
+==1== To see them, rerun with: --leak-check=full --show-leak-kinds=all
+==1==
+==1== For counts of detected and suppressed errors, rerun with: -v
+==1== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
+```
+
+The important part is
+
+```
+==1== 32 bytes in 1 blocks are definitely lost in loss record 1 of 4
+==1==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
+==1==    by 0x52076B1: swift_slowAlloc (in /usr/lib/swift/linux/libswiftCore.so)
+==1==    by 0x5207721: swift_allocObject (in /usr/lib/swift/linux/libswiftCore.so)
+==1==    by 0x108E58: $s4test12MemoryLeakerCACycfC (in /tmp/test)
+==1==    by 0x10900E: $s4test28myFunctionDoingTheAllocationyyF (in /tmp/test)
+==1==    by 0x108CA3: main (in /tmp/test)
+```
+
+which can demangled by pasting it into `swift demangle`:
+
+```
+==1== 32 bytes in 1 blocks are definitely lost in loss record 1 of 4
+==1==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
+==1==    by 0x52076B1: swift_slowAlloc (in /usr/lib/swift/linux/libswiftCore.so)
+==1==    by 0x5207721: swift_allocObject (in /usr/lib/swift/linux/libswiftCore.so)
+==1==    by 0x108E58: test.MemoryLeaker.__allocating_init() -> test.MemoryLeaker (in /tmp/test)
+==1==    by 0x10900E: test.myFunctionDoingTheAllocation() -> () (in /tmp/test)
+==1==    by 0x108CA3: main (in /tmp/test)
+```
+
+So valgrind is telling us that the allocation that eventually leaked is coming from `test.myFunctionDoingTheAllocation` calling `test.MemoryLeaker.__allocating_init()` which is correct.
+
+### Limitations
+
+- `valgrind` doesn't understand the bit packing that is used in many Swift data types (like `String`) or when you create `enum`s with associated values. Therefore `valgrind` sometimes claims a certain allocation was leaked even though it might not have
+- `valgrind` will make your program run _very slow_ (possibly 100x slower) which might stop you from even getting far enough to reproduce the issue.
+
+## Debugging leaks with `Leak Sanitizer`
+
+If you build your application using
+
+    swift build --sanitize=address
+
+it will be built with [Address Sanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer) enabled. Address Sanitizer also automatically tries to find leaked memory blocks, just like `valgrind`.
+
+The output from the above example program would be
+
+```
+=================================================================
+==478==ERROR: LeakSanitizer: detected memory leaks
+
+Direct leak of 32 byte(s) in 1 object(s) allocated from:
+    #0 0x55f72c21ac8d  (/tmp/test+0x95c8d)
+    #1 0x7f7e44e686b1  (/usr/lib/swift/linux/libswiftCore.so+0x3cb6b1)
+    #2 0x55f72c24b2ce  (/tmp/test+0xc62ce)
+    #3 0x55f72c24a4c3  (/tmp/test+0xc54c3)
+    #4 0x7f7e43aecb96  (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
+
+SUMMARY: AddressSanitizer: 32 byte(s) leaked in 1 allocation(s).
+```
+
+which shows the same information as `valgrind`, unfortunately however not symbolicated due to [SR-12601](https://bugs.swift.org/browse/SR-12601).
+
+You can symbolicate it using `llvm-symbolizer` or `addr2line` if you have `binutils` installed like so:
+
+```
+# /tmp/test+0xc62ce
+addr2line -e /tmp/test -a 0xc62ce -ipf | swift demangle
+0x00000000000c62ce: test.myFunctionDoingTheAllocation() -> () at crtstuff.c:?
+```
+## Debugging transient memory usage with `heaptrack`
+[Heaptrack](https://github.com/KDE/heaptrack) is very useful for analyzing memory leaks/usage with less overhead than `valgrind` - but more importantly is also allows for analyzing transient memory usage which may significantly impact performance by putting to much pressure on the allocator. 
+
+In addition to command line acccess, there is a graphical front-end `heaptrack_gui`.
+
+A key feature is that it allows for diffing between two different runs of your application, making it fairly easy to troubleshoot differences in `malloc` behavior between e.g. feature branches and main.
+
+A short how-to run on Ubuntu 20.04 (using a different example than above, as we look at transient usage in this example), install `heaptrack` with:
+
+```
+sudo apt-get install heaptrack
+```
+
+Then run the binary with `heaptrack` two times — first we do it for `main` to get a baseline:
+```
+> heaptrack .build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
+heaptrack output will be written to "/tmp/.nio_alloc_counter_tests_GRusAy/heaptrack.test_1000_autoReadGetAndSet.84341.gz"
+starting application, this might take some time...
+...
+heaptrack stats:
+    allocations:              319347
+    leaked allocations:       107
+    temporary allocations:    68
+Heaptrack finished! Now run the following to investigate the data:
+
+  heaptrack --analyze "/tmp/.nio_alloc_counter_tests_GRusAy/heaptrack.test_1000_autoReadGetAndSet.84341.gz"
+```
+Then run it a second time for the feature branch:
+```
+> heaptrack .build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
+heaptrack output will be written to "/tmp/.nio_alloc_counter_tests_GRusAy/heaptrack.test_1000_autoReadGetAndSet.84372.gz"
+starting application, this might take some time...
+...
+heaptrack stats:
+    allocations:              673989
+    leaked allocations:       117
+    temporary allocations:    341011
+Heaptrack finished! Now run the following to investigate the data:
+
+  heaptrack --analyze "/tmp/.nio_alloc_counter_tests_GRusAy/heaptrack.test_1000_autoReadGetAndSet.84372.gz"
+ubuntu@ip-172-31-25-161 /t/.nio_alloc_counter_tests_GRusAy> 
+```
+Here we could see that we had 673989 allocations in the feature branch version and 319347 in `main`, so clearly a regression.
+
+Finally, we can analyze the output as a diff from these runs using `heaptrack_print` and pipe it through `swift demangle` for readability:
+
+```
+heaptrack_print -T -d heaptrack.test_1000_autoReadGetAndSet.84341.gz heaptrack.test_1000_autoReadGetAndSet.84372.gz | swift demangle
+```
+`-T` gives us the temporary allocations (as it in this case was not a leak, but a transient alloaction - if you have leaks remove `-T`).
+
+The output can be quite long, but in this case as we look for transient allocations, scroll down to:
+
+```
+MOST TEMPORARY ALLOCATIONS
+307740 temporary allocations of 290324 allocations in total (106.00%) from
+swift_slowAlloc
+  in /home/ubuntu/bin/usr/lib/swift/linux/libswiftCore.so
+43623 temporary allocations of 44553 allocations in total (97.91%) from:
+    swift_allocObject
+      in /home/ubuntu/bin/usr/lib/swift/linux/libswiftCore.so
+    NIO.ServerBootstrap.(bind0 in _C131C0126670CF68D8B594DDFAE0CE57)(makeServerChannel: (NIO.SelectableEventLoop, NIO.EventLoopGroup) throws -> NIO.ServerSocketChannel, _: (NIO.EventLoop, NIO.ServerSocketChannel) -> NIO.EventLoopFuture<()>) -> NIO.EventLoopFuture<NIO.Channel>
+      at /home/ubuntu/swiftnio/swift-nio/Sources/NIO/Bootstrap.swift:295
+      in /tmp/.nio_alloc_counter_tests_GRusAy/.build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
+    merged NIO.ServerBootstrap.bind(host: Swift.String, port: Swift.Int) -> NIO.EventLoopFuture<NIO.Channel>
+      in /tmp/.nio_alloc_counter_tests_GRusAy/.build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
+    NIO.ServerBootstrap.bind(host: Swift.String, port: Swift.Int) -> NIO.EventLoopFuture<NIO.Channel>
+      in /tmp/.nio_alloc_counter_tests_GRusAy/.build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
+    Test_test_1000_autoReadGetAndSet.run(identifier: Swift.String) -> ()
+      at /tmp/.nio_alloc_counter_tests_GRusAy/Sources/Test_test_1000_autoReadGetAndSet/file.swift:24
+      in /tmp/.nio_alloc_counter_tests_GRusAy/.build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
+    main
+      at Sources/bootstrap_test_1000_autoReadGetAndSet/main.c:18
+      in /tmp/.nio_alloc_counter_tests_GRusAy/.build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
+22208 temporary allocations of 22276 allocations in total (99.69%) from:
+    swift_allocObject
+      in /home/ubuntu/bin/usr/lib/swift/linux/libswiftCore.so
+    generic specialization <Swift.UnsafeBufferPointer<Swift.Int8>> of Swift._copyCollectionToContiguousArray<A where A: Swift.Collection>(A) -> Swift.ContiguousArray<A.Element>
+      in /home/ubuntu/bin/usr/lib/swift/linux/libswiftCore.so
+    Swift.String.utf8CString.getter : Swift.ContiguousArray<Swift.Int8>
+      in /home/ubuntu/bin/usr/lib/swift/linux/libswiftCore.so
+    NIO.URing.getEnvironmentVar(Swift.String) -> Swift.String?
+      at /home/ubuntu/swiftnio/swift-nio/Sources/NIO/LinuxURing.swift:291
+      in /tmp/.nio_alloc_counter_tests_GRusAy/.build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
+    NIO.URing._debugPrint(@autoclosure () -> Swift.String) -> ()
+      at /home/ubuntu/swiftnio/swift-nio/Sources/NIO/LinuxURing.swift:297
+...
+22196 temporary allocations of 22276 allocations in total (99.64%) from:
+```
+
+And here we could fairly quickly see that the transient extra allocations was due to extra debug printing and querying of environment variables:
+
+```
+NIO.URing.getEnvironmentVar(Swift.String) -> Swift.String?
+  at /home/ubuntu/swiftnio/swift-nio/Sources/NIO/LinuxURing.swift:291
+  in /tmp/.nio_alloc_counter_tests_GRusAy/.build/x86_64-unknown-linux-gnu/release/test_1000_autoReadGetAndSet
+NIO.URing._debugPrint(@autoclosure () -> Swift.String) -> ()
+```
+
+And this code will be removed before final integration of the feature branch, so the diff will go away.

docs/performance.md

@@ -107,11 +107,11 @@ Let's assume we have compiled the above code using `swift build -c release` into
 
 ```
 # Step 1: Record the stack frames with a 99 Hz sampling frequency
-sudo perf record -F 99 -g -- ./slow
+sudo perf record -F 99 --call-graph dwarf -- ./slow
 # Alternatively, to attach to an existing process use
-#     sudo perf record -F 99 -g -p PID_OF_SLOW
+#     sudo perf record -F 99 --call-graph dwarf -p PID_OF_SLOW
 # or if you don't know the pid, you can try (assuming your binary name is "slow")
-#     sudo perf record -F 99 -g -p $(pgrep slow)
+#     sudo perf record -F 99 --call-graph dwarf -p $(pgrep slow)
 
 # Step 2: Export the recording into `out.perf`
 sudo perf script > out.perf
@@ -125,6 +125,16 @@ sudo perf script > out.perf
 
 The resulting file will look something like:
 
-![](perf-issues-flamegraph.svg)
+![](../images/perf-issues-flamegraph.svg)
 
 And we can see that almost all of our runtime is spent in `isFavouriteNumber` which is invoked from `addFavouriteNumber`. That should be a very good hint to the programmer on where to look for improvements. Maybe after all, we should use `Set<Int>` to store the favourite numbers, that should get is an answer to if a number is a favourite number in constant time (_O(1)_).
+
+## Alternate `malloc` libraries
+For some workloads putting serious pressure on the memory allocation subsystem, it may be beneficial with a custom `malloc` library.
+It requires no changes to the code, but needs interposing with e.g. an environment variable before running your server.
+It is worth benchmarking with the default and with a custom memory allocator to see how much it helps for the specific workload.
+There are many `malloc` implementations out there, but a portable and well-performing one is [Microsofts mimalloc](https://github.com/microsoft/mimalloc). 
+
+Typically these are simply enabled by using LD_PRELOAD:
+
+`> LD_PRELOAD=/usr/bin/libmimalloc.so  myprogram`

docs/setup-and-ide-alternatives.md

@@ -0,0 +1,27 @@
+## Setting up  
+
+The [supported platforms](https://swift.org/platform-support/) for running Swift on the server and the [ready-built tools packages](https://swift.org/download/) are all hosted on swift.org together with installation instructions. There you also can find the [language reference documentation](https://swift.org/documentation/).
+
+## IDE and Swift-aware editor alternatives
+
+There are a few different options for doing the actual coding with completions etc depending on what platform you are using, a non-exhastive list:
+
+[Atom IDE support](https://atom.io/packages/ide-swift)
+
+[CLion](https://www.jetbrains.com/help/clion/swift.html)
+
+[Emacs plugin](https://github.com/swift-emacs/swift-mode)
+
+[VIM plugin](https://github.com/keith/swift.vim)
+
+[Visual Studio Code](https://code.visualstudio.com)
+
+[Visual Studio Code optional code completion plugin](https://marketplace.visualstudio.com/items?itemName=vknabel.vscode-swift-development-environment)
+
+[Xcode](https://developer.apple.com/xcode/ide/)
+
+## Support for other editors using LSP
+
+Additionally, the [SourceKit-LSP project](https://github.com/apple/sourcekit-lsp) has both an [extensive list of editors that supports it](https://github.com/apple/sourcekit-lsp/tree/main/Editors) and instructions on how to get completion in your favourite editor.
+
+_Do you know about another IDE or IDE plugin that is missing? Please submit a PR to add them here!_

docs/testing.md

@@ -0,0 +1,29 @@
+# Testing 
+
+Running `swift test` from the terminal will trigger the unit tests (or triggering the test action in Xcode).
+
+SwiftPM is integrated with [XCTest, Apple’s unit test framework](https://developer.apple.com/documentation/xctest). Test results will be displayed in Xcode or printed out to the terminal.
+
+Like building on Linux, testing on Linux requires the use of Docker. For example:
+
+`$ docker run -v "$PWD:/code" -w /code swift:latest swift test`
+
+The above command will run the tests using the latest Swift Docker image, utilizing bind mounts to the sources on your file system.
+
+Swift supports architecture-specific code. By default, Foundation imports architecture-specific libraries like Darwin or Glibc. While developing on macOS, you may end up using APIs that are not available on Linux. Since you are most likely to deploy a cloud service on Linux, it is critical to test on Linux.
+
+A historically important detail about testing for Linux is the `Tests/LinuxMain.swift` file. 
+
+- In Swift versions 5.4 and newer tests are automaticlaly discovered on all platforms, no special file or flag needed.
+- In Swift versions >= 5.1 < 5.4, tests can be automaticlaly discovered on Linux using `swift test --enable-test-discovery` flag.
+- In Swift versions older than 5.1 the `Tests/LinuxMain.swift` file provides SwiftPM an index of all the tests it needs to run on Linux and it is critical to keep this file up-to-date as you add more unit tests. To regenerate this file, run `swift test --generate-linuxmain` after adding tests. It is also a good idea to include this command as part of your continuous integration setup.
+
+### Testing for production
+
+- For Swift versions between Swift 5.1 and 5.4, always test with `--enable-test-discovery` to avoid forgetting tests on Linux.
+
+- Make use of the sanitizers. Before running code in production, do the following:
+    * Run your test suite with TSan (thread sanitizer): `swift test --sanitize=thread`
+    * Run your test suite with ASan (address sanitizer): `swift test --sanitize=address` and `swift test --sanitize=address -c release -Xswiftc -enable-testing`
+
+- Generally, whilst testing, you may want to build using `swift build --sanitize=thread`. The binary will run slower and is not suitable for production, but you might be able to catch threading issues early - before you deploy your software. Often threading issues are really hard to debug and reproduce and also cause random problems. TSan helps catch them early.

docs/ubuntu.md

@@ -36,7 +36,7 @@ docker run --rm \
      cp -P /usr/lib/swift/linux/lib*so* .build/install/'
 ```
 
-> Tip: If you are building this project for production, use `swift build -c release`, see [building for production](README.md#building-for-production) for more information.
+> Tip: If you are building this project for production, use `swift build -c release`, see [building for production](building.md#building-for-production) for more information.
 
 Notice that Swift's shared libraries are being included. This is important since Swift is not ABI stable on Linux. This means Swift programs must run against the shared libraries they were compiled with. 
 
@@ -111,7 +111,7 @@ sudo apt install clang libicu-dev build-essential pkg-config
 
 This guide will install Swift 5.2. Visit the [Swift Downloads](https://swift.org/download/#releases) page for a link to latest release. Copy the download link for Ubuntu 18.04.
 
-![Download Swift](images/swift-download-ubuntu-18-copy-link.png)
+![Download Swift](../images/swift-download-ubuntu-18-copy-link.png)
 
 Download and decompress the Swift toolchain.
 
@@ -170,7 +170,7 @@ cd swift-nio
 swift build
 ```
 
-> Tip: If you are building this project for production, use `swift build -c release`, see [building for production](README.md#building-for-production) for more information.
+> Tip: If you are building this project for production, use `swift build -c release`, see [building for production](building.md#building-for-production) for more information.
 
 ### Run
 

memory-leaks.md

@@ -1,140 +0,0 @@
-# Debugging Memory Leaks
-
-Debugging memory leaks isn't straightforward on Linux, it's much easier done on macOS. The best tool is usually the Xcode's [Memory Graph Debugger](https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/debugging_with_xcode/chapters/special_debugging_workflows.html#//apple_ref/doc/uid/TP40015022-CH9-DontLinkElementID_1).
-[Instruments](https://help.apple.com/instruments/mac/10.0/#/dev022f987b) and `leaks` can also be very useful. If you cannot reproduce the problem on macOS, not all is lost, find a few suggestions below. The fine folks at IBM have also put together a [guide on diagnosing memory leaks](https://developer.ibm.com/swift/2018/01/26/diagnosing-server-side-swift-memory-leaks/).
-
-## Example program
-
-The following program doesn't do anything useful but leaks memory so will serve as the example:
-
-```swift
-public class MemoryLeaker {
-    var closure: () -> Void = { () }
-
-    public init() {}
-
-    public func doNothing() {}
-
-    public func doSomethingThatLeaks() {
-        self.closure = {
-            // This will leak as it'll create a permanent reference cycle:
-            //
-            //     self -> self.closure -> self
-            self.doNothing()
-        }
-    }
-}
-
-@inline(never) // just to be sure to get this in a stack trace
-func myFunctionDoingTheAllocation() {
-    let thing = MemoryLeaker()
-    thing.doSomethingThatLeaks()
-}
-
-myFunctionDoingTheAllocation()
-```
-
-## Debugging leaks with `valgrind`
-
-If you run your program using
-
-    valgrind --leak-check=full ./test
-
-then `valgrind` will output
-
-```
-==1== Memcheck, a memory error detector
-==1== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
-==1== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
-==1== Command: ./test
-==1==
-==1==
-==1== HEAP SUMMARY:
-==1==     in use at exit: 824 bytes in 4 blocks
-==1==   total heap usage: 5 allocs, 1 frees, 73,528 bytes allocated
-==1==
-==1== 32 bytes in 1 blocks are definitely lost in loss record 1 of 4
-==1==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
-==1==    by 0x52076B1: swift_slowAlloc (in /usr/lib/swift/linux/libswiftCore.so)
-==1==    by 0x5207721: swift_allocObject (in /usr/lib/swift/linux/libswiftCore.so)
-==1==    by 0x108E58: $s4test12MemoryLeakerCACycfC (in /tmp/test)
-==1==    by 0x10900E: $s4test28myFunctionDoingTheAllocationyyF (in /tmp/test)
-==1==    by 0x108CA3: main (in /tmp/test)
-==1==
-==1== LEAK SUMMARY:
-==1==    definitely lost: 32 bytes in 1 blocks
-==1==    indirectly lost: 0 bytes in 0 blocks
-==1==      possibly lost: 0 bytes in 0 blocks
-==1==    still reachable: 792 bytes in 3 blocks
-==1==         suppressed: 0 bytes in 0 blocks
-==1== Reachable blocks (those to which a pointer was found) are not shown.
-==1== To see them, rerun with: --leak-check=full --show-leak-kinds=all
-==1==
-==1== For counts of detected and suppressed errors, rerun with: -v
-==1== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
-```
-
-The important part is
-
-```
-==1== 32 bytes in 1 blocks are definitely lost in loss record 1 of 4
-==1==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
-==1==    by 0x52076B1: swift_slowAlloc (in /usr/lib/swift/linux/libswiftCore.so)
-==1==    by 0x5207721: swift_allocObject (in /usr/lib/swift/linux/libswiftCore.so)
-==1==    by 0x108E58: $s4test12MemoryLeakerCACycfC (in /tmp/test)
-==1==    by 0x10900E: $s4test28myFunctionDoingTheAllocationyyF (in /tmp/test)
-==1==    by 0x108CA3: main (in /tmp/test)
-```
-
-which can demangled by pasting it into `swift demangle`:
-
-```
-==1== 32 bytes in 1 blocks are definitely lost in loss record 1 of 4
-==1==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
-==1==    by 0x52076B1: swift_slowAlloc (in /usr/lib/swift/linux/libswiftCore.so)
-==1==    by 0x5207721: swift_allocObject (in /usr/lib/swift/linux/libswiftCore.so)
-==1==    by 0x108E58: test.MemoryLeaker.__allocating_init() -> test.MemoryLeaker (in /tmp/test)
-==1==    by 0x10900E: test.myFunctionDoingTheAllocation() -> () (in /tmp/test)
-==1==    by 0x108CA3: main (in /tmp/test)
-```
-
-So valgrind is telling us that the allocation that eventually leaked is coming from `test.myFunctionDoingTheAllocation` calling `test.MemoryLeaker.__allocating_init()` which is correct.
-
-### Limitations
-
-- `valgrind` doesn't understand the bit packing that is used in many Swift data types (like `String`) or when you create `enum`s with associated values. Therefore `valgrind` sometimes claims a certain allocation was leaked even though it might not have
-- `valgrind` will make your program run _very slow_ (possibly 100x slower) which might stop you from even getting far enough to reproduce the issue.
-
-## Debugging leaks with Leak Sanitizer
-
-If you build your application using
-
-    swift build --sanitize=address
-
-it will be built with [Address Sanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizerLeakSanitizer) enabled. Address Sanitizer also automatically tries to find leaked memory blocks, just like `valgrind`.
-
-The output from the above example program would be
-
-```
-=================================================================
-==478==ERROR: LeakSanitizer: detected memory leaks
-
-Direct leak of 32 byte(s) in 1 object(s) allocated from:
-    #0 0x55f72c21ac8d  (/tmp/test+0x95c8d)
-    #1 0x7f7e44e686b1  (/usr/lib/swift/linux/libswiftCore.so+0x3cb6b1)
-    #2 0x55f72c24b2ce  (/tmp/test+0xc62ce)
-    #3 0x55f72c24a4c3  (/tmp/test+0xc54c3)
-    #4 0x7f7e43aecb96  (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
-
-SUMMARY: AddressSanitizer: 32 byte(s) leaked in 1 allocation(s).
-```
-
-which shows the same information as `valgrind`, unfortunately however not symbolicated due to [SR-12601](https://bugs.swift.org/browse/SR-12601).
-
-You can symbolicate it using `llvm-symbolizer` or `addr2line` if you have `binutils` installed like so:
-
-```
-# /tmp/test+0xc62ce
-addr2line -e /tmp/test -a 0xc62ce -ipf | swift demangle
-0x00000000000c62ce: test.myFunctionDoingTheAllocation() -> () at crtstuff.c:?
-```