Add user-facing API for Streaming Lambda functions that receives JSON
events
### Motivation:
Streaming Lambda functions developed by developers had no choice but to
implement a handler that receives incoming data as a `ByteBuffer`. While
this is useful for low-level development, I assume most developers will
want to receive a JSON event to trigger their streaming Lambda function.
Going efficiently from a `ByteBuffer` to a Swift struct requires some
code implemented in the `JSON+ByteBuffer.swift` file of the librray. We
propose to further help developers by providing them with a new
`handler()` function that directly receives their `Decodable` type.
### Modifications:
This PR adds a public facing API (+ unit test + updated README) allowing
developers to write a handler method accepting any `Decodable` struct as
input.
```swift
import AWSLambdaRuntime
import NIOCore
// Define your input event structure
struct StreamingRequest: Decodable {
let count: Int
let message: String
let delayMs: Int?
}
// Use the new streaming handler with JSON decoding
let runtime = LambdaRuntime { (event: StreamingRequest, responseWriter, context: LambdaContext) in
context.logger.info("Received request to send \(event.count) messages")
// Stream the messages
for i in 1...event.count {
let response = "Message \(i)/\(event.count): \(event.message)\n"
try await responseWriter.write(ByteBuffer(string: response))
// Optional delay between messages
if let delay = event.delayMs, delay > 0 {
try await Task.sleep(for: .milliseconds(delay))
}
}
// Finish the stream
try await responseWriter.finish()
// Optional: Execute background work after response is sent
context.logger.info("Background work: processing completed")
}
try await runtime.run()
```
This interface provides:
- **Type-safe JSON input**: Automatic decoding of JSON events into Swift
structs
- **Streaming responses**: Full control over when and how to stream data
back to clients
- **Background work support**: Ability to execute code after the
response stream is finished
- **Familiar API**: Uses the same closure-based pattern as regular
Lambda handlers
Because streaming Lambda functions can be invoked either directly
through the API or through Lambda Function URL, this PR adds the
decoding logic to support both types, shielding developers from working
with Function URL requests and base64 encoding.
We understand these choice will have an impact on the raw performance
for event handling. Those advanced users that want to get the maximum
might use the existing `handler(_ event: ByteBuffer, writer:
LambaStreamingWriter)` function to implement their own custom decoding
logic.
This PR provides a balance between ease of use for 80% of the users vs
ultimate performance, without closing the door for the 20% who need it.
### Result:
Lambda function developers can now use arbitrary `Decodable` Swift
struct or Lambda events to trigger their streaming functions. 🎉
---------
Co-authored-by: Tim Condon <0xTim@users.noreply.github.com>
Streaming Codable Lambda function
This example demonstrates how to use the StreamingLambdaHandlerWithEvent protocol to create Lambda functions that:
- Receive JSON input: Automatically decode JSON events into Swift structs
- Stream responses: Send data incrementally as it becomes available
- Execute background work: Perform additional processing after the response is sent
The example uses the streaming codable interface that combines the benefits of:
- Type-safe JSON input decoding (like regular
LambdaHandler) - Response streaming capabilities (like
StreamingLambdaHandler) - Background work execution after response completion
Streaming responses incurs a cost. For more information, see AWS Lambda Pricing.
You can stream responses through Lambda function URLs, the AWS SDK, or using the Lambda InvokeWithResponseStream API.
Code
The sample code creates a StreamingFromEventHandler struct that conforms to the StreamingLambdaHandlerWithEvent protocol provided by the Swift AWS Lambda Runtime.
The handle(...) method of this protocol receives incoming events as a decoded Swift struct (StreamingRequest) and returns the output through a LambdaResponseStreamWriter.
The Lambda function expects a JSON payload with the following structure:
{
"count": 5,
"message": "Hello from streaming Lambda!",
"delayMs": 1000
}
Where:
count: Number of messages to stream (1-100)message: The message content to repeatdelayMs: Optional delay between messages in milliseconds (defaults to 500ms)
The response is streamed through the LambdaResponseStreamWriter, which is passed as an argument in the handle function. The code calls the write(_:) function of the LambdaResponseStreamWriter with partial data written repeatedly before finally closing the response stream by calling finish(). Developers can also choose to return the entire output and not stream the response by calling writeAndFinish(_:).
An error is thrown if finish() is called multiple times or if it is called after having called writeAndFinish(_:).
The handle(...) method is marked as mutating to allow handlers to be implemented with a struct.
Once the struct is created and the handle(...) method is defined, the sample code creates a LambdaRuntime struct and initializes it with the handler just created. Then, the code calls run() to start the interaction with the AWS Lambda control plane.
Key features demonstrated:
- JSON Input Decoding: The function automatically parses the JSON input into a
StreamingRequeststruct - Input Validation: Validates the count parameter and returns an error message if invalid
- Progressive Streaming: Sends messages one by one with configurable delays
- Timestamped Output: Each message includes an ISO8601 timestamp
- Background Processing: Performs cleanup and logging after the response is complete
- Error Handling: Gracefully handles invalid input with descriptive error messages
Build & Package
To build & archive the package, type the following commands.
swift package archive --allow-network-connections docker
If there is no error, there is a ZIP file ready to deploy.
The ZIP file is located at .build/plugins/AWSLambdaPackager/outputs/AWSLambdaPackager/StreamingFromEvent/StreamingFromEvent.zip
Test locally
You can test the function locally before deploying:
swift run
# In another terminal, test with curl:
curl -v \
--header "Content-Type: application/json" \
--data '{"count": 3, "message": "Hello World!", "delayMs": 1000}' \
http://127.0.0.1:7000/invoke
Or simulate a call from a Lambda Function URL (where the body is encapsulated in a Lambda Function URL request):
curl -v \
--header "Content-Type: application/json" \
--data @events/sample-request.json \
http://127.0.0.1:7000/invoke
Deploy with the AWS CLI
Here is how to deploy using the aws command line.
Step 1: Create the function
# Replace with your AWS Account ID
AWS_ACCOUNT_ID=012345678901
aws lambda create-function \
--function-name StreamingFromEvent \
--zip-file fileb://.build/plugins/AWSLambdaPackager/outputs/AWSLambdaPackager/StreamingFromEvent/StreamingFromEvent.zip \
--runtime provided.al2 \
--handler provided \
--architectures arm64 \
--role arn:aws:iam::${AWS_ACCOUNT_ID}:role/lambda_basic_execution
Important
The timeout value must be bigger than the time it takes for your function to stream its output. Otherwise, the Lambda control plane will terminate the execution environment before your code has a chance to finish writing the stream. Here, the sample function stream responses during 10 seconds and we set the timeout for 15 seconds.
The --architectures flag is only required when you build the binary on an Apple Silicon machine (Apple M1 or more recent). It defaults to x64.
Be sure to set AWS_ACCOUNT_ID with your actual AWS account ID (for example: 012345678901).
Step 2: Give permission to invoke that function through a URL
Anyone with a valid signature from your AWS account will have permission to invoke the function through its URL.
aws lambda add-permission \
--function-name StreamingFromEvent \
--action lambda:InvokeFunctionUrl \
--principal ${AWS_ACCOUNT_ID} \
--function-url-auth-type AWS_IAM \
--statement-id allowURL
Step 3: Create the URL
This creates a URL with IAM authentication. Only calls with a valid signature will be authorized.
aws lambda create-function-url-config \
--function-name StreamingFromEvent \
--auth-type AWS_IAM \
--invoke-mode RESPONSE_STREAM
This call returns various information, including the URL to invoke your function.
{
"FunctionUrl": "https://ul3nf4dogmgyr7ffl5r5rs22640fwocc.lambda-url.us-east-1.on.aws/",
"FunctionArn": "arn:aws:lambda:us-east-1:012345678901:function:StreamingFromEvent",
"AuthType": "AWS_IAM",
"CreationTime": "2024-10-22T07:57:23.112599Z",
"InvokeMode": "RESPONSE_STREAM"
}
Invoke your Lambda function
To invoke the Lambda function, use curl with the AWS Sigv4 option to generate the signature.
Read the AWS Credentials and Signature section for more details about the AWS Sigv4 protocol and how to obtain AWS credentials.
When you have the aws command line installed and configured, you will find the credentials in the ~/.aws/credentials file.
URL=https://ul3nf4dogmgyr7ffl5r5rs22640fwocc.lambda-url.us-east-1.on.aws/
REGION=us-east-1
ACCESS_KEY=AK...
SECRET_KEY=...
AWS_SESSION_TOKEN=...
curl --user "${ACCESS_KEY}":"${SECRET_KEY}" \
--aws-sigv4 "aws:amz:${REGION}:lambda" \
-H "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
--no-buffer \
--header "Content-Type: application/json" \
--data '{"count": 3, "message": "Hello World!", "delayMs": 1000}' \
"$URL"
This should output the following result, with configurable delays between each message:
[2024-07-15T05:00:00Z] Message 1/3: Hello World!
[2024-07-15T05:00:01Z] Message 2/3: Hello World!
[2024-07-15T05:00:02Z] Message 3/3: Hello World!
✅ Successfully sent 3 messages
Undeploy
When done testing, you can delete the Lambda function with this command.
aws lambda delete-function --function-name StreamingFromEvent
Deploy with AWS SAM
Alternatively, you can use AWS SAM to deploy the Lambda function.
Prerequisites : Install the SAM CLI
SAM Template
The template file is provided as part of the example in the template.yaml file. It defines a Lambda function based on the binary ZIP file. It creates the function url with IAM authentication and sets the function timeout to 15 seconds.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: SAM Template for StreamingFromEvent Example
Resources:
# Lambda function
StreamingNumbers:
Type: AWS::Serverless::Function
Properties:
CodeUri: .build/plugins/AWSLambdaPackager/outputs/AWSLambdaPackager/StreamingFromEvent/StreamingFromEvent.zip
Timeout: 15
Handler: swift.bootstrap # ignored by the Swift runtime
Runtime: provided.al2
MemorySize: 128
Architectures:
- arm64
FunctionUrlConfig:
AuthType: AWS_IAM
InvokeMode: RESPONSE_STREAM
Outputs:
# print Lambda function URL
LambdaURL:
Description: Lambda URL
Value: !GetAtt StreamingNumbersUrl.FunctionUrl
Deploy with SAM
sam deploy \
--resolve-s3 \
--template-file template.yaml \
--stack-name StreamingFromEvent \
--capabilities CAPABILITY_IAM
The URL of the function is provided as part of the output.
CloudFormation outputs from deployed stack
-----------------------------------------------------------------------------------------------------------------------------
Outputs
-----------------------------------------------------------------------------------------------------------------------------
Key LambdaURL
Description Lambda URL
Value https://gaudpin2zjqizfujfnqxstnv6u0czrfu.lambda-url.us-east-1.on.aws/
-----------------------------------------------------------------------------------------------------------------------------
Once the function is deployed, you can invoke it with curl, similarly to what you did when deploying with the AWS CLI.
curl -X POST \
--data '{"count": 3, "message": "Hello World!", "delayMs": 1000}' \
--user "$ACCESS_KEY":"$SECRET_KEY" \
--aws-sigv4 "aws:amz:${REGION}:lambda" \
-H "x-amz-security-token: $AWS_SESSION_TOKEN" \
--no-buffer \
"$URL"
Undeploy with SAM
When done testing, you can delete the infrastructure with this command.
sam delete