Comparing AWS stream-like tools

I love AWS, unironically. I have been using it since it came into existence, and my admiration for the platform has only grown with time. Even though I spend a lot of time wiring up different services in AWS, I still have trouble keeping all of the streaming facilities straight. Today I got stuck on a JetBlue flight with broken WiFi, so of course I took the opportunity to scrape through the AWS CLI documentation to create a comparison table and get this sorted once and for all.

Service Model Volume FIFO Put From Push To Fan-In Fan-Out Default Limit Rsvd Cap?
SNS pub-sub Low No λ, Cloudwatch λ, http, email, sms, sqs 100,000 No
SQS message queue Med No λ, sns λ 1 No
SQS FIFO FIFO msg queue Med Yes λ 1 No
Kinesis Stream data stream High Kinda λ, KAnalytics λ 50/region Yes
Kinesis Firehose managed data delivery High No λ, KAnalytics S3, ES, Splunk, Redshift 1 50/region No
Kinesis Analytics stream analytics Med Yes? KStream, KFirehose KStream, KFirehose, λ 1 3 50/region No*
Dynamo Stream change log Med Yes? Dynamo λ 1 No
S3 k/v store High KFirehose λ No
Logs Immutable log stream High No λ KStream, KFirehose, λ No

What’s clear from this table, and will be familiar to AWS veterans, is that Lambda really is the glue that holds all of these services together. In some cases, you’ll find yourself resorting to a Lambda when you need to connect two elements in your pipeline that don’t know how to talk to each other on their own.

A few notes:

  • Astute readers will see some category errors and unfair comparisons.
    • S3 is not a stream in any normal sense; I’ve included it because with event notifications it can often be an alternative to some of the other tools here, especially if retention is desirable.
    • Kinesis Analytics is kind of an odd duck as well, much more specialized and compute-centric than the others here. But because it can read from and write to the other Kinesis services, and also write directly to Lambda, it felt worth including here.
    • Dynamo Streams are very specific, and do only one thing. I’m pretty sure they are Kinesis Streams under the hood, though.
  • Fan-in and fan-out are approximate. Nothing is truly infinity, of course, and I may update this post with some of the practical limits.
  • Kinesis streams have per-shard FIFO if you’re careful
  • Volume entries are super vague, of course, and are relative to the highest-volume services. SNS can handle lots of notifications by most measures; it’s just slow compared to monsters like Kinesis and S3.

And here is a prettier version if you’d like:

AWS Stream Comparison