Amazon Timestream – Fast, scalable, fully managed time series database

At my day job, I build a lot of machine learning systems that require data to be fed in a time series manner[1].<p>Often this means building systems to analyze terabytes of logs [semi]-realtime. All I have to say is - thank god! This is going to make my job a lot easier, and likely empower us to remove our current infrastructure setup.<p>I know at one point we actually considered building our own time series database. Instead, we ended up utilizing a Kafka queue with an SQL based backend after we parsed and paired down the data, because it was the only one quick enough to do the queries.<p>Should make a lot of the modeling I&#x27;ve worked on a bit easier[1].<p>[1] <a href="https:&#x2F;&#x2F;medium.com&#x2F;capital-one-tech&#x2F;batch-and-streaming-in-the-world-of-data-science-and-data-engineering-2cc029cdf554" rel="nofollow">https:&#x2F;&#x2F;medium.com&#x2F;capital-one-tech&#x2F;batch-and-streaming-in-t...</a>

This is not cheap for the &quot;DevOps&quot; use case.<p>Imagine you have 1000 servers submitting data to 100 timeseries each minute. That&#x27;s 100,000 writes a minute (unless they support batch writes across series) At $0.50 per million writes that&#x27;s $72 a day or $26k a year.<p>Now imagine you want to alert on that data. Say you have 100 monitors that each evaluate 1GB of data once a minute. At $10 per TB of data scanned, that&#x27;s $1,440 a day or $525k a year!

I&#x27;m actually impressed at how incredibly expensive they made this. $0.50 per million 1KB writes, which is <i>20x</i> what aurora charges, since aurora allows 8KB writes. And Aurora is already expensive if you actually read&#x2F;write to it.

Quite excited for this! We have currently been experimenting with using DynamoDB, and managing our own rollups of our incoming data (previously on an RDS, which is not a good choice for this kind of data).<p>---<p>I&#x27;ve seen a lot of people complain about pricing, so I thought I&#x27;d share a little why we are excited about this:<p>We have approximately 280 devices out, monitoring production lines, sending aggregated data every 5 seconds, via MQTT to AWS IoT. The average messages published that we see is around ~2 million a day (equipment is often turned off, when not producing). The packet size is very small, and highly compressable, each below 1KB, but let&#x27;s just make it 1KB.<p>We then currently funnel this data into Lambda, which processes it, and puts it into DynamoDB and handles rollups. The costs of that whole thing is approximately $20 a day (IoT, DynamoDB, Lambda and X-Ray), with Lambda+DynamoDB making up $17 of that cost.<p>Finally, our users look at this data, live, on dashboards, usually looking at the last 8 hours of data for a specific device. Let&#x27;s throw around that there will be 10,000 queries each day, looking at the data of the day (2GB&#x2F;day &#x2F; 280devices = 0.007142857 GB&#x2F;device&#x2F;day).<p>---<p>Now, running the same numbers on the AWS Timestream pricing[0] (daily cost):<p>- Writes: 2million * $0.5&#x2F;million = $1<p>- Memory store: 2 GB * $0.036 = $0.072<p>- SSD store: (2GB * 7days) * $0.01 (GB&#x2F;day) * 7days = $0.98<p>- Magnetic store: (2 GB * 30 days) * $0.03 (GB&#x2F;month) = $1.8<p>- Query: 10,0000 queries * 0.007142857GB&#x2F;device&#x2F;day --&gt; 71GB = free until day 14, where it&#x27;ll cost $10, so $20 a month.<p>Giving us: $1 + $0.072 + $0.98 + $1.8 + ($20&#x2F;30) = $4.5&#x2F;day.<p>From these (very) quick calculations, this means we could lower our cost from ~$20&#x2F;day to ~$4.5&#x2F;day. And that&#x27;s not even taking into account that it removes our need to create&#x2F;maintain our own custom solution.<p>I am probably missing some details, but it does look bright!<p>[0] <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;timestream&#x2F;pricing&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;timestream&#x2F;pricing&#x2F;</a>

It&#x27;s got to be a rough day for the team at <a href="https:&#x2F;&#x2F;www.influxdata.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.influxdata.com&#x2F;</a> . This could become serious competition for their InfluxCloud hosted offering.

Nice to see, this has felt like a gap in cloud offerings for a while... and the open source options have difficulties.<p>From the little that was said, going to guess this uses something like Beringei (<a href="https:&#x2F;&#x2F;code.fb.com&#x2F;core-data&#x2F;beringei-a-high-performance-time-series-storage-engine&#x2F;" rel="nofollow">https:&#x2F;&#x2F;code.fb.com&#x2F;core-data&#x2F;beringei-a-high-performance-ti...</a>) under the hood

The financial read cost of this database makes it practically unusable for customer facing dashboards, disappointing.

A place to put the timestamped data they download from yesterday&#x27;s Amazon Ground Station.

Been searching for years to a good alternative to postgres for storing gobs of weather timeseries data. So far we have been running postgres system for many years in production and have hired multiple contractors to implement a &#x27;real timeseries solution&#x27;. All of which have been utter shit and complete failures. The AWS services are expensive as all hell. With a little bit of imagination we created a unique schema for timeseries data that doesnt require terabytes of space, and processes billions of data points a day, and has blazing fast queries into said data.

So what will this compare wrt boundary, signalfx, stackdriver, etc types of previous services...<p>Ill have to go look into this, because if aws historic pricing for any large volume stream, quickly becomes untennable.<p>Its very easy to have gobs and gobs of time series points... aws might make using this way too expensive for anything at relative scale for a small startup?

I wonder how this compares to KDB

Seems positioned to compete with Azure Data Explorer (MSFT&#x27;s log&#x2F;time series optimized service). I know Azure runs a lot of services on top of Data Explorer (previously called Kusto) I wonder if this is a true internal battle tested product or a me-too offering.

Seems like this could be a great remote storage backend for Prometheus.

Honest question: when dealing with time-series data, do you actually need every data point? Is that level of granularity really necessary?<p>IMO, it makes way more sense to decide the aggregations you want ahead of time (e.g. &quot;SELECT customer, sum(value) FROM purchases GROUP BY customer&quot;). That way, you deal with substantially less data and everything becomes a whole lot simpler.

We had applications where we were tracking guests in a venue through various means. We tried a number of queuing systems to manage the flood of events, but they&#x27;d all fall over. I&#x27;ll love to run my old &quot;venue simulator&quot; through this and see if it can stand up to actual guest load as they walk around, ride, purchase things.

I&#x27;m wondering if this shares any technology with the CloudWatch metrics backend. They&#x27;ve been making improvements there all year, and most of them generally align with what&#x27;s announced here.<p>CloudWatch metrics are also very expensive for what you get, so that&#x27;s another similarity to Timestream ;)

I couldn’t tell from the page is this SQL based similar to timescale or a more similar to influxdb?

This is looking like a managed druid... that would be very nice to have.

Anyone know if this is what CloudWatch Insights uses? If so, it doesn&#x27;t even come close to competing with Elasticsearch performance (with a tiny cluster), it seemed quite slow.

There have been a lot of amazon links this week

where is the docs?

There are at least 7 Amazon-related stories on the HN front page right now, what’s going on?

A quick look at the Hacker News frontpage shows a bit of a problem,<p><pre><code> 1. Amazon Timestream (amazon.com) 3. Amazon Quantum Ledger Database (amazon.com) 8. Amazon FSx for Lustre (amazon.com) 13. AWS DynamoDB On-Demand (amazon.com) 14. Amazon&#x27;s homegrown Graviton processor was very nearly an AMD Arm CPU (theregister.co.uk) 21. Building an Alexa-Powered Electric Blanket (shkspr.mobi) 30. Amazon FSx for Windows File Server (amazon.com)</code></pre>

jesus christ six amazon articles in a day? AWS is undeniably the body of christ for HN but am i missing something? FSX, blockchain, timestream, Graviton, ground station, and cloudwatch... all of these articles are advertisements for mundane shit.

I count 7 separate Amazon posts on the front of HN. Is this some conspiracy? #NotAmused #ShouldBeBundled