TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Saving cloud costs by writing our own database

211 点作者 wolframhempel大约 1 年前

40 条评论

RHSeeger大约 1 年前
&gt; we’ve replaced our $10k&#x2F;month Aurora instances with a $200&#x2F;month Elastic Block Storage (EBS) volume.<p>Without any intent to insult what you&#x27;ve done (because the information is interesting and the writeup is well done)... how do the numbers work out when you account for actually implementing and maintaining the database?<p>- Developer(s) time to initially implement it<p>- PjM&#x2F;PM time to organize initial build<p>- Developer(s) time for maintenance (fix bugs and enhancement requirements)<p>- PjM&#x2F;PM time to organize maintenance<p>The cost of someone to maintain the actual &quot;service&quot; (independent of the development of it) is, I assume, either similar or lower, so there&#x27;s probably a win there. I&#x27;m assuming you have someone on board that was on charge of making sure Aurora was configured &#x2F; being used correctly; and it would be just as easier if not easier to do the same for your custom database.<p>The cost of 120,000&#x2F;year for Aurora seems like it would be less than the cost of development&#x2F;organization time for the custom database.<p>Note: It&#x27;s clear you have other reasons for needing your custom database. I get that. I was just curious about the costs.
评论 #39954615 未加载
评论 #39954067 未加载
评论 #39954058 未加载
评论 #39954052 未加载
评论 #39953940 未加载
评论 #39954661 未加载
评论 #39957242 未加载
评论 #39961477 未加载
评论 #39954080 未加载
评论 #39953938 未加载
评论 #39954094 未加载
jrockway大约 1 年前
Everyone seems fixated on the word database and the engineering cost of writing one. This is a log file. You write data to the end of it. You flush it to disk whenever you&#x27;ve filled up some unit of storage that is efficient to write to disk. Every query is a full table scan. If you have multiple writers, this works out very nicely when you have one API server per disk; each server writes its own files (with a simple mutex gating the write out of a batch of records), and queries involve opening all the files in parallel and aggregating the result. (Map, shuffle, reduce.)<p>Atomic: not applicable, as there are no transactions. Consistent: no, as there is no protection about losing the tail end of writes (consider &quot;no space left on device&quot; halfway through a record). Independent: not applicable, as there are no transactions. Durable: no, the data is buffered in memory before being written to the network (EBS is the network, not a disk).<p>So with all of this in mind, the engineering cost is not going to be higher than $10,000 a month. It&#x27;s a print statement.<p>If it sounds like I&#x27;m being negative, I&#x27;m not. Log files are one of my favorite types of time series data storage. A for loop that reads every record is one of my favorite query plans. But this is not what things like Postgres or Aurora aim to do, they aim for things like &quot;we need to edit past data several times per second and derive some of those edits from data that is also being edited&quot;. Now you have some complexity and a big old binary log file and some for loops isn&#x27;t really going to get you there. But if you don&#x27;t need those things, then you don&#x27;t need those things, and you don&#x27;t need to pay for them.<p>The question you always have to ask, though, is have you reasoned about the business impacts of losing data through unhandled transactional conflicts? &quot;read committed&quot; or &quot;non-durable writes&quot; are often big customer service problems. &quot;You deducted this bill payment twice, and now I can&#x27;t pay the rent!&quot; Does it matter to your end users? If not, you can save a lot of time and money. If it does, well, then the best-effort log file probably isn&#x27;t going to be good for business.
评论 #39954247 未加载
评论 #39954350 未加载
评论 #39955859 未加载
mdaniel大约 1 年前
Anytime I hear &quot;we need to blast in per-second measurements of ...&quot; my mind jumps to &quot;well, have you looked at the bazillions of timeseries databases out there?&quot; Because the fact those payloads happen to be (time, lat, long, device_id) tuples seems immaterial to the timeseries database and can then be rolled up into whatever level of aggregation one wishes for long-term storage<p>It also seems that just about every open source &quot;datadog &#x2F; new relic replacement&quot; is built on top of ClickHouse, and even they themselves allege multi-petabyte capabilities &lt;<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39905443">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39905443</a>&gt;<p>OT1H, I saw the &quot;we did research&quot; part of the post, and I for sure have no horse in your race of NIH, but &quot;we write to EBS, what&#x27;s the worst that can happen&quot; strikes me as ... be sure you&#x27;re comfortable with the tradeoffs you&#x27;ve made in order to get a catchy blog post title
评论 #39953974 未加载
评论 #39954882 未加载
评论 #39953816 未加载
评论 #39953915 未加载
yau8edq12i大约 1 年前
Wasn&#x27;t this already discussed here yesterday? The main criticism of the article is that they didn&#x27;t write a database, they wrote an append-only log system with limited query capabilities. Which is fine. But it&#x27;s not a &quot;database&quot; in the sense that someone would understand when reading the title.
评论 #39954077 未加载
评论 #39954112 未加载
评论 #39954203 未加载
评论 #39954220 未加载
评论 #39954973 未加载
评论 #39959010 未加载
评论 #39954156 未加载
评论 #39954248 未加载
评论 #39954118 未加载
zX41ZdbW大约 1 年前
Sounds totally redundant to me. You can write all location updates into ClickHouse, and the problem is solved.<p>As a demo, I&#x27;ve recently implemented a tool to browse 50 billion airplane locations: <a href="https:&#x2F;&#x2F;adsb.exposed&#x2F;" rel="nofollow">https:&#x2F;&#x2F;adsb.exposed&#x2F;</a><p>Disclaimer: I&#x27;m the author of ClickHouse.
评论 #39961234 未加载
MuffinFlavored大约 1 年前
&gt; We want to be able to handle up to 30k location updates per second per node. They can be buffered before writing, leading to a much lower number of IOPS.<p>&gt; This storage engine is part of our server binary, so the cost for running it hasn’t changed. What has changed though, is that we’ve replaced our $10k&#x2F;month Aurora instances with a $200&#x2F;month Elastic Block Storage (EBS) volume. We are using Provisioned IOPS SSD (io2) with 3000 IOPS and are batching updates to one write per second per node and realm.<p>I would be curious to hear what that &quot;1 write per second&quot; looks like in terms of throughput&#x2F;size?
评论 #39953900 未加载
time0ut大约 1 年前
Good article.<p>&gt; EBS has automated backups and recovery built in and high uptime guarantees, so we don’t feel that we’ve missed out on any of the reliability guarantees that Aurora offered.<p>It may not matter for their use case, but I don&#x27;t believe this is accurate in a general sense. EBS volumes are local to an availability zone while Aurora&#x27;s storage is replicated across a quorum of AZs [0]. If a region loses an AZ, the database instance can be failed over to a healthy one with little downtime. This has only happened to me a couple times over the past three years, but it was pretty seamless and things were back on track pretty fast.<p>I didn&#x27;t see anything in the article about addressing availability if there is an AZ outage. It may simply not matter or maybe they have solved for it. Could be a good topic for a follow up article.<p>[0] <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;blogs&#x2F;database&#x2F;introducing-the-aurora-storage-engine&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;blogs&#x2F;database&#x2F;introducing-the-aurora...</a>
kumarm大约 1 年前
I have built similar system in 2002 using JGroups (JavaGroups at the time before open source project was acquired by JBoss) while persisting asynchronously to DB (Oracle at the time). Our scale even in 2002 was much higher than 13,000 vehicles.<p>The project I believe still appears in success story on JGroups website after 20+ years. I am surprised people are writing their own databases for location storage in 2024 :). There was no need to invent new technology in 2002 and definitely not in 2024.
评论 #39959925 未加载
afro88大约 1 年前
These two sentences don&#x27;t work together:<p>&gt; [We need to cater for] Delivery companies that want to be able to replay the exact seconds leading up to an accident.<p>&gt; We are ok with losing some data. We buffer about 1 second worth of updates before we write to disk<p>Impressive engineering effort on it&#x27;s own though!
xyst大约 1 年前
This seems like they rewrote Kafka to me.<p>Even moderately sized Kafka clusters can handle the throughput requirement. Can even optimize for performance over durability.<p>Some limited query capability with components such as ksqldb.<p>Maybe offload historical data to blob storage.<p>Then again, Kafka is kind of complicated to run at these scales. Very easy to fuck up.
评论 #39957020 未加载
评论 #39954227 未加载
the_duke大约 1 年前
I don&#x27;t know what geospatial features are needed, but otherwise time series databases are great for this use case.<p>I especially like Clickhouse, it&#x27;s generic but also a powerhouse that handles most things you throw at it, handles huge write volumes (with sufficient batching), supports horizontal scaling, and offloading long-term storage to S3 for much smaller disk requirements. The geo features in clickhouse are pretty basic, but it does have some builtin geo datatypes and functions for eg calculating the distance.
kaladin_1大约 1 年前
I love the attitude, we didn&#x27;t see a good fit so we rolled ours.<p>Sure it won&#x27;t cover the bazillion cases the DBs out there do but that&#x27;s not what you need. The source code is small enough for any team member to jump in and debug while pushing performance in any direction you want.<p>Cudos!
CapeTheory大约 1 年前
It&#x27;s amazing what can happen when software companies start doing something approximating real engineering, rather than just sitting a UI on top of some managed services.
yunohn大约 1 年前
This is more a bespoke file format than a full blown database. It’s optimized for one table schema and a few specific queries.<p>Not a negative though, not everything needs a general purpose database. Clearly this satisfies their requirements, which is the most important thing.
评论 #39955145 未加载
diziet大约 1 年前
As others had mentioned, probably hosting your own clickhouse instance could yield major savings while allowing for much more flexibility in the future for querying data. If your use case can be served by what clickhouse offers, gosh is it an incredibly fast and reliable open source solution that you can host yourself.
bawolff大约 1 年前
Kind of misleading to not include the cost of developing it yourself.<p>I think everything is cheaper than cloud if you do it yourself when you don&#x27;t count staffing cost.
评论 #39954044 未加载
Simon_ORourke大约 1 年前
I&#x27;ve no doubt this is true, however, anyone I&#x27;ve ever met who exclaimed &quot;let&#x27;s create our own database&quot; would be viewed as dangerous, unprofessional or downright uneducated in any business meeting. There&#x27;s just too much can go badly wrong, for all the sunk cost in getting anything up and running.
评论 #39954196 未加载
评论 #39956256 未加载
评论 #39954056 未加载
rstuart4133大约 1 年前
A lot of people here are making very confident sounding assertions, yet some as saying it&#x27;s just an append only log file and some imply it&#x27;s sharded. Something everyone does agree on is they are very vague about what geospartial features they need.<p>The one thing they do say is &quot;no ACID&quot;. That implies no b-trees, because an unexpected stop means a corrupted b-tree. Perhaps they use a hash instead, but it would have to be a damned clever hash tree implementation to avoid the same problem. Or perhaps they just rebuild the index after a crash.<p>Even a append only log file has to be handled carefully without ACID. An uncontrolled shutdown in more file systems will at leave blocks of nulls in the file and 1&#x2F;2 written blocks if they cross disk block boundaries.<p>It&#x27;s a tantalising headline, but after reading the 1,200 words I&#x27;m none the wiser on what they built or whether it meets their own specs. A bit of a disappointment.
INTPenis大约 1 年前
That is such an insane headline.<p>You might as well say &quot;we saved 100% of cloud costs by writing our own cloud&quot;.
endisneigh大约 1 年前
It would be interesting to see a database built from the ground up for being trivial to maintain.<p>I use managed databases, but is there really that much to do for maintaining a database? The host requires some level of maintenance - changing disks, updating the host operating system, failover during downtime for machine repair, etc. if you use a database built for failover I imagine much of this doesn’t actually affect the operations that much assuming you slightly over provision.<p>For a database alone I think the work needed to maintain is greatly exaggerated. That being said I still think it’s more than using a managed database, which is why my company still does so.<p>In this case though, an append log seems pretty simple imo. Better to self host.
fifilura大约 1 年前
Would building a data lakehouse be an option?<p>Stream the events to s3 stored as Parquet or Avro files, maybe in Iceberg format.<p>And then use Trino&#x2F;Athena to do the long term heavy lifting. Or for on-demand use cases.<p>Then only push what you actually need live to a Aurora.
评论 #39954240 未加载
kroolik大约 1 年前
I could be missing something, but I can&#x27;t really wrap my head around &quot;unlimited paralelism&quot;.<p>What they say is that the logic is embedded into their server binary and they write to a local EBS. But what happens when they have two servers? EBS can&#x27;t be rw mounted in multiple places.<p>Won&#x27;t adding the second and more servers cause trouble like migrating data when new server joins the cluster, or a server leaves the cluster?<p>I understand Aurora was too expensive for them. But I think it is important to note their whole setup is not HA at all (which may be fine, but the header could be misleading).
评论 #39958290 未加载
评论 #39955010 未加载
rvba大约 1 年前
&gt; So - given that we don’t know upfront what level of granularity each customer will need, we store every single location update.<p>Maybe Im cynical but interesting that &quot;the business&quot; didnt start to check it to cut costs. I know that customers love this feature. Cynically I can see it costing more, so some customers would drop in.<p>Also it looks they rewrote a log &#x2F; timeseries &quot;database&quot; &#x2F; key value store? As pthers mention sounds like reinventing the wheel to get a cool blog post and boost career solving &quot;problems&quot;.
rad_gruchalski大约 1 年前
&gt; we’ve replaced our $10k&#x2F;month Aurora instances with a $200&#x2F;month Elastic Block Storage (EBS) volume<p>Reminds me how I implemented mssql active-active log replication over dropbox shares back in 2010 to synchronise two databases in the Us and in the UK. Worked perfectly fine except of that one hurricane that took them out for longer than 14 days. This was more than the preconfigured log retention period.
pheatherlite大约 1 年前
How fast can reads be thou? Even if skipping along a fixed offset, reading 4 byte identifiers to filter out location updates for vehicles, that&#x27;s still a sequential scan of a massive file. Wouldn&#x27;t this read issue become a choking point to a degree that would make growth a curse? Then you get into weird architectures that exist solely to facilitate predigested reads?
remram大约 1 年前
They mention all those features of databases, presenting them as important:<p>&gt; Databases are a nightmare to write, from Atomicity, Consistency, Isolation, and Durability (ACID) requirements to sharding to fault recovery to administration - everything is hard beyond belief.<p>Then talk about their geospatial requirements, PostGIS etc, making it seems they need geospatial features (&quot;PostGIS for geospatial data storage&quot; -- wtf? you need PostGIS for geospatial <i>query</i> not merely <i>storage</i>...)<p>In reality, they did not require any of the features they mention throughout the article. What a weird write-up!<p>I guess the conclusion is &quot;read the F*-ing specs&quot;. Don&#x27;t grab a geospatial DBMS just because you heard the words &quot;longitude&quot; and &quot;database&quot; once.
评论 #39957986 未加载
nikonyrh大约 1 年前
Very interesting, it must feel great to get to apply CS knowledge at work, rather than writing basic CRUD apis &#x2F; websites.
评论 #39954306 未加载
trebecks大约 1 年前
if i&#x27;m reading the op right, they kind of use ebs as a buffer for fresh data until it ages out to s3. they use a &quot;local&quot; disk to hold the stuff used by the queries that people actually make and the queries run quick. they let the old stuff rot in s3 where its almost never used. that sounds like a good idea to save money plus the stuff that&#x27;s done often is fast.<p>the ebs slas look reasonable to a non expert like me and you can take snapshots. it sound like you need to be careful when snapshotting to avoid inconsistencies if stuff is only partially flushed to disk. so you&#x27;d need to pause io while it snapshots if those inconsistencies matter. that sounds bad and would encourage you to take less frequent snapshots...? you also pay for the snapshot storage but i guess you wouldn&#x27;t need to keep many. i like that aws defines &quot;SnapshotAPIUnits&quot; to describe how you get charged for the api calls.<p>with aurora, it looks like you can synchronously replicate to a secondary (or multiple secondaries) across azs in a single region. it sounds nice to have a sync copy of stuff that people are using. op says the&#x27;yre ok with a few seconds of data loss so i&#x27;m wondering how painful losing a volume right before taking a snapshot would be.<p>i wonder if anything off the shelf does something similar. it sounds like people are suggesting clickhouse. i saw buffer table in their docs and it sounds similar <a href="https:&#x2F;&#x2F;clickhouse.com&#x2F;docs&#x2F;en&#x2F;engines&#x2F;table-engines&#x2F;special&#x2F;buffer" rel="nofollow">https:&#x2F;&#x2F;clickhouse.com&#x2F;docs&#x2F;en&#x2F;engines&#x2F;table-engines&#x2F;special...</a>. it looks like it has stuff to use s3 as cold storage too. i even see geo types and functions in the docs. i&#x27;ve never used clickhouse so i don&#x27;t know if i&#x27;m understanding what i read, but it sounds like you could do something similar to whats described in the post with clickhouse if the existing geo types + functions work and you are too lazy to roll something yourself.
loftsy大约 1 年前
Apache Cassandra could be a good fit here. Highly parallel frequent writes with some consistency loss allowed.
exabrial大约 1 年前
Why is everyone dead set on “must use aws” these days? One can cut their cloud costs by 100x with colo.<p>And if you write your own db as they did here, it can 100% take advantage of your setup.
zinodaur大约 1 年前
Very cool! When I started reading the article I thought it was going to end up using an LSM tree&#x2F;RocksDB but y&#x27;all went even more custom than that
mavili大约 1 年前
That&#x27;s called engineering; you had a problem, you came up with a solution THAT WORKS for your needs. Nicely done and thanks for sharing.
selimnairb大约 1 年前
Seems like DuckDB or TileDB backed by S3 may meet your needs and be a lot cheaper than Aurora.
awinter-py大约 1 年前
we have invented write concern = 0
halayli大约 1 年前
They talk about what they store but zero mention on their retreival requirements.
tshanmu大约 1 年前
&quot;Of course, that’s an unfair comparison, after all, Postgres is a general purpose database with an expressive query language and what we’ve built is just a cursor streaming a binary file feed with a very limited set of functionality - but then again, it’s the exact functionality we need and we didn’t lose any features.&quot;
icsa大约 1 年前
How is it possible to save more than 100% ?
评论 #39953831 未加载
评论 #39929869 未加载
评论 #39953780 未加载
评论 #39954976 未加载
评论 #39954319 未加载
bevekspldnw大约 1 年前
“We are running a cloud platform that tracks tens of thousands of people and vehicles simultaneously”<p>…that’s not something to brag about.
评论 #39955806 未加载
brianhama大约 1 年前
Honestly, this doesn’t seem like that high of requirements. There are tens of thousands of companies that are doing more spatial data processing and are using standard cloud databases just fine.
SmellTheGlove大约 1 年前
I&#x27;m surprised to see the (mostly) critical posts. My reaction before coming to the comments was:<p>- This is core to their platform, makes sense to fit it closely to their use cases<p>- They didn&#x27;t need most of what a full database offers - they&#x27;re &quot;just&quot; logging<p>- They know the tradeoffs and designed appropriately to accept those to keep costs down<p>I&#x27;m a big believer in building on top of the solved problems in the world, but it&#x27;s also completely okay to build shit. That used to be what this industry did, and now it seems to have shifted in the direction of like 5-10% of large players invent shit and open source it, and the other 90-95% are just stitching together things they didn&#x27;t build in infrastructure that they don&#x27;t own or operate, to produce the latest CRUD app. And hell, that&#x27;s not bad either, it&#x27;s pretty much my job. But it&#x27;s also occasionally nice to see someone build to their spec and save a few dollars. It&#x27;s a good reminder that costs matter, particularly when money isn&#x27;t free and incinerating endless piles of it chasing a (successful) public exit is no longer the norm.<p>I get the arguments that developer time isn&#x27;t free, but neither is running AWS managed services, despite the name. And they didn&#x27;t really build a general purpose database, they built a much simpler logger for their use case to replace a database. I&#x27;d be surprised if they hired someone additional to build this, and if they did, I&#x27;d guess (knowing absolutely nothing) that the added dev spends 80% of their time doing other things. It&#x27;s not like they launched a datacenter. They just built the software and run it on cheaper AWS services versus paying AWS extra for the more complex product.
评论 #39954504 未加载