TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Why Not to Build a Time-Series Database

204 点作者 dgildeh超过 6 年前

11 条评论

tuukkah超过 6 年前
TLDR: &quot;Why Not to Build a Time-Series Database? Because we&#x27;re building one and you should pay us.&quot;<p>&gt; <i>Hopefully our story will make you think twice before trying to build your own TSDB in house using open-source solutions, or if you’re really crazy, building a TSDB from scratch. Building and maintaining a TSDB is a full time job, and we have dedicated expert engineers who are constantly improving and maintaing our TSDB, and no doubt will iterate the architecture again over time as we hit an even higher magnitude of scale down the line.</i><p>&gt; <i>Given our experience in this complex space, I would sincerely recommend you don’t try and do this at home, and if you have the money you should definitely outsource this to the experts who do this as a full time job, whether its Outlyer or another managed TSDB solution out there. As so many things turn out in computing, it’s harder than it looks!</i>
评论 #18405075 未加载
评论 #18404570 未加载
评论 #18405724 未加载
评论 #18404361 未加载
评论 #18405238 未加载
manish_gill超过 6 年前
As I was reading through the post I kept wondering why they weren&#x27;t using some warehousing technique for older data - either dump it to S3 or better yet, Google BigQuery, which is amazingly fast at that scale. They only did it after doing lots of fire-fighting and per-tenant clusters.<p>Clickhouse would also be a good option for doing aggregating queries that TSDBs are mostly used for.<p>One of my wishlist items in the data space is a Managed Clickhouse offering. :-)
bra-ket超过 6 年前
We used a combination of Kafka + Hbase+ Phoenix (<a href="http:&#x2F;&#x2F;phoenix.apache.org&#x2F;" rel="nofollow">http:&#x2F;&#x2F;phoenix.apache.org&#x2F;</a>) for similar purpose. It takes some effort to setup initial Hbase cluster but once you do it manually once and automate with Ansible &#x2F;systemd it&#x27;s pretty robust in operation.<p>All our development was around query engine using plain JDBC&#x2F;SQL to talk to Hbase via Phoenix. Scaling is as simple as adding a node in the cluster.
评论 #18404898 未加载
评论 #18404104 未加载
camel_gopher超过 6 年前
In case anyone is interested in a video on this, here&#x27;s the talk presented at the local San Francisco monitoring group, monitorSF (<a href="https:&#x2F;&#x2F;www.meetup.com&#x2F;MonitorSF&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.meetup.com&#x2F;MonitorSF&#x2F;</a>)<p><a href="https:&#x2F;&#x2F;youtu.be&#x2F;lA85vs6e3UA" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;lA85vs6e3UA</a>
marshf超过 6 年前
Time-series data handling&#x2F;storage seems a mostly solved problem in the mining, oil, and manufacturing industries. Deployed in the field since the 80&#x27;s,<p><a href="https:&#x2F;&#x2F;www.osisoft.com&#x2F;about-osisoft&#x2F;#more-about-pi-system" rel="nofollow">https:&#x2F;&#x2F;www.osisoft.com&#x2F;about-osisoft&#x2F;#more-about-pi-system</a><p>Discourse: Industry user now OSIsoft employee
评论 #18405034 未加载
评论 #18404244 未加载
statictype超过 6 年前
Nice article.<p>&gt;its not uncommon for some of our customers to send us millions of metrics every minute<p>What kind of customers&#x2F;services generate millions of points a minute?
评论 #18403398 未加载
评论 #18403420 未加载
评论 #18403332 未加载
评论 #18403520 未加载
评论 #18403354 未加载
评论 #18405150 未加载
评论 #18403589 未加载
评论 #18403334 未加载
评论 #18404586 未加载
评论 #18403341 未加载
评论 #18403310 未加载
manigandham超过 6 年前
&quot;time-series database&quot; is some of the most overhyped nonsense since noSQL.<p>Time-series is just data with time as a primary component. It comes in all shapes and volume, but if you have a lot of data and are running heavy OLAP queries than we already have an entire class of capable databases.<p>Use any modern distributed relational column-oriented database, set primary key to metric id + timestamp, and you&#x27;ll be able to scale easily with full SQL and joins. You can keep your other business data there too, along with JSON, geospatial, window functions, and all the other rich analytical queries available with relational databases.<p>We have trillion row tables that work great. No special &quot;TSDB&quot; needed.
评论 #18403551 未加载
评论 #18403683 未加载
评论 #18403718 未加载
评论 #18403743 未加载
评论 #18404077 未加载
评论 #18403810 未加载
评论 #18404600 未加载
评论 #18404821 未加载
评论 #18404349 未加载
评论 #18403577 未加载
Daneel_超过 6 年前
I’m surprised no one has brought up Splunk in here (that I could see at a cursory glance).<p>They manage to do time-series storage on a pretty large scale (over 5PB&#x2F;day for their largest customer).
评论 #18404964 未加载
objektif超过 6 年前
Not very knowledgable in the area but can someone please explain how does kdb fit within this class of time series dbs and whether there are any alternatives available to kdb.
评论 #18404224 未加载
dgildeh超过 6 年前
As the blog author, great to see the discussion and feedback, so appreciate it!<p>Without going through comments one by one, the main ones about this being a solved problem or there&#x27;s already solutions out there that do this, I would just say those comments remind me exactly of the type of conversations I had years ago with my team. We all thought it would be much easier or thought there would be something off the shelf that could do everything, and after several years of fire fighting, the reality was the problem looks much simpler than it really is, by a long mile.<p>Now that we&#x27;ve been doing this for a few years, and spoken directly with creators of many other TSDBs, we take a very skeptical view of all claims made about any database. They all sound amazing when you first read about them, maybe even work great in testing, till you hit scale and then you find all the limitations. If there was a perfect TSDB out there, everyone would be using it and there wouldn&#x27;t be a new one announced on a weekly basis!<p>I think the one comment on query loads being different sums things up - I&#x27;ve no doubt all the other options thrown out there work well for data historians, but for monitoring tools with loads of concurrent users, loading dashboards with 10&#x27;s or 100&#x27;s of queries each, and alerting systems polling every few seconds in parallel, the query load can get very high quickly, and making those fast while still writing metrics in at scale, is a hard problem and I don&#x27;t think any individual TSDB has really solved that properly, which is why we ended up building our own distributed architecture ourselves.
OldHand2018超过 6 年前
It&#x27;s pretty ridiculous that &quot;Time-Series Database&quot; has come to mean ingesting massive amounts of streaming data. They&#x27;ve been around a long time and have many use cases.<p>They&#x27;re a great way to store data efficiently, accessing specific data if you know the time range you are looking for is very fast and simple, and you can roll your own in a few dozen lines of C if that&#x27;s what you want to do. If that&#x27;s all you need, why not?
评论 #18405324 未加载