TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

How To Make An Infinitely Scalable RDBMS

154 点作者 jpmc超过 11 年前

11 条评论

glogla超过 11 年前
There was very interesting presentation by one professor. I&#x27;m not sure about what university, but he seemed to know his work.<p>He talked about how databse world is about to change. ACID is really expensive in terms of resources, and so are the more difficult things about relational schema (foreign keys, checks, etc). And architecture of classic RDBMSes is pretty wasteful -- they use on-disk format but cache it in memory.<p>He talked about how there are basically three new paths for DBMSes to follow. 1) Some drop the restrictions to become faster. This is the NoSql stuff, because you don&#x27;t really need ACID for writing to Facebook wall.<p>This is called NoSql database.<p>2) OLAP, in data warehousing, the usual way to do things is that you load ridiculous amount of data into database, and then run analytical queries, that tend to heavily use aggregation and sometimes use just few dimmensions, while the DWH data tend to be pretty wide.<p>For this, column store makes perfect sense. It is not very quick on writes, but it can do very fast aggregation and selection of just few columns.<p>This is called Column store.<p>3) In OLTP, you need throughtput, but the question is, how big are your data, and how fast do they grow? Because RAM tends to get bigger exponentially, while how many customers you have will probably grow linearly or maybe fuster, but not much. So your data could fit into memory, now, or in future.<p>This allows you to make very fast database. All you need to do is to switch the architecture to memory-based, store data in memory format in memory and on disk. You don&#x27;t read the disk, you just use it to store the data on shutdown.<p>This is called Main memory database.<p>No, that was the presentation. It was awesome, and if someone can find it, please give us a link! My search-fu was not strong enouhg.<p>...<p>What interests me is that we have NoSql databases for some time already, and we have at least one huge (are very expensive) column store: Teradata. But this seems to be first actual Main memory database.<p>My dream would be to switch Postgres to main memory or column store mode, but I guess that&#x27;s not happening very soon :)
评论 #6797647 未加载
评论 #6796734 未加载
评论 #6796714 未加载
评论 #6796838 未加载
评论 #6796699 未加载
评论 #6800633 未加载
评论 #6798488 未加载
评论 #6799494 未加载
评论 #6799966 未加载
alanctgardner2超过 11 年前
I&#x27;m a little skeptical:<p>- a bunch of the novel components (the UPS aware persistence layer, for example) aren&#x27;t actually built yet<p>- they&#x27;re pushing for people to build businesses on it already. I would characterize it as &quot;bleeding-edge with bits of glass glued on&quot;, so this doesn&#x27;t seem entirely honest.<p>- there&#x27;s mostly a lot of breathless talk about how great and fast and scalable it is, but no mention of CAP theorem. To boil down their feature set, it&#x27;s an in-memory RDBMS using the Actor model.
评论 #6795868 未加载
yid超过 11 年前
&gt; UPS systems will stay active for a few minutes, based on their capacity, and the manager process will gracefuly shut down each daemon and write data to disk storage. This will ensure durability--even against power failure or system crash--while still maintaining in memory performance.<p>How does a UPS ensure durability against system or program crashes, disk corruption in large clusters, and other failures that can affect a simple write()?<p>&gt; The real killer for database performance is synchronous transaction log writes. Even with the fastest underlying storage, this activity is the limiting factor for database write performance. InfiniSQL avoids this limiting factor while still retaining durability<p>How do you plan to implement this (since it appears it hasn&#x27;t been implemented)? What is your fundamental insight about synchronous transaction logs that makes InifiSQL capable of being durable while (presumably) not having a synchronously written transaction log? If your answer is the UPS, please see my first question.<p>Edit: I don&#x27;t see any mention of Paxos anywhere. Could you explain what you&#x27;re using for consensus?
评论 #6796981 未加载
gopalv超过 11 年前
GNU AFPL?<p>Clause 13 is a real pain to deal with when exposing this over the network.<p>I guess the developer wants to sell a license (like the mysql java client GPL&#x27;ing).<p>Can&#x27;t blame him, he needs to get paid.
评论 #6795649 未加载
评论 #6795632 未加载
MichaelGG超过 11 年前
In-memory distributed database? VoltDB is already way past 500Ktx&#x2F;sec on a 12-node cluster.<p>On their site though, it says no sharding and that it can do these 500Ktx&#x2F;sec even when each transaction involves data on multiple nodes. Does this performance degrade directly in relation to the number of nodes a tx needs to touch?<p>A simple, straightforward, wire-level description of how things work when coordinating and performing transactions across would be very useful. There&#x27;s a lot of excited talk about actors, but nothing that really examines why this is faster, or any sort of technical analysis.
评论 #6796108 未加载
eksith超过 11 年前
Looking at the &quot;About and Goals&quot; section of their docs <a href="http://www.infinisql.org/docs/overview/#idp37033600" rel="nofollow">http:&#x2F;&#x2F;www.infinisql.org&#x2F;docs&#x2F;overview&#x2F;#idp37033600</a><p>I can&#x27;t seem to find the word &quot;Reliable&quot; or any variation thereof anywhere in there.<p>In fact, that word is no where to be found on the blog post or on the entire InfiniSQL page (not in the Overview, Guides, Reference or even FAQ). I find this quite remarkable since reliability is the true virtue of an RDBMS, not speed or even capacity. At least that&#x27;s what PostgreSQL aims for and this being another RDBMS, and is also open source, I see it as InfiniSQL&#x27;s only direct competitor.<p>It&#x27;s nice that this is scalable, apparently, to ridiculous levels, but if I can&#x27;t retrieve what I store in exactly the same shape as I stored it, then that&#x27;s a bit of a buzz kill for me.<p>Can we have some assurance that this is the case?<p>There&#x27;s a note on &quot;Durability&quot; and a shot at log file writing for transactions, and presumably InfiniSQL uses concurrency and replicas, to provide it. In the Data Storage section, it mentions that InfiniSQL is still an in-memory database for the most part <a href="http://www.infinisql.org/docs/overview/#idp37053600" rel="nofollow">http:&#x2F;&#x2F;www.infinisql.org&#x2F;docs&#x2F;overview&#x2F;#idp37053600</a><p>What they&#x27;re describing is a massively redundant, UPS backed, in-memory cache.<p>Am I wrong?
评论 #6799094 未加载
leif超过 11 年前
The write bottleneck for traditional databases has never been the write-ahead log, with group commit and a battery-backed RAID controller you&#x27;ll have a hard time saturating the disk with log writes. The bottleneck has always been random I&#x2F;O induced by in-place updating indexes based on B-trees. You don&#x27;t need to be in-memory if you use better data structures. TokuDB and TokuMX are proof of that.
评论 #6798673 未加载
brianberns超过 11 年前
An in-memory RDBMS hardly seems to be &quot;infinitely scalable&quot;. How would this work with DBs in the terabyte size or larger?
评论 #6797940 未加载
评论 #6797765 未加载
评论 #6796691 未加载
amalag超过 11 年前
This is what Clustrix (YC company) claims to do.
评论 #6798178 未加载
jeremycole超过 11 年前
What&#x27;s up with the weird coding standards? Include files named infinisql_*.h and #line statements... strange.
评论 #6798448 未加载
flatfilefan超过 11 年前
What is the difference to Teradata or Netezza except this is open source and lack the burden of universality yet?
评论 #6805193 未加载