TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Turning the IDE Inside Out with Datalog

275 点作者 arjunnarayan将近 5 年前

24 条评论

brabel将近 5 年前
Jetbrain IDEs (e.g. IntelliJ) have structural search which is basically a query language for code: <a href="https:&#x2F;&#x2F;www.jetbrains.com&#x2F;help&#x2F;idea&#x2F;structural-search-and-replace.html#to_search_structurally" rel="nofollow">https:&#x2F;&#x2F;www.jetbrains.com&#x2F;help&#x2F;idea&#x2F;structural-search-and-re...</a><p>I know there&#x27;s more to just queriability in the post, but I thought this should&#x27;ve been mentioned when discussing what existing IDEs can offer.
评论 #23889494 未加载
评论 #23889162 未加载
sriram_malhar将近 5 年前
Niko Matsakis had an insightful observation about this approach in a similar post (which is referenced by OP):<p><a href="http:&#x2F;&#x2F;smallcultfollowing.com&#x2F;babysteps&#x2F;blog&#x2F;2017&#x2F;01&#x2F;26&#x2F;lowering-rust-traits-to-logic&#x2F;" rel="nofollow">http:&#x2F;&#x2F;smallcultfollowing.com&#x2F;babysteps&#x2F;blog&#x2F;2017&#x2F;01&#x2F;26&#x2F;lowe...</a><p>The key observation to me was that the traditional Datalog&#x2F;Prolog way of unifying is through syntactic equality, which is a bit too simple to express the kind of equality needed in Rust and elsewhere. You can express it in Datalog, but as it gets farther away from the source, error-generation suffers.
评论 #23888563 未加载
评论 #23888397 未加载
kibwen将近 5 年前
In the Rust ecosystem, I believe the next-generation IDE support is already beginning to incorporate some ideas like this, although I believe it uses Prolog rather than Datalog (I&#x27;d be interested to learn more about how they differ). The logic library is called Chalk ( <a href="https:&#x2F;&#x2F;github.com&#x2F;rust-lang&#x2F;chalk" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;rust-lang&#x2F;chalk</a> ), and it&#x27;s designed to be a full trait resolution engine for Rust (trait resolution being a necessary step to e.g. autocomplete methods on types); rust-analyzer has been using it for quite a while now, and it&#x27;s slowly being incorporated into rustc itself. Not as impressive as basing the entire IDE on queries, of course!<p>The Rust project is also using the differential-datalog library mentioned in the OP to underlie their third-generation borrow checker: <a href="https:&#x2F;&#x2F;github.com&#x2F;rust-lang&#x2F;polonius" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;rust-lang&#x2F;polonius</a>
评论 #23888121 未加载
评论 #23887705 未加载
simplify将近 5 年前
Fantastic work! Datalog&#x2F;Prolog are old-yet-futuristic technologies that I wished every developer knew about, as it solves a lot of problems in very elegant ways (as attested to by this post).<p>I noticed you (assuming you&#x27;re the author) used what looks like a non-traditional Datalog syntax – which makes sense to me as, IMO, Datalog&#x2F;Prolog desperately need first-class support for a record-like syntax to finally break into mainstream. Is there any prior work to this syntax, or did you just develop it as you needed it?
评论 #23888494 未加载
评论 #23888233 未加载
pjmlp将近 5 年前
Rejuvenating that many devs are finally looking into this kind of workflows.<p>For prior work on IDE as database check &quot;Energize&#x2F;Cadillac &amp; Lucid&#x27;s Demise&quot;, <a href="https:&#x2F;&#x2F;www.dreamsongs.com&#x2F;Cadillac.html" rel="nofollow">https:&#x2F;&#x2F;www.dreamsongs.com&#x2F;Cadillac.html</a><p>You can see it in action on this 1993 marketing video from Lucid, <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=pQQTScuApWk" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=pQQTScuApWk</a><p>IBM also tried the same concept on Visual Age for C++, version 4, which was one of the very last versions of the product.<p><a href="http:&#x2F;&#x2F;www.edm2.com&#x2F;index.php&#x2F;VisualAge_C%2B%2B_4.0_Review" rel="nofollow">http:&#x2F;&#x2F;www.edm2.com&#x2F;index.php&#x2F;VisualAge_C%2B%2B_4.0_Review</a><p>Both suffered from too heavy hardware requirements for what most companies were willing to pay for, otherwise we could have had Smalltalk like tooling, with incremental compilation and reloading for C++ already during the early 90&#x27;s.
jierenchen将近 5 年前
This is very cool! It&#x27;s really awesome to see this code as data concept gaining a lot of traction recently. Hope to see this project developed further.<p>I&#x27;m working on a similar project here: <a href="https:&#x2F;&#x2F;sourcescape.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;sourcescape.io&#x2F;</a>, but intended for use outside the IDE on larger collections of code (like the codebase of a large company.)<p>Agreed on the Prolog&#x2F;Datalog approach of expressing a query as a collection of facts. CodeQL does the same. From one datastore nerd to another, I actually think this is a relatively unexplored area of querying knowledge graphs (code being a very complex, dense knowledge graph.)<p>Very excited to see where you go next with this &quot;percolate search&quot; functionality in the IDE.
评论 #23888430 未加载
sideeffffect将近 5 年前
This is a similar idea to that of SemanticDB, which is used in the Scala community<p><a href="https:&#x2F;&#x2F;scalameta.org&#x2F;docs&#x2F;semanticdb&#x2F;guide.html" rel="nofollow">https:&#x2F;&#x2F;scalameta.org&#x2F;docs&#x2F;semanticdb&#x2F;guide.html</a><p>It is a queryable database of semantic information about the program which is generated by the compiler (compiler plugin, to be precise). Once generated, other tools which need semantic information, like linters or language servers, can consume it without having to worry about how to actually generate it.<p>You might enjoy a talk about it: How We Built Tools That Scale to Millions of Lines of Code by Eugene Burmako<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=C770WpI_odM" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=C770WpI_odM</a><p>Kythe by Google is also a similar thing: <a href="https:&#x2F;&#x2F;kythe.io&#x2F;" rel="nofollow">https:&#x2F;&#x2F;kythe.io&#x2F;</a>
sandermvanvliet将近 5 年前
I would have expected a mention of Roslyn which powers Visual Studio (Not code). Sort of similar to the IntelliJ approach but it is also what drives the C# compiler, it maintains the code model (actually 2 of them, syntactic and semantic) which makes it pretty powerful. It is multi language (VB, C# and I think F# now too?) but it’s perhaps not as universal as the Language Server Approach
评论 #23888642 未加载
评论 #23888811 未加载
afarrell将近 5 年前
One challenge to this that languages which easily allow metaprogramming can encourage codebases where it is hard to write tools to gain insight into the structure of a codebase.<p>You can say “don’t do that”, but I didn’t. Past coworkers did.
评论 #23888420 未加载
disposedtrolley将近 5 年前
Are there any examples of &quot;programs as databases&quot; being applied to structured text like JSON or YAML? Something like a generic system which takes a set of rules&#x2F;facts and the source data, and transforms these into a queryable data structure.<p>I&#x27;m working with a lot of OpenAPI [1] specifications currently, some of which span tens of thousands of lines. Heaps of parent, child, sibling type relationships, and objects which are defined-once and referenced in many places. It would be nice if I could perform a search like &quot;select all schemas used in this path&quot; or &quot;select the count of references to this parameter&quot;.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;OAI&#x2F;OpenAPI-Specification&#x2F;blob&#x2F;master&#x2F;versions&#x2F;3.1.0.md" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;OAI&#x2F;OpenAPI-Specification&#x2F;blob&#x2F;master&#x2F;ver...</a>
评论 #23890121 未加载
ilaksh将近 5 年前
Reminds me a little of &quot;Intentional Programming&quot; or Structured Editors.
nine_k将近 5 年前
This is really nice. Key points:<p>- Parse the program like an IDE would, but expose the data in an open queryable database format (both line and unlike a language server).<p>- Use Datalog for storing the facts and <i>inferring</i> new facts about data.<p>(A fun fact: the Datalog implementation they use is written in Haskell and generates programs in Rust.)
评论 #23887351 未加载
habosa将近 5 年前
This is extremely cool, great work! You&#x27;re right, IDEs can do so much magic but they&#x27;re limited by the &#x27;spells&#x27; their creators have imagined. Many times I&#x27;ve wanted to do a simple semantic refactor that I could express as logic but the IDE can&#x27;t handle it.<p>(Also hi Pete!)
评论 #23887892 未加载
评论 #23892747 未加载
Davidbrcz将近 5 年前
This idea of turning a program into a database and using prolog&#x2F;dalalog on it is not new.<p>The most successful example is Semmle (bought by Github), which has been doing it for years now, with a SQL-like syntax for requests (named &quot;.QL&quot;).
gavinray将近 5 年前
This is how Rust Analyzers intellisense works, it uses a query engine called Salsa that stores the symbols in a database and the linting and completion&#x2F;semantics are entirely query-driven:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;salsa-rs&#x2F;salsa" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;salsa-rs&#x2F;salsa</a><p>Good video describing it&#x27;s use for working with Rust&#x27;s AST:<p><a href="https:&#x2F;&#x2F;youtu.be&#x2F;i_IhACacPRY?t=1348" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;i_IhACacPRY?t=1348</a>
z3t4将近 5 年前
I&#x27;ve written an IDE in JS where all internal structures are available in the runtime which can be hooked into. I can however not see the usability in what the article describe. So I would like to se some practical examples. What if you could do queries like in the article, what sort of queries would you run !?
YeGoblynQueenne将近 5 年前
Nice article and great idea, but as is traditional there are some slight fudges of what Datalog is or what Horn clauses are etc, that I&#x27;d like to unfudge, slightly. It&#x27;s Sunday! What better than to start our day with a very quick and almost not completely fudgy intro to logic programming? Le&#x27;ts go!<p>To begin with, Datalog is not a &quot;cousin&quot; of Prolog as stated in the section &quot;Interlude: Brief Intro to Datalog&quot;. Datalogs (there are many variants!) are subsets of Prolog. For example, a typical datalog is the language of definite clauses with no function symbols [¹] and with no negation as failure [²]. Another datalog may allow only the cons function in order to handle lists; etc.<p>Otherwise the syntax of datalog is identical to Prolog, but there is a further difference, in that Prolog is evaluated from the &quot;top down&quot; whereas Datalog is evaluated from the &quot;bottom up&quot;. What that means is that given a &quot;query&quot; (we&#x27;ll come to the scare quotes in a moment) Prolog will try to find &quot;rules&quot; whose heads unify with a literal in the query (A literal is an atom, or the negation of an atom; &quot;p(χ,α)&quot; is an atom.) whereas datalog will first generate the set of all ground atoms that are consequences of the program in the context of which the query was made, then determine whether the atoms in the query are in the set of consequences of the program [³]. The reason for the different execution model is that the bottom-up evaluation is guaranteed to terminate [⁴] whereas Prolog&#x27;s top-down evaluation can &quot;go infinite&quot; [⁵]. There is of course another, more subtle difference: Prolog can &quot;go infinite&quot; because of the Halting problem, from which datalog does not suffer because, unlike Prolog, it does not have Universal Turing Machine expressivity [⁶].<p>So in short, datalog is a restricted subset of Prolog that has the advantage of being decidable, while Prolog in general is not, but is also incomplete while Prolog is complete [⁷].<p>Now, the other slight fudge in the article is about &quot;rules&quot;, &quot;facts&quot; and &quot;queries&quot;. Although this is established and well-heeled logic programming terminology, it fudges the er fact that those three things are the <i>same kind of thing</i>, namely, they are, all three of them, Horn clauses [⁸].<p>Specifically, Horn clauses are clauses with a single positive literal.<p>Crash course in FOL: an atom is a predicate symbol followed by a set of terms in parentheses. Terms are variables, functions or constants (constants are functions with 0 arity, i.e. 0 arguments). A literal is an atom, or the negation of an atom. A clause is a disjunction of literals. A clause is Horn when it has at most 1 positive literal. A Horn clause is a definite clause when it has exactly 1 positive literal.<p>The following are Horn clauses:<p><pre><code> ¬P(χ) ∨ ¬P(υ) P(χ) ∨ ¬Q(χ) Q(α) Q(β) </code></pre> In logic programming tradition, we write clauses as implications (because ¬A ∨ B ≡ A → B) and with the direction of the implication arrow reversed to make it easier to read long implications with multiple premises. So the three clauses above are written as:<p><pre><code> ←P(χ), P(υ) (a) P(χ) ← Q(χ) (b) Q(α)← (c) Q(β)← (d) </code></pre> And those are a &quot;query&quot;, (a), a &quot;rule&quot;, (b) and two &quot;facts&quot;, (c) and (d).<p>Note that (b,c,d) are <i>definite</i> clauses (they have exactly one positive literal, i.e. their head literal, which is what we call the consequent in the implication). Facts have only a positive literal; I like to read the dangling implication symbol as &quot;everything implies ...&quot;, but that&#x27;s a bit idiosyncratic. The bottom line is that definite clauses with no negative literals can be though of as being always true, hence &quot;facts&quot;. Queries, i.e. Horn clauses with no positive literals, are the opposite: &quot;nothing implies&quot; their body literals (my idiosyncratic reading) so they are &quot;always false&quot;. Queries are also called &quot;goals&quot;. Finally, definite clauses with both positive and negative literals can be thought of as &quot;conditionally true&quot;.<p>Prolog and datalog programs are written as sets of definite clauses, i.e. sets of &quot;facts&quot; and &quot;rules&quot;. So, when we want to reason about the &quot;facts&quot; and &quot;rules&quot; in the program, we make a &quot;query&quot;. Then, the language interpreter, which is a top-down resolution theorem prover [⁹] in the case of Prolog, or bottom-up fixpoint calculation in the case of datalog [¹⁰], determines whether our &quot;query&quot; is true. If the query includes any variables then the interpreter also returns evey set of variable substitutions that make the query true.<p>In the example above, (a) has two variables, χ and υ and evaluating (a) in the context of (b,c,d) would return a &quot;true&quot; result with the variable substitution {χ&#x2F;α,υ&#x2F;β}, i.e. (a) is true iff χ = α and υ = β.<p>And that&#x27;s how Horn clauses and definite clauses become &quot;rules&quot;, &quot;facts&quot; and &quot;queries&quot;.<p>Next time: how the leopard got its stripes and the hippopotamus learned to love the first order predicate calculus.<p>_________________<p>[¹] This is my (first) slight fudge because constants are also functions, with 0 arguments. So, to be formal, the typical datalog above has &quot;no functions of arity more than 0&quot;.<p>[²] Negation-as-failure makes a language non-monotonic, in the sense that introducing new &quot;facts&quot; can change the meaning of a theory, i.e. a program.<p>[³] So, its Least Herbrand Model, or its Least Fix-Point (LFP).<p>[⁴] <i>Because</i> it finds the LFP of the query and the program.<p>[⁵] Unless evaluated by SLG resolution, a.k.a. tabling, similar to memoization.<p>[⁶] Although <i>higher-order</i> datalogs, that allow for predicate symbols as terms of literals <i>have</i> UTM expressivity, indeed a UTM can be defined in a higher-order datalog fragment where clauses have up to two body literals with at most two arguments:<p><pre><code> utm(S,S) ← halt(S). utm(S,T) ← execute(S,S₁), utm(S₁,T). execute(S,T) ← instruction(S,P), P(S,T). </code></pre> Originally in:<p>Tärnlund, S.-A. (1977). Horn clause computability. BIT Numerical Mathematics, 17(2), 215–226.<p>[⁷] Less fudgy, definite programs are refutation complete under SLD resolution, meaning that any atom that is entailed by a definite program can be derived by SLD resolution. A definite program is a set of definite clauses, explanation of which is coming right up.<p>[⁸] Long time ago, I explained this to a colleague who remarked that all the nice syntactic elegance in query languages falls apart the moment you try to make a query, which usually has a very different syntax than the actual rows of the tables in the database. So I said &quot;that&#x27;s the point! Queries are also Horn clauses!&quot; and his immediate remark was &quot;That just blew my mind&quot;. It&#x27;s been so long and I&#x27;m so used to the idea that I haven&#x27;t a clue whether this is really mind blowing. Probably, being my usual excited self, I just said it in a way that it sounded mind blowing (gesticulating widely and jumping up and down enthusiastically, you can picture the scene) so my colleague was just being polite. That was over drinks at the pub after work anyway.<p>[⁹] Resolution is an inference rule that allows the derivation of new atoms from a set of clauses. In theorem proving it&#x27;s used to refute a goal clause by deriving the empty clause, □. Since a goal is a set of negated literals, refuting the goal means basically that the negated literals are true. So our query is true in the context of our program.<p>[¹⁰] Datalog&#x27;s bottom-up evaluation uses something called a TP operator. It basically does what I said above, starts with the ground atoms in a program and then derives the set of consequences of the clauses in the program. In each iteration, the set of consequences are added to the program and the process is repeated, until no new consequences are derived. As stated above, the process is guaranteed to complete because every datalog definite program has a least fixpoint, which is also its Least Herbrand Model (we won&#x27;t go into Herbrand Models and Herbrand interpretations, but, roughly, an LHM is the smallest set of atoms that make the union of a definite program and a goal true). A more complete introduction to LHMs and LFPs and how they are used in bottom-up evaluation for datalog can be found here:<p><a href="https:&#x2F;&#x2F;www.doc.ic.ac.uk&#x2F;~mjs&#x2F;teaching&#x2F;KnowledgeRep491&#x2F;Fixpoint_Definite_491-2x1.pdf" rel="nofollow">https:&#x2F;&#x2F;www.doc.ic.ac.uk&#x2F;~mjs&#x2F;teaching&#x2F;KnowledgeRep491&#x2F;Fixpo...</a>
评论 #23891361 未加载
评论 #23894465 未加载
评论 #23888669 未加载
coderdd将近 5 年前
With <a href="https:&#x2F;&#x2F;github.com&#x2F;TreeTide&#x2F;underhood" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;TreeTide&#x2F;underhood</a>, my goal is to provide a read-only view of code, geared for understanding and debugging. Having to maintain the ability to edit comes with constraints.
geordimort将近 5 年前
Sorry to disappoint but for people doing programming language semantics there’s nothing new here. The hard part is always to go from the made-up language to a set of multiple full-blown languages.
tom_mellior将近 5 年前
&gt; I want to see how easy it is to add more advanced features to FP, like generics<p>This will be interesting to see, since the straight-forward implementation of generics for functional programming languages uses unification, which is not available in Datalog. It will probably be possible to encode things for any given program, since its universe of types should be finite. But it will involve jumping through hoops to encode something equivalent to unification.
pgt将近 5 年前
Ask yourself: what is the difference between a database and a programming language? The question is worth exploring.
airocker将近 5 年前
I think the biggest win here could be transactions. One commit can be worked upon by more than one person at once.
LeonB将近 5 年前
The .net dependency and code maintenance tool “NDepend” has a feature called CodeQuery that this reminded me of.<p>You can write queries such as:<p>from m in Application.Methods where m.NbLinesOfCode &gt; 30 &amp;&amp; m.IsPublic select m
jtwaleson将近 5 年前
I was very confused as I had read the title as &#x27;... with Datadog&quot; (the monitoring tool).<p>Anyway, I think both are a good idea. Datadog -&gt; integrating monitoring information back into the IDE. Datadog -&gt; working with the actual semantics of the code rather than just blobs of text.