TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Autodoc: Toolkit for auto-generating codebase documentation using LLMs

166 点作者 funfunfunction大约 2 年前

16 条评论

SilverBirch大约 2 年前
I have a conceptual problem with this. Documentation is meant to describe stuff thats&#x27;s <i>not in the code</i>. Sure, there&#x27;s the odd occasion where you&#x27;ve done some super weird optimization where you want to say &quot;Over here I&#x27;ve translated all my numbers into base 5, bit reversed them and then added them, mathematically this is the same as just adding them in base 9, but fits our custom logic cell better&quot;. But that&#x27;s the exception, the general purpose of documentation is to describe <i>why</i> it&#x27;s doing hwat it&#x27;s doing, not how. Tell me that this module does X this way because that helps this other module do Y&quot;. Tell me why you&#x27;ve divided the problem this way. You&#x27;re giving information about why certain design decisions were made, not just describe what they are.<p>It doesn&#x27;t matter how good your LLM is, the information simply isn&#x27;t there for it to know the information it needs to document. You&#x27;re never going to get a comment out of this that says &quot;This interface is meant to be backwards compatible with the interface Bob once wrote on a napkin in the pub on a particularly quiet friday afternoon when he decided to reinvent Kafka&quot;.
评论 #35301307 未加载
评论 #35302291 未加载
评论 #35302317 未加载
评论 #35306810 未加载
评论 #35303620 未加载
divan大约 2 年前
Just tested it on a side-project codebase.<p>Main impression - it does hallucinate like crazy. I asked &quot;How does authorization of HTTP request work?&quot; and it started spitting explanation of how user bcrypt hash is stored in SQlite database and token is stored in Redis cache. There are no signs of SQLite or Redis whatsover on this project.<p>In other query it started confidently explaining how `getTeam` and `createTeam` functions work. There are no such entity or a word &quot;team&quot; in the entire codebase. To add to the insult, it said that this whole team handling logic is stored in `&#x2F;assets&#x2F;sbadmin2&#x2F;scss&#x2F;_mixins.scss`.<p>Other time it offered extremely detailed explanation of some business-logic related question, linking to a lot of existing files from the project, but that was completely off.<p>Sometimes it offered meaningful explanations, but was ignoring the question. Like I ask to explain relation between two entities and it started showing how to display that entity in a HTML template.<p>But I guess it&#x27;s just a question of time when tools like this become a daily assistant. Seems invaluable for the newcomers to the codebase.
评论 #35300979 未加载
评论 #35301787 未加载
评论 #35302173 未加载
评论 #35302028 未加载
ch33zer大约 2 年前
Most interesting part to me, the prompts:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;context-labs&#x2F;autodoc&#x2F;blob&#x2F;83f03a3cee62d6e00a440c20890badfe0cf1aa96&#x2F;src&#x2F;cli&#x2F;commands&#x2F;index&#x2F;prompts.ts#L3">https:&#x2F;&#x2F;github.com&#x2F;context-labs&#x2F;autodoc&#x2F;blob&#x2F;83f03a3cee62d6e...</a><p>&gt; You are acting as a code documentation expert for a project called ${projectName}. Below is the code from a file located at \`${filePath}\`. Write a detailed technical explanation of what this code does. Focus on the high-level purpose of the code and how it may be used in the larger project. Include code examples where appropriate. Keep you response between 100 and 300 words. DO NOT RETURN MORE THAN 300 WORDS. Output should be in markdown format. Do not say &quot;this file is a part of the ${projectName} project&quot;. Do not just list the methods and classes in this file. Code: ${fileContents} Response:
评论 #35299568 未加载
评论 #35298944 未加载
hedora大约 2 年前
This has far surpassed my dystopian predictions of how people would misuse LLMs.<p>Self spamming your own code base with comments that are either obvious, misleading or wrong was previously unfathomable to me.<p>Most people think I’m unrealistically pessimistic.<p>Well done.
评论 #35299303 未加载
评论 #35298921 未加载
评论 #35298699 未加载
评论 #35299373 未加载
评论 #35298706 未加载
评论 #35299813 未加载
评论 #35299814 未加载
评论 #35298675 未加载
评论 #35298910 未加载
andrewmcwatters大约 2 年前
I&#x27;m far, far more interested in having an LLM tell me where particular functionality is in a codebase, and how it works from a high level.<p>Autogenerating function documentation seems like such a low bar by comparison. It&#x27;s like taking limited creativity and applying it with high powered tools.<p>Literally like asking for a faster horse.<p>Tell me how WebKit generates tiles for rasterizing a document tree. Show me specifically where it takes virtualized rendering commands and translates them into port specific graphics calls.<p>Show me the specific binary format and where it is written for Unreal Engine 5 .umaps so that I can understand the embedded information for working between different types of software or porting to different engines.<p>Some codebases are so large that it literally doesn&#x27;t matter if individual functions are documented when you have to build a mental model of several layers of abstraction to understand how something works.
评论 #35299789 未加载
评论 #35300564 未加载
golem14大约 2 年前
Today, LLMs learn from a codebase that has mostly insightful comments from well-meaning humans.<p>In the future, the training sets will contain more and more automatically generated stuff I believe will not be curated well, leading to a spiral of ever declining quality.
评论 #35301446 未加载
评论 #35300305 未加载
ch33zer大约 2 年前
One thing I find worse than no docs is wrong docs.<p>It would be really cool if we could take code + docs, feed it into an LLM and get a determination of whether the code matches what&#x27;s in the docs. It could also be a good way to evaluate the correctness of the generated docs from the linked tool (assuming it works).
userbinator大约 2 年前
If docs can be generated, they&#x27;re not worth reading.
评论 #35299351 未加载
评论 #35302429 未加载
评论 #35299202 未加载
rkagerer大约 2 年前
Come talk to me when it can reverse engineer as good as Ken Shirriff, develop a complete understanding of the whole codebase, and generate authoritatively accurate and useful output. Oh, and uncover bugs while it&#x27;s at it.
评论 #35299313 未加载
评论 #35300214 未加载
评论 #35299370 未加载
评论 #35301459 未加载
评论 #35300946 未加载
zx8080大约 2 年前
How to verify the meaning of docs? How to deal with the model hallucinations?<p>It would be hell to lose trust to api docs due to those risks.
评论 #35299688 未加载
评论 #35298697 未加载
评论 #35298663 未加载
petesergeant大约 2 年前
Example output: <a href="https:&#x2F;&#x2F;github.com&#x2F;context-labs&#x2F;autodoc&#x2F;blob&#x2F;master&#x2F;.autodoc&#x2F;docs&#x2F;markdown&#x2F;src&#x2F;index.md">https:&#x2F;&#x2F;github.com&#x2F;context-labs&#x2F;autodoc&#x2F;blob&#x2F;master&#x2F;.autodoc...</a>
评论 #35299143 未加载
verdverm大约 2 年前
Please run this on the BuildKit code base which has almost no comments but a huge usage footprint. It&#x27;s also (obviously) a non-trivial test case
评论 #35298840 未加载
评论 #35298661 未加载
smrtinsert大约 2 年前
I&#x27;m more interested for LLMs to get the point when it can look at the several hundred codebases in your company and tell you who sets what value in your local data model, and why they set it. There&#x27;s always slack instead of poorly generated documentation.
splatzone大约 2 年前
This is extremely interesting. I have a few monstrous repositories I’d like to try it on.<p>The thing I’m wondering about is the cost. How much would it cost to run this on the entire WordPress source, for example?
评论 #35298718 未加载
whiplash451大约 2 年前
Very cool stuff.<p>I think people who dismiss this kind of tool because it can hallucinate stuff are off topic.<p>The AI will get better and better, but more importantly we will evolve and learn to work with this kind of tool.
ImageDeeply大约 2 年前
Granting that some hallucination is likely, still seems like a step in the right direction.