Personal computing paves the way for personal library science

I work in a very interdisciplinary, and somewhat niche, tech&#x2F;engineering field. For the past 15 years, I&#x27;ve been saving every relevant PDF that I can find -- mostly studies of the sort published by Elsevier and Springer, but also books and presentations. I now have around 10k, which probably makes it the largest private library focused on this particular domain of expertise.<p>It has been extremely useful, especially because it&#x27;s text-searchable and the really important papers are properly categorized.<p>A local LLM will make it 100x more useful. Also, it might not even need be &quot;local.&quot; If I make it available via the web, I can probably sell access to other scientists and engineers in my field.<p>Recent advances <i>really</i> benefit data hoarders out there.<p>I&#x27;d add that these days it totally makes sense to download libgen&#x27;s entire archive, because (1) storage has never been cheaper, and (2) you can use it to train local LLMs.

&gt; The &quot;during&quot; is hard work, and very lonely work. There are no promises of success, and indeed, the path is one where you can&#x27;t see more than three feet ahead of you and you exist on the cliff&#x27;s edge of extinction by any silly mishap. The work of &quot;during&quot; is exhausting, and it constantly holds you taut and alert, afraid of the shadows that lurk beyond the campfire&#x27;s edge.<p>Well said. All anyone can do is to do the lonely work till you can&#x27;t anymore or you find friends to not be lonely at that work anymore.

there may be an important divergence implied by this essay .. people here ask about using an LLM.. but the essay refers to &quot;different photographs of the same scene from different photographers&quot; or other personal collection items that are related but subjective or not-authoritative<p>There is a rush in public to condense and summarize many authoritative publications to find patterns, or to replace a human expert with automated results.. yet that is fundamentally different than taking multiple incomplete perspectives to add to a human library-owners knowledge and investigations.<p>It is subtle to speak it but not subtle in its implications.. taking &quot;data as facts&quot; and condensing them or reordering them or rewriting an output based on them, using automation, is different than a human mind taking in many inputs for human mind knowledge and enabling new outputs from a human author.

I&#x27;ve had related ideas lurking at the back of my mind for a while now. Essentially, I want to save more things locally and and interact with it. For example, I have a bunch of book notes stored in Bear. I&#x27;d like to be able to ask questions about those notes, and also show the pages of the book itself.

Personal computing has stagnated for such a long time, it creates substantial uncertainty about what state it might evolve to if and when the next step actually happens.<p>In this respect local LLM&#x27;s are simply the tip of the iceberg, pointing out the vast amount of personal information processing that is available in principle but does not actually happen.

<i>&gt; Personal Library Science is the leverage of LLM technology, applied to a personal library. A personal library differs from a impersonal library in the fact that a personal library is an interpretation of a source material. These interpretations include: photographs from different photographers at the same event, or favorite scenes from a movie, or favorite passages from books, parts of songs that bring you to tears, etc. Importantly, these interpretations create unique sets that go on to create unique problems which require unique, idiosyncratic solutions.</i><p>Would an LLM-driven &quot;Personal Library&quot; require manually annotated textual interpretation of each curated item, or could it derive personal interpretations from user history and the uniqueness of curated items&#x2F;sets?<p>For those who have been using local, offline LLMs with a manually curated text&#x2F;image corpus, what have been the most valuable or surprising use cases?<p>Author demo video (2023), <a href="https:&#x2F;&#x2F;youtube.com&#x2F;watch?v=7TgqMRz2r3M" rel="nofollow">https:&#x2F;&#x2F;youtube.com&#x2F;watch?v=7TgqMRz2r3M</a> &amp; tooling comment (2024), <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39789712">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39789712</a><p><i>&gt; Inspired by the commonplace book format, I take highlights from Kindle and embed them in a DB. From there I build (multiple) downstream apps but the central one, Commonplace Bot is a bot that serves as a retrieval and transformer for said highlights.</i><p>Related: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Lifelog" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Lifelog</a>

Years ago I spent thousands of hours trying to figure out how to organize a digital library.<p>My final answer was to use the Library of Congress catalog system. They need to add some sub-categories for how-to explanations.<p>Then have a field for media type (video vs. PDF vs. image)<p>Then note the style of presentation (academic vs. folksy vs. a manual vs. a dad showing you how to do this)<p>Then note the language

Not too long ago, I managed to pretty much ruin the wiki for a small (and at that time opensource) CNC machine by using it as my personal notebook --- probably my usage of it thus was a big part of why it was left off-line when the person hosting it moved.<p>You can see it on the Wayback Machine:<p><a href="https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20211127090321&#x2F;https:&#x2F;&#x2F;wiki.shapeoko.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20211127090321&#x2F;https:&#x2F;&#x2F;wiki.shap...</a><p>In retrospect, I should have put some of that effort into:<p><a href="https:&#x2F;&#x2F;en.wikibooks.org&#x2F;wiki&#x2F;Hobbyist_CNC_Machining" rel="nofollow">https:&#x2F;&#x2F;en.wikibooks.org&#x2F;wiki&#x2F;Hobbyist_CNC_Machining</a><p>although since then, a machine owner worked up:<p><a href="https:&#x2F;&#x2F;shapeokoenthusiasts.gitbook.io&#x2F;shapeoko-cnc-a-to-z" rel="nofollow">https:&#x2F;&#x2F;shapeokoenthusiasts.gitbook.io&#x2F;shapeoko-cnc-a-to-z</a><p>I still regret a bunch of stuff I didn&#x27;t keep copies of, esp. the scans of Barry Hughart&#x27;s notes for his novels.<p>The irony is that one can see a bit of the result of discussion of this sort of thing at the top of one&#x27;s browser window --- the URL bar, where URL == &quot;Uniform Resource Locator&quot; --- the originally proposed term was &quot;Universal Resource Locator&quot;, but the argument against that was that people were not librarians, and that unlike Ted Nelson&#x27;s Xanadu, there wouldn&#x27;t an over-arching data structure and organization, so a given document wouldn&#x27;t have a single canonical location.<p>Anyone interested in this sort of thing who hasn&#x27;t read it, should read Tim Berner-Lee&#x27;s book:<p><a href="https:&#x2F;&#x2F;www.w3.org&#x2F;People&#x2F;Berners-Lee&#x2F;Weaving&#x2F;Overview.html" rel="nofollow">https:&#x2F;&#x2F;www.w3.org&#x2F;People&#x2F;Berners-Lee&#x2F;Weaving&#x2F;Overview.html</a>

Best quote from the article: <i>&quot;...personal library science is focused on your relationship with your information. How do we store information so that it useful at a later date? How do we transform our information into new valuable assets in different creative domains? How do we do all of this while being flexible enough for the idiosyncrasies, proclivities, likes and dislikes of eight billion distinct individuals? How do we chronicle the information diet of a single person as they learn new things, interact with the world at different phases in their life? How do we make sure we can pass down our best knowledge to generations below?&quot;</i>

For me, on macOS, FoxTrot Professional has been my personal file data indexer. Its essential go-to feature for me is sophisticated searching, including a form of regex. Also, true wildcards, no stopwords, and proximity searches put it far out in front of anything else I have tried, including many recent LLM local-docs tools. I have millions of files in dozens of formats in hundreds of GB gathered over decades (many digitized by me) and it handles it all like a champ, though an SSD drive and a late-model Mac is a must at that size. And backups, cause ain&#x27;t nobody want to lose that.

If one remembers NeXT included all sorts of non-computer documents and literature. The idea of storing vasts amounts of data for personal use was at the dawn of the PC era.