Show HN: Local Node.js app to save everything you browse and serve it offline

It&#x27;s a bit disappointing to see other people showcasing their own work without a single mention of the link above. Perhaps make your own submission instead if that&#x27;s the intention?<p>As a casual reader and others obvious interest in this area I&#x27;d very much prefer a sentence or two about the quality of the work presented, feel free to link your own stuff afterwards, it&#x27;s a bit offputting to see such blatant self-promotion.

Congrats on getting your project to the front page of HN. With that said I think you are going to need to change your approach if you want this project to be usable as more than a toy project in the long run.<p>From what I can tell it essentially saves a map of url -&gt; response in memory as you browse. Every 10 seconds this file is serialized to json and dumped to a cache.json file. This is going to be very inefficient as the number of web pages indexed grows since you are rewriting the entire cache every 10 seconds even if only a few pages have been added to it. It also will eventually exceed the memory of the computer running the app if the content of every page ever visited needs to be loaded into memory. I highly recommend looking into some of the other suggestions mentioned here, either sqlite or mapping a local directory structure to your caching strategy so that you can easily query a given url without keeping the entire cache in memory, and also add &#x2F; update urls without rewriting the entire cache.

You could store the data in a git repo per domain, so that implicit de-duplication happens on re-visits &amp; for shared resources.<p>You could have a raw dir (the files you receive from the server) and a render dir that consists of snapshots of the DOM + CSS with no JS &amp; external resource complexity.<p>When the global archive becomes too big, history could be discarded from all the git repos by discarding the oldest commit in each repo, and so on.<p>SOLR is probably the right tool for the index but there is something undeniably appealing about staying in the pure file paradigm - you could use sqlite&#x27;s FTS5 module to do that too.

what are the security implications of permanently running chrome in remote debug mode?<p>a bit more than half a year ago I started playing around with this and was surprised how on the one hand there are really really good tools nowadays for self archiving but how on the other hand there hasn&#x27;t been any progress in implementing these in a, for end users, comfortable way<p>My working theory right now is that saving every request&#x2F;response as well as every interaction on a page should allow us to completely restore web site state at any point in time and will open up some super interesting use cases around our interaction with information found online<p>But in order to do this it seems necessary to go through the remote debug protocol like this project here is doing. And since this is somewhat of an unusual approach I could not find much information about the security aspect of running every site at any time with remote debugging activated. Common web scrapers&#x2F;archiving tools will instead only use remote chrome debug to open and capture specific urls<p>Storage is so dirt cheap today that there is zero reason why we shouldn&#x27;t have reliable historic website state for everything we have ever looked at<p>And judging by the HN front pages of the last months, many here are interested in this and related use cases (search&#x2F;index&#x2F;annotations&#x2F;collaborative browsing)

Like 20 years ago I&#x27;ve used a program called Teleport Pro, to do something similar.<p>I would dial up with my phone modem when the internet access was cheap (during the night), it would automatically browse a page I provided, and in the morning I would have the page ready to read.<p>Fun times with 10 to 20 kb&#x2F;s speeds.

I’m curious of why did you go on the path of using chrome debugging functionality instead of implementing an HTTP proxy which would provides the benefit of behind browser-agnostic too.<p>May you expand on that please?

Great feature. Though it feels like a UI misstep that the user had to use npm to switch between recording and browsing. A nicer solution could be a chrome extension button, or access the archived version via a synthetic domain. E.g. example.com.archived

Great idea! I like the concept.<p>One of the things I miss most about the old web was how trivial it was to local mirror any website. It was great!

I remember I tried something similar a long time ago but decided it wasn&#x27;t worth it.<p>2MB per page at 100 page a day, 200MB &#x2F; day. That is 73GB per year.<p>May be once a year I get the problem where I remember reading something about it but could not google back the exact page in my memory. So I had a proxy solution set up, but the maths ends up it wasn&#x27;t worth paying the Storage cost just for this one time convenient.

Between this project and the others mentioned in the discussion these are excellent resources for anyone needing to have a forensic record of their how they assembled evidence from browsing open sources on the internet. Package this as a VM that can be quickly spun up new and fresh per case and sell support for LE types and you’ve got a business.

Nitpicking but am I the only one who hates &quot;serve&quot; being used in strange contexts? IMHO to serve is to send something over a network. If it&#x27;s all happening locally, the verb should be &quot;load&quot; because it&#x27;s just taking a file and loading it into a browser at that point.

Really brilliant implementation concept.<p>I love how it uses the browsers debug port to save literally everything. I have often dreamed of “a Google for everything I’ve seen before”.<p>I recently spent some time making something like this and hope to release it soon as FOSS. However, it differs in some critical ways.<p>I desire to:<p>- save pages of interest, but not a firehose of everything I ever see<p>- save from anywhere on any internet device (eg mobile phone)<p>- Archive rich content like YouTube videos or songs even if I do not watch the entire (or any of) the video, and supporting credentials (eg .netrc)<p>Looking forward to digging deeper into this thread and your project for more ideas!

You should add &quot;upload &#x2F; sync with decentralized storage&quot; to the future goals.<p>Seems like a logical next step to have it sync to an IPFS or Dat drive. Not sure how it would be implemented though.

I love how there&#x27;s only a single browser or two in the entire world, lol (safari I&#x27;ve got bo clue about) that&#x27;s while assuming chrome and firefox&#x27;s debugging streams would be compatible....<p>you assume I don&#x27;t use any forks, or custom versions, what if I use an Electron based browser? what about Pale Moon or other forks which have older, if any, such interfaces? what about Opera? etc etc, you get the point...I hope...

Bump for my related project: <a href="https:&#x2F;&#x2F;github.com&#x2F;CGamesPlay&#x2F;chronicler" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;CGamesPlay&#x2F;chronicler</a><p>I&#x27;m actually in the process of rewriting this. I like your approach of using DevTools to manage the requests, the approach taken in Chronicler is to hook into Chrome&#x27;s actual request engine.<p>You might like to look at Chronicler to see some attempts at UI for a project like this, particular decisions around what to download and how to retrieve it.

Related: I&#x27;m making a program in Go to inline all the resources for a web page so it ends up being a single file that you can work with offline more easily: <a href="https:&#x2F;&#x2F;github.com&#x2F;ijt&#x2F;inline" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;ijt&#x2F;inline</a>.

I&#x27;ve been building something similar, but that uses Firefox sync to grab history and bookmarks. <a href="https:&#x2F;&#x2F;github.com&#x2F;jimktrains&#x2F;ffsyncsearch" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;jimktrains&#x2F;ffsyncsearch</a>

If anyone would be interested in the next major version, please add your email to this list to be notified: <a href="https:&#x2F;&#x2F;forms.gle&#x2F;FJmsXCDy18RrbFtt9" rel="nofollow">https:&#x2F;&#x2F;forms.gle&#x2F;FJmsXCDy18RrbFtt9</a>

Nice job, I think this is promising but there has got to be a better way than having people enable their debugger. Is there any reason you can&#x27;t just copy the contents of each page and then post it somewhere?

Why not use a proxy?

Related wwwoffle: <a href="http:&#x2F;&#x2F;www.gedanken.org.uk&#x2F;software&#x2F;wwwoffle&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.gedanken.org.uk&#x2F;software&#x2F;wwwoffle&#x2F;</a>

could this beat google? local search of anything i have seen, plus silo search sites for specific purpose like amazon and HN. would you miss anything given that google results are either bought or gamed? maybe need a better social media

Obligatory bump for my project ReadableWebProxy (<a href="https:&#x2F;&#x2F;github.com&#x2F;fake-name&#x2F;ReadableWebProxy" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;fake-name&#x2F;ReadableWebProxy</a>) that was originally intended to do this.<p>At this point, it does a GIANT pile of additional things most of which are specific to my interests, but I think it might be at least marginally interesting to others.<p>It does both full autonomous web-spidering of sites you specify, as well as synchronous rendering (You can browse other sites through it, with it rewriting all links to be internal links, and content for unknown sites fetched on-the-fly).<p>I solve the javascript problem largely by removing all of it from the content I forward to the viewing client, though I do support remote sites that load their content through JS via headless chromium (I wrote a library for managing chrome that exposes the entire debugging protocol here: <a href="https:&#x2F;&#x2F;github.com&#x2F;fake-name&#x2F;ChromeController" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;fake-name&#x2F;ChromeController</a> <a href="https:&#x2F;&#x2F;pypi.org&#x2F;project&#x2F;ChromeController&#x2F;" rel="nofollow">https:&#x2F;&#x2F;pypi.org&#x2F;project&#x2F;ChromeController&#x2F;</a>).