HTTrack Website Copier

This brings back so many fond memories. I grew up in a rural part of Kenya where the internet was scarce and tech practically non-existent. I was interested in web dev and taught myself PHP using HTTrack to download the php manual site, then the cprogramming.com website. I remember writing these site contents onto a thick notebook to read in school. Cprogamming.com imho was my programming foundation as I treated it as programming gospel. That kid back then would be shocked at how far I've come, now a dev at MS. Not sure how I came across httrack back then but I am so glad I did

I’ve used it a few times to “secure” an old but relevant dynamic website site. Like a site for a mature project that shouldn’t disappear from the internet but it’s not worth upgrading 5 year old code that wont pass our “cyber security audit” due to unsupported versions of php or rails so we just convert to a static site and delete the database. Everything pretty much works fine on the front end, and the CMS functionality is no longer needed. It’s great for that niche use case.

Not sure about the context on why this is on HN but it surely put a smile on my face. Used to use it during 56K era when I just download everything and read it. Basically using it as RSS before RSS was a thing.

I&#x27;ve been using HTTrack for almost two decades to create static archives of a yearly website for an annual event.<p>It doesn&#x27;t do the job 100% but it&#x27;s a start. In particular, HTTrack does not support srcset, so only the default (1x) pixel-density images were archived (though I manually edited the archives to inject the high pixel-density images, as well as numerous other necessary fix-ups).<p>The benefit of the tool is fine control over the crawling process as well as which files are included. Included files have their URLs rewritten in the archived HTML (and CSS) to account for querystrings, absolute vs. relative URLs, external paths, etc.; non-included files <i>also</i> have their URLs rewritten to change relative to absolute links; thus, you can browse the static archive, and non-included assets still function if they are online at their original URL, even if the static archive is on local storage or hosted at a different domain than the original site.<p>It was more work each year as the website gradually used script in more places, leading to more and more places I would need to manually touch-up the archive to make it browsable. The website was not itself an SPA, but contained SPAs on certain pages; my goal was to capture the snapshot of the initial HTML paint of these SPAs but not to have them functional beyond that. This was (expectedly) beyond HTTrack&#x27;s capabilities.<p>At least one other team member wanted to investigate <a href="https:&#x2F;&#x2F;github.com&#x2F;Y2Z&#x2F;monolith" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;Y2Z&#x2F;monolith</a> as a potential modern alternative.

Funny timing. Just yesterday I was looking for an easy Windows tool to do a simple stress-test on a website (legally ofc). A requirement of mine was to just give it the root URL and the tool should discover the rest automatically (staying on the same domain). Also, parameters like parallelism had to be easily manageable. After trying some crawlers &#x2F; copiers and other tools I went back to a simple one I already knew from saving static copies of websites in the past: HTTrack. It fit the bill perfectly! You can add the root URL, set it to &quot;scan only&quot; (so it doesn&#x27;t download everything) and tweak the settings like connections and speed (and even change some settings mid-run, save settings, pause, ...). So thanks xroche for HTTrack! :)

I used this recently to download websites, stuffed them into a sqlite db, processed them with Mozllia&#x27;s readability library, and then used the result and an llm to ask questions of the webpage itself.<p>It was helpful to take each step in chunks, as I didn&#x27;t have a complete processing pipeline when I started.<p>I had wondered if there was an easier or better way to do this, as I probably would have liked to get the sitemap, pass the sitemap to an llm, then only download selected html pages vs the entire website.

Or just do:<p><pre><code> wget -rkpN -e robots=off https:&#x2F;&#x2F;www.example.com&#x2F;</code></pre>

This isnt 1998 anymore so downloading the files from modern websites doesn&#x27;t really work if youre trying to maintain your own private local &#x2F; re-hosted copy of a site. especially ones with dynamically loaded content. Some additional processing is needed to fix the files. I have never been able to find a modern scraping solution that works with most modern websites. I suppose the existence of this sort of tool is in conflict of interest of Big Tech, for it would make the creation of visually identical looking phishing sites that easier much.

I used this all the time twenty years ago. Tried it out again for some reason recently, I think at the suggestion of ChatGPT (!), for some archiving, and it actually did some damage.<p>I do wish there was a modern version of this that could embed the videos in some of my old blog posts so I could save them entire locally as something other than an HTML mystery blob. None of the archive sites preserve video, and neither do extensions like SingleFile. If you&#x27;re lucky, they&#x27;ll embed a link to the original file, but that won&#x27;t help later when the original posts go offline.

Long time ago, HTTrack came very handy for me at work. We created a PHP&#x2F;Mysql application to store data for a census of industrial sites and related info. Some day my boss tell me the customer wants this census being delivered to them in a auto-startup CD-ROM which was very fashionable at that time, I used HTTrack to download every page of our PHP database and all be browseable offline from the CD-ROM, the auto-startup just launch the browser at the index page.<p>Very handy.

A few years ago my workplace got rid of our on-premise install of fogbugz. I tried to clone the site with HTTrack but did not work due to client-side JavaScript and authentication issues.<p>I was familiar with C#&#x2F;webview2 and used that: generate the URL’s, load the pages one by one, wait for it to build the HTML, and then save the final page. Intercept and save the css&#x2F;image request.<p>If you have ever integrated a browserview in a dektop or mobile app, you already know how to do this.

I thought there was an update but it just looks like a random share…so I’ll share something random!<p>My favorite forum back in the day was the Rockman.EXE Online forums and they had server issues a few times. I was afraid of it going offline for good and came across HTTrack. My laptop was crappy as hell so maybe that’s why I didn’t have the best experience with it?<p>Or maybe trying to backup a forum from the front-end wasn’t a good idea LOL.

Every techie that lived through dialup days was constantly downloading stuff to read offline. I&#x27;m usually not very nostalgic as I think even through crazy times like these, we are still improving, but I feel I had much better focus back then.<p>Of course this doesn&#x27;t translate into better productivity because we have way better tools today but it was nice to read, say, the gcc manual in one go.

I can&#x27;t recall the details, but this tool had quite some friction last time I tried downloading a site with it. Too many new definitions to learn too many knobs it asks you to tweak. I opted to use `wget` with the `--recursive` flag, which just did what I expected it to do out of the box: crawl all links you can find and download them. No tweaking needed, and nothing new to learn.

Related: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=27789910">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=27789910</a>

I still remember Netvampire <a href="https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;19990125091054&#x2F;http:&#x2F;&#x2F;netvampire.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;19990125091054&#x2F;http:&#x2F;&#x2F;netvampire...</a> , an old application I used to do the same in winNT days.

I used the hell out of this, back in the day. We would copy websites down, then use Swish-E from the command line to index everything from the file system, then somehow make a web interface out of it. Good times.

How does it compare to SiteSucker (&lt;<a href="https:&#x2F;&#x2F;ricks-apps.com&#x2F;osx&#x2F;sitesucker&#x2F;index.html" rel="nofollow">https:&#x2F;&#x2F;ricks-apps.com&#x2F;osx&#x2F;sitesucker&#x2F;index.html</a>&gt;)?

omg I used to use httrack to archive interesting sites at home, usual some sort of hobbyist boardgame thing or historical resource. I never kept them but the originals are long gone now, I should have!

The good ole days this vlc lots of other free yet functional quirky designed little tools with critical thought for use cases

Never found a great alternative of this for Mac.

No one has mentioned firecawl? Can anyone compare to archivebox, httrack?

In old time, I used to download entire website using HTTrack and read it later.

This doesn&#x27;t really work with most sites anymore, does it? It can&#x27;t run JavaScript (unlike headless browsers with Playwright&#x2F;Puppeteer, for example), has limited supported for more modern protocols, etc.?<p>Any suggestions for an easy way to mirror modern web content, like an HTTrack for the enshittifed web?

Nostalgia !!

Is this<p>wget --mirror<p>?

Time to add AI mode to this :).