Is there a tool/script that will allow me to filter out ads from a page when downloading it using curl. (Similar to how uBlock Origin works for a browser).<p>Basically, what I am doing is downloading a snapshot of a site using curl. But the sites have advertisements in them which I want to filter out. So is there a tool that will let me do that from the command line so that the output file doesn't have ads in it?<p>In short, I want something like uBlock Origin but for html files that I will be converting to PDF's or epubs. Something like:<p>curl https://www.google.com | AdRemover.sh | htmltopdf<p>Most of the solutions I found require you to update the /etc/hosts file to stop showing the ads but would rather avoid that if possible.
After taking a break and stepping away for a bit, I realized that I was recreating an archiving system for websites and that there are existing solutions that do the same thing.<p>I found <a href="https://github.com/ArchiveBox/ArchiveBox/">https://github.com/ArchiveBox/ArchiveBox/</a> which is a self hosted web archiving system. It covers most of my usecases (and I can extend it for additional functionality) so I am going to set this up and try it out.<p>Thanks all for the help.
Do you have to use Curl? It wouldn't render a lot of sites correctly anyway (anything that uses JS for rendering).<p>Can you run a puppeteer/playwright instance (which control real browsers) and add an ad blocker to that? e.g. <a href="https://github.com/ghostery/adblocker">https://github.com/ghostery/adblocker</a> or <a href="https://github.com/microsoft/playwright-python/issues/782">https://github.com/microsoft/playwright-python/issues/782</a>
Editing /etc/hosts is going to be the easiest option.<p>The best option would be to use a programming language and a good HTML parser to do the job. eg: Use Python and BeautifulSoup to dig through the tree looking for any HTML tag which references an ad-serving network.