A Raspberry Pi as a decent residential proxy

&gt; Don’t expose the proxy to the world or attackers will be able to interact with your home devices<p>Or use it to get stuff on the internet that they don&#x27;t want to get directly themselves. Once upon a time I accidentally enabled open proxying on the web server on my Linux box. This was before smart phones and tablets and such, when my Linux box was the only network client I had, and so I didn&#x27;t have a NAT router. Just modem straight to Linux box.<p>One day I checked my Apache logs to see if my extremely low traffic web site had any visitors.<p>First impression: &quot;What the hell is all this traffic? This is orders of magnitude more than I should have!&quot;.<p>Second impression: &quot;Why the hell is someone trying to get horse_fucking.mov from my server?&quot;.<p>Third impression: &quot;That didn&#x27;t give a 404. Why is my server actually providing horse_fucking.mov?&quot;.<p>(No, the orders of magnitude extra traffic were not all people getting horse_fucking.mov. There was a whole plethora of bestiality titles, plus a lot of other porn that you would not want to fetch directly from your own IP address either out of embarrassment if you got caught or because it was almost certainly illegal).

I run a number of news website scrapers for various projects. Originally I ran them all on AWS EC2 instances and racked up sizeable bills. Recently though, I pulled an unused Raspberry Pi from my desk drawer, installed them all onto the Pi, and now run them for near free from my desk! Granted I need to be a little bit more efficient with memory and processor cycles, but that’s a fun little challenge. I find if you respect the websites in terms of not hitting them multiple times a minute (or only once an hour which I do) then you’ll very seldom get banned.

Have you looked at running wireguard on your server and connecting to it from the pi. Then you have a vpn between server and the pi where you can send data in any protocol

Why not just use the SSH dynamic SOCKS5 proxy?<p>Just ssh -D 12345 raspberry and then just use localhost:12345 as the scrapers SOCKS5 proxy. No need to involve a http proxy.<p>EDIT: AAh, ssh port is not accessible from outside and the raspi does a reverse connect.

For more reliable ssh tunnels you can use autossh.<p><a href="https:&#x2F;&#x2F;www.harding.motd.ca&#x2F;autossh&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.harding.motd.ca&#x2F;autossh&#x2F;</a>

That is a very retro raspberry pi :-D

Is there a solid solution to the SD card problem? Every Rpi I&#x27;ve tried to use as a server eventually kills its SD card, probably as running an os on SD is destined for failure with write wear. I&#x27;m guessing using SD for boot only and then using a home NAS would be an option, but I&#x27;ve not come across a dummies guide to making a Rpi reliable long term.

&gt; Surprisingly, 256 MB in RAM have been enough to maintain the app running for months until now.<p>What is total RSS of all your proxy processes?

I setup an Openvpn server on my LAN, with username&#x2F;password credentials so I can access my LAN from anywhere. I have multiple vpn configurations for my cell phone&#x2F;laptop to either map only 192.x through the VPN or all traffic through it. It&#x27;s very very useful, and seems like a much more secure option in my opinion.

I still keep a server running in my basement doing WireGuard since Netflix et al. all block cloud IPs&#x2F;ASNs

Why not use a normal HTTP proxy like Squid? It would be more easy to use it since almost all languages and clients support HTTP proxy built-in.