Ask HN: Are there forgotten servers out there?

Yes - in my network admin days back in 2007, we were decommissioning our datacenter as we&#x27;d virtualized everything and as we were gutting the room we found our PABX&#x2F;Voicemail server under the tiles, happily running away with OS&#x2F;2. Nobody ever knew where it was and it had been installed 13 years before. The floor was on a UPS and generator so the power had never been disconnected.<p>It was happily chugging away, running our shitty phone system, and hadn&#x27;t been restarted in 10+ years.<p>Edit: This also happened to me at a utility company I worked at. We had a server that ran some critical calculations for us but nobody could recall where it was physically located. It&#x27;s way worse now that everything&#x27;s virtualized - the cruft just sits there for years until you suddenly run out of resources and start looking closely.

At a place I worked we had a service that relied on a cron job running a script every hour. One day someone wanted to know where the server was so he could connect an external hard drive to it and copy some files in. Not a single person knew where the server was or even what it&#x27;s IP address was. Since it had been setup years ago, everyone who had worked on it had since left the company and no one ever documented it. So somewhere either in our office or in one of three public clouds we used was a server happily running a script every hour that no one could locate or stop. We eventually found it when we moved offices and a super old Dell box in a tiny dusty closet was unplugged and the script stopped running. Now I always document the physical locations of services too.

Oh, Yes! In a previous job we developed VoIP servers (pbxes). One day a customer started experiencing some really weird problems and I was asked to debug it.<p>I logged into the server and started analyzing traffic, turned out that traffic on a upstream VoIP switch didn&#x27;t always match the traffic leaving the server I was debugging. It was as if a identical system was getting and responding to parts of the traffic. - and after some more debugging I discovered that there was a older identical system online somewhere in their server room. Years ago - all services had been stopped, the system backed up and migrated to a new server. One day there had been a power loss, and when the servers rebooted, the old system everyone had forgotten about launched it&#x27;s previously stopped services - causing the customer all sorts of weird VoIP problems.

I set up one of these deliberately in younger days...<p>As a bored government employee in the early 1990s, I become fascinated with the WWW. I was a network admin (NetWare 2.15c), and there was a fat, unused internet pipe and several unused phone lines. I started to mess around with Linux (Slackware kernel 0.97 I think), and after two weeks I had it talking to a Hayes modem. Voila, instant ISP! A little while later I installed Slirp, and it became my personal dial-up connection for many, many years.<p>Before I left that job in 1995, I moved the server (headless) to a broom closet (wrong of me, I know, I know). Knowing gov&#x27;t culture, no one would mess with something like that. It was up and running at least until 1998 or so, at which point I moved to another country, and when I moved back, I no longer had the dial-up number. I like to think it is running to this day.

There are many (possibly apocryphal) stories about servers lost in ceilings, locked closets, etc that just kept on serving, sometimes critical services.<p>I think we as an industry have gotten much better about this. In the old days, as small minicomputers and micros expanded into less and less technical businesses, wiring standards and server room design guides were not well-known or followed.<p>Often, people just sort of winged it. Some employees are more naturally methodical, have better memories, and are longer-tenured than others. Also, many of the stories are from universities, where long-term thinking isn&#x27;t guaranteed (but embarrassing stories have always been popular to share!).<p>A quick DDG finds this story from the University of North Carolina:<p><a href="http:&#x2F;&#x2F;www.bradleymonk.com&#x2F;wp&#x2F;weve-lost-server-54&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.bradleymonk.com&#x2F;wp&#x2F;weve-lost-server-54&#x2F;</a><p>USENET archives would be a great source for more of these.

This guy&#x27;s server: <a href="http:&#x2F;&#x2F;bash.org&#x2F;?5273" rel="nofollow">http:&#x2F;&#x2F;bash.org&#x2F;?5273</a><p>&lt;erno&gt; hm. I&#x27;ve lost a machine.. literally _lost_. it responds to ping, it works completely, I just can&#x27;t figure out where in my apartment it is.

This week I got a support call from one customer: &quot;hey, we have a maintenance scheduled for our Dell servers, is it ok if we shut down the two pre-prod servers?&quot; &quot;what pre-prod servers?&quot; [all the non-production servers were virtualized about 4 years ago] &quot;well, there are two servers here, with pre-prod labels and running Linux; no applications are running on them&quot;<p>So I guess 4 years ago somebody migrated the servers to a virtual environment and then just forgot about them. The IPs were migrated to the VMs and these servers were left without an IP address on the network connections, so apart from going to the data room and checking every machine (what they did during the maintenance) there was no way to find them. They were still up and running after 4 years.

Oh god yes. A few years ago some labmates and I discovered that a post doc had set up an entire computer cluster of 30 workstations &#x2F; associated infrastructure, colocated it at a data center nearby, then left without leaving any documentation. Everyone that knew the cluster existed left soon after, and so labmates that arrived later were running simulations on their personal laptops while this cluster sat idle. We discovered it only because my PI received a large bill for rack space and remembered it existed.

Sure, happens now and then.<p>I work at an ISP &#x2F; housing &#x2F; colocation company, and occasionally hardware goes missing (nobody knows where it is anymore, it&#x27;s not where it&#x27;s supposed to be). Maybe some of them are broken, some might be stolen, but I&#x27;m fairly sure others are still running, not serving any purpose.<p>And from time to time we stumble over some virtual machine (or even phyiscal server) where nobody knows anymore what it&#x27;s supposed to do; the standard procedure seems to be firewall it off, wait for a few weeks or months to see if anybody complains, and if not, shut it down, maybe archive it.

My father has an email account hosted by Verizon with the domain @gte.net. General Telephone &amp; Electric merged with Bell Atlantic in 2000 to become Verizon.<p>My father&#x27;s @gte.net email delivery has recently become spotty. After hours of phone calls with Verizon, no one at any level of support can seem to find the old GTE mail servers.<p>Some Googling for smtp.gte.net found this IP address, 207.115.153.29, which seems to respond to pings but not smtp traffic.

I am sure there are tens of thousands if not more. Server admins are very risk averse and don&#x27;t tend to turn off machines just to check if someone&#x27;s using it.<p>And in large organizations, it&#x27;s hard to keep track of who&#x27;s using what and if they still need it.<p>Imagine working at a hospital or a bank, emailing people about a server, and everyone says it&#x27;s not being used. Then you turn it off and something critical gets broken at 3am a week later resulting in an emergency. Who gets blamed, you or the people who forgot the server was being used? The people who set it up may not even be working there anymore.

Years ago I shut down a start-up, and as part of some business partnership, we had a physical server hosted in some big company&#x27;s datacenter. They forgot about it, and I didn&#x27;t have time or incentive to drive over there to fetch one old machine. So it just kept running there for a couple years, at first just continuing to serve my failed company&#x27;s website, and then doing nothing, other than serving as a personal download proxy for me. That lasted for about 2-3 years until they finally found it and shut it off, so I took it home.

We came across one at work (NASA) a few years back, a server running in a far off room that no one remembered anything about. It had probably last been used by a grad student at least a decade ago, but was still up and running and doing, well, something. We eventually had it removed and turned the room into a collaboration space.

My 1996 homepage is still online under its original URL. Which likely means the system hosting it is still online. Except the original ISP no longer exists, nor does the ISP that bought the original ISP.<p>Next year I&#x27;m going to find that system on the 20th anniversary of that page.

I&#x27;ve always wanted to intentionally setup servers in hidden places just to see how long they stay online. For example, putting a Raspberry Pi in the back of a rack in a server closet. Maybe run a Tor relay on it or something like that.

Back in the late 80s &#x2F; early 90s there was a BBS in my hometown (Windsor, Ontario, Canada) called GB Hotel. It wasn&#x27;t very popular, but if all the other BBS lines were busy, it was something to do.<p>The owner of the BBS, handle &quot;Kilroy&quot; as I recall, hadn&#x27;t logged into his own BBS in _years_. Nor anyone else&#x27;s, for that matter.<p>We used to joke that he&#x27;d probably died, and Mom and Dad just left his Commodore 64 sitting there, wondering why their phone bill was twice as much as their neighbours.

Seen it a few times. Company acquires another company or an employees leaves. And then you find out years later during an office refit that the PC under their desk was running a critical business process.

<a href="http:&#x2F;&#x2F;www.theregister.co.uk&#x2F;2001&#x2F;04&#x2F;12&#x2F;missing_novell_server_discovered_after&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.theregister.co.uk&#x2F;2001&#x2F;04&#x2F;12&#x2F;missing_novell_serve...</a><p><i>One of the university&#x27;s Novell servers had been doing the business for years and nobody stopped to wonder where it was - until some bright spark realised an audit of the campus network was well overdue.</i><p><i>According to a report by Techweb it was only then that those campus techies realised they couldn&#x27;t find the server. Attempts to follow network cabling to find the missing box led to the discovery that maintenance workers had sealed the server behind a wall.</i>

I know of one interesting story here. Early on in (what would eventually become a very successful) tech company&#x27;s life, they acquired some collocated space for free thanks to a friend (call him Brad) of an early employee.<p>Eventually Brad left that particular DC operator, leaving the startup with no inside contact - but the servers stayed up for years to come (and the company was never charged for the resources).<p>These machines were eventually &quot;replaced&quot; with newer hardware in a nearby facility, but the DC operator has no idea where these old machines are physically located within the facility (thus cannot remove them), so they remain active to this very day, sitting idly by...

Previous company I was at had a small server setup to run a few automated tasks every morning. It must have been running for at least 10 years before a new employee asked where it was physically located.<p>They eventually found it covered in dust in a small utility closest. None of the fans were working either (other than the CPU&#x2F;PSU) yet it still chugged along doing its job.

I&#x27;m always fascinated by StorJ.<p><a href="http:&#x2F;&#x2F;garzikrants.blogspot.com&#x2F;2013&#x2F;01&#x2F;storj-and-bitcoin-autonomous-agents.html" rel="nofollow">http:&#x2F;&#x2F;garzikrants.blogspot.com&#x2F;2013&#x2F;01&#x2F;storj-and-bitcoin-au...</a><p>TLDR; a SaaS system that not only auto-scales, but does so autonomously. When it has excess funds, it buys more VMs and provisions them with itself. If cashflow is low, it shuts some down. Add in the ability to create accounts (using bitcoin as currency) and you could very quickly lose the paper trail and have a SaaS owned by the Æther.

There&#x27;s the story about the novell server left running in a walled-off room:<p><a href="http:&#x2F;&#x2F;www.theregister.co.uk&#x2F;2001&#x2F;04&#x2F;12&#x2F;missing_novell_server_discovered_after&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.theregister.co.uk&#x2F;2001&#x2F;04&#x2F;12&#x2F;missing_novell_serve...</a>

Not really &quot;doing anything&quot;, but related: I had to set up a voip server for a test project in 2003, which was supposed to be terminated half a year later. Being a test project, I was advised not to document anything formally, just to place it in the rack and ignore everything else. I left this job soon after.<p>12 years later, the phone number still is working, even though the companies phone system has been relocated to another room and was replaced by another vendor. No idea how this happened.

I heard a company forgot about their third datacenter (second backup), as several sys-admins left the company. It was found in an old bunker below the basement of a building.

Intriguing question; I would think that a certain percentage of servers might have survived their owners post-mortem. Of course there could be many additional reasons for abandoned servers.

AWS instances on the free tier that starts to charge your credit card after a year.

We have several gorgeous AS&#x2F;400s still going strong, quietly gathering dust while they wait for the next time we need to test something on them.<p>I think these servers will outlive me.

Well, every six months or so I review my various VPSes and I usually find at least one I&#x27;d forgotten about, so unless I&#x27;m unusually forgetful I&#x27;d say &quot;yes&quot; :)

Apart from normal servers, lots of embedded devices probably also have integrated servers. There must be many, many of them idling away somewhere.

Right after I was hired by a large portal part of a much larger telco, we were decommissioning one of our data centers. In one of the planning meetings there was a discussion about an undocumented class C range that nobody knew what was, yet had some significant traffic. I never knew what that network was or what was running there. As far as I know, someone may have lost their spam relays or their phishing hosts.<p>Another fun story happened years before. There was a large power outage at the data center of another large Brazilian portal. Three days later, someone calls our office, asking if we remember what kind of hardware the server was running on. The machine didn&#x27;t boot and they needed to get to its console. Unfortunately, nobody knew where the machine was physically located or what did it look like. In the end, it was found inside a Cubix chassis, an early blade-like machine.

I took over a job at a marketing firm once who had lost their senior developer about a year before. They had been running on luck and a whatever quick fixes their front-end developers could hack together when anything went down. Needless to say, no one had any idea where the servers where or even how many servers we had. Took a while to track them down.<p>Almost a year after I had thought that I had tracked everything down we had an odd error start cropping up on our sites where the domains on a couple older sites wouldn&#x27;t resolve on just some computers. After digging through it we found that there was some name-server that I had missed but we still didn&#x27;t know where it was until a thunderstorm killed out office router. In the process of replacing it we tracked one cable to a closet in the basement where, lo, the mystery name server was sitting.

Many years ago I did a job for a big corporation with a big IT department. Servers were in the data center of course and to get one you had to fill out a lot of different forms. When the project was on fire we were allowed to bring in our own machine and use it as a server.<p>Later, when the project ended no one bothered to go through the whole formal process to get the machine out of the premises again. The machine was just left there connected to the network.<p>In the months after the project ended different departments moved into the building. Different people with different tasks. Yet the machine stayed connected. I guess no one had the courage to shut down a machine they didn&#x27;t know about.<p>For a year or so I still used to check occasionally if I still could log into the machine. It disappeared eventually but unfortunately I don&#x27;t remember how long exactly it lasted.

Maybe some remember an UK website with &quot;celebs&quot; in the URL, in the late 1990s. (black background with small stars, website used frames)<p>It was probably the largest image database back then. That website had thousands of photos from every female celebrity categorized by name. The website owner collected all celebrity images from public news groups. One day the updates stopped and everyone wondered what happened. Many months later the website vanished (ca. 2002) and I read in the news that a friend found the dead body of the owner and no one paid the bills. As usual a domain grabber replaced it with a scam website shortly afterwards. I remember only a few frame-pages were backed up by Archive.org - though I forgot the URL of the website. Does anyone remember that former rather famous site?

Of course there are, the question though is how &quot;forgotten&quot; and what is a server. There are web sites out there running on servers where they haven&#x27;t been changed or updated in years. Presumably they are forgotten in one sense or another.

I&#x27;ve got one, that I set up for a client but who decided to not need it. Still running and paid for. I imagine this happens a lot.<p>Edit: Now that I think of it, I &#x27;ve got 3 for 2 different clients.

The reason I enjoy these kind of stories is the uptime. I know it is not a real indicator of anything but I enjoy seeing machines that have been running for years. (I used to frequent a website were people actually tracked their uptimes).<p>One of my favorites was this one, on ARS:<p><a href="http:&#x2F;&#x2F;arstechnica.com&#x2F;information-technology&#x2F;2013&#x2F;03&#x2F;epic-uptime-achievement-can-you-beat-16-years&#x2F;" rel="nofollow">http:&#x2F;&#x2F;arstechnica.com&#x2F;information-technology&#x2F;2013&#x2F;03&#x2F;epic-u...</a>

A few years ago when I upgraded to a new more powerful box at my hosting provider I noticed half a year later the old one (which I had whiped and done some testing on after transfering) was still chugging along, serving the silly 403 page I set up. After a few days of talking to the confused support it got shut down.

Relevant <a href="http:&#x2F;&#x2F;bash.org&#x2F;?5273" rel="nofollow">http:&#x2F;&#x2F;bash.org&#x2F;?5273</a>

I have a forgotten server happily chugging away somewhere, it still send me mail sometimes. it might be lonely.

Yes, it happens. From the small company to a corportation, those can be: vms, old servers, racks of servers...<p>Having a perfect CMDB for hardware and following all procedures step by step could identify &#x2F; prevent cases like that. But in reality, in normal environment it&#x27;s going to happen.

I&#x27;d assume like 20% :-)

I used to administer a website that was running on a server set up by the original site owner at a hosting company he worked at. The site kept running for 5 or 6 years after he stopped working there.

Yeah, a department I worked in had over 1000 vm&#x27;s up with nothing installed. That&#x27;s what you get when you have high turnover, huge cash flow, and management that didn&#x27;t care.

Somehow this thread reminds me of Wall-e.

I wonder what the Carbon Footprint of all this unused compute power is?