When Your Fiber ISP's 'Dumb' Device Screws Your TCP Sessions

FWIW I&#x27;m also on CenturyLink FTTH and just a week or two ago noticed latency spikes and packet loss which magically went away after 15 minutes. Good to read this analysis for future reference. I really wish end users had more control over ONT boxes similar to how we can use own modems for cable&#x2F;DSL. A DOCSIS-like provisioning by ISP should be possible.<p>Off topic, but CenturyLink Fiber still uses PPPoE and 6rd instead of native dual stack in many markets and are unwilling to upgrade to more modern configurations.<p>EDIT: I do not use Tor at all.

I was never happy with the performance of Calix CPE. We used them heavily at my last job, and indeed customers would have all sorts of trouble that we could never reproduce when we sent a tech. My favorite little hack was that I wanted live stats from the OLTs to be in our own database so that it could show up in our support portal and internal CRM and be aggregated for general network health statistics. (i.e. when someone went out to repair a fiber, they could instantly see the customers come back online, or more often... know while they were still out in the field that they didn&#x27;t fix it) I wrote a program to scrape it (by ssh-ing in, thanks golang.org&#x2F;x&#x2F;crypto&#x2F;ssh!, because their SOAP API returned no useful details), and after running for many days... it caused the OLT to stop routing packets entirely. No Internet routing, no management interface, it just flat out died unrecoverably. Anyway, they blamed my app, so I built them a static binary of the scraper that could run on Windows (they didn&#x27;t have any Linux boxes) and after much back and forth they traced it down to a race condition between the two redundant processor modules in the OLT. So much whining how it was my fault, when it was their fault.<p>At the ISP before that we made our own CPE. The leads on that project really understood the Internet and managed to get reasonable latency, even over WiFi. But the incumbents still seem to not know about fq_codel, or how to put more than 4MB of RAM in their devices, and the users suffer as a result. This article reminded me of how mad it makes me, sorry for the rant. (I switched to a different industry where less lasers are involved.)

I remember something about this from a few years back. Can&#x27;t recall the link now though.<p>His ssh sessions were constantly timing out. It only happened when he left the SSH session to idle. It turns out his router was dropping the TCP sessions because it considered them dead. He got around it by implementing a &quot;keep alive&quot; packet, of sorts. Very interesting stuff. I don&#x27;t really work at such a low level in the stack regularly, so it&#x27;s quite fascinating to see the strange issues people encounter with these tools. Especially when ISP&#x27;s meddle around with stable protocols.<p>Also reminds me of how some ISP DNS servers totally ignore TTL values from DNS records[0].<p>[0]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=29568510" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=29568510</a>

Quoting the article, the cause is identified &quot;The Calix 716GE-I ONT device is working as designed by activating Denial of Service (DOS) attack prevention when too many connections are established, which includes jumbo or small packets&quot;. Sounds like a reasonable feature for residental devices, even if it isn&#x27;t compatible with the niche usecase of running a Tor relay.

So ISP delivers router that breaks your internet, and they won&#x27;t replace it with a real ONT? Then why not simply replace it yourself? As long as it isn&#x27;t PON, but just plain AON, that should be relatively straight forward.

&gt; But what if a large number of TCP connections is intentional?<p>Sorry, that ship sailed long ago. Carriers have forever put restrictions on how their customers can use their internet connections, such as &quot;no hosting servers&quot; or even not getting a routable IP address. Traffic shaping is part of the deal too.<p>I think the only means we have to change the situation (in the face of a lack of competition) is to lobby for municipal internet. Or start a company.

Who’d think I’d be <i>happy</i> to have a Huawei ONT for my FTTH setup?<p>But reading this, clearly one can have much worse.

Glad I didn&#x27;t pick CenturyLink for fiber when I moved here, but Wave G&#x27;s incredibly unreliable in its own way which makes me wonder if they&#x27;re using the same hardware. Kinda wish I picked Google Fiber.

How do I check what the limit is on century link? I routinely use more than 1000 connections and want to know what my cap is

I’m not sympathetic to the author at all. You’re essentially using a home ISP for commercial purposes by hosting Tor relays. If you need resilience, then you really ought to colocate at a DC. 10 gbit is not that expensive these days, and you would provide your own switch like mikrotik.