Stack Overflow has solved a similar problem using WebSockets -- most of their pages will automatically update (e.g, to display the presence of edits, new answers, and added comments) immediately using a socket.
i would think the answer is fairly obvious. 4chan needs an official API that allows for http connects to remain open. this way extension and 3rd party application developers have something to write software around that doesn't impede the ability to serve standard page GET requests to web users.
The main problem seems to be revenue. With 4chan, you could probably do a unique form of advertising -- companies pay ~$100/cpm to NOT have their products or brand featured on any 4chan pages. :)
Thats interesting. And 4chan had only one guy (moot) managing all of this makes it even more impressive.<p>By the way, some of the posters commenting on 4chan degrading morality. This is an anon forum and its just a reflection of our society! The Guardian summarized 4chan as "lunatic, juvenile... brilliant, ridiculous and alarming". Perhaps the best description for 4chan.<p>And remember 4chan gave us LolCats, Memes, Rage faces etc.<p>P.S. By the way, why did mods kill the other thread?
HTTP 304, Not Modified?<p>There's no accounting for an inconsiderate user base. This has always been a big problem for 4chan. I remember several instances where the site was effectively taken offline for days by a JavaScript virus embedded in PNG images that <i>required the user to follow directions in the image</i> in order to spread.
Image preloading and autogif are fantastic features, so getting rid of them would be a net negative. Instead, why not have them be controlled by some client side js, so that images/gifs are only preloaded for the next 20 or so posts from what the user is currently reading? If the user stops scrolling, stop preloading. Preload the final 5 or so images to cover skip-to-bottom, but the vast majority of the preloaded images will not be viewed, and so should not be loaded.
"despite having a bunch of fast drives in RAID 10 and 64 GB of RAM, our image server simply cannot keep up. "<p>Wait, so he only has a single image server?
The key take away is: if you don't look at YOUR server logs, but relay on 3rd party solutions, you are missing the picture. By large.<p>That Google analytics "guess" was abysmal. Client side logging is fine to complement server side.
Meh, I'm always super unimpressed when simple text based websites have trouble scaling. Everything that's highly requested should be available in memory, and it should be trivial to spit it out instantly.<p>I'm not a scaling wizard, but I'd guess 99/100 times the reason CRUD apps have problems scaling is because they are over-engineered, and there is a tendency to solve scaling issues by adding another layer of complexity instead of optimizing the root application.