Some fun facts from scaling and optimizing the dominant school management site in our country that are used by schools, kindergartens, parents and kids. Schools years ago no longer may keep physical journals around because they use this system.<p>Peak was usually when semester end was coming. Currently, daily, we must sustain 2x of that peak, but peaked at 3x of that on the first day "remote schools opened". The everyday traffic is currently like almost 3x it would have been, measuring by requests per second received on frontends.<p>We were struggling usually at end of semester. Luckily we stated to do some upgrades and optimizations to handle that, just before the lockdown started. In the end, we can now sustain many more times traffic we have. The hardest part wasn't adding more resources (that was a MUST for sure) but it was much more effort to handle stuff that scales vertically (SQL) and some file share issues.<p>So there were some intermittent issues that were frustrating the users, sometimes bringing whole site down for minutes. (Sometimes is the trickiest part to handle). That includes hunting down expensive queries (not so hard), calling less or more optimal queries, taming SQL plan cache and dealing with some NTFS stuff for file share.<p>Some of the issues couldn't be solved solely throwing more hardware in.<p>I'm still puzzled for the file share issue on Windows. Yeah, not so clever to store millions of files in a single folder within NTFS filesystem. We have append-only share, no deletes ever happening. Stuff like timestamps and short paths were disabled, defragmented $mft and the likes, but... every ~24 hours the drive would become sooo painfully slow and inacessible. Some access denied errors get thrown, etc. And it continues for some minutes. Maybe 15. Sometimes more. But between that ~15min period it works, so it's like a wave with some period. But the thing is, outside of this window, the file share works very good (with all those millions of files within a folder). They were never deleted and we didn't need to enumerate - just fetch file by file.<p>No, Antivirus wasn't at fault, no backup system wasn't messing around. Is there stuff NTFS may do under the hood on SSD drives? I'm aware there is a TRIM process, but as I understand it has to deal when the stuff is being deleted from the SSD?<p>We moved to rotating disks (!) and split those files across few folders and got rid of those issues. But still, 1 folder contains way more files you would hear anyone saying is healthy on NTFS.