I love these as an example of what I've called "scaling deeper". Elasticsearch is a fabulous technology for a general set of use-cases. But if you can define certain tasks specifically, there's often a very fast/cheap way to accomplish it outside of the general.<p>I remember a case where we needed to pull all incoming IP addresses for a given 24 hour period for certain common user-facing queries, and the specialized FPGA driven scalable database appliance was returning those queries in minutes. So instead we ran the query once a day, and then just flipped bits in a big bit array (IPv4 is half a GB of bits) and just let each array act as a standing query. A query for a certain day just mapped to a specific file, and some simple math would answer the query 0 for IP never seen 1 for IP seen. It was nearly instant.<p>It transformed a certain team's workflow from asking a few times a day for yesterday's IP visits to many times a day and then they started asking for the data over a period of months instead to look for patterns. Queries against this structure for the same thing over a year took less than a second. It all ran on a 4GB VM with a few GB of spinning rust storage. Pretty soon this was built into a bunch of custom tools across a bunch of teams and was feeding the data tens of thousands of times per day, then we started enriching it with geodata, reputation data, and other things. It also reduced the query load on the main dB substantially.<p>Very simple engineering, transformative, and required simply trying to find a way to scale deeper instead of building a bigger multimillion dollar dB.