A bit surprised to see no support for DPDK. While using AF_PACKET makes the project a bit more portable, you'll be able to save a lot of cycles by using skipping the kernel all together.
Interesting!<p>An open source high performance rolling packet dump with packet index for incident response.<p>To be honest, I had imagined Google already had solutions like this internally :)<p>These have been commercially available for a few years, look at RSA Security Analytics (formerly NetWitness), or BlueCoat Security Analytics (formerly Solera). Or the (open source) Bro Time Machine. I used to work with one of these products in the past.<p>What make systems like this a lot more powerful is more and easier search and retrieval.
While indexing IP numbers and port numbers is good, it will get much more useful if you can connect it to something like 'bro' and get session level data and then index filenames, user-agents, file hashes, and others pieces of information. I'm sure you can see the use cases.<p>Having an easy way to query 'all traffic with this particular user agent', together with the full packet capture, which allows you to write new rules, can significantly increase the efficiency of a security team.<p>Apart from the streaming analytics, once the PCAP data is stored, you can use mapreduce type operations on them to search through yesterday's data with today's IDS signatures (look at PacketPig/what Packetloop does). Maybe a lambda architecture is the way to go, or just reprocess old data through the same stream processing.<p>Cool work though! I'm curious where this will go next.
The design document is a good read and includes high-level details of how they're grabbing packets (AF_PACKET), the packet index format (leveldb), and defensive action they took (fuzz testing via AFL, setcap, setcomp):<p><a href="https://github.com/google/stenographer/blob/master/DESIGN.md" rel="nofollow">https://github.com/google/stenographer/blob/master/DESIGN.md</a>
They probably don't want to give away too much (like security details of their network), but I think it'd be more compelling with some examples of how to use this for Intrusion Detection.<p>It's a topic I don't know much about, and I think it'd reinforce the claim this isn't for user monitoring.