I suppose I'm a modern app developer, so here's what I would actually do:<p>Postgres.<p>1) I'm not going to invent my own system. I am not the first person who wants to store addresses and boolean values. If I'm looking at this occasionally over many months, I don't want to keep relearning my elegant bit-structure.<p>2) SQL is great, Postgres is even better with built-in network operators[1] for the more specific queries (e.g. subnets.)<p>3) Bitfields are all fun and games until you want to change things. I'm already having flashbacks to rails `has_bitfield` columns. Also, fewer people can confidently twiddle bits than can understand boolean SQL columns.<p>4) The author mentions possible compression methods. I'd rather a tested project like Postgres think about everything to do with storing my data.<p>I understand the criticism, but with cloud storage so cheap it feels like optimization just to show off. I'd rather save time than bits. If there is a satire that involves me using Postgres when I shouldn't, that would be welcome.<p>[1] <a href="http://www.postgresql.org/docs/current/static/functions-net.html" rel="nofollow">http://www.postgresql.org/docs/current/static/functions-net....</a>
I'm not really sure what the point of this article is. If you're clever, you can represent any data as a bit array. And once you're there, counting bits or XORing them together is easy. But is that system easy to understand? Is the code easy to read for the rest of the developers who work on the project? Or for future hires? What happens if the data set becomes too large to fit in memory on a single machine? Storage is cheap and getting cheaper every day. The whole data set described in this problem (done the "inefficient" way) fits on a flash drive at today's capacities.<p>There are always tradeoffs when it comes to architecture design. Speed and storage space are only two of the many factors that require consideration.
"I can perform a simple AND operation on the 3 monthly bit arrays, and then count the number of “1” bits."<p>I think an old-timer would tell you that he'd be using an "OR" operation to count the number of hosts seen in the last three months.<p>As a long-time embedded systems engineer, you learn to guard every CPU cycle and memory location jealously. That behavior, along with avoiding needless abstractions that complicate the system AND cost CPU and memory, have served me well as I've switched to higher-level software.<p>As an aside, memory resident databases are now practical even for problems of this size. I've used the CERN Colt library to provide sparse arrays stored in memory in place of databases several times now. My current pet project is processing a mouse genome for my daughter's lab. With the amount of data being read, I'd always be I/O bound if I wasn't packing a lot of working data into memory. This is the programmers equivalent of the CPU's cache, but one we can easily control. Give it a try!
Why wouldn't you want to use an SQL database with an index to store this data? Just have a column for the ports and a boolean flag for the status of the port?<p>Honestly, I'd probably use bash/nmap/ping and psql to insert data. Want to query? psql and grep.<p>These examples seem like a great way to re-invent the wheel with modern buzzwords.
Business Guy shakes his head and gets a subscription on <a href="https://www.shodan.io/" rel="nofollow">https://www.shodan.io/</a>
Ah. Thank god the imaginary old school guy is better than the imaginary straw man modern guy! I'd better go learn Go immediately.<p>Personally I'd start off by looking into using Cython or Numpy and maybe pickle to disk for storage.<p>Does that make me the old guy or the new guy or is it a false dichotomy?
Funny quote from the article: "This is a big data problem." Hmm maybe on a raspberry pi, or some kind of retrocomputing challenge (Do this on a simulated IBM 1130!).<p>I don't know how an app developer thinks, but I know the article characterization of the old developer is wrong, we'd chuck it into the existing relational DB, maybe spin up a new cloud instance and reserve some NAS space if necessary but this is a pretty small data set by modern standards so probably nothing special is required. All the reporting devolves into silly SQL onliner competition. The first question is a COUNT(*) and GROUP BY. The second question is a ridiculously simple SELECT. The third question is another simple select if you stored your ip addr bytes in separate columns, even if you allow non /24 addrs. The fourth is another GROUP BY.<p>The article does fit the stereotype I've seen that new programmers prioritize ease of storing data over ease of reporting, and old timers vice versa. Like the difference between coming up with the fastest next move in checkers vs "solving" checkers in the game theory sense.
A couple of errors near the end for "How Many Hosts Were “Up” Last Month But Now It’s “Down”" and vice versa. The error is that using xor on the same item twice is a no-op...<p>(A xor B) xor A == B.<p>The correct equations would be,<p>!this_month and last_month = "up" last month, down now.<p>and<p>this_month and !last_month = "down" last month, up now.
This just compares a decent programmer with a terrible one. Not a modern app developer with an old timer. This is especially evident where the systems programmer is aware enough to use Go (which is a debatable choice, but certainly displays good awareness), but the modern app dev doesn't know when (not) to use Big Data and when to try optimizing things (e.g. the JSON choice).<p>The only "mistake" I see a modern app developer making might be using Python, but hey, computers are fast (so python should still be fine). Most would probably come up with a similar bit-twiddly solution or a simple DB-based solution. No biggie.<p>There's no dearth of horrible systems code out there either. It might be less likely for a systems programmer (even a horrible one) to mess up this exercise, but you could probably choose an exercise which would have the reverse effect. In the end, it's a toy exercise, not one where you actually design a significant piece of software.<p>There is a point buried in all of this; which is that learning systems programming will probably make you a better modern app developer since you get the correct mindset to tackle problems like this (also, vice versa?). But there's too much hyperbole obscuring it.<p>There's also the other point about <i>how</i> systems programming is done vs modern app dev. It's a valid one, but there are benefits to both approaches, and it boils down to each being useful in its own domain.
Couple of defensive, bruised egos here in the responses. Lighten up, it's funny! We all know that one guy for whom the answer to every question is HADOOP MAP REDUCE!! I thought the writing was fun.
Unfortunately our hero the array developer got a bit too clever, pun intended, and produced an incorrect program. The last two of the XOR tricks won't work the way he thought.
Point of this article is not the point of this article. The clever bit was:<p>> ...the IPv4 address will convert into a number...<p>Many fancy, post-modern app developers might insist, "You won't do math with an address; it's not a number." But some things <i>are</i> numbers, with a scheme and pattern you can exploit.
There's no simple solution without a lot of context.
<a href="http://mockingeye.com/a-classic-of-soviet-engineering/" rel="nofollow">http://mockingeye.com/a-classic-of-soviet-engineering/</a>
It's fascinating how many people here are defending the "modern app developer" approach, mostly with arguments about flexibility, maintainability, ability to pass on the code to junior developers, and the like. If you think about what the "old-timer's" code here would <i>actually be</i>, these kinds of objections make no sense at all.<p>Assuming the code to run nmap itself would be equivalent either way and we're interested in the data storage and analysis functions here, the old-timer would write that entire part of the program in about one screen of any decent programming language in perhaps 10 minutes. The functions would be short and simple, needing only basic iteration and bitwise arithmetic. Any junior programmer who's going to get anywhere in this industry would be able to understand that code in moments with no specialist knowledge or additional training. Nothing about the code would get in the way of any reasonable commenting or testing policy either.<p>If in the future someone didn't find the compact data structure appropriate for some new application, they could easily convert the data to a more suitable alternative format, because the current format would be well-specified, simple, efficient, and without external dependencies.<p>Hypothetical arguments about scalability are silly. The problem is fundamentally built around <i>IPv4 addresses</i>, which have been 32 bits wide since they were devised and will still be 32 bits wide tomorrow and next year. Designing for something more scalable is some horrible combination of scope creep, YAGNI violation, and worst of all, not giving even cursory thought to what the requirements actually mean. (I await the seemingly inevitable unintentionally amusing response about IPv6...)<p>The only thing I really quibble with here is the characterisation of the two types of developer. I don't think this is really about modern vs. old-timer. It's just about a good programmer -- who looks at each problem on its merits, chooses suitable tools for the job, and leaves their options open -- and the bad programmer, who does not.<p>Well, that and the fact that no self-respecting old-timer would misuse the word "performant" so heinously, but I digress. :-)
That's actually a good comparison between different mindsets. Even if it reads a bit biased pro-go (or pro-old-time-system-developer) it doesn't tell what's the best way to implement this - and in my opinion it's neither one.<p>So what's actually the best way to implement this? What are the motivations for choosing one way?<p>"Modern App Developer Way":
+ adoptable, scalable, readable data formats
- storage, computation<p>"Old-Time System Developer Way":
+ storage, computation
- "locked-in" data formats<p>"Database Developer Way":
+ ease of implementation
? storage, computation<p>Any other options/ideas?
As a relatively junior developer, I have been told my some of my senior colleagues that I sometimes miss the forest for the trees, and so this may be another instance of that.<p>But it doesn't seem "trivial" to me how one gets the output of nmap into whatever database, data structure one chooses here. I know nmap can produce XML, presumably there is a csv format, but it would seem like the XML/CSV -> JSON conversion (following our intrepid Modern App Developer) would be an easier more maintainable way to go, versus XML -> to bit array (memory map file). Also, is managing the nmap or masscan and whatever other ancillary processes required to execute this plan equally as onerous in either paradigm? Finally, and this is likely controversial, this particular problem "feels" like its stacked against the Modern App Developer, given that it isn't trying to solve a problem most Modern Apps try to solve (or try to solve as an end rather than a means to an end)
Pentester: I'll use mass scan, someone else already solved this problem.
<a href="https://github.com/robertdavidgraham/masscan" rel="nofollow">https://github.com/robertdavidgraham/masscan</a>
I'm betting on the modern app developers system being much easier to maintain and understand ( and pass off to junior programmers), at the cost of speed.
If anybody is curious, the zmap project from the University of Michigan does this and more.<p><a href="https://zmap.io" rel="nofollow">https://zmap.io</a>
One could differentiate the two proposed solutions by the level of involvement of third parties (e.g., cloud hosting providers, authors of garbage collected or scripting languages, etc.). Some of these third parties have been involved since "old times", others have not.
Based on my recent experiences, the modern app developer would write a slow, bloated web app that takes ages to produce a result but has a slick UI with awesome animations.
Am I the only one who thinks that there are far more efficient data structures than a bit array to represent this data ?<p>How about an RLE encoded list of hosts, for instance ? (since they're consecutive 32-bit integers). There'd be way less data than in a bitfield which will make most of these queries far faster than iterative bitfield lookups. Also, much more of the data would fit and stay in memory, which means that all queries that iterate over it will be 10x faster or maybe more.<p>Of course experimenting with data structures is not something you can do very efficiently in Go, as it'll be painfully verbose code. C++ would be far more useful.<p>But this is basically the old argument for/against optimizing code. The real problem with the "old timer" programmer is that there are quick ways to break his program. When new queries present themselves, the "old timer" will quickly find his datastructures not optimized for the queries, or that they require complex calculations, which means that for data analysis the modern app developer will probably "win".<p>When it comes to putting a product in production, it needs to be fast and cheap. Any company that doesn't hire an old-timer developer for that will quickly find their costs exploding. This may be acceptable for a few weeks when trying to find product/market fit but it won't last long.
My background is much closer to the "old timer" but I think I'm definitely on the side of modernity here, along with everybody shouting "sql".<p>Also I don't see why the old timer needs both a bit array and a uint64 port array, they can put the up/down bit in the high bit of each uint64.