Applying the Unix Philosophy to Object-Oriented Design

The post is essentially another take on the Single Responsibility Principle. It's a good principle, and Rails model classes are also a good choice of an example where it's frequently violated.<p>Rails practically <i>begs</i> you to disregard SRP in your models. It entices you to put all your validation logic, relationship specifications, retrieval logic and business logic into one gigantic basket. Under these circumstances "fat models" become "morbidly obese models".<p>Rails 3 introduced "concerns" as a means of organizing the bloat into mixins. These are a testability nightmare. There doesn't seem to be any clean way of testing concerns without instantiating one of the model classes that include them.<p>The only way to use ActiveRecord without sacrificing testability, SRP, and the OP's application of "the Unix philosophy to Object Design" seems to be by limiting usage of it to to a pure persistence layer. But doing so seems to preclude using most of the ActiveRecord features that make it so convenient in the first place. Can anybody set me straight on this somehow? I would love to be completely wrong about everything I've just written here.

&#62; UserContentSpamChecker.new(content).spam?<p>Noooooooooo !<p>Sorry. There is no purpose for these layers of indirections since you're never passing the instance around to share state. If UserContentSpamChecker where a value object you could have argued that you're doing a form of casting but it's not. Best is to stay with a simple function:<p><pre><code> module UserContent extend self TRIGGER_KEYWORDS = %w(viagra acne adult loans xrated).to_set def is_spam?(content) flagged_words(content).present? end protected def flagged_words(content) TRIGGER_KEYWORDS &#38; content.split end end </code></pre> Now you can just call the function directly:<p>&#62; UserContent.is_spam?(content)

The tldr of the article is to write modular code. This is great advice, I hope most intermediate software engineers already live by this.<p>If I recall my CS history correctly, Multics/Unix was one of the first major software projects that embraced the modular philosophy. But modularity is a natural approach when designing large systems, and has been embraced by electrical and mechanical engineers for way longer than software systems have existed. Modularity even extends to processes; Henry Ford made his millions by modularizing the assembling process of his cars.

It's a nice idea, but look at how much code you have to write to obtain this decoupling.<p>The "mental" abstraction is nice to have, but when you end up typing <i>more</i> characters, it's just counterproductive.<p>Once you start repeating yourself, then it makes sense. To do it from the start is overengineering—"you ain't gonna need it".

Maybe it's because I've been reading about DCI lately, but I think this example could go even further in moving business logic out of the AR model.<p><pre><code> class GuestbookLibrarian def initialize(rate_limiter, tweeter, spam_checker) @rate_limiter = rate_limiter @tweeter = tweeter @spam_checker = spam_checker end def add_entry(name, msg, ip_addr) raise PostingTooSpammy if @spam_checker.is_spammy? msg raise PostingTooFast if @rate_limiter.exceeded? ip_addr entry = GuestbookEntry.create(:name =&#62; name, :msg =&#62; msg, :ip_addr =&#62; ip_addr) @rate_limiter.record(ip_addr) @tweeter.tweet_new_guestbook_post(name, msg) entry end end class GuestbookController &#60; ApplicationController ... SNIP ... rescue_from PostingTooSpammy, :with =&#62; :some_spam_handler rescue_from PostingTooFast, :with =&#62; :some_other_spam_handler def create #maybe these should be globals in an initializer somewhere if we #use them elsewhere? or in a :before_filter at least :) rate_limiter = UserContentRateLimiter.new tweeter = Tweeter.new spam_checker = UserContentSpamChecker.new librarian = GuestbookLibrarian.new(rate_limiter, tweeter, spam_checker) entry = librarian.add_entry(params[:name], params[:msg], params[:ip_addr]) redirect_to entry, :notice =&#62; "Thanks for posting" end end </code></pre> Something like that? Too Java-y? Feedback appreciated :)

Yet the refactored example is still showing application domain logic being built within the web framework. For simple applications that are only ever going to be delivered via the web and only ever rely on one particular web framework this might be ok. That being said, I'd like to see movement away from this back to using web frameworks or anything else simply as the communication platform between your application and the user.

The Microsoft Word example is not entirely accurate. Most office application <i>are</i> in fact having well defined interfaces through OLE [1]. This works in several languages like Perl, VB, Delphi or Python.<p>[1]: <a href="http://www.adp-gmbh.ch/perl/word.html" rel="nofollow">http://www.adp-gmbh.ch/perl/word.html</a>

Wow, I didn't realize the McIlroy who critiqued Knuth's famous literate program also conceived of shell pipes in the first place: <a href="http://www.leancrew.com/all-this/2011/12/more-shell-less-egg" rel="nofollow">http://www.leancrew.com/all-this/2011/12/more-shell-less-egg</a>

I don't particularly see the unix philosophy ever shining through in object oriented design. The problem is that objects are too inflexible - you can't just glue methods together, piping arbitrary objects into methods like you can do with data at a command line prompt.

While this is certainly useful, it's pretty basic. Something that I find is a much closer fit to UNIX pipes is iterators. UNIX pipes work similarly in that all of the commands in the pipeline are executed in "parallel" and the OS passes data incrementally between each process.<p>I primarily work on a Python codebase, and I've found that using iterators for complex, fault-tolerant data pipelines allows decoupled design without many of the performance &#38; additional complexity drawbacks often encountered with cleanly abstracted, decoupled code. For example, when executing a multi-get for objects by primary key, the pipeline looks roughly like this:<p>1. Fetch from heap cache<p>2. Fetch from remote cache<p>3. Fetch from backing store<p>4. Backfill the heap cache<p>5. Backfill the remote cache<p>6. Apply basic filters (e.g. deleted == False, etc)<p>At each step there are usually two or three layers of abstraction underneath. Much of the space requirements, and some of the overhead time at each step can be collapsed to O(1) instead of O(N).<p>For example, a cache multiget abstraction on-top of memcache might look something like this:<p><pre><code> def deserialize_user(serialized_user): return json.loads(serialized_user) def build_prefixed_memcache_key(prefix, key): return "%s:%s" % (prefix, key) def get_users_from_remote_cache(user_keys): cached_users = get_from_memcache(user_keys, "user") deserialized_users = {deserialize_user(value) for key, value in cached_users.iteritems()} return deserialized_users def get_from_memcache(keys, prefix): rekeyed = {build_prefixed_memcache_key(prefix, key): key for key in keys} from_memcache = [] for chunk in chunks(rekeyed.keys(), 20): results = memcache_mget(chunk) from_memcache.extend(results) unkeyed = {rekeyed[key]: value for key, value in from_memcache} return unkeyed </code></pre> Notice how at each step there is a large amount of "buffering" that causes allocation, copying, and quite a bit of additional work. Each layer of abstraction adds a pretty large cost to the step. Using an iterator implementation, we can clean up this code and make it more performant:<p><pre><code> def get_users_from_remote_cache(user_keys): for key, user in get_from_memcache(user_keys, "user"): yield deserialize_user(user) def get_from_memcache(keys, prefix): for chunk in ichunks(keys, 20): rekeyed = {build_prefixed_memcache_key(prefix, key): key for key in chunk} for key, value in memcache_mget(rekeyed.keys()): yield (rekeyed[key], value) </code></pre> It's clear how much cleaner this code is. Notice how this snippet avoids the large amount of "buffering" of data between steps and short-circuits quite a bit of code when possible (for instance, if all of the fetches miss). In real code that's heavily abstracted &#38; layered, avoiding all of this work translates into significant performance &#38; cost advantages.<p>Iterators also allow building portions of the pipeline with built-in functions that avoid the interpeter.<p><pre><code> def deserialize_users(users): return imap(pickle.loads, users) def get_users_from_remote_cache(user_keys): cached_users = get_from_memcache(user_keys, "user") only_values = imap(itemgetter(1), cached_users) return deserialize_users(only_values) </code></pre> This block of code executes in only one pass through the interpreter (imap, itemgetter, and pickle.loads are all implemented as native functions). This is incredibly powerful because it means that iterator-based abstractions can be built by combining these native building blocks without the overhead of recursion within the interpreter at each step.<p>Pushing all the recursion into native code:<p><pre><code> def gprefix(prefix, delimiter): prefix_with_delim = prefix + delimiter prefixfn = partial(add, prefix_with_delim) unprefixfn = itemgetter(slice(len(prefix_with_delim), None)) return prefixfn, unprefixfn def memcache_mget_chunked(keys): chunks = ichunks(keys, 20) result_blocks = imap(memcache_mget, chunks) flattened_results = chain.from_iterable(result_blocks) return flattened_results def get_values_from_memcache(keys, prefix): prefixfn = partial(add, "%s:" % prefix) prefixed_keys = imap(prefixfn, keys) key_pairs_from_cache = memcache_mget_chunked(prefixed_keys) values_from_cache = imap(itemgetter(1), key_pairs_from_cache) return values_from_cache </code></pre> * I apologize for any code errors, this code wasn't tested in it's entirety.

This is the principle to always create loosely coupled components that can be glued together.<p>I think the OO community has already learned that interfaces, categories, traits, mixins, or whatever they are called in your favorite language, favor more re-usability than plain inheritance.<p>The trick is to learn when inheritance, delegation or composition make sense.<p>As with everything in life, it takes time to proper learn them with lots of failed experiences along the way.

Quite a long (although interesting) article. tldr version: <a href="http://tldr.io/tldrs/50b62f8abb22039977000471" rel="nofollow">http://tldr.io/tldrs/50b62f8abb22039977000471</a>

&#62; write programs that do one thing and do it well. Write programs to work together.<p>ASIDE: Just been reading <i>The Wealth of Nations</i>, and Smith talks about the "division of labour", which is similar to this specialisation. By concentrating on one task, a workman can increase his dexterity at it, not waste time switching between tasks, and find ways to do it better. "Do one thing and do it well" is a pithy exaggeration of this specialisation.<p>This division of labour is only possible because of trade: other workmen do the tasks that you aren't doing, and you barter with each other to get what you each need. Because you specialised, you each do your own task more efficiently, so you are both better off. Programs that "work together" is similar, because if you couldn't use another program, you'd <i>have</i> to include it in the first program. "Programs" would also include 3rd party libraries and frameworks I think.<p>He goes onto say that one limitation on how much labour can be divided is the market size: you to be able to trade what you make. If people don't need much of what you produce, you've over-specialised. However, if you have access to a large enough market, then in aggregate, with many people using a little bit, you can survive. Larger markets therefore allow more specialisation, and therefore more wealth. Rich civilisations grew up around navigable rivers (especially with canals), and inland seas (calmer and safer to navigate than the open ocean), because water-carriage facilitates trade with more people (larger market), over larger distances, at lower transport cost and less risk.<p>Does the analogy fall down with free programs, since you won't go out of business just because few people use it? Open source projects <i>do</i> seem to need users, for encouragement, bug reports, spreading the word, contributing bug fixes etc - users pay in attention, not money. Without attention, projects die.<p>Does the analogy fall down when it's the same person writing all the separate programs? You can certainly specialise, and therefore do a better job; and also "exchange" data between those parts. (Provided of course that specialisation is actually more efficient, and outweighs the costs of exchange). But the motivation doesn't apply in quite the same way, since the parts of the code don't get paid - not in money, not in attention. You're more like a communistic planned economy (or, within a firm).<p>But the interesting point I've been leading up to is: does a larger market for programs lead to greater specialisation? For example, Java has a large number of users, and has a ton of 3rd party programs - and that does seem to include very specialised libraries.<p>Or, are other factors at work, leading to more one-package programs, such as Word; or dividing the market into smaller ecosystems, such as Java itself, and also other languages python/ruby/PHP, and even frameworks, like RoR. There are barriers between these "markets", in the extra difficulty of using library from one platform in another. For example, for JS, I get the impression there are many libraries for doing the same web/forms/template etc. While competition is healthy, and these frameworks are definitely improving on each other, they aren't the "division of labour" of interest to this particular discussion.<p>In programs, does a larger "market" of fellow coders result in more specialisation in the "one thing" done well?

Unix philosophy has nothing to do with objects. It is about <i>interfaces</i>, <i>streams</i> and common intermediate data representation - <i>plain text</i>.<p>Scheme or Erlang are <i>real</i> examples of following similar philosophy.<p>Modularity is much more basic concept, underlying any software. It is about spiting code into blocks for later reuse, named, in case of procedures, or anonymous in case of blocks or lambdas.<p>The notion that smaller, share-nothing procedures doing only one well-defined task are better than bloated and close-coupled ones is a general one.<p>So, do not try to mislead others. There is nothing from UNIX Philosophy there, just a basic concept of modularity.

Well , maybe the author should have figured out that the way Unix works is closer to functional programming than object oriented programming, before writing this blog.