How to Replace IMAP

Not to spend too much effort defending IMAP (which does, to be fair, kinda suck) but... come on. It's not <i>that</i> hard. It works just fine, enjoying pervasive support with very high quality clients and servers available freely on essentially all platforms.<p>A new protocol would be ... prettier, I guess. But to pretend that the reason facebook and IM are replacing email is IMAP (and not, y'know, spam) is borderline delusional.

This is a collection of the author's favourite technologies/gripes, not a useful proposal. A large number of errors can be found by considering that it'll have to talk SMTP on the back end.<p>I'll only point to some "highlights".<p>OAuth isn't nearly secure enough (<a href="http://hueniverse.com/2009/04/explaining-the-oauth-session-fixation-attack/" rel="nofollow">http://hueniverse.com/2009/04/explaining-the-oauth-session-f...</a>) for your e-mail (which, don't forget, gives access to <i>everything</i> via password resets). Let's just ignore the part about third-party access.<p>Encoding everything as UTF-8 is nice, but breaks the first time some idiot mail client puts Shift-JIS encoded Japanese in the Subject: field. Without any charset marker, obviously. (This goes right back to the "where does the mail come from?" issue.)<p>IMAP has a THREAD extension (solving #4), a SEARCH extension (solving #9 and parts of others), and all sensible servers support IDLE (<a href="http://en.wikipedia.org/wiki/IMAP_IDLE" rel="nofollow">http://en.wikipedia.org/wiki/IMAP_IDLE</a>). It also supports downloading MIME parts separately, solving the part of #7 that can actually be solved (again, you'll have to accept mail.)

I can't say whether his implementation makes sense.<p>Regardless, I would like to see public key encryption of all email. There's no reason why messages should be plain text anywhere except at the two ends.<p>I realize key management would be a problem, but it's probably a solvable problem. The solution would have to allow most people to be practically unaware of encryption issues.

First the solution doesn't really fulfill all of imaps use cases (especially push mail) but that is unimportant.<p>What is important is that the IMAP protocol isn't difficult, and I don't think that is what limits "email innovation." It is more so that it has been around since the 70s and email clients haven't been able to remove themselves from their past. It isn't the protocol limiting them from moving forward, it is the shear number of other clients that wouldn't support any of the "innovations" one would try to bring to email.

There are some good points, but:<p>1. HTTP / HTTPS - please don't... There are stupid/broken transparent proxies in many places that will effectively block this protocol. Also custom TLS connection gives some hope for domain-specific certificates instead of ip-specific - there is enough problem with them in HTTPS as it is.<p>3. People still don't know how to set the encoding (or what the encoding is) and will happily send you Greek in latin1 or something equally silly.<p>4. Would it really work for mailing lists where conversations naturally fork into many branches?<p>5. IMAP supports labels... even if it's crazy number-based labelling where every client needs to set its own names for the labels. We'd just need to make it not suck this time ;)<p>So it's not all that bad in IMAP - there are many areas where we can do worse by accident. But yes - it would be nice to have server-side text searching...

My responses to your Design Outline.<p>1. I'm not convinced that http/https is the best protocol. I agree that it's the defacto standard, but if you're going to go through this, there might be a better way. (Push, in particular, is a problem).<p>2. Stateless is wonderful, agreed.<p>3. I agree with JSON over XML, but what about binary data? That is the weakest part of JSON, in my opinion.<p>4. I like where you're going with this, but I'm not sure this is the exact right way to handle it.<p>5. Agreed.<p>6. how about SHA-1 hashes of content? Done properly, that would eliminate any duplicate messages, and could make it very easy to include other messages. This might also allow content to be transfered between accounts only once, dropping bandwidth and speeding delivery.<p>7. Don't know enough about MIME to comment.<p>8. According to your comments on this page, you're referring to long polling. I agree that this use case needs to be addressed, but I'm not sure that this is the way to do it. Perhaps multiple options here?<p>9. Good idea, but do you think that stemming/language differences is going to be a problem? What about other types of content? attachments?<p>The biggest thing missing from this list is some sort of Public Key Encryption. Anyway, it looks like a good start.

&#62; All data that's ever sent to or received from the server would be in JSON format. JSON is much more human-readable than XML.<p>I don't need my protocols to be human readable. I need my computers to read them and know that I've got good, bug-free, reusable, efficient, and easy-to understand implementations.<p>This isn't an argument for XML over JSON -- I've made good use of both, but I don't want to consider how easy it is to type up communication protocols when I'm defining stuff the replacement for something I expect to take over the world.<p>It's <i>by far</i> easier to implement the memcached binary protocol as a client or a server than it is to implement the text protocol. Easier to get right, more regular, and just generally more pleasant.

Ah, yes. A new email scheme that would require us to replace all clients and servers in one fell swoop. I can see that taking off.

First, I love reMail. Gabor breathes email and search. If you own an iPhone and use Gmail or Google Apps, get reMail--you'll thank him.<p>That said, it's easy to skim this article and nod, "Yeah! Yeah!", but when we're discussing these, laymen may need to know a few of these seem over-simplified.<p>&#62; "TCP connections are great, but for transferring large amounts of email securely, HTTP is the way to go."<p>1. <a href="http://wiki.answers.com/Q/What_is_the_difference_between_tcp_and_http" rel="nofollow">http://wiki.answers.com/Q/What_is_the_difference_between_tcp...</a> - HTTP uses a TCP connection. For that matter so does SMTP and IMAP and the proposed REMAP.<p>2. I generally have at least 5 devices using IMAP (on Google Apps) at once, and interleave my interactions with these devices arbitrarily, yet expect each to show me the exact same state when I look at it. Keeping them all in the apparently same state is the sort of intractable software problem bedeviled with implementation details, and I agree throwing out IMAP and starting over with lessons learned could be easier than, say, the RFC linked below.<p>3. <a href="http://ajaxian.com/archives/json-vs-xml-the-debate" rel="nofollow">http://ajaxian.com/archives/json-vs-xml-the-debate</a> - Seems this point is something like GIF vs JPG vs PNG, with a bit of "Markdown vs HTML" thrown in. Meanwhile, both XML and JSON are missing an important concept given labels/folders and conversations: multiple references to single objects.<p>4. Conversations or threads exist today as In-Reply-To: GUID and References: GUID headers, among others, but see the JSON problem above. We don't realize these headers are there because clients generally ignore these and try to group on Subject instead. Gmail grouping respecting existing headers is much better, but from usability point of view, Gmail's approach makes it challenging to bulk delete individual matching messages across a broad set of conversations. For example, to bulk move or mark all messages on a particular date results in the full conversations being moved or marked. So I like this idea as long as I can still optionally operate on sets of messages without affecting the rest of the conversation.<p>5. <a href="http://googlesystem.blogspot.com/2007/08/organizing-chaos-folders-labels-search.html" rel="nofollow">http://googlesystem.blogspot.com/2007/08/organizing-chaos-fo...</a> - It's not yet clear that people prefer tags to folders. After all, monkeys don't expect a banana to be in two boxes at once. I don't want to label anything with keywords. I want semantic search.<p>6. Excellent point. Changing GUIDs out from under us is just lame. Having had to resort to <a href="http://github.com/rgrove/larch" rel="nofollow">http://github.com/rgrove/larch</a> recently to move a decade of email onto Google Apps because Thunderbird was putting new IDs on each moved message, I fully agree this problem is evil.<p>7. Seems we're stuck with MIME until we get rid of every legacy email server in the world, or control what servers our recipients use. Between the new server and new clients, sure. But the server will have to know it to send it. Granted, IMAP is for receiving, and SMTP is for sending, but something in the chain has to know MIME.<p>8. "Call" an HTTP endpoint that "returns" when messages arrive? Just nomenclature, since IDLE is a "call" that "returns" when state changes. Either way, the socket is held open, so neither of these is "push". Maybe Gabor meant "posts back" instead of "returns"? In any case, under the hood, iPhone push and Microsoft ActiveSync are exploiting out-of-band channel such as SMS to notify the phone it should poll (or in case of ActiveSync, maybe doing a "long pull"). See <a href="http://msexchangeteam.com/archive/2004/04/26/120520.aspx" rel="nofollow">http://msexchangeteam.com/archive/2004/04/26/120520.aspx</a> versus <a href="http://tools.ietf.org/internet-drafts/draft-maes-lemonade-p-imap-12.txt" rel="nofollow">http://tools.ietf.org/internet-drafts/draft-maes-lemonade-p-...</a><p>9. Will the RFC specify a search results algorithm, or will the same search return different results depending on the implementation of mail server the client is connected to? We manipulate a "client" tool to perform the search and access the email, so our natural mental model is of the client at hand, not of the remote server. This lets the brain wrap itself around Gmail's web search versus Apple Mail's search box vs OS X Spotlight vs Gabor's reMail all returning different results. Practially speaking, search at the server does make far more sense from both a data retention and security standpoint. I have 509 MB of index in Gabor's "reMail" app on my iPhone, and would rather not.<p>Thanks for brainstorming this, and thanks for a kickass iPhone app.

The description for #8 sounds more like long polling to me. For push email, I'd envision something like registering a location to send notifications to (web hooks).

You could implement this as a service on top of couchdb. The oreilly couchdb book that has just been released shows examples of developing apps using the javascript engine that's inside couchdb. You'd get easy access to some functionality by doing this - tagging, email versioning.