How IMAP works under the hood

I started writing a guide to IMAP back when I was working on an email client: <a href="https:&#x2F;&#x2F;github.com&#x2F;rmccue&#x2F;griffin&#x2F;tree&#x2F;master&#x2F;docs&#x2F;imap" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;rmccue&#x2F;griffin&#x2F;tree&#x2F;master&#x2F;docs&#x2F;imap</a><p>Pulling large amounts of data for things like threading can be difficult on certain servers; my preferred approach ended up being to pull every ID and thread ID to maintain an in-memory tree. (This was, iirc, partially because Gmail’s implementation was slightly crippled with relation to threading.)<p>(I never finished the guide because I stopped on the project, alas - if IMAP were easier to work with, I might have finished it! And sadly, no JMAP support on Gmail, and the gateway was broken.)

For the extra interested… reminded me of this one: <a href="https:&#x2F;&#x2F;explained-from-first-principles.com&#x2F;email&#x2F;" rel="nofollow">https:&#x2F;&#x2F;explained-from-first-principles.com&#x2F;email&#x2F;</a>

I&#x27;ve been working on some email stuff and I think probably four things are vexing about IMAP:<p>- The grammar is hard. I built a parser using lpeg and I&#x27;m incredibly glad I did--doing it ad hoc won&#x27;t lead to good results.<p>- It&#x27;s an asynchronous protocol. You can send lots of requests to a server and you have to tag them so you can match them up with responses later. You don&#x27;t generally want to do that in a client; i.e. you don&#x27;t want to do these transactional things over an async connection and track state across all of it. You want to like, block on deleting things, renaming things, sending things, etc.<p>- IMAP is multi-user--it&#x27;s built around the idea of multiple clients accessing the same mailbox at the same time and streaming updates. Another thing you really don&#x27;t want to deal with when building an email client.<p>- There&#x27;s functionality that you basically shouldn&#x27;t use; the big one is search. Even the specs more or less say &quot;good luck using this&quot;.<p>You can group all this under the a heading of &quot;we thought people would use this over telnet&quot;, but attachments and non-plain-text email I think made that non-viable.<p>I think this all means probably every non-web email client treats IMAP like POP and keeps its own store. I haven&#x27;t done a survey or anything, but I&#x27;d be surprised if that weren&#x27;t true.

Interesting no attempt has been made to make it at least be less heavy on networked bytes. Especially since it is old and was meant to be used on a connection with no compression or encryption.<p>HasChildren could have been Parent, HasNoChildren could have been Leaf or Child. And so many more things.

I&#x27;ve been looking to migrate from Exchange-Outlook, but there really aren&#x27;t any options. There just isn&#x27;t an open source solution to have an integrated email &#x2F; tasks &#x2F; events &#x2F; contacts, with consistent labels across item types and reminders.<p>With Outlook, i can use a custom view to see every item in a category flagged for follow-up. I can also set a reminder on a contact, or drag an email or contact onto my calendar to create an event.

Of course these days the mega-corp walled garden email providers don&#x27;t really follow standards like IMAP. IMAP will not work with, say, Google&#x27;s gmail or Microsoft office365, or AT&amp;T ISP email, etc, etc. They have each implemented their own proprietery out-of-band authentication system that only works over HTTPS using the OAuth2.0 toolkit to build it. Any email client that does not explicitly design for each particular OAuth2.0 implementation (each megacorp&#x27;s is slightly different) will not be able to connect over IMAP (unless they login via HTTPS using a web browser and set up &quot;app passwords&quot; for google, or similar for others).

Can IMAP be used as a file server system? Outlook has this functionality, where files can be stored directly outside of emails.

I&#x27;ve been trying to get approval from Google for the sensitive scopes to use IMAP, and they classified us as needing &quot;CASA Tier 3 Security Assessment&quot;. It looks like it is going to be a long, tedious, opaque, and expensive process.

That was a pretty interesting read. I didn&#x27;t realize one could interact directly with an IMAP server like we can with FTP and telnet + HTTP.

This is the kind of thing I would have expected to read in 2600 back in the day. And why I _always_ looked for 2600 at the bookstore.