[brongondwana]

You put a lot of effort into your response, and I appreciate that.

But hey - that was a 36 message folder in that video. Show me an IMAP client that would do better and I'll be sufficiently impressed. They don't exist, because you can't do that.

I can show you, right now, a 500,000 message view across multiple folders infinite scrolling with JMAP and I can jump to the middle of that view and load a screenful of messages.

You CAN NOT do an IMAP sort without fetching one record for every message in the folder. If you have a million messages in a folder (yes, we have customers who do that) then you need to fetch a million records.

QRESYNC does improve matters a lot - except almost nobody implements it (we do: I fixed the initial implementation to be exactly correct in all the edge cases) - and JMAP's synchronisation is based on QRESYNC with extra protections against a flood of traffic if the server has many more changes than you expected.

I suspect your allergy to all things JSON and HTTP has coloured your attitude to the protocol, which is a pity.

(IMAP isn't a batch protocol - it supports pipelining, and that's good - you'll notice we also tag commands in JMAP)

Anyway, I look forward to your demonstration of an IMAP client fetching and preparing a view of a 36 message folder alongside our web interface loading the same folder from scratch, no local data present.

Until then, screw your calling bullshit. Go get yourself an iPhone (or ANYTHING with an IMAP client on it) and create yourself a FastMail free trial and reproduce the scenario in the video yourself. You don't need to take my word for it, it's trivial to recreate.

Basically, put up or shut up if you think IMAP is so good. If you think we're missing an IMAP extension that would make your demo work better, let me know and I'll implement it. If you need more than a month, let me know and I'll extend your trial as long as you like.

[jstedfast]

It'll take me longer than a month to implement my own IMAP client to do things efficiently.

As it just so happens, the company I work for is having a "work on anything you want as long as it dog-foods our products" week next week and I was already planning on writing an IMAP mail client for iOS that didn't suck. Unfortunately, due to the complexity of writing a mail client, I doubt I'll finish in a week.

That said, thinking about the problem a bit more since last night, there's 1 major performance bottleneck in the IMAP protocol for this particular use-case but it could be solved with an extension to SORT.

So, as I'm sure you know, the SORT command (even with ESORT) only allows returning MIN, MAX, MODSEQ, and COUNT (from memory there might be 1 more, but I forget what it is) in addition to the matching UIDs/indexes. It's not that these aren't useful, but to optimize the "open a mailbox for the first time and display a message list to the user in the fastest way possible", they aren't all that helpful. Instead, what is needed is something more akin to the following (made up) command:

TAG001 UID SORT RETURN (OFFSET 0 LIMIT 50 FETCH (ENVELOPE FLAGS ...)) UTF-8 REVERSE DATE ALL

And this command could return untagged FETCH responses with the requested info - since we'd need a way to determine order, we could say that the FETCH results should be in the order requested or we could also have it return an untagged ESEARCH response like extended SORT commands do now to define the UID ordering.

In other words, make it so that SORT (and SEARCH, for that matter) return the information that we ultimately care about rather than the list of UIDs which we then would have to use to fetch the information we want.

(in hindsight, this is probably what you meant by "chatty" whereas I interpreted it to mean "verbosity" last night)

Basically, make it a bit more SQL-like.

Problem solved.

[brongondwana]

Yeah, OK - so you're doing what JMAP does but with a ton of search extension which STILL don't get you cross-folder support. Nice, problem solved - except you don't have a month to implement a good IMAP client, and neither does anyone else in the world.

Has ANYONE ever implemented a good IMAP client, in your opinion? If not, why not?

(JMAP is basically IMAP but more SQL-like, and without a bunch of custom syntax, yay)

[jstedfast]

I think plenty of people have time to implement a good IMAP client, just not me because I have a job that is completely unrelated to writing mail clients.

[brongondwana]

http://trojita.flaska.net/

two developers, 5 years, not yet feature complete - but it's the closest thing to someone attempting to write a really good IMAP client based on the RFCs, and they've been active in mailing lists and IRC too.

Funny how nobody has actually built this mythical IMAP client you say is so easy.

[jstedfast]

Wow, way to attack a straw man there.

[jstedfast]

> I can show you, right now, a 500,000 message view across multiple folders infinite scrolling with JMAP and I can jump to the middle of that view and load a screenful of messages. You CAN NOT do an IMAP sort without fetching one record for every message in the folder. If you have a million messages in a folder (yes, we have customers who do that) then you need to fetch a million records.

If I've got the sort order (from a UID SORT REVERSE DATE command), and a user jumps to the middle of the list, I can FETCH just the visible subset of messages by calculating what the UIDs would be and then doing:

TAG UID FETCH <visible-uid-set> (ENVELOPE FLAGS ...)

I'm not sure why you think you'd have to issue a FETCH command for each message individually.

[brongondwana]

Yeh, no shit - but you need to either have downloaded EVERY UID in that sort order (no updates either - if ANYTHING changes in the folder while you were disconnected you need to re-fetch the entire sort list) or you need sort extensions which don't even exist yet.

[jstedfast]

Then why did you say you couldn't if "yea, no shit" you knew you could?

If you've already grabbed ENVELOPE info for all of the older messages (i.e. all of the messages from a previous session), you can just download the ENVELOPE info of new messages and then sort locally.

I mean, if only a few messages have arrived since the previous session, it's probably faster to grab the ENVELOPEs and then sort locally.

If there's a lot of new mail, it might be more efficient to sort server side and just fetch the ENVELOPEs for the messages in view and lazy-load the rest when the user is idle (which is most of the time).

A dead-simple heuristic would be if the number of new messages is fewer than the number of messages you can show in a list, then it's probably not worth doing server-side sorting, right? Does that make sense? Especially as mobile devices get more and more powerful. You just need to design your sqlite database such that it's setup to do optimal sorting by whatever sorting method you are using (by date should be easy, right?).

I've actually done this sort of thing before (although not with IMAP), so I know it's doable. It just takes thought to design it and time to implement it.

I've studied the IMAP specs in my spare time over the past year or so and know them pretty well and I'm confident I could get good results if I had the time to implement it, but it's not like anyone in the world could implement a solid IMAP client in a week or even a month working on it full time and seeing as how I couldn't spend full-time working on writing a mail client due to the fact that I have a job that doesn't involve me writing said mail client, I don't see how you could possibly expect me to implement one in a month.