pfsockopen will open a persistent socket connection like persistent database connections, which might be useful if we can figure out how to leverage it correctly. My guess is it may be more complicated than anything else we do.  

Another option is to have a helper program running as a daemon to handle the imap connections. That runs into portability options unless we can write that in PHP itself, but that seems ugly to me.  That leaves us back where we are I think. 

I think SASL CRAM MD5 is fairly efficient at doing multiple authentications as apart of a single "session" for web apps and the like. 

I think the pfsockopen route is impractical, but I only just now spent a few minutes looking at some info. As for a daemon that is loaded with pitfalls but may be the best route. I would rather do that in Perl than PHP (maybe), but that does create an additional server requirement. There are existing IMAP proxies that could be leveraged as well. The real issue for me on this is understanding the protocol enough to know how to handle the communications. Here is a relevant snippet from the RFC:

   An IMAP4rev1 connection consists of the establishment of a
   client/server network connection, an initial greeting from the
   server, and client/server interactions. These client/server
   interactions consist of a client command, server data, and a server
   completion result response.

Followed in a subsection by this:

  Server data MAY be sent as a result of a client command, or MAY be
  sent unilaterally by the server. There is no syntactic difference
  between server data that resulted from a specific command and server
  data that were sent unilaterally.

So, does that mean that as a client we should be prepared to receive unsolicited server information to our existing IMAP session, or does it mean that in the course of the client command -> server data -> server completion result the server can just insert additional information over and above the standard response data? If it's the former we have I think quite a bit more work if we want to take advantage of IMAP session persistence.

The client must send a command to the server to respond to, however the IDLE command changes things a little. When the client issues an IDLE command it allows the server to send unsolicited messages to the client (ie- you have new mail).  For a webclient, though, we want to be sure NOT to issue that command, as we dont have an IMAP session long enough to worry about it.

If we use a helper daemon, though, it might be worth using IDLE so we dont have to poll for new messages with each request. If we go with Perl, we maybe dont want to reinvent the wheel and just use some CPAN modules (and maybe improve it as we go). I think though its worth making a helper be optional, since there are times it will just get in the way (imagine the imap server is the web server, why would we want yet another daemon running?)

PS- there is no way to attribute comments to an author here. 

So if the client does not issue an IDLE command, we can code with the understanding that even if we have a proxy holding a persistent connection to the server open that it will not be required to listen and possibly queue unsolicited data from the server. Unfortunately not being able to use IDLE means we lose a good reason for being persistent in the first place. Really we then have only authentication overhead being saved by persistence. I remember emailing the IMAP proxy developers years ago to ask if their software would queue unsolicited server data. I don't remember getting a response, and my guess was that it did not.

I agree on optional for a helper, but what that could mean is some significantly different code paths for helper vs non-helper setups. Especially if the helper allows us to use features such as IDLE.

I understand that the IMAP server is capable of holding open lots of sessions at a time, but I worry about performance when the client (daemon) which usually runs on the same machine as the IMAP server, is the client to all those connections.

The other factor here is that while doing the quick connect->get data->disconnect thing is contrary to the design of IMAP, it is also proven to work pretty darn well. 

Its not really all that contrary to IMAP to connect->get_data->disconnect, its just inefficient.  The idea of having to rebuild TCP sessions, SSL/TLS negotiation, and do authentication every time you want to check for new messages is just wasteful, but wastefulness aside, it follows the spec.  Some installations may use remote IMAP servers where all that overhead is too much, others may keep the imap serveron the same host.  And even there, if you use some funky authentication scheme, maybe authenticating 600 times in 20 min is a bad thing. 

So the more I consider this, the more I realize the "helper" daemon is just going to end up being a caching IMAP proxy.  This gives us a good logical separation between the hastymail code and the "helper" and makes the helper completely optional.  It also means we dont have to do any of the work since there are already imap proxys out there (though not many). 

So I vote for keeping the IMAP code clean and to spec, not worrying about IDLE, and just try to get good SASL support, since rfc2831 describes the "subsequent auth" which can help some auth systems.

So if we don't have IDLE we are stuck with IMAP STATUS, and the SELECT response assuming we are selecting a mailbox during our page load, from which to determine if we need to run more costly queries regarding the mailbox contents. The STATUS response is unfortunately sometimes quite costly to run across many folders, which is why we only check for new mail in configured folders on the new mail page itself.

This is assuming a request for folder status is not being made behind the scenes via an ajax request, in which case who cares how long it takes UW to run a STATUS on 100 folders hosted over NFS :P

Seriously however if persistence is off the table (and it seems to have become no more realistic or beneficial than it was 5 years ago) then we can at least bring our response parsing up to speed to handle possibly unrequested server information being slipped into the data as well as the unsuspected literal (Ah EIMS, those were the days).

As for STARTTLS my guess is that maybe there is something in PHP5 but I am not aware of a mechanism in PHP4 that can take a non-TLS socket and after reading/writing to it make it a TLS connection without disconnecting.

Its best to support both. Some mail clients don't know how to do one or the other, some firewall admins like to keep ports to a minimum, others still like to block things that are not SSL, so regardless what the IMAP guys think, both is good.

 Its easy enough to do, too:

<?

 $fp = stream_socket_client("tcp://imap.example.com:143", $errno, $errstr, 30);
if (!$fp) {
  die("Unable to connect: $errstr ($errno)");
}

/* do stuff here */

/* Turn on encryption */

if($use_tls) {

        fwrite($fp, "STARTTLS\r\n");

         $response = fgets($fp);

         if($response == "PROCEED") { 

        stream_socket_enable_crypto($fp, true, STREAM_CRYPTO_METHOD_SSLv23_CLIENT);
       }
}
fwrite($fp, "LOGIN 'username' 'password'\r\n");

/* continue on ... */
fclose($fp);
?>

(god, typing code in here sucks, please excuse all the errors)