UID Missing?

[RyanThompson]

Hello - I have been wondering how I can retrieve a specific message outside of it's temporary queue position (message number)?

I don't see the UID / SE_UID in the get_header_array().

Is there a way to pick out a message like this using Peeker?

Thanks a ton!

[sophistry]

Hi RyanThomson,

Thanks for the question. As you noted, the peeker class is currently designed around the temporary message number. It would mean a few changes to the core code to make it use the more "permanent" UID. I put permanent in quotes because the UID can change per message - specifically per mailbox. Also, I have not done strict testing on this, but the comments in the PHP docs indicate that the UID can actually change. http://php.net/manual/en/function.imap-uid.php

Finally, since the original peeker library was built around a POP3 server core, I have tried to minimize the use of functions that do not work on both IMAP and POP3 (UID is not applicable in POP3).

However, I think most people using this library are using it on IMAP and the UID can be an important data point.

So, all it would take to get the UID into the data returned by get_header_array() would be to add the imap_uid() function to the message() function.

At line 162 of peeker.php imap_uid() could insert an array element with the UID. Also, you would have to add a $uid property to the peeker_header class so the _set_class_vars() function brings in that data and makes it available to the rest of the code.

What do you think about adding it, testing it, and sending a pull request?

[RyanThompson]

I certainly will do that! That's interesting the way it behaviea though (the uid). Whatnot the standard method of pinpointing a message on the server? Cause that's really what I'm after.

The most I've seen is using a combination of mailbox / uid search. The examples I ran into included to and from values too but seems that a unique uid within a given box would be fine. As long as the app is mindful that the uid may very well change when it's moved to a different box.

Am I understanding that correctly?

Great work. Very nice API. Ill get that PR in soon ^_^

On a second note, how can I pull entire messages for storing without marking as SEEN? The docs only briefly mentions it but that's all. Would I just have to loop and remark as UNSEEN?

Thanks again,

Ryan

[sophistry]

It seems that it's up to the IMAP server as to how it implements SEEN/UNSEEN - I discovered that gmail has its own methods and you can only really pull certain bits of data before gmail thinks the message is SEEN. Then gmail will do what it is told to do with the message.

This server-side marking pretty much precludes any "pull entire message and keep marked UNSEEN" design except if you do what you mentioned: pull everything you need from the message and then explicitly mark it UNSEEN. I have not tried that but there's no reason why it wouldn't work except if a particular IMAP server has its own rules about marking messages.

Thanks for the feedback! Much appreciated.

[RyanThompson]

Got the UID in and it's working perfectly - added a couple similar helper functions as well.

One question - Seems like this is rather slow.. Is other webmail apps seem a bit faster, I wonder if my code is inefficient. Im don't need to download the entire message. Just headers and perhaps the body text. No attachments.

Here is my code, peeker in it's simplest form:

$messages = $this->peeker->get_message(0, 10);

When I ramp that up to 25 and higher for archiving tasks, it takes like 15 seconds. I would think it can grab necessary info in no time. BUT I am not sure if it is grabbing the entire message or what. Seems like the docs reference to peeking without disturbing seen status is not clear.

I tried this:

$this->peeker->set_message_class('peeker_header');
$messages = $this->peeker->get_message(0, 10);

And it 500s at me.

And lastly.. Do you know of a good synching methodology? Or have any reference you can point me to? Basically just synch if a message(s) has been deleted, flagged, moved, etc. Seems like searching and updating my entire database using "where in" is the only way..

Thanks again for all the help. Once I get my app working properly and test it out Ill send that PR. Seems fine as it is though.

[RyanThompson]

I think this is all because I don't quite understand detectors.. Very foggy to me. All I can get from different peeker_headers classes n stuff is 500 errors.

[sophistry]

Hi Ryan,

To get the set_message_class('peeker_header'); command to work you need to pair it with $this->peeker->message(); method. The get_message() command automatically loads the peeker_parts class - it's a wrapper to make the peeker library easier to get started with. Hoever, as you discovered, it can be a little bit slow to download attachments when you don't need them! If you use the set_message_class() you can tell it to load the peeker_header, the peeker_body, or the peeker_parts class. If you use set_message_class() you need to only use the message() method.

I suppose the docs need a little bit of detail in that area. So, I'll add this information.

IMAP connections to gmail (at least) are slow. Any caching you can do will speed up how fast it seems. The IMAP projects that ship with UIs attached all implement a message cache - even though IMAP is supposed to make it so you don't have to cache!

As far as synching, it sounds like you are talking about keeping your local copies of messages in synch with the messages that remain on the IMAP server. I do this by relying on the IMAP server "seen" rules and just having the IMAP server move the message from the INBOX. Then I only check the INBOX and load in new messages. To have a failsafe, peeker gives you the ability to "fingerprint" a message (defaults to a hash of the to from subject and date headers).

Detectors are a way to structure code so that it is easy to add new functionality to peeker and contribute back. It's basically the plugin architecture for the peeker email framework. It encourages you to create modular code.

[RyanThompson]

Wonderful! This library is really turning out to be very nice.

There is one huge glaring issue though. UID support. I added in some quick fixes to do ALL of the same functions using UID also if desired. The reason being is that there is no way to safely pick out a message on the server. Msgno always changes, fingerprint is local only, etc. So I added a pretty powerful set of features by simply upping a few of yours lol.

Cache is really the issue too. Storing data locally in a db only makes sense. IMAP only seems to cache very little. So being able to search, manipulate and mirror changes to the server.. MUCH faster.

So I will test a bit more and send in 1 big PR - And we can discuss it from there. I highly suggest adding the ability to do these things by UID otherwise the library seems really limited in a weird way. Not limited.. but limited you know?

There are only two things more I would suggest, and will in turn build:

1. Optionally leave //unseen flags when fetching the body
2. Related to above - But return ONLY a preview text value built from plain / html values and again, leaving //unseen or whatever that flag is as is. Doing this by fecthing parts 0, 1, 1,1, 1,2 I think they are. That way any attachments are ignored.

Thoughts?

[RyanThompson]

PS - With the changes above the app works great:

1. Syncing only inserts the mailbox and UID of messages. Mark all as deleted first, then insert / update seen status and not-deleted on duplicate entries. Effectively telling us WHAT is in a given mailbox.
2. Pulling headers and body text when previews are needed for a given message and storing those too in a db.

The rest is pretty self explanatory. But the UID is mandatory because when fetching what messages to read / display from the db, UID is used as it is unique per mailbox. Then I can get_message_by_uid(array of UIDs) and store them real quick before rendering. Then they are there for later too.

Can you think of any holes? I am moving into testing now that my brain is dead from all this learning so fast lol.

Thanks again for your efforts. Way cool.

[sophistry]

sounds great! i am looking forward to seeing what you put together. i agree that proper UID support is the best feature to add.

[RyanThompson]

Ok, scrapped my changes and approaching with a clean view. Learning a lot though. Have some questions.

1. I think it would be best to duplicate the message() method to message_by_uid() ONLY because the way it loops. Everything else can use $this->peek_parent->use_uid?FT_UID:0 type stuff in the options. Then the only difference would be that message_by_uid would be single only and the Msgno would be treated as UID per PHP IMAP instrucitons.
2. I would like to be able to get number of attachments or boolean has_attachments without downloading them.. Seems like the only way those methods work is if the entire message is fetched which defeats the purpose.
3. Adding a top() type function to grab the first 250 characters from the body - thoughts?
4. No flags are given / detected? I would prefer to use only peek_header and there really should be an ability to nab flag statues from there.. I see there are Flagged / Unseen / Recent variables in the peeker_header.php file, perhaps I am not utilizing it properly?

That's it for now - Adding my CLEAN changes tonight.

[RyanThompson]

Oh and should be able to grab by parts / like grab body content but not messages or omit CSV, ZIP, etc file types.. Thoughts or ideas? Still very much learning.

I have bounced back and forth from the capable IMAP_Client by Atmail using MailParser and it has a great feature set and is QUICK (Uses PEAR/Net/Socket) but would really like to use Peeker if I can get the features out of it.

I think they are quite easy to put in - Just real new at this stuff yet..

[sophistry]

I know what you are saying about the loop in the message() method - it would have to be re-written to use UIDs. But, this is fine and an internal change like this should not affect the usage of the libraries - you'd still have MsgNo around if needed.

Some of the things you mention are better dealt with in layers classes (basically set up as traits because there weren't traits when i made this library!). It involves just breaking out the functionality that you want and putting the methods in a layers class. Then name that class per the naming convention ($layer_name = 'peeker_'.$layer.'_methods';) and put it in the directory with the peeker files. So for the top() method you mention it would be a method in a new class that is layered in and depends on the body class being loaded so that peeker can grab the data needed to perform the substr().

Flags should be set into the properties when the message is loaded if they are set in the message data returned by the imap lib call in the message() method. They won't be set if there are no flags sent by the imap server.

Thanks for the feeback! Keep it coming.

[RyanThompson]

First commit - Starting simple: https://github.com/ryanthompson/peeker
Adds basic UID options to most methods. I keep adding them as I find / need them lol.

[RyanThompson]

So flags.. I still can't get those to work. Only the headers (peeker_header class_ is needed to determine flags right?

So after get_messages() in a foreach($messages as $message) for example - how would the flags work? $message->Unseen and the others are never set?

[RyanThompson]

How can I pull the parts of the message only? Like what's there. There needs to be a way to get a list of parts AND get only specific parts. Loading massive ZIP attachments just to read the email shouldn't be required. Is this possible? If not, where would you suggest I start building it at? The peeker_parts I would assume. Ideas?

Anything you know of to improve performance? Just a tich slower than Im used to but I wonder if that's the IMAP library with PHP. Specifically when loading the body.

[RyanThompson]

Seems related: https://github.com/Witti/peeker/commit/80f8142c81a75b4dcbdc9b7b9cc39d30b51b1655

We should probably have methods for all those goodies right?

[sophistry]

Hi Ryan,

This was a productive thread for a while. Any development on your front?

Thanks!

[RyanThompson]

Not just yet - I had to put the project on the back burner for the time being - However the CURRENT project I am working on calls for this library to be maxed out very soon. I'll be picking up activity then.

It's by far, when features and simplicity are considered, the best wrapper I've found thus far. Good stuff ^_^