sb_imapfilter with Hotmail causes loss of emails

[alexeip0]

Tried using sb_imapfilter from 987bb19 with Hotmail account. Noticed that it occasionally deletes emails from the filtered folder (Inbox in my case) such that they cannot be recovered.

Tried to repro. I believe these were the steps.

1. Had 6 old emails in Inbox. Received 3 new ones.
2. Moved those 3 into training ham folder, restarted imapfilter - it reported learning them
3. Moved those 3 back into Inbox, restarted imapfilter - and 7 out of 9 emails disappeared.

This was the log from the last run:

47:17.74 > LIEA8 SELECT Inbox
47:17.79 < * 9 EXISTS
47:17.79 < * 3 RECENT
47:17.79 < * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
47:17.79 < * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
47:17.79 < * OK [UIDVALIDITY 14] UIDVALIDITY value
47:17.79 < * OK [UIDNEXT 138291] The next unique identifier value
47:17.79 < LIEA8 OK [READ-WRITE] SELECT completed.
47:17.79 > LIEA9 UID SEARCH UNDELETED
47:17.80 < * SEARCH 138256 138258 138262 138266 138270 138276 138288 138289 138290
47:17.80 < LIEA9 OK SEARCH completed.
47:17.80 > LIEA10 UID FETCH 138256 RFC822.HEADER
47:17.83 < * 1 FETCH (RFC822.HEADER {9673}
47:17.83 read literal size 9673
47:17.83 < UID 138256)
47:17.83 < LIEA10 OK FETCH completed.
[imapfilter] already classified: 138256
47:17.83 > LIEA11 UID FETCH 138258 RFC822.HEADER
47:17.85 < * 2 FETCH (RFC822.HEADER {8826}
47:17.85 read literal size 8826
47:17.85 < UID 138258)
47:17.86 < LIEA11 OK FETCH completed.
[imapfilter] already classified: 138258
47:17.87 > LIEA12 UID FETCH 138262 RFC822.HEADER
47:17.88 < * 3 FETCH (RFC822.HEADER {8898}
47:17.88 read literal size 8898
47:17.88 < UID 138262)
47:17.89 < LIEA12 OK FETCH completed.
[imapfilter] already classified: 138262
47:17.89 > LIEA13 UID FETCH 138266 RFC822.HEADER
47:17.91 < * 4 FETCH (RFC822.HEADER {8811}
47:17.91 read literal size 8811
47:17.91 < UID 138266)
47:17.92 < LIEA13 OK FETCH completed.
[imapfilter] already classified: 138266
47:17.92 > LIEA14 UID FETCH 138270 RFC822.HEADER
47:17.94 < * 5 FETCH (RFC822.HEADER {8810}
47:17.94 read literal size 8810
47:17.94 < UID 138270)
47:17.95 < LIEA14 OK FETCH completed.
[imapfilter] already classified: 138270
47:17.95 > LIEA15 UID FETCH 138276 RFC822.HEADER
47:17.97 < * 6 FETCH (RFC822.HEADER {8882}
47:17.97 read literal size 8882
47:17.97 < UID 138276)
47:17.98 < LIEA15 OK FETCH completed.
[imapfilter] already classified: 138276
47:17.98 > LIEA16 UID FETCH 138288 RFC822.HEADER
47:18.01 < * 7 FETCH (RFC822.HEADER {19919}
47:18.01 read literal size 19919
47:18.01 < UID 138288)
47:18.01 < LIEA16 OK FETCH completed.
[imapfilter] already classified: 138288
47:18.01 > LIEA17 UID FETCH 138289 RFC822.HEADER
47:18.04 < * 8 FETCH (RFC822.HEADER {18721}
47:18.04 read literal size 18721
47:18.04 < UID 138289)
47:18.04 < LIEA17 OK FETCH completed.
[imapfilter] classified as None: 138289
47:18.04 > LIEA18 UID FETCH 138289 (BODY.PEEK[])
47:18.07 < * 8 FETCH (BODY[] {112973}
47:18.07 read literal size 112973
47:18.08 < UID 138289)
47:18.08 < LIEA18 OK FETCH completed.
47:18.12 > LIEA19 UID FETCH 138289 (FLAGS INTERNALDATE)
47:18.13 < * 8 FETCH (FLAGS (\Seen \Recent) INTERNALDATE "30-Jan-2021 18:45:00 -0800" UID 138289)
47:18.20 < LIEA19 OK FETCH completed.
47:18.21 > LIEA20 APPEND Inbox (\Seen) "30-Jan-2021 18:45:00 -0800" {112925}
47:18.22 < + Ready for additional command text.
47:18.22 write literal size 112925
47:18.63 < * 4 RECENT
47:18.63 < * 10 EXISTS
47:18.63 < LIEA20 OK [APPENDUID 14 138291] APPEND completed.
47:18.63 > LIEA21 UID STORE 138289 +FLAGS.SILENT (\Deleted \Seen)
47:18.70 < LIEA21 OK STORE completed.
[imapfilter] found saved message 138289 in iteration 0
47:18.70 > LIEA22 UID SEARCH (UNDELETED HEADER X-Spambayes-MailId "26")
47:18.72 < * SEARCH 138258 138262 138266 138270 138276 138288 138290 138291
47:18.72 < * 8 EXPUNGE
47:18.72 < * 9 EXISTS
47:18.72 < LIEA22 OK SEARCH completed.
47:18.72 > LIEA23 UID STORE 138258 +FLAGS.SILENT (\Deleted \Seen)
47:18.83 < LIEA23 OK STORE completed.
47:18.83 > LIEA24 UID STORE 138262 +FLAGS.SILENT (\Deleted \Seen)
47:18.91 < * 2 EXPUNGE
47:18.91 < * 8 EXISTS
47:18.91 < LIEA24 OK STORE completed.
47:18.91 > LIEA25 UID STORE 138266 +FLAGS.SILENT (\Deleted \Seen)
47:19.00 < * 2 EXPUNGE
47:19.00 < * 7 EXISTS
47:19.00 < LIEA25 OK STORE completed.
47:19.00 > LIEA26 UID STORE 138270 +FLAGS.SILENT (\Deleted \Seen)
47:19.08 < * 2 EXPUNGE
47:19.08 < * 6 EXISTS
47:19.08 < LIEA26 OK STORE completed.
47:19.08 > LIEA27 UID STORE 138276 +FLAGS.SILENT (\Deleted \Seen)
47:19.16 < * 2 EXPUNGE
47:19.16 < * 5 EXISTS
47:19.16 < LIEA27 OK STORE completed.
47:19.16 > LIEA28 UID STORE 138288 +FLAGS.SILENT (\Deleted \Seen)
47:19.23 < * 2 EXPUNGE
47:19.23 < * 4 EXISTS
47:19.23 < LIEA28 OK STORE completed.
47:19.23 > LIEA29 UID STORE 138290 +FLAGS.SILENT (\Deleted \Seen)
47:19.30 < * 2 EXPUNGE
47:19.30 < * 3 EXISTS
47:19.30 < LIEA29 OK STORE completed.
47:19.30 > LIEA30 UID FETCH 138290 RFC822.HEADER
47:19.32 < * 2 EXPUNGE
47:19.32 < * 2 EXISTS
47:19.32 < LIEA30 NO Some of the requested messages no longer exist.
47:19.32 NO response: Some of the requested messages no longer exist.
SpamBayes IMAP Filter Version 1.1b3 (Nov 23, 2017).

Loading database C:\Users\alexei\Documents\SpamBayes\hammie.db... Done.
User interface url is http://localhost:8880/
Account: outlook.office365.com:993
Training
Classifying
........�*Traceback (most recent call last):
File "C:\Python27\Scripts\sb_imapfilter.py", line 4, in
import('pkg_resources').run_script('spambayes==1.1b3', 'sb_imapfilter.py')
File "C:\Python27\lib\site-packages\pkg_resources_init_.py", line 666, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "C:\Python27\lib\site-packages\pkg_resources_init_.py", line 1462, in run_script
exec(code, namespace, namespace)
File "c:\python27\lib\site-packages\spambayes-1.1b3-py2.7.egg\EGG-INFO\scripts\sb_imapfilter.py", line 1324, in
run()
File "c:\python27\lib\site-packages\spambayes-1.1b3-py2.7.egg\EGG-INFO\scripts\sb_imapfilter.py", line 1306, in run
imap_filter.Filter()
File "c:\python27\lib\site-packages\spambayes-1.1b3-py2.7.egg\EGG-INFO\scripts\sb_imapfilter.py", line 1111, in Filter
self.unsure_folder, self.ham_folder)
File "c:\python27\lib\site-packages\spambayes-1.1b3-py2.7.egg\EGG-INFO\scripts\sb_imapfilter.py", line 955, in Filter
for msg in self:
File "c:\python27\lib\site-packages\spambayes-1.1b3-py2.7.egg\EGG-INFO\scripts\sb_imapfilter.py", line 811, in iter
yield self[key]
File "c:\python27\lib\site-packages\spambayes-1.1b3-py2.7.egg\EGG-INFO\scripts\sb_imapfilter.py", line 842, in getitem
"fetch %s rfc822.header" % (key,), response)
File "c:\python27\lib\site-packages\spambayes-1.1b3-py2.7.egg\EGG-INFO\scripts\sb_imapfilter.py", line 295, in check_response
raise BadIMAPResponseError(command, IMAP_response)
main.BadIMAPResponseError: The command 'fetch 138290 rfc822.header' failed to give an OK response.
('NO', ['Some of the requested messages no longer exist.'])

[smontanaro]

Not sure I can help much with this. I don't use SpamBayes in an interactive setting anymore (still use it to train the SpamBayes database for mail.python.org but don't contact a live mail server - and have never used sb_imapfilter) and don't use Hotmail. By specifying Hotmail are you implying that you have used sb_imapfilter with other IMAP servers and not experienced message lossage? My expectation is that SpamBayes should never delete email, but again, I've never used sb_imapfilter.

One thing to be aware of with IMAP (I only recently learned this) is that UIDs are per-folder. By moving messages back and forth, I suppose it's possible that either sb_imapfilter or the Hotmail server is getting confused.

Can you attach your config file with your Hotmail credentials xxxxxx-d out?

[alexeip0]

are you implying that you have used sb_imapfilter with other IMAP servers and not experienced message lossage

No, this is my first time trying sb_imapfilter

My expectation is that SpamBayes should never delete email

I suspect this comment in sb_imapfilter points to what might be happening:

"""Warnings:
o We never delete mail, unless you use the -e/purge option, but we do
mark a lot as deleted, and your mail client might remove that for
you. We try to only mark as deleted once the moved/altered message
is correctly saved, but things might go wrong. We strongly
recommend that you try this script out on mail that you can recover
from somewhere else, at least at first.
"""

Now if I look at gmail vs hotmail IMAP settings there are these extra options that gmail provides and hotmail doesn't:

When I mark a message in IMAP as deleted:
Auto-Expunge on - Immediately update the server. (default)
Auto-Expunge off - Wait for the client to update the server.

When a message is marked as deleted and expunged from the last visible IMAP folder:
Archive the message (default)
Move the message to the Trash
Immediately delete the message forever

I have a feeling hotmail behaves like gmail with "Auto-Expunge on" and "Immediately delete the message forever".

I'm wondering if someone with IMAP expertise can comment on the above theory or maybe propose a workaround for hotmail.

Can you attach your config file with your Hotmail credentials xxxxxx-d out

[imap]
server:outlook.office365.com
use_ssl:True
username: xxxxxx
filter_folders:Inbox
spam_folder:Inbox/spam.90
unsure_folder:Inbox/spam
ham_train_folders:Inbox/train.ham
spam_train_folders:Inbox/train.spam

[smontanaro]

Thanks. I suspect you are correct about the quoted warning. I'm a Gmail user. It behaves differently, not really having folders, but labels. A message can have any number of labels, which is more-or-less like it residing in multiple folders simultaneously. Maybe I'm missing something (again, never having used this particular program), but if I wanted to assemble a set of ham and spam on my IMAP server, I wouldn't want any messages to be deleted, particularly ham. I can understand that you might eventually want to delete some messages, but not without user control. Again, considering my status as a Gmail user, I use its filter features a lot. I have some mail I never delete. I would, at most, want such messages marked as ham or spam. I wouldn't want them moved anywhere, though with Gmail's label feature that wouldn't be a terrible problem. I wouldn't want to have to rummage around in messages moved to a ham folder and manually put them back where they belong. I use a program to generate XKCD 936 passwords. It uses IMAP4 to peek at suitably tagged mail (messages containing the "polly" label) to construct its dictionary. It doesn't modify any messages and doesn't mark anything as seen: https://github.com/smontanaro/polly Other than adding suitable message headers, it's not clear to me why sb_imapfilter has to modify messages. I'll see if I can set it up to talk to my Gmail account.