JMAP: Internal Error on Email/query

[jasonmunro]

Hello!
I'm working on implementing JMAP for my webmail client and while things are going quite well (great work on this!), I ran into an issue today replicating IMAP SEARCH with email/Query. I'm filtering a specific mailbox, with a subject match, and an "after" utcdate requirement.

Request:

POST /jmap/ HTTP/1.1
Host: localhost:8080
User-Agent: hm3
Authorization: Basic ...
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: application/json
Accept: application/json
Content-Length: 238

{"using":["urn:ietf:params:jmap:core","urn:ietf:params:jmap:mail"],"methodCalls":[["Email\/query",{"accountId":"testuser","filter":{"after":"2019-01-09T06:00:00Z","subject":"php","inMailbox":"77125b4f-d681-457f-b50e-3e1089e12d2e"}},"s"]]}

Response:

HTTP/1.1 200 OK
Date: Thu, 17 Jan 2019 04:43:59 GMT
Vary: Accept-Encoding
Content-Type: application/json; charset=utf-8
Content-Length: 108

{"methodResponses":[["error",{"type":"serverError","description":"Internal Error"},"s"]],"sessionState":"0"}

Syslog:
Jan 16 22:44:58 shop cyrus/http[10207]: jmap: _emailsearch_run: Internal Error

Git version:
git checkout at b7f171c197bdcd26e4a8207c67ebad1dd5d7e1af

If there is anything else I can provide to help troubleshoot (or if I am doing something daft) just let me know. Thanks for your time!

[zvasilev]

Hello Jason,

I think you need to add operator if you use more than one filter parameters.
For example this query work for me.

{"methodCalls":[["Email/query",{"calculateTotal":false,"collapseThreads":false,"filter":{"conditions":[{"inMailbox":"9684553c-ef62-4ba6-82a9-0a462228a64c"}],"operator":"OR"},"limit":25,"position":0,"sort":[{"isAscending":false,"property":"receivedAt"}]},"XzT7WA==#1"],"using":["urn:ietf:params:jmap:core","urn:ietf:params:jmap:mail"]}

[rsto]

I can't reproduce the issue on my local build, which correctly parses and executes the filter you specified:

      "Email/query",
        {
            "accountId": "testuser",
            "filter": {
                "after": "2019-01-09T06:00:00Z",
                "inMailbox": "77125b4f-d681-457f-b50e-3e1089e12d2e",
                "subject": "php"
            }
        },

The internal error in your syslog indicates that the search backend on your system is broken. Are you using Xapian, did you configure it in imapd.conf and did you run squatter to index your mail? You can verify this by running your filter with a text-only search, omitting any other search criteria, e.g.

      "Email/query",
        {
            "accountId": "testuser",
            "filter": {
                "subject": "php"
            }
        },

Depending on the outcome of this operation, we can investigate further why the query isn't working on your system.

@zvasilev An OR filter operator with a single argument is redundant (but the query executor will optimize it away). An equivalent filter to the original filter would be e.g.

    "filter": {
        "conditions": [
            {
                "inMailbox": "77125b4f-d681-457f-b50e-3e1089e12d2e"
            },
            {
                "after": "2019-01-09T06:00:00Z"
            },
            {
                "subject": "php"
            }
        ],
        "operator": "AND"
    },

[rsto]

@zvasilev Rereading you comment I now understand that you posted your filter just an example for a FilterOperator. Sorry for the confusion. Still: you don't need the OR operator in your example (but it doesn't hurt either).

[jasonmunro]

The internal error in your syslog indicates that the search backend on your system is broken. Are you using Xapian, did you configure it in imapd.conf and did you run squatter to index your mail? You can verify this by running your filter with a text-only search, omitting any other search criteria, e.g.

I apologize I was not aware of this extra setup! Once I configured Xapian, added squatter to cyrus.conf, manually ran squatter to do the indexing - it's working as expected. Thank you @rsto for the pointer on that, and again sorry for another issue that turns out to be bad configuration on my end!

[jasonmunro]

As this was just a configuration problem on my end, please feel free to close
this issue. While it's still open, I just want to say thanks for the great support!

I'm more than halfway through implementing JMAP in my webmail client, and it's
going way better than I expected. What I really appreciate about JMAP is that I
can use it in a very similar way to IMAP, which is great for a first pass at
integration, but I can already see how I can improve performance using JMAP
specific techniques.

Having a sorted and paginated query for messages in a mailbox is a HUGE win
for an IMAP client writer, but IMO the real special-sauce of JMAP is multi-method
requests with back references. Not only can I get a sorted paginated list, but
I can also get everything I need to populate a search result or mailbox page
listing all in the same request.

I'm sure I will submit another issue (or two or more) that turns out to be a
configuration problem on my end, so please accept my advanced apology. And
thank you once again for your quick follow ups and concise replies. Oh, and
thanks for JMAP - after many years of wrestling with IMAP I'm really enjoying
learning a new approach to E-mail :)

[rsto]

Great to hear that you got it sorted out! Feel free to post further issues, you can also post on the cyrus mailing list if you have questions.