Show SMTP Queue ID in Outgoing Emails

[freescout-helpdesk]

In the "Outgoing Emails" pop-up for a sent reply, when the status is "Accepted for delivery", show the ID that the receiving SMTP server gave when accepting it. This makes it much easier to trace, especially on busy SMTP servers. Plus, if the SMTP ID is known, then custom code can be written that hooks into the mailserver in order to show the log entries from there.

#3327 (comment)

[freescout-helpdesk]

https://stackoverflow.com/questions/27656923/how-to-get-email-queue-id-with-swiftmailer
https://stackoverflow.com/questions/33863328/how-to-set-swifmailer-plugins-when-using-the-laravel-mail-class

[freescout-helpdesk]

It is possible to receive an SMTP Queue ID via Swiftmailer plugin:

250 2.0.0 Ok: queued as 6mJpz6J4RdhGh

But we can't determine to which thread this SMTP Queue ID belongs to as Swift_Events_ResponseEvent does not have any extra data allowing to do it:
https://github.com/freescout-helpdesk/freescout/blob/dist/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Events/ResponseEvent.php

[naturaw-canine-food]

I did this for another project internally. You gotta first Create the SwiftGetSmtpQueueId class where the responseReceived method extracts the SMTP queue ID from the response and then associates it with the email thread.

...and then implement the logic to associate the SMTP queue ID with the email thread.

namespace App\Misc;

use Swift_Events_ResponseEvent;
use Swift_Events_ResponseListener;

class SwiftGetSmtpQueueId implements Swift_Events_ResponseListener
{
    public function responseReceived(Swift_Events_ResponseEvent $evt)
    {
        $response = $evt->getResponse();
        $response_parts = explode(' ', $response);
        $smtp_queue_id = end($response_parts);
    
        // Get the thread_id or conversation_id from the $evt object.
        // The $evt object may not have this information directly, 
        // so you may need to modify the Freescout application 
        // to include this information in the $evt object when it is created.
        $thread_id = $evt->getThreadId();
        $conversation_id = $evt->getConversationId();
    
        // Store the $smtp_queue_id in the database associated with the thread_id or conversation_id.
        // You might create a new table in the database to store this information,
        // or you might store it in an existing table.
    
        // Here is an example of how you might store it in a new table:
        $smtpQueueId = new SmtpQueueId();
        $smtpQueueId->thread_id = $thread_id;
        $smtpQueueId->conversation_id = $conversation_id;
        $smtpQueueId->smtp_queue_id = $smtp_queue_id;
        $smtpQueueId->save();
    }
}

...you would need to create a new SmtpQueueId model and a new table in the database to store the thread_id, conversation_id, and smtp_queue_id. You would also need to modify the FS to include the thread_id and conversation_id in the $evt object when it is created.

You will prob need to edit FS and the database schema to accommodate this new functionality. Also, you will need to handle potential errors, such as the $evt object not having the thread_id or conversation_id, or the database operation failing.

[freescout-helpdesk]

@DavidAnderson684 We are trying to figure out why you may need SMTP Queue ID. Normally if a mail server can not deliver an email to the recipient it sends a bounce email containing all the details to FreeScout. Your mail server does not send bounce emails?

[DavidAnderson684]

We get bounce mails. But the cases that are more time-consuming are when the recipient doesn't see the reply for some other reason than a bounce - e.g. their spam filter silently consumed the email (far too many commercial providers do this). This results in someone having to trace the logs to see exactly whose systems lost the mail. Anything that makes this a quicker process is helpful. Without the acceptance ID showing in Freescout, someone has to read the logs and pick out the conversation via searching via non-unique fields (time, email address), so it adds inefficiency to the process (and makes it harder to automate in code).

[freescout-helpdesk]

@DavidAnderson684 Can you give an example of your SMTP Queue ID to choose proper DB field type.

[freescout-helpdesk]

Also what about Message-ID? Maybe it will be enough to track "lost" emails?

[DavidAnderson684]

Example of an SMTP queue ID from exim (www.exim.org - the most common SMTP server on the Internet according to http://www.securityspace.com/s_survey/data/man.201801/mxsurvey.html) is 1qdTQv-009wPw-37. The first character is always a 1.

The Message-ID is better than nothing; it's theoretically unique (though nothing guarantees this since it is client-generated and clients can't communicate with eachother), and a search on the logs can then identify lines that match it. But the queue ID is the "primary key" as far as the mailserver is concerned, and can be searched quickly especially if storing your logs in a structured format, so is much faster.

Here is an example of the log entries for an uncomplicated mail that came in an out on exim. The Message-ID is there, but one would have to first search for it to extract the ID the mailer has used (the one it gives as the queue ID in the SMTP conversation), and then perform a second search on that.

2023-09-05 10:37:23 1qdTQv-009wPw-37 <= [email protected] H=support.example.com (support.example.com) [1.2.3.4] P=esmtpsa X=TLS1.3:TLS_AES_256_GCM_SHA384:256 CV=no A=authenticate_virtual:[email protected] S=19534 [email protected] T="[#12345] Example.com Support Request"
2023-09-05 10:37:23 1qdTQv-009wPw-37 => support-agent-1 <[email protected]> R=virtual_router T=dovecot_virtual_delivery_lmtp C="250 2.0.0 <[email protected]> sJGIDeME92TPISQAj9TpAg Saved"
2023-09-05 10:37:23 1qdTQv-009wPw-37 Completed

[freescout-helpdesk]

Implemented in the master branch.