Review the whole process/code/documentation for Cypht to work for Google and Microsoft's email offerings. (was: Error Ocurred when attempting gmail oauth2 setup)

[Terramoto]

I'm running the docker version, i've:

• setup the oauth2.ini file in app_data path specified in hm3.ini
• php script/config_gen.php

The file is detected during the config_gen.php, i then proceed to the interface:

• pick up the gmail option from the drop-down list.
• Setup the email address and name,
• Proceed to the Google authentication, permissions page,
• Check emails and calendars access permissions and then it redirects me to the webmail url but signed out. I type in login details but a "Error Ocurred" is displayed and the url reverts to the default webmail url path.

I've notice the redirect URL sends two scopes:
?state=nux_authorization&code=(Authorization-Code)&scope=https://mail.google.com/%20https://www.googleapis.com/auth/contacts

I've tried removing the second scope and this time i got the "Error Ocurred" message but it didn't signed me off....

[marclaporte]

Thank you @Terramoto for this report.

Can you try with debug mode activated as per https://cypht.org/install.html ?

Thanks!

[Terramoto]

Thank you @Terramoto for this report.

Can you try with debug mode activated as per https://cypht.org/install.html ?

Thanks!

I'm afraid it doesn't seem to work, i'm using the docker version provided in the README:
https://hub.docker.com/r/sailfrog/cypht-docker

# ln -s /usr/local/share/cypht /var/www/html/mail-debug
ln: /var/www/html/mail-debug: No such file or directory

I've tried to set the display_errors on in the PHP.ini file but that hasn't made much of a difference.

[marclaporte]

@rodriguezny How can we activate errors in Cypht Docker?

[rodriguezny]

@Terramoto did you try to visualize the container log ?

[Terramoto]

@rodriguezny Yes, i'm afraid it was only showing the access logs, as in POST GET requests and response codes.

I went to check this morning and found there's a php.ini-development in /usr/local/etc/php/, replaced the existing with it and after attempting to add the account i'm now getting the following Notice but no errors:

11.11.11.17 - - [04/Oct/2023:09:45:30 +0000] "GET /?state=nux_authorization&code=4/0AfJohXmsmSbycXNtvcr-_-_OlcsMc0GLsi6s7WvcrI24HlbtmrjGXhfA6XsNnA&scope=https://mail.google.com/%20https://www.googleapis.com/auth/contacts HTTP/1.1" 200 1562 "https://accounts.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
NOTICE: PHP message: PHP Notice:  Trying to access array offset on value of type bool in /usr/local/share/cypht/modules/nux/modules.php on line 104
NOTICE: PHP message: PHP Notice:  Trying to access array offset on value of type bool in /usr/local/share/cypht/modules/nux/modules.php on line 104
NOTICE: PHP message: PHP Notice:  Trying to access array offset on value of type bool in /usr/local/share/cypht/modules/nux/modules.php on line 104
NOTICE: PHP message: PHP Notice:  Trying to access array offset on value of type bool in /usr/local/share/cypht/modules/nux/modules.php on line 105
127.0.0.1 -  04/Oct/2023:09:45:34 +0000 "POST /index.php" 303

[rodriguezny]

Thanks @Terramoto for the report.
The notices are due to php version, no need to be worried about that, but the log doesn't show the error that causes the "error occurred" message when attempting gmail auth2 message.
Adding a Gmail account seems broken or there is an issue we need to fix. Give us some time to investigate on the issue on cypht without docker and cypht docker version, will come back to you later.

Did you try to setup gmail auth2 on cypht outside docker (no docker version) ? If no, you can try it and provide us feedback waiting for us to come back to you with the fix/solution ?

[marclaporte]

For the record: https://github.com/cypht-org/cypht/wiki/OAUTH2-over-IMAP

[rodriguezny]

Thank you @marclaporte !

[Terramoto]

I've managed to enable some logging:

2023/10/05 11:56:20 [error] 44#44: *35 FastCGI sent in stderr: "PHP message: Array
(
    [0] => Using Hm_DB_Session with Hm_Auth_DB
    [1] => Using file based user configuration
    [2] => Already registered module for message re-attempted: imap_show_message
    [3] => Already registered module for message re-attempted: imap_message_list_type
    [4] => Already registered module for message re-attempted: imap_server_ids
    [5] => Using sapi: fpm-fcgi
    [6] => Request type: HTTP
    [7] => Request path: /
    [8] => TLS request: 0
    [9] => Mobile request: 0
    [10] => Page ID: servers
    [11] => Using Hm_Cache for cache
    [12] => Redis enabled but not supported by PHP
    [13] => Memcached enabled but not supported by PHP
    [14] => CACHE backend using: noop
    [15] => Connecting to dsn: mysql:host=db;dbname=cypht
    [16] => LOGGED IN
    [17] => HTTP header fingerprint check failed
    [18] => Deleting cookie: name: hm_session, lifetime: 1696503380, path: /, domain: webmail.terramoto.xyz, secure: , html_only 1
    [19] => Deleting cookie: name: hm_id, lifetime: 1696503380, path: /, domain: webmail.terramoto.xyz, secure: , html_only 1
    [20] => Deleting cookie: name: hm_reload_folders, lifetime: 1696503380, path: , domain: webmail.terramoto.xyz, secure: , html_only 
    [21] => Deleting cookie: name: hm_msgs, lifetime: 1696503380, path: /, domain: webmail.terramoto.xyz, secure: , html_only 1
    [22] => Deleting cookie: name: hm_msgs, lifetime: 1696503380, path: /, domain: webmail.terramoto.xyz, secure: , html_only 1
    [23] => TRANSLATION NOT FOUND :Cypht:
    [24] => TRANSLATION NOT FOUND :An Error Occurred:
    [25] => PHP version 7.4.33
    [26] => Zend version 3.4.0
    [27] => Peak Memory: 4096
    [28] => PID: 74
    [29] => Included files: 78
)" while reading response header from upstream, client: 11.11.11.17, server: localhost, request: "GET /?page=servers HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "webmail.terramoto.xyz", referrer: "https://webmail.terramoto.xyz/?state=nux_au

It looks like the cause is:
[17] => HTTP header fingerprint check failed

cypht/lib/session_base.php

Line 42 in c240489

public function check_fingerprint($request) {

I've disabled fingerprint in h3m.ini, now i get the following:

2023/10/05 12:22:29 [error] 46#46: *104 FastCGI sent in stderr: "PHP message: Array
(
    [0] => Using Hm_DB_Session with Hm_Auth_DB
    [1] => Using file based user configuration
    [2] => Already registered module for message re-attempted: imap_show_message
    [3] => Already registered module for message re-attempted: imap_message_list_type
    [4] => Already registered module for message re-attempted: imap_server_ids
    [5] => Using sapi: fpm-fcgi
    [6] => Request type: HTTP
    [7] => Request path: /
    [8] => TLS request: 0
    [9] => Mobile request: 0
    [10] => Page ID: servers
    [11] => Using Hm_Cache for cache
    [12] => Redis enabled but not supported by PHP
    [13] => Memcached enabled but not supported by PHP
    [14] => CACHE backend using: noop
    [15] => Connecting to dsn: mysql:host=db;dbname=cypht
    [16] => LOGGED IN
    [17] => IDLETIMER: timer exceeded, logged out
    [18] => Deleting cookie: name: hm_session, lifetime: 1696504949, path: /, domain: webmail.terramoto.xyz, secure: , html_only 1
    [19] => Deleting cookie: name: hm_id, lifetime: 1696504949, path: /, domain: webmail.terramoto.xyz, secure: , html_only 1
    [20] => Deleting cookie: name: hm_reload_folders, lifetime: 1696504949, path: , domain: webmail.terramoto.xyz, secure: , html_only 
    [21] => Deleting cookie: name: hm_msgs, lifetime: 1696504949, path: /, domain: webmail.terramoto.xyz, secure: , html_only 1
    [22] => Deleting cookie: name: hm_msgs, lifetime: 1696504949, path: /, domain: webmail.terramoto.xyz, secure: , html_only 1
    [23] => TRANSLATION NOT FOUND :Cypht:
    [24] => TRANSLATION NOT FOUND :An Error Occurred:
    [25] => PHP version 7.4.33
    [26] => Zend version 3.4.0
    [27] => Peak Memory: 2048
    [28] => PID: 74
    [29] => Included files: 78
)" while reading response header from upstream, client: 11.11.11.29, server: localhost, request: "GET /?page=servers HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "webmail.terramoto.xyz", referrer: "https://webmail.terramoto.xyz/?state=nux_

How to enable debug:

cypht/index.php

Line 18 in c240489

define('DEBUG_MODE', true);

[Terramoto]

I'm reopening this issue because i had been told previously the docker image was not properly maintained, so i started from fresh.
Unfortunately, after dealing with other issues where the login was failing because open_basedir wasn't allowing the sqlite2 file access. I've managed to sort this how and get to the point of configuring the oauth2 again.

The issue with Oauth2 repeats on this new setup, a normal imap account works. What i'm noticing with this one is that the authorization token takes a while, after signing in to the webmail, and this seems to be because it's calling sendmail.

[apezio]

I had an issue - Google would redirect me back to Cypht after I approved oauth2 access, BUT the Cypht webpage was only showing a login page even though I had been logged in moments ago. If i entered my username/pass and logged in, the oauth2 (gmail) account would not have been added to Cypht.

After many identical failures, i noticed when I was redirect back to Cypht after authorizing access in Gmail, I would be at the login prompt but the URL in my browser address bar was obviously an approval of some sort. So I copied the URL and instead of logging into Cypht I just hit back on my browser a few times until I was back in Cypht (still logged in!) and pasted the URL into my address bar. Hitting enter would finish adding Gmail to Cypht, success at last!

I believe when adding my oauth2 app at google, i used my Cypht install URL (https://webmail.domain.tld/) as the redirect_uri (i have never understood if this is redirect_uri or redirect_urL) and I think the issue is related. Nothing mentions which URL to use.

Another note - if you don't add the redirect_uri when creating the oath2 app/auth at Google, but instead add it later, it doesn't work. It only works (for me) when you add it during the original oauth creation process even though it allows you to add one later. It could be that it would eventually work - there is a delay of some sort.

[marclaporte]

@kambereBr Please review the whole process/code/documentation for Cypht to work for Google and Microsoft's email offerings.

Here is the current documentation: https://github.com/cypht-org/cypht/wiki/OAUTH2-over-IMAP

It states:
"Next you need to edit the modules/imap/oauth2.ini file, and move it to the "app_data_dir" as defined in your hm3.ini file."
"Now re-run the config_gen.php script which will find your oauth2.ini file and combine it into the site settings."

This is too complicated. Please explore a way for users to do this via the web interface (like for IMAP accounts). FYI, Henock is working on a revamp of the whole setup process (Making it easier to set up IMAP, SMTP and connect them together) so you should be in touch with him to converge efforts. Ref: https://avan.tech/item81120 (internal link, sorry if you don't have access).

First make it work with the standard installation procedure. Once this is working well, please revisit the Docker install with @rodriguezny

Thanks!

[kambereBr]

@marclaporte, Noted. Thank you!

[marclaporte]

https://workspaceupdates.googleblog.com/2023/09/winding-down-google-sync-and-less-secure-apps-support.html

[marclaporte]

@christer77 You closed this. Is it simple now?

The https://github.com/cypht-org/cypht/wiki/OAUTH2-over-IMAP is still too complicated (ex.: re-run the config_gen.php) and it's still mentioning hm3.ini (which no longer exists in Cypht 2+)

I suggest you take a non geeky friend or family member and ask them to set this up, while you watch.

[marclaporte]

From: https://gitter.im/cypht-org/community

[marclaporte]

Related issue: #1351

[marclaporte]

@Bienvenumoringa @christer77 #1429