Problem with API - /host/XXX reports 'not authenticated' but all other APIs work after authentication

[pliablepixels]

There seems to be an increasing number of people who are reporting issues that a part of the APIs are failing when authentication is enabled. Specifically, everything under the /host/ route fails with a 401.

See here for help needed to debug this issue.

Occurrence
Seems to be only on some systems. Don't have a pattern yet

Steps to reproduce

a) login to ZM (OPT_AUTH is on) using a browser, Open a new tab
b) Go to http://server/zm/api - shows green
c) Go to http://server/zm/api/monitors.json, /events.json etc all checkout fine - they return valid JSON data
d) Go to http://server/zm/api/host/getVersion.json and pretty much any /host/XXXX.json function returns 401 Not authenticated

I don't have a sense on what exactly is causing this - whether its an OS version problem, a Cake PHP problem, a PHP problem, a webserver config problem or something else.

Why its failing - the cause (but not root cause)
What we did figure out is the reason this happens is because the HostController.php file gets an empty Session Object, while all other Controllers get a valid session object. The API auth layer relies on the Session object to validate if you have logged in.

Why is the Session vars empty just for this one controller, I haven't a clue. I am setting up this thread so that I can invite folks who are facing this problem to report their OS/cake/PHP versions and see if we can debug something. I'll soon be tagging folks to contribute.

Temporary Workaround (has security issues, see below)
The workaround, for now is this: disable Auth checking on HostController. This will however expose the following details to any user who can access your server (as there is no auth in this controller). Note that you can keep Basic Auth enabled - it will continue to work and will honor basic auth. The problem is only with ZM Auth, for those who are affected.
a) API version
b) System load
c) Your server Time Zone (only applicable for ZM 1.30.2 and above)
d) disk percent (used/free) on a per monitor basis (this API also slows your system down significantly in addition to exposing your list of monitor names. You can comment out this function - see note below)

Note - if you are facing this problem, you can completely comment out the getDiskPercent API in HostController.php to remove open access. zmNinja does not use this API and neither does ZoneMinder - so there is no functionality loss

The workaround code is to modify HostController.php inside api/app/controller/ and add the public function beforeFilter(){} code just below the class HostController extends AppController line like so:

class HostController extends AppController {
public function beforeFilter() {}

[pliablepixels]

Folks seeing this issue, please post here:

INFORMATION NEEDED (See point g - it will greatly help if someone facing this issue can let me access that system. So far, I can't debug much as I can't reproduce it)

a) Your cakePHP version: cat /usr/share/zoneminder/www/api/lib/Cake/VERSION.txt | grep -v '//' (change /usr/share/zoneminder to your installed path)
b) Your PHP version: php --version
c) Your ZoneMinder version (you can see it from web console)
d) How did you install ZM ? did you d.1) compile from source d.2) use a package, and if so, which package
e) OS version
f) Please post your apache/nginx config
g) If you are facing this problem and there is a possibility for you to give me SSH access (need root) to a test ZM box with this problem, please shoot me an email (pliablepixels@gmail) - I'd like to debug - I am not able to reproduce this issue.

thx

[ghost]

i will try to remove the ZM auth and just use apache basic auth to see if it makes a difference, i remarked that the apache auth login is passed on the url inside the zmninja app... so i guess we could still have some minor security while having a working zmninja... testing this before making any code changes. BRB

[pliablepixels]

@itpcss @dakota @FrancoisDesfosses can you please report your system configuration as per the data requested here.

Would love to get to the core of this issue. @FrancoisDesfosses - lets focus on the API part for now and keep zmNinja out of it (as we are on the ZoneMinder GitHub issue list)

[ghost]

a) 2.8.0
b) PHP 5.4.16 (cli) (built: Nov 6 2016 00:29:02)
c) v1.30.0 zoneminder-1.30.0-4.el7.centos.x86_64
d) used ZM repository, using yum install
e) CentOS Linux release 7.3.1611 (Core)

[pliablepixels]

Based on what @dakota and I saw earlier:
a) If you remove OPT_AUTH (zm auth) it should work (basic auth should work too)
b) Please let me know if the workaround to HostController.php works for you with basic auth enabled

To test, please don't use zmNinja. Just launch a browser, login, open a new tab and try the /host/getVersion.json API in the browser URL.

[ghost]

a) opt_auth removed from ZM
b) applied file changes
c) enabled apache basic auth ;-)
d) this time, all api work fine. using work arround proposed.

[pliablepixels]

okay, thanks for the confirmation. Hopefully we can figure out something about the core issue. In the meantime, you can comment out getDiskPercent completely - no one uses it (I modified my first post)

[ghost]

my current hostcontroller.php does not contain the gettimezone function, should i add it ?

[pliablepixels]

its optional - it will be part of next ZM release

[knight-of-ni]

hmmmm, this appears to be working fine for me.
Each machine below was tested using Chrome and Firefox:

Machine 1
cake 2.8.0
php 5.4.16
zm 1.30.2 rc1
centos 7.3
apache 2.4

Machine 2
cake 2.8.0
php 5.6.30
zm 1.30.2 rc1
nginx 1.10
Fedberry 24 (Fedora 24 Spin)

In each case, when I enter https://myserver/zm/api/host/getVersion.json

I see:

{
    "version": "1.30.2",
    "apiversion": "1.0"
}

A couple times I got a 404 error message, but that turned out to be me fat fingering the wrong url.

[pliablepixels]

Yes I'm not entirely sure what is the reason why this fails for some folks. It's some server side issue - because I tried some of these portals with my browser and had the same issue.

[knight-of-ni]

Just to provide another data point with a version of zoneminder matching an earlier post, I tried this from our publicly available demo server:

cake 2.8.0
php 5.4.16
zm 1.30.0 release
centos 7.3
apache 2.4.6

https://demo.zoneminder.com/zm/api/host/getVersion.json

{
    "version": "1.30.0",
    "apiversion": "1.0"
}

Note that I performed this test using both my admin login and the read-only zmuser login. Both were successful.

[dakota]

a) 2.8.0
b) PHP 5.6.29
c) ZM 1.30.0
d) FreeBSD PKG (using pkg install zoneminder)
e) FreeBSD 10.3
f)

server {
                root /usr/local/www/zoneminder;
                server_name localhost,zoneminder;
                try_files $uri $uri/ /index.php$is_args$args;
                index index.php;

                location = /cgi-bin/nph-zms {
                        include fastcgi_params;
                        fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
                        fastcgi_pass    unix:/var/run/fcgiwrap/fcgiwrap.sock;
                }

    location ~ \.php$ {
        try_files $uri =404;
        include             /usr/local/etc/nginx/fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_index       index.php;
        fastcgi_pass       unix:/var/run/php-fpm.sock;
    }

        location /api {
                rewrite ^/api(.+)$ /api/app/webroot/$1 break;
                try_files $uri $uri/ /api/app/webroot/index.php?$args;
        }

    location ~ \.(jpg|jpeg|gif|png|ico)$ {
        access_log          off;
        expires         33d;
    }
}

[pliablepixels]

@dakota, @FrancoisDesfosses instead of doing an empty beforeFilter in HostController, can you try explicitly calling parent::beforeFilter() inside it?

class HostController extends AppController {
public function beforeFilter() {
   parent::beforeFilter();
}

[ghost]

Fatal error: Call to undefined function beforeFilter() in /usr/share/zoneminder/www/api/app/Controller/HostController.php on line 6 { "success": false, "data": { "code": 500, "name": "Call to undefined function beforeFilter()", "message": "Call to undefined function beforeFilter()", "url": "/zm/api/host/getVersion.json", "exception": { "class": "FatalErrorException", "code": 500, "message": "Call to undefined function beforeFilter()", "trace": [ "#0 /usr/share/zoneminder/www/api/lib/Cake/Error/ErrorHandler.php(213): ErrorHandler::handleFatalError(1, 'Call to undefin...', '/usr/share/zone...', 6)", "#1 [internal function]: ErrorHandler::handleError(1, 'Call to undefin...', '/usr/share/zone...', 6, Array)", "#2 /usr/share/zoneminder/www/api/lib/Cake/Core/App.php(970): call_user_func('ErrorHandler::h...', 1, 'Call to undefin...', '/usr/share/zone...', 6, Array)", "#3 /usr/share/zoneminder/www/api/lib/Cake/Core/App.php(943): App::_checkFatalError()", "#4 [internal function]: App::shutdown()", "#5 {main}" ] }, "queryLog": { "default": { "log": [ { "query": "SELECT Config.Id, Config.Name, Config.Value, Config.Type, Config.DefaultValue, Config.Hint, Config.Pattern, Config.Format, Config.Prompt, Config.Help, Config.Category, Config.Readonly, Config.Requires FROM zm.Config AS Config WHERE Config.Name = 'ZM_OPT_USE_API' LIMIT 1", "params": [ ], "affected": 1, "numRows": 1, "took": 0 }, { "query": "SELECT Config.Id, Config.Name, Config.Value, Config.Type, Config.DefaultValue, Config.Hint, Config.Pattern, Config.Format, Config.Prompt, Config.Help, Config.Category, Config.Readonly, Config.Requires FROM zm.Config AS Config WHERE Config.Name = 'ZM_OPT_USE_AUTH' LIMIT 1", "params": [ ], "affected": 1, "numRows": 1, "took": 0 } ], "count": 2, "time": 0 } } } }

[pliablepixels]

that's surprising. Can you post your modified HostController.php file in a GitHub gist?

[dakota]

Adding parent::beforeFilter(); brings back the issue with unauthorised access.

[ghost]

does that has something to do with ZM settings option,
AUTH_RELAY Method used to relay authentication information (?) hashed plain none

currently set to none

[knight-of-ni]

The following is a total random guess based on some other cakephp weirdness I observed today.

Try setting Cake's debug back to 2 in your core.php file:
https://github.com/ZoneMinder/ZoneMinder/blob/master/web/api/app/Config/core.php.default#L34

Apparently the debug variable controls much more than the verbosity of cake's output.

[dakota]

@knnniggett, doesn't change a thing :) The only things that the debug level changes (In CakePHP 2.x) is the amount of information reported, how much detail is in error messages, and how long schema caching is (10 seconds instead of 1000 days). Source: I'm a CakePHP core developer :)

I have absolutely zero idea what could possibly be causing this issue though, and why it's only on that one controller.

[knight-of-ni]

@dakota Turning off debug does indeed do more than lower the verbosity of cake's output. It also stops cakephp from creating several folders under the tmp folder.

Perhaps you already know, this is by design:
cakephp/cakephp#7987

If debug is off and the folders mentioned in that thread don't already exist, then cake will throw an error, rather than create them during runtime. I didn't mention that because this particular thread isn't about the cake tmp folders, nor do we need to discuss the "why" behind this. It is what it is.

Since no one yet knows the root cause of the problem described here, I'm simply going back through our recent commits to see what was touched last. We didn't change much relating to the zoneminder API, but one recent change was to set cake debug from 2 -> 0.

[dakota]

@knnniggett You'll notice that's for CakePHP 3.1, only applies for the File Log and File cache engines, and not for sessions (And can only be a problem when using CakePHP to handle sessions). In this case Zoneminder is using the default php session handler.

This problem manifests itself regardless of the value of debug, and ONLY for the HostsController, all the other controllers work 100% as expected.

[knight-of-ni]

No, the problem manifests itself only with debug set to 0. This is very easy to duplicate so I encourage you try it yourself. These errors appear in the logs with debug set to 0:

[Wed Mar 29 12:13:06.537104 2017] [:error] [pid 2587] [client ::1:33048] PHP Warning:  _cake_core_ cache was unable to write 'cake_dev_en-us' to File cache in /usr/share/zoneminder/www/api/lib/Cake/Cache/Cache.php on line 327
[Wed Mar 29 12:13:06.537173 2017] [:error] [pid 2587] [client ::1:33048] PHP Warning:  /usr/share/zoneminder/www/api/app/tmp/cache/persistent/ is not writable in /usr/share/zoneminder/www/api/lib/Cake/Cache/Engine/FileEngine.php on line 385
[Wed Mar 29 12:13:06.537256 2017] [:error] [pid 2587] [client ::1:33048] PHP Fatal error:  Uncaught CacheException: Cache engine "_cake_core_" is not properly configured. Ensure required extensions are installed, and credentials/permissions are correct in /usr/share/zoneminder/www/api/lib/Cake/Cache/Cache.php:186\nStack trace:\n#0 /usr/share/zoneminder/www/api/lib/Cake/Cache/Cache.php(151): Cache::_buildEngine('_cake_core_')\n#1 /usr/share/zoneminder/www/api/app/Config/core.php(375): Cache::config('_cake_core_', Array)\n#2 /usr/share/zoneminder/www/api/lib/Cake/Core/Configure.php(72): include('/usr/share/zone...')\n#3 /usr/share/zoneminder/www/api/lib/Cake/bootstrap.php(431): Configure::bootstrap(true)\n#4 /usr/share/zoneminder/www/api/app/webroot/index.php(90): include('/usr/share/zone...')\n#5 {main}\n  thrown in /usr/share/zoneminder/www/api/lib/Cake/Cache/Cache.php on line 186

I've seen additional errors reporting other folders such as cache/models and logs are not writeable as well.
The problem immediately goes away if we change debug back to 2, but as reported by another user that can cause cake to reveal sensitive information, thus an alternative solution is desired.

What we did is add the cake cache folders to the tmpfiles.d service to ensure all of cake's tmp folder exist and are owned by the web account user:
e46f641

If you have a better idea, then please let us know.

[dakota]

@knnniggett That's not the problem being discussed in this issue though.

This issue is about api requests to /api/hosts/getVersion.json always getting a 401 Not Authorized response, but all other requests (like /api/monitors.json) working perfectly. This particular problem exists regardless of the value of debug.

As for the tmp file issue, the options are

1. Tell users to make api/tmp/* owned or writable by the web user.
2. The fix done in e46f641
3. Don't use the file cache engine, but one of the others (Like Memcached or APCu)

Ideally, the api should be upgraded to CakePHP 3 and leverage the CRUD plugin's api functionality (Which now includes JSON API support). I'll see if I can do that sometime soon if I can find myself some time!

[knight-of-ni]

Thanks for the help, and I understand the time factor (or lack thereof). The original author of the zoneminder API hasn't been very active with the project. This has left us in a bit of a knowledge gap when it comes to problems requiring a strong familiarity with cakephp. And yes, this is a shameless attempt at asking for help from anyone who might read this.

[pliablepixels]

if someone who is facing this problem is able to get me SSH access to his/her instance, I'll debug it and spend time. Without access to an instance that has this problem, its near impossible for me to even try.

[JamesM85]

I believe I am experiencing the issue if you'd like to look at it?

I'm using zoneminder 1.30.4 running on Centos 6.9 installed from repository

[pcollaog]

I have this error when I call the API

Error: Uncaught exception 'FatalErrorException' with message '[MissingPluginException] Plugin Crud could not be found. #0 /usr/share/php/Cake/Core/App.php(227): CakePlugin::path('Crud') #1 /usr/share/php/Cake/Core/App.php(549): App::path('Lib', 'Crud') #2 [internal function]: App::load('CrudControllerT...') #3 /usr/share/zoneminder/www/api/app/Controller/AppController.php(34): spl_autoload_call('CrudControllerT...') #4 /usr/share/php/Cake/Core/App.php(567): include('/usr/share/zone...') #5 [internal function]: App::load('AppController') #6 [internal function]: spl_autoload_call('AppController') #7 /usr/share/php/Cake/Error/ExceptionRenderer.php(152): class_exists('AppController') #8 /usr/share/php/Cake/Error/ExceptionRenderer.php(92): ExceptionRenderer->_getController(Object(CacheException)) #9 /usr/share/php/Cake/Error/ErrorHandler.php(126): ExceptionRenderer->__construct(Object(CacheException)) #10 [internal function]: ErrorHandler::handleException(Object(CacheException)) #11 {main}' in /usr/share/php/Cake/Error/E	
File: /usr/share/php/Cake/Error/ErrorHandler.php	
Line: 138

Additional to this... I have other error when I call the same endpoint and maybe the zm.conf wan't parse o load the ZM_* variables, so... I replace all the ZM_* variables in database.php and the error disapear. This last one I can't handle :( any workarround?

Some additional info:

OS: Debian stretch
ZM: 2:1.30.4-dmo1+deb9u1
PHP: PHP 7.0.19-1 (cli) (built: May 11 2017 14:04:47) ( NTS )
Cake: 2.8.5

[knight-of-ni]

I mistakenly closed this issue out yesterday.

@pcollaog The particular API error you are experiencing is not the same issue reported in this thread. Since we were unsuccessful yesterday in our IRC channel, you might want to try building your own zoneminder package. There are couple documented ways to do that, which are shown here:
http://zoneminder.readthedocs.io/en/latest/installationguide/index.html

Please contact me in our IRC channel for further details.

In the long run, I do hope you file a bug report against the package your are using. The package maintainer needs to get involved to fix the root cause.
https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=zoneminder;dist=unstable

[martinamw]

I also had this issue until yesterday. I could login to Zoneminder using it's built-in method, but calling a the version api resulted in not authorised every time.

In my case I as overwrote the default php setting: output_buffering in global apache php.ini (the reason for this was for another web app - pydio - running on same server). I set this back to default 4096, and no more issues with API or zmninja iPhone app.

[mcdamo]

AFAICT the extra whitespace in Models/Host.php was causing PHP to prematurely output to the browser and thus interfering with the sessions or HTTP headers.

[knight-of-ni]

@mcdamo wow, nice catch. I never would have thought of that. I'm going to close this out since #2062 was merged. All, if there are any new/other issues related to api authentication, please start a new issue.

[pliablepixels]

Interesting. I have the same space in my host.php Model but I don't have this issue. But if it solves the issue for those who are facing it, I'm not complaining.