Report tracking into wrong Site ID and missing token auth

[tsteur]

fix #12922

• When a tracking request is invalid for any of the 2 reasons, we log the request as a failure
• Any failed request older than 2 days ago is automatically deleted again
• Weekly we check for tracking failures and email super users only (doing this for admins is bit tricky)
• Admins and SuperUsers see a new widget on Admin Home screen plus it can be used in the dashboard:
  
• New diagnostics page that shows tracking failures in depth
  

I was also thinking about showing a box in the reporting toolbar, eg on the right... but then thought it may not be needed for now and might be rather annoying. May be good to first see how many false reports maybe come in etc.

I reckon it is a good working solution v1 for this tool and see from there how/where it goes. Heaps could be done there.

Todo: We should write FAQ articles for those 2 particular errors and adjust the links in the code

[mattab]

@tsteur just thought of another use case which occurs from time to time, if Matomo is in an unstable state, there may be columns missing, this happened to me regularly, such errors:

DEBUG: Exception: Error query: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'last_idlink_va' in 'field list' In query: SELECT visit_last_action_time, visit_first_action_time, idvisitor, idvisit, user_id, visit_exit_idaction_url, visit_exit_idaction_name, visitor_returning, visitor_days_since_first, visitor_days_since_order, visitor_count_visits, visit_goal_buyer, location_country, location_region, location_city, location_latitude, location_longitude, referer_name, referer_keyword, referer_type, idsite, visit_entry_idaction_url, visit_total_actions, visit_total_interactions, visit_total_searches, config_device_brand, config_device_model, config_device_type, visit_total_events, visit_total_time, location_ip, location_browser_lang, campaign_content, campaign_id, campaign_keyword, campaign_medium, campaign_name, campaign_source, last_idlink_va, custom_var_k1, custom_var_v1, custom_var_k2, custom_var_v2, custom_var_k3, custom_var_v3, custom_var_k4, custom_var_v4, custom_var_k5, custom_var_v5   FROM log_visit WHERE visit_last_action_time >= ? AND visit_last_action_time <= ? AND idsite = ? AND config_id = ?
DEBUG:                 ORDER BY visit_last_action_time DESC
DEBUG:                 LIMIT 1 Parameters: array (
DEBUG:   0 => '2018-10-03 05:21:30',
DEBUG:   1 => '2018-10-03 06:21:30',
DEBUG:   2 => 1,
DEBUG:   3 => '/ƫK����',
DEBUG: )

So it would be really great if we could detect these problems and also report them in the page 👍

[tsteur]

Not really so much possible. Cause it was said it is not needed to support anything else and therefore it is not built to include metadata etc. If something like this is possible, it would be only possible to say a column was missing, but not which column or table etc.

[diosmosis]

Does this mean the exception cannot be caught later? If so, could log the failure there.

[tsteur]

Yes it would otherwise not be caught later. and logging wouldn't work as people wouldn't see the logged error. Like nobody would really look there and eg on cloud people wouldn't even be able to see the logs. We have heaps of logs like idsite=0 already and want to make users aware of this problem with this PR. It's really the most common tracking error.

[diosmosis]

I meant log through Failures::logFailure(), like catch in Tracker::main() and log that failure. But if it can't be caught then that's not an option I guess.

[tsteur]

Catching it there wouldn't 100% work eg when using bulk request or queued tracking and eg one request contains idsite=0...

The tracker handler starts processing each tracking request within $handler->process in the Tracker.:track method... and basically the only way be to handle it otherwise in Tracker::trackRequest() but by then this error might have already occurred.

[diosmosis]

Left some comments & need to resolve some conflicts. Haven't tested locally yet, will do so next review.

[tsteur]

@diosmosis applied the review feedback and resolved the conflicts. I might need to update some UI tests...

[diosmosis]

Left one further comment, but will probably merge this today. if my comment is important can be done in a new PR

[tsteur]

@diosmosis just realising a feature to log not auth request regressed due to another PR... because isVisitExcluded seems to no longer work basically when CIP is set... be good to not merge.

[tsteur]

@diosmosis just pushing a fix, and also fixing the failed system and ui tests with this commit

[diosmosis]

Looks like there are some other test failures now? the novisit tests are failing (and possibly a unit test). Maybe some others too.

[tsteur]

hoping to have fixed them @diosmosis will check the results tomorrow.

[tsteur]

this was tricky... hopefully this time it fixes most or all of the tests :)

[mattab]

Will be very useful 👍

Not really so much possible. Cause it was said it is not needed to support anything else and therefore it is not built to include metadata etc. If something like this is possible, it would be only possible to say a column was missing, but not which column or table etc.

Would it be possible to catch these sql error cases, and report them as failures in this new tool's UI? (currently they may go un-detected for a few hours or days until someone realises tracking does not anymore)

[tsteur]

No, at least not as part of 3.8 release.

[diosmosis]

Almost working, looks like BulkTracking & QueuedTracking have some failing integration tests

[tsteur]

@diosmosis fixed the bulk and queued tracking tests. Looks like MarketingCampaignsReporting isn't directly failing because of this change. Hopefully can be finally merged :)