Customizeable / Extendable Visitor Profile & Visitor Log #11579

sgiehl · 2017-04-03T20:03:25Z

Proposed changes to the way how visitor log and profile are created:

Introduces new classes (named VisitorDetails) that can be implemented in every plugin.
With this classes the visitor and action data used for the Live plugin can be provided, filtered and manipulated. Additionally the classes can define extra content to be rendered in visitor log & profile

Additional changes / improvements

Actions for visits are now queried as bulk requests, instead of for each visit seperatly
Actions of a visit in Visitor Log are hidden by default but can be expanded on click
Show Browser Language in Country Flag Tooltip (refs Piwik: Visitor Log and browser language #11707)
Display Content Impressions & Interactions in Visitor Log & Profile (fixes Content-tracking: display content views and interactions in the Visitor log #6416 refs Content-tracking: Enrich other reports / API's #6268)
Display a overview of used devices in Visitor Profile (refs Visitor profile: display detailed user devices info #6161)
Show visit detail icons in visit list of user profile (fixes display Device Type/Model in Visitor Profile pop-up #11520)
Show top visited pages in Visitor Profile (fixes Visitor profile summary: show number of visits of unique pages & repeated visit pages #8912)
Do not group duplicate page views, but show them with a refresh icon instead, to prevent loss of additional information (fixes visitor log: grouping hides information #9586)
Always show visitor profile link in visitor log (fixes Always show the link to "Visitor Profile" in the Visitor log #10272)
Display custom Dimensions in Visitor Log & Profile (refs piwik/plugin-CustomDimensions#28, Show custom dimensions in visitor log & profile plugin-CustomDimensions#61)
Display detailed campaign data in Visitor Log & Profile (refs Display all campaign channels tags in the Visitor Log and Visitor Profile plugin-MarketingCampaignsReporting#3, Show campaign data in visitor log & profile plugin-MarketingCampaignsReporting#11)
Show bandwidth data in Visitor Log & Profile (Show bandwidth data in visitor log & profile plugin-Bandwidth#13)
Reverse enumeration of visits in visitor profile (fixes Visitor Profile: order of visit numeration is confusing #8491)
Show idvisit in the visitor log and user profile (IP tooltip)
Always return raw referrer url for visitor details (fixes always return full Referrer URL in Live.getLastVisitDetails API #11976)

fixes #6111
refs #9507

layout changes also fixes #11589

tsteur · 2017-05-04T19:05:47Z

FYI: An issue I ran a few times into when extending the visitor logs eg via event or so is that most of the events are methods / hooks are triggered per visitor. When showing like 250 or up to 500 visits on one page, we might do 500 times a SQL query for each plugin that hooks into. For example plugins would check if there is a custom dimension, or media views, form interactions, recorded sessions, etc. They are all in different tables etc and query per visitor.

It would be great if there was like an event or method / hook that gives you the row of visitors so it is more efficiently possible to fetch in one query the data (eg using idvisit in (....)) instead of having heaps of individual queries, then render it later individually. I haven't looked much into PR or the other PRs if something like this is already implemented but be good to have.

tsteur · 2017-05-04T19:07:08Z

plugins/Actions/VisitorDetails.php

+					log_link_visit_action.interaction_position
+					" . $sqlCustomVariables . ",
+					log_action_event_category.name AS eventCategory,
+					log_action_event_action.name as eventAction


FYI: Not needed for now but eventually we also need to fetch log_link_visit_action.idpageview as other plugins will likely hook into individual actions based on the idpageview. I will add this likely once needed

I'll add log_link_visit_action.idpageview

mattab · 2017-05-08T22:07:43Z

CHANGELOG.md

@@ -18,6 +18,7 @@ The Product Changelog at **[piwik.org/changelog](http://piwik.org/changelog)** l
 ### Breaking Changes
 * New config setting `enable_plugin_upload` lets you enable uploading and installing a Piwik plugin ZIP file by a Super User. This used to be enabled by default, but it is now disabled by default now for security reasons.
 * New Report class property `Report::$supportsFlatten` lets you define if a report supports flattening (defaults to `true`). If set to `false` it will also set `ViewDataTable\Config::$show_flatten_table` to `false`
+* The event `Live.getAllVisitorDetails` has been deprecated and will be removed in Piwik 4. Use a `VisitorDetails` class instead (see Live plugin). 


Let's add a test for this deprecation in tests/PHPUnit/Unit/DeprecatedMethodsTest.php

How to test a deprecated Event? Did we do that for other Events before?

probably we can't do this. So maybe we just create an issue in Piwik 4 milestone instead?

mattab · 2017-05-09T08:12:17Z

This looks promising 👍

What are your thoughts regarding Thomas comment on performance?
if possible, It would be great to include in this PR the refactoring of all template code into relevant plugins. This could validate the new APIs and finalise the project.

We might need to move this into 3.1.0 release as I assume it won't be finished in time for 3.0.4?

* update piwik-icons submodule * fix some system tests * update screenshots

mattab

Looks much better, nicely done 👍
Noticed the XML export contains invalid XML ie. <Apple iPhone>1</Apple iPhone>, example url

	<devices>
		<Smartphone>
			<count>1</count>
			<icon>plugins/Morpheus/icons/dist/devices/smartphone.png</icon>
			<devices>
				<Apple iPhone>1</Apple iPhone>
			</devices>
		</Smartphone>
	</devices>

mattab · 2017-08-09T23:38:15Z

plugins/Actions/lang/en.json

@@ -42,7 +42,9 @@
        "PagesReportDocumentation": "This report contains information about the page URLs that have been visited. %s The table is organized hierarchically, the URLs are displayed as a folder structure.",
        "PageTitlesReportDocumentation": "This report contains information about the titles of the pages that have been visited. %1$s The page title is the HTML %2$s Tag that most browsers show in their window title.",
        "PageUrls": "Page URLs",
+        "PageViewsByVisitor": "Number of times this page was seen in all visits by this visitor",


suggest instead: Number of times this page was viewed by this visitor across all visits.

(edit: this was an old comment I forgot to mention, but maybe this string is not used anymore)

mattab · 2017-08-31T04:50:06Z

We reviewed with Thomas and here is additional feedback:

Is it possible to left-align the text even when icons may be different sizes here?
also make the font size smaller (the same as the default font size in the visitor profile)
Should add some more spacing on the left/right. we reckon adding 5px on each side might be enough, then 3px above and below the "goal number / date" and "Visit SitesManager plugins: added group managment to admin UI #4" maybe 3px smaller. (also possibly add 1 px above and/or below each action as it is hard to separate them and to read because they are quite close to each other.) Like this:
Speed: over many tests on demo2, it takes 1.5s for server to generate response for 100 visits on 3.x-dev, while it takes server 10s to load 100 visits on the branch. New code is about 5 to 10 times as slow as before. Had you reverted the mysql logic to run one query and fetch all data at once? If you did already, could you run a profiler session quickly to see whether any obvious thing could be improved? especially wondering the time spent in php vs mysql and whether we could run more optimised sql and run less sql queries.

sgiehl · 2017-08-31T20:09:06Z

I've fixed all layout issues. Regarding the speed issue: No, I hadn't reverted the sql changes. But the query is now slower for sure, as it contains four more joins for selecting the content interactions.
And the query for fetching the actions did change much in order get them for all visits instead of only one (idvisit = ? vs idvisit IN (?))

You could try to disable the contents plugin and compare the speed then (that should remove the extra joins). I'm not sure if I will have some time to profile that tomorrow.

mattab · 2017-09-04T04:54:05Z

Merged! 🎉

Still we do need to look at the performance issue because it's important that Visitor log loads fast 100 visits at once. Please share what you find in Profiling? if we need to set it up on demo2 let me know.

… merged This reverts commit d833d30.

sgiehl added the Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. label Apr 3, 2017

sgiehl added this to the 3.0.3 milestone Apr 3, 2017

mattab modified the milestones: 3.0.3, 3.0.4 Apr 3, 2017

sgiehl force-pushed the visitordetails branch 6 times, most recently from d37d772 to b34f7fc Compare April 10, 2017 07:54

sgiehl force-pushed the visitordetails branch 4 times, most recently from 9505fa3 to 393c88e Compare April 20, 2017 19:47

sgiehl mentioned this pull request Apr 24, 2017

Display all campaign channels tags in the Visitor Log and Visitor Profile matomo-org/plugin-MarketingCampaignsReporting#3

Closed

sgiehl force-pushed the visitordetails branch from 393c88e to 5875421 Compare April 30, 2017 13:24

This was referenced May 4, 2017

Show custom dimensions in visitor log & profile matomo-org/plugin-CustomDimensions#61

Merged

Show campaign data in visitor log & profile matomo-org/plugin-MarketingCampaignsReporting#11

Merged

tsteur reviewed May 4, 2017

View reviewed changes

sgiehl force-pushed the visitordetails branch 2 times, most recently from 71e882c to aa12678 Compare May 8, 2017 08:40

mattab mentioned this pull request May 8, 2017

Request for events in Live plugin #11560

Closed

mattab reviewed May 8, 2017

View reviewed changes

mattab modified the milestones: 3.0.4, 3.0.5 May 9, 2017

sgiehl force-pushed the visitordetails branch 2 times, most recently from dcd2e48 to 1af4204 Compare August 28, 2017 09:40

sgiehl and others added 4 commits August 28, 2017 12:20

Refactor profile summaries into additional classes

209f74b

update expected test files

62374d9

update screenshots

b9bedb4

submodule update

f6d491d

sgiehl force-pushed the visitordetails branch from 1af4204 to f6d491d Compare August 28, 2017 10:21

update piwik-icons submodule (#11904)

706234a

* update piwik-icons submodule * fix some system tests * update screenshots

mattab requested changes Aug 31, 2017

View reviewed changes

sgiehl added 4 commits August 31, 2017 18:17

make device list xml compatible

07dd1a4

improve translation

70c1b34

improve icon position

bd86ced

improves spacings

1e7e505

sgiehl added 2 commits August 31, 2017 22:21

update test files

c16234b

small css improvement

6a2b408

sgiehl force-pushed the visitordetails branch 2 times, most recently from 35b917d to c83597e Compare September 1, 2017 10:34

update screenshots

ea43e85

sgiehl force-pushed the visitordetails branch from c83597e to ea43e85 Compare September 1, 2017 11:22

tsteur mentioned this pull request Sep 4, 2017

Visits numbering is wrong in Visitor Profile bug #11996

Closed

mattab merged commit 660df51 into 3.x-dev Sep 4, 2017

fengkaijia mentioned this pull request Sep 9, 2017

Show Tibet as part of China on map #11930

Merged

fengkaijia referenced this pull request in matomo-org/matomo-icons Sep 11, 2017

temporarily revert "Remove CTA flag" until matomo-org/matomo#11930 is…

bbf7642

… merged This reverts commit d833d30.

mattab added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. labels Sep 11, 2017

mattab mentioned this pull request Sep 18, 2017

Document how plugins can extend Visitor Log and Visitor Profile #12073

Closed

sgiehl deleted the visitordetails branch October 3, 2017 12:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Customizeable / Extendable Visitor Profile & Visitor Log #11579

Customizeable / Extendable Visitor Profile & Visitor Log #11579

sgiehl commented Apr 3, 2017 •

edited

Loading

tsteur commented May 4, 2017

tsteur May 4, 2017

sgiehl May 8, 2017

mattab May 8, 2017

sgiehl May 8, 2017

mattab May 8, 2017

mattab commented May 9, 2017

mattab left a comment

mattab Aug 9, 2017 •

edited

Loading

mattab commented Aug 31, 2017 •

edited

Loading

sgiehl commented Aug 31, 2017

mattab commented Sep 4, 2017

Customizeable / Extendable Visitor Profile & Visitor Log #11579

Customizeable / Extendable Visitor Profile & Visitor Log #11579

Conversation

sgiehl commented Apr 3, 2017 • edited Loading

Proposed changes to the way how visitor log and profile are created:

Additional changes / improvements

tsteur commented May 4, 2017

tsteur May 4, 2017

Choose a reason for hiding this comment

sgiehl May 8, 2017

Choose a reason for hiding this comment

mattab May 8, 2017

Choose a reason for hiding this comment

sgiehl May 8, 2017

Choose a reason for hiding this comment

mattab May 8, 2017

Choose a reason for hiding this comment

mattab commented May 9, 2017

mattab left a comment

Choose a reason for hiding this comment

mattab Aug 9, 2017 • edited Loading

Choose a reason for hiding this comment

mattab commented Aug 31, 2017 • edited Loading

sgiehl commented Aug 31, 2017

mattab commented Sep 4, 2017

sgiehl commented Apr 3, 2017 •

edited

Loading

mattab Aug 9, 2017 •

edited

Loading

mattab commented Aug 31, 2017 •

edited

Loading