Automatically run test against minimum/maximum supported Piwik version defined in plugin.json

[tsteur]

Follow up on #8660

Background: Defining supported Piwik version

In plugin.json a developer can specify supported Piwik version. Minimum supported Piwik version defaults to 2.0 and maximum supported Piwik versions defaults to the latest stable.

This is described in http://developer.piwik.org/guides/distributing-your-plugin#pluginjson-file . Versions can be specified like

"require": {
    "piwik": ">=2.0.3" // requires at least Piwik 2.0.3
} 

or

"require": {
    "piwik": ">=2.0.0,<=2.2.0" // requires a Piwik version 2.0.0 up to 2.2.0
}

A full list of possible version definitions can be seen here: https://getcomposer.org/doc/articles/versions.md

Problem: Tests on Travis

They currently run against the minimum defined version in plugin.json automatically and against the maximum supported version (currently named PIWIK_LATEST_STABLE_TEST_TARGET). The maximum supported version in .travis.yml is however not automatically updated but it should. This means, currently, a plugin developer has to always update .travis.yml automatically which can be easily forgotten and plugin developers might not be even aware that they have to do this. Having to specify such things twice (plugin.json and .travis.yml) is not good experience and is fragile. Especially since the name suggests that it always runs against latest stable Piwik version. This can end in broken Piwik installations for thousands of users.

Expected solution (and more on the problem):

Tests on travis should automatically read the defined version constraints in plugin.json and run one build against minimum supported version and one build against maximum supported version. This has to be automatic. One should not have to define it in plugin.json and .travis.yml. The whole point of having tests run on Travis is to get notified when something doesn't work anymore. It's expected that a build fails if we release a new version and the plugin is not compatible with that version anymore. This will help the developer to maintain the supported versions in plugin.json. If a build fails because we release a new version the developer has to get notified and take action: Either change the required Piwik version in plugin.json to not support the latest stable version anymore (by specifying eg <2.X.X) or to make the plugin compatible. Afterwards a plugin developer needs to release a new version otherwise users that use this plugin end up in having a broken Piwik that might no longer work at all.

All this part about what is the correct required Piwik version is a very critical part see eg the issue I created in #6149 since it's basically not possible to know for a developer in which versions a plugin will work (maybe will be a bit better once we do #8547 ) and people easily forget to update the required Piwik version when they work on their plugin.

More about this:

Can be found in the discussion of #8660

I do not think this is too complicated to implement but in case there are maybe some more steps in between...

• We should maybe rename PIWIK_LATEST_STABLE_TEST_TARGET. The name is why I thought it would be always the latest stable automatically. In Travis etc one often doesn't see the full variable but only PIWIK_LATEST_STABLE or even if one can see the full variable in travis it still kinda suggests it will be tested against latest stable. Maybe a better name could be PIWIK_TARGET_VERSION.
• A simple first step could be also to email plugin developers (at least the once we know or a mailing list etc) to check their plugins re compatibility with latest beta eg 5 days before a new release and we could mention in this email that they have to update .travis.yml to do this.
• Piwik PRO developers vs Community / 3rd party developers:

From what I understand the expected solution is not implemented yet because it's possibly not working for Piwik PRO developers. I think this is not the right way to go as it ends in thousands of broken Piwik installations and I think if we find the right solution then it is good for both. Eg Piwik PRO developers can still define a minimum/maximum supported version via plugin.json. By default no maximum version is defined but Piwik PRO developers could always specify a max supported version and increase it manually and it should work for them too. For community plugin developers no maximum supported Piwik version should be specified for now as they will be usually still compatible when a new Piwik version is released and if it breaks, we need to make sure they get notified when we run a test.

If we don't do this because it just doesn't work for Piwik PRO developers we should maybe remove the generate:travis-yml command from Piwik core and either provide it via a separate "ProDevelopment" plugin or rename the command in core to pro:generate:travis-yml or generate:travis-yml-piwikpro. This way it is clear it is not intended to be used by the community. We can then maybe at some point add a second command that generates a different travis-yml that will automatically update versions.

[diosmosis]

Do you know how many 3rd party plugins have tests and use travis?

[mattab]

Tests on travis should automatically read the defined version constraints in plugin.json and run one build against minimum supported version and one build against maximum supported version. This has to be automatic.

👍

From what I understand the expected solution is not implemented yet because it's possibly not working for Piwik PRO developers.

What makes it "not working" for Piwik PRO developers?

I do not think this is too complicated to implement but in case there are maybe some more steps in between...

If it's not complicated, +1 to improve this in 2.15.0

[diosmosis]

What makes it "not working" for Piwik PRO developers?

The piwik pro workflow is:

1. Generate plugin w/ test against == latest stable (or specific version if for specific version)
2. Finish plugin against latest stable.
3. When Piwik stable is about to be released (ie release candidate), make plugin work on new stable in branch.
4. When Piwik stable is released, change explicit version and merge branches so new version of plugin works w/ new Piwik.
5. Until cloud/client is updated, provide support by fixing old versions that have test against version set to older piwik (thus making the travis build usable).

The "core developer creating plugin against unstable dev-master" workflow is:

1. Generate plugin w/ test against == master.
2. Finish plugin against master.
3. When its finished and a new Piwik stable released, set test against to new latest stable.
4. Hand of maintenance to pro.

I don't know what the community developer workflow is, but if no one in the community writes tests or uses travis builds, I don't see why we should support this use case.

If it's not complicated, +1 to improve this in 2.15.0

The issue is what to do when generating. If we generate for pro development, we have to make sure there's a maximum supported version so the build does not become useless after a new Piwik stable is released (developers can change this manually themselves, but I would prefer to enable them). This will help ensure pro development goes smoothly.

If we don't generate a maximum by default (and silently handle the case where there is no maximum, which is the core of @tsteur's suggestion I believe), then it helps us remind users to update their 3rd party plugins (since their builds will fail). This happens, of course, only if they use tests and the travis build, which I think they don't. And I also don't think the argument about broken Piwiks applies. Yes Piwiks might be broken if developers never work on their plugin again, but I don't see this as a problem, just another difference between using a free product (Piwik w/ 3rd party plugins) vs a product backed w/ professional QA (ie, linux vs. redhat linux).

The only complete solution I would be in favor of is forcing plugins to define a maximum supported version.

[tsteur]

From what I see the only problem is the not having defined a maximum supported version for Piwik Pro developers. If I understand this right it's already done wrong there as the maximum supported version should be defined in plugin.json via require and this seems to be not the case (for Piwik Pro developers). So basically there are currently many plugins that have not specified a maximum supported version, only via tests. This means it has to be set in plugin.json for Piwik Pro developers anyway. This could be done manually, or we add an option to the config eg [Development]prodev=1. When this is set, we automatically create plugin with max supported version specified in plugin.json.

Yes Piwiks might be broken if developers never work on their plugin again, but I don't see this as a problem, just another difference between using a free product (Piwik w/ 3rd party plugins) vs a product backed w/ professional QA (ie, linux vs. redhat linux).

This is a big problem for the experience of our users. We need to develop a world class platform otherwise the platform aspect won't work. It's important to have an easy & stable platform to make developers happy. The community is what made us successful so far and what will make it even more in the future.

I think there's no way around that Piwik Pro developers specify min/max supported version via require in plugin.json. We even check those kinda versions in Piwik whether a plugin is still compatible etc. This should solve the problem. If not, we could by default have eg PIWIK_TARGET_VERSION=max_supported and if someone wants to specify a version manually this could be possible via PIWIK_TARGET_VERSION = 2.15.0

[diosmosis]

We need to develop a world class platform otherwise the platform aspect won't work. It's important to have an easy & stable platform to make developers happy. The community is what made us successful so far and what will make it even more in the future.

I would call linux a world class platform. And yet, in many ways, it has a terrible user experience, compared to OS X or even (shudder) windows. Which is why people pay for quality.

And the community exists to support itself. What its generosity confers to the project is not nearly enough for me to live on. As long as I work for pro, my livelihood is in the hands of others doing pro development, so it is in that way I will direct my efforts.

I think there's no way around that Piwik Pro developers specify min/max supported version via require in plugin.json.

The issue for me isn't whether they have to specify both in plugin.json. I'm fine w/ that. The problem for me, is if you make the default (ie, what is generated after generate:travis-yml is run) work for "community developers that use travis" (whoever they actually are) it will no longer work as well for pro development.

If you force all plugins to have a max version specified, then it solves the problem nicely to me. The workflow will support pro development, and when users find the plugins they want won't work with a new piwik, they'll complain to the plugin developers. So the plugin developers will find out, we don't have to do any work, and it will work regardless of whether users write tests and use travis (which as far as I can tell by looking at plugins on the market place is NO ONE).

[mattab]

If you force all plugins to have a max version specified, then it solves the problem nicely to me. The workflow will support pro development

fyi: we cannot force plugins to have a max version specified, even PRO plugins. This would break the platform usability. from comment:

They should not have to define a maximum version, otherwise we failed being a platform. It's different for major releases where a plugin can break but it should not for any minor or patch releases. This is crucial otherwise we won't have any plugins that support the latest version and if one is using a plugin one couldn't probably upgrade anymore to the latest Piwik version since one is dependent on the Piwik versions that supports that plugin.

[tsteur]

I issued a PR to automatically run tests against a maximum Piwik version defined in plugin.json here matomo-org/travis-scripts#21 . This means a developer no longer has to edit plugin.json and .travis.yml to specify supported versions. Also it means if a developer supports all Piwik versions from a certain version (eg >=2.14.0) one does no longer always have to adjust the .travis.yml when a new Piwik version is released as the maximium supported version is automatically fetched.

This should result in less work for all developers (including Piwik PRO) and make it sure to run tests against latest Piwik version (it gets otherwise easily forgotton to adjust .travis.yml)

[mattab]

Also it means if a developer supports all Piwik versions from a certain version (eg >=2.14.0) one does no longer always have to adjust the .travis.yml when a new Piwik version is released as the maximium supported version is automatically fetched.

This sounds really useful and would save time and effort 👍 We'll be testing over next few days/weeks that this work as expected.