The Big 3.0 Update 💥 #70

swissspidy · 2018-10-24T20:27:24Z

This is a first pass at making Traduttore way more flexible and working for (almost) all sorts of source code repositories.

I got a bit carried away during implementation, so this contains a lot of changes—but I think they're worth it.

Here's a breakdown:

Uses Traduttore Registry to make the plugin translatable using our own translation platform running Traduttore (inception!). (Use Traduttore Registry in this plugin #73).
Right now only the plugin meta data and the Slack event names are translatable. In a next step we can extend this to the actual Slack notification messages and perhaps more.
Adds source code loaders for Mercurial and Subversion repositories.
This means Mercurial repositories on Bitbucket can be properly downloaded.
While no Subversion repository provider (e.g. SourceForge) is directly supported right now, the loader could be used by developers with their own repository or developers who prefer using GitHub with Subversion. This can be achieved by hooking into the various filters to override the various class instances.
Adds support for incoming webhooks from Bitbucket, GitHub, and GitLab (GitLab.com and self-managed)
This is by far the biggest change. It should especially allow using self-managed GitLab repositories without necessarily having to write any custom code to override URLs or things like that.
Upon handling a webhook, repository information like name and URLs are stored as meta data in the database. This means a self-managed GitLab repository will point to the right host and everything after the first incoming webhook.
When scheduling code updates, it first unschedules any existing events.
Lots of documentation changes.
This is far from complete, but is a good start.

Things I still need to do (any help appreciated):

Manual Testing
This includes testing with actual Bitbucket and GitLab repositories.
Documentation improvements
Document and verify steps for repository setup, as well as adding docs for all new filters.
Getting the Loader tests to work
We don't want to test if Git/Mercurial/Subversion are actually working, but whether the behavior of the classes is correct. Otherwise tests fail if the software isn't installed on your machine (or the CI provider).
I've looked into https://github.com/php-mock/php-mock-phpunit and https://github.com/mikehaertl/php-shellcommand for mocking exec() calls and some sort of Command dependency injection.
Analyze existing class logic
Might need to move some stuff around. For example delete_local_repository() might make more sense in the Updater instead of Runner.

Fixes #64, #65, #66, #73.

Reduces Loader class to one per VCS

swissspidy · 2018-10-25T10:32:08Z

The current WIP includes docs for setting up private GitLab repositories.

I think we should try to automate this though, see #57.

swissspidy · 2018-10-25T19:17:20Z

Made some changes:

When receiving a webhook, the following information about the repository is now stored in project meta:

URL
VCS type (git, hg, svn, etc.)
type (gitlab, github, etc.)
visibility (private/public)

This way when trying to clone a repository and update translations, we don't have to check for visibility again (no extra HTTP request), plus we know the repo type even for self-hosted repositories because of the webhook.

Only if no webhook was received you do have to hook into the filters to manually set the repo type or something.

However, with #57 we could mitigate this. Plus with these changes here we'd already have the storage mechanism for such a new settings field.

swissspidy · 2018-10-25T19:24:11Z

Linting is failing because PHPCS apparently doesn't pick up @inheritdoc... I guess I have to duplicate all affected docs.

This matches the wording used by GitLab and other sites

We use the `ZipArchive` class which is part of the zip extension.

swissspidy · 2018-10-26T08:30:40Z

I guess I really need to copy all docblocks. Ugh.

squizlabs/PHP_CodeSniffer#214
squizlabs/PHP_CodeSniffer#1846

codecov-io · 2018-10-27T19:55:58Z

Codecov Report

Merging #70 into master will increase coverage by 1.14%.
The diff coverage is 76.64%.

@@             Coverage Diff              @@
##             master      #70      +/-   ##
============================================
+ Coverage     67.93%   69.08%   +1.14%     
- Complexity      189      338     +149     
============================================
  Files            17       26       +9     
  Lines           499      828     +329     
============================================
+ Hits            339      572     +233     
- Misses          160      256      +96

Impacted Files	Coverage Δ	Complexity Δ
inc/Loader/Git.php	`0% <0%> (-92.69%)`	`7 <0> (-6)`
inc/CLI/CacheCommand.php	`0% <0%> (ø)`	`5 <5> (+1)`	⬆️
inc/Loader/Subversion.php	`0% <0%> (ø)`	`11 <11> (?)`
inc/CLI/UpdateCommand.php	`0% <0%> (ø)`	`6 <0> (+1)`	⬆️
inc/ZipProvider.php	`87.34% <0%> (-7.18%)`	`25 <2> (+2)`
inc/Loader/Mercurial.php	`0% <0%> (ø)`	`7 <7> (?)`
inc/WebhookHandlerFactory.php	`100% <100%> (ø)`	`5 <5> (?)`
inc/LoaderFactory.php	`100% <100%> (ø)`	`4 <4> (-1)`	⬇️
inc/Updater.php	`94.73% <100%> (+0.61%)`	`22 <2> (+2)`	⬆️
inc/Loader/Base.php	`100% <100%> (ø)`	`2 <2> (?)`
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c0855ad...0ca812a. Read the comment docs.

grappler

Generally it looks good. I have not tested it yet.

I have added a few comments to the documentation.

docs/bitbucket.md

docs/configuration.md

docs/gitlab.md

docs/bitbucket.md

grappler · 2018-10-29T08:08:20Z

docs/bitbucket.md

+2. Click on "Add webhook".
+3. Set `https://<url-to-your-glotpress-site>.com/wp-json/traduttore/v1/incoming-webhook` as the payload URL.
+4. Choose `application/json` as the content type.
+5. Enter and remember a secret key.


It seems like Bitbucket does not support a secret key in cloud version. https://bitbucket.org/site/master/issues/14683/add-webhook-secret

I still need to test this on Bitbucket to actually write accurate docs. But good to know 👍

Co-Authored-By: swissspidy <[email protected]>

See #73.

swissspidy · 2018-10-29T13:39:12Z

Updated the description to give a better picture of the changes this involves.

@grappler @ocean90 If you don't mind giving some feedback, any help is appreciated.

I know some people are eagerly waiting for GitLab support, hence the big changes here.

Only Bitbucket Server (self-managed) supports webhook secrets

ocean90 · 2018-10-30T14:25:33Z

inc/WebhookHandler/Bitbucket.php

+			return hash_equals( $token, $payload_signature );
+		}
+
+		return true;


Looks scary having a permission callback which returns true "by default". Can we change this to false by default?

Returning false by default doesn't work because that means you can't easily use Bitbucket webhooks. Bitbucket.org doesn't support secrets, so 🤷‍♂️

ocean90

LGTM. Once merged, please update our install for some real-world testing.

swissspidy added 5 commits October 24, 2018 20:40

First pass at dedicated webhook handlers

aea957c

Split Git loader into two separate ones

b1a1e7a

Update docs

6a5a43e

Use separate classes for repository data

c46fb29

Reduces Loader class to one per VCS

Begin with doc improvements

d2248d5

swissspidy added 5 commits October 25, 2018 12:51

More docs

a4d83c7

Store project repository info in meta table

453bedb

Update project meta in webhook handlers

08f9d31

Leverage new meta data in repository classes

3d4eae2

Cleanup

f2e07e0

Update some docs

50892ac

swissspidy added 4 commits October 26, 2018 10:07

self-hosted -> self-managed

aec4d47

This matches the wording used by GitLab and other sites

Composer normalize

6106d3c

Specify zip extension requirement in composer.json

dd1b474

We use the `ZipArchive` class which is part of the zip extension.

Update slack plugin suggestion text

cc2c092

swissspidy mentioned this pull request Oct 26, 2018

Support self-hosted Git repositories (e.g. GitLab) #65

Closed

swissspidy mentioned this pull request Oct 26, 2018

Delete translation set meta and project meta on uninstall #71

Closed

swissspidy added 5 commits October 26, 2018 11:55

Inline docs updates

4ae00ab

Store more repository info in project meta

ed00497

Move more stuff around

7a37b12

Return types

229638c

Lint fix

d301277

swissspidy changed the title ~~[WIP] General webhook handler~~ [WIP] Webhooks, Custom Repositories, Bug Fixes, 💥 Oct 26, 2018

swissspidy added 2 commits October 27, 2018 21:49

Begin with updating tests

4028a3c

Lint fixes

324ea43

grappler reviewed Oct 29, 2018

View reviewed changes

grappler and others added 14 commits October 29, 2018 10:12

Update step in Bitbucket webhook description

5f02487

Co-Authored-By: swissspidy <[email protected]>

s/GitLab/Bitbucket

b9aadb5

Co-Authored-By: swissspidy <[email protected]>

Fix secret name in constants docs

75378d6

Co-Authored-By: swissspidy <[email protected]>

Update GitLab webhooks docs

9b465e2

Co-Authored-By: swissspidy <[email protected]>

Remove unnecessary step in GitLab webhook docs

eca717c

Co-Authored-By: swissspidy <[email protected]>

Clarify secret token step

31ff541

Co-Authored-By: swissspidy <[email protected]>

Further clarify webhook docs for GitLab

ecbc761

Co-Authored-By: swissspidy <[email protected]>

Remove unnecessary step in Bitbucket webhooks docs

efb8f66

Co-Authored-By: swissspidy <[email protected]>

Test that the caching in the webhook handlers works

38fdcfe

Set up translation loading using Traduttore Registry

0918fdb

See #73.

Fix local path in Loader class

54e32be

Collapse dirname() calls

eeb53ba

Add PoC subversion loader

8a64c78

Doc fixes

30dc418

swissspidy changed the title ~~[WIP] Webhooks, Custom Repositories, Bug Fixes, 💥~~ The Big 3.0 Update 💥 Oct 29, 2018

swissspidy requested review from ocean90 and grappler October 29, 2018 13:37

swissspidy requested a review from derpaschi October 29, 2018 15:01

swissspidy added 2 commits October 29, 2018 22:43

Update GitLab docs

746a54a

Update docs for Bitbucket

0ca812a

Only Bitbucket Server (self-managed) supports webhook secrets

ocean90 reviewed Oct 30, 2018

View reviewed changes

ocean90 approved these changes Oct 30, 2018

View reviewed changes

This was referenced Oct 30, 2018

Tests for Source Code Loaders #75

Open

Analyze class structure #76

Open

swissspidy merged commit af7023c into master Oct 30, 2018

swissspidy deleted the webhooks branch October 30, 2018 19:13

swissspidy mentioned this pull request Oct 31, 2018

Protect route for incoming Bitbucket webhooks #84

Open

swissspidy added this to the 3.0.0 milestone Nov 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The Big 3.0 Update 💥 #70

The Big 3.0 Update 💥 #70

swissspidy commented Oct 24, 2018 •

edited

Loading

swissspidy commented Oct 25, 2018

swissspidy commented Oct 25, 2018

swissspidy commented Oct 25, 2018

swissspidy commented Oct 26, 2018

codecov-io commented Oct 27, 2018 •

edited

Loading

grappler left a comment

grappler Oct 29, 2018

swissspidy Oct 29, 2018

swissspidy commented Oct 29, 2018

ocean90 Oct 30, 2018

swissspidy Oct 30, 2018

ocean90 left a comment

The Big 3.0 Update 💥 #70

The Big 3.0 Update 💥 #70

Conversation

swissspidy commented Oct 24, 2018 • edited Loading

swissspidy commented Oct 25, 2018

swissspidy commented Oct 25, 2018

swissspidy commented Oct 25, 2018

swissspidy commented Oct 26, 2018

codecov-io commented Oct 27, 2018 • edited Loading

Codecov Report

grappler left a comment

Choose a reason for hiding this comment

grappler Oct 29, 2018

Choose a reason for hiding this comment

swissspidy Oct 29, 2018

Choose a reason for hiding this comment

swissspidy commented Oct 29, 2018

ocean90 Oct 30, 2018

Choose a reason for hiding this comment

swissspidy Oct 30, 2018

Choose a reason for hiding this comment

ocean90 left a comment

Choose a reason for hiding this comment

swissspidy commented Oct 24, 2018 •

edited

Loading

codecov-io commented Oct 27, 2018 •

edited

Loading