Acceptance/Regression Testing

[jag3773]

Story

As an admin|developer I want to be sure that new releases don't re-introduce problems that have already been fixed so that we don't have to redo work.

Update

Mostly this is a research task to identify the delta between the following:

• Upstream Gitea integration tests
• Opsview DCS acceptance test script ( https://github.com/unfoldingWord-dev/nagios_plugins/blob/master/gogs_acceptance_test.sh )
• Desired set of tests (https://github.com/unfoldingWord-dev/gogs/wiki/Items-for-Acceptance-Testing )

Notes

Especially since we have a list of customizations that we merge into our Gogs releases, it would be good if we had regression tests setup so that we can verify that we haven't forgot to merge something or reverted certain changes.

We've made a list of things to test for in https://github.com/unfoldingWord-dev/gogs/wiki/Items-for-Acceptance-Testing .

[ethantkoenig]

The admins of gitea want to have a comprehensive set of unit tests before adding end-to-end tests, so I'll first work on adding unit tests (which I assume would be of at least some value to us as well?)

[jag3773]

Most of things we need to test for that are listed in the document should not be merged upstream but should be apart of our fork.

[jag3773]

I have a basic acceptance test for Gogs at https://github.com/unfoldingWord-dev/nagios_plugins/blob/master/gogs_acceptance_test.sh . This is run by Opsview 1x per hour against production. Obviously, this won’t catch anything before production, but it is at least a step in the right direction so we have a better battery of tests running against prod. Hopefully it can also serve as a starting point for writing something to run on travis.

Note that this would run on Travis just fine if we setup a "fixture" that loads the database with a user and token. At that point we could extend the script to create a repo add some commits, etc.

[ethantkoenig]

I recommend we use go-gitea/gitea#1290 for our integration tests; there are still a few things to iron out, but it should provide an accessible and maintainable framework for integration/acceptance/regression/whatever-you-want-to-call-them tests.

[jag3773]

👍

[jag3773]

What's the latest status on this @ethantkoenig ?

[jag3773]

We will review on 2/19