Add functional tests

[izqui]

Given the increased complexity with new features, modifying a command can have side-effects in other commands that rely on it.

My proposal is that the initial test suite for the CLI should have a bunch of tests that interact with the CLI directly. Then each test case would pass if, it exits without an error and the side-effects of the command are what was expected (e.g. check the chain to see if a DAO or APM package is at the correct state, or check artifacts against the expected state)

[0x-r4bbit]

I looked into this and I think it'd make sense to decouple the command tasks from the actual command implementations.

For example, if we'd wanna write a test for aragon init, it comes with the side effect that a template repo is being cloned and npm dependencies are being installed. This can slow down the testing experience, so I think it'd be nice if we could test the tasks in an isolated fashion (not necessarily through the CLI). For that however, the tasks need to be extracted into their own module (or at least exported separately).

Would you be okay with that?

[izqui]

Do you mean the individual tasks or the command tasks? Most of the commands export their TaskList at command.task, so they can already be tested without testing the CLI itself.

It would be great to see how what you have in mind would look like, just an example of how a small command would change to allow this would be great

[0x-r4bbit]

Most of the commands export their TaskList at command.task, so they can already be tested without testing the CLI itself.

Exactly. Even that however would mean, one either tests the whole task list as a whole or nothing, unless one can access specific tasks from that list (not super familiar with the TaskList API yet).

So in my head a command could look like this:

exports.handler = (...) => {
  const tasks = new TaskList([
    {
      ...
      task: async someTaskFunction
    }
  ])

  tasks.run(...)
}

someTaskFunction would be defined somewhere (probably imported). That way someTaskFunction can be tested without even knowing its part of a CLI. Obviously one can break it down further into smaller bits and pieces as different tasks may have different "sub" tasks.

Basically where I'm trying to get is a setup where for example the init tasks can be tested, without necessarily having to test (and therefore execute) the tasks of init that introduce the "negative" side effects (like npm install, git clone etc).

This is also not a solution to the problem, more a "how about we go this way for now, instead of not having any tests at all"

Hope this makes sense!

[sohkai]

I'm definitely a fan of how pando's structured their cli code, that sounds similar to what you're suggesting @PascalPrecht.

[0x-r4bbit]

Yes, that's kind of what I had in mind. Thing is just that handler right now is mostly composed of a TaskList, so just having the handler function separated won't be enough, as we'd still end up having all of the tasks in there.

I'll give it more thought.

[0x-r4bbit]

Here's a WIP POC, just to get out the idea: #168

[Quazia]

Any updates on this currently? Getting tests into CLI is going to be a priority for us soon.

[0x-r4bbit]

@Quazia #168 is pending since quite a while (and needs a rebase), but it implements some first couple of tests.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for contributing to Aragon! 🦅