Usage of azure as cloud provider for CI

[artem-sidorenko]

it can be done with kitchen-azurerm

Advantages:

• billing per minute instead of per hour (DO)
• enterprise linux destros available, so we can test on real RH and/or Suse

disadvantages:

• bigger spin-up time for VMs (DO 30 secs vs 2-3 mins Azure)
• a bit more complicated setup for test-kitchen (2 more environment variables)

I also checked the AWS and Google Cloud. AWS has billing per hour and in my tests the spin-up times were 5-7 mins. Google Cloud can be used only for organisations within Europe.

@chris-rock @atomic111 opinions?

[atomic111]

this sounds good. is there a possibility that the pr from a fork are also be checked?

[artem-sidorenko]

@atomic111 unfortunately not. This has nothing to do with cloud provider itself, see details below:

• auth token is required to spawn VMs. This token has to be secret, otherwise somebody would consume resources and we would pay for it
• Travis enables this secret token only for the repository, where the token is configured (see docs on secret variables). In our case this is a base repo, everybody can configure own token for its fork.
• Travis PR runs are a bit complicated, you have two CI runs: so-called push run (CI run on the git branch of our fork) and so-called merge run (git branch gets merged to the master and CI run is executed). Your secret token gets visible for the push run if it is configured for your fork. The secret token gets visible for the merge run only of the source branch of PR is located within base repository.

All this things basically make sense - you do not want to pay for somebody, you do not want to run unknown/unreviewed code automatically, which could expose your secrets. The only one option for us is: to push branches to the base repo. For contributors outside of dev-sec: we have to repush the PR (it takes only two git commands, so it fine from my POV)

[majormoses]

Is there some reason to not use kitchen-docker, kitchen-dokken, or similar? Pretty sure we would be able to test anything we want leveraging travis-ci. No cost other than someone's time which would be true either way. This removes the need to expose any external provider keys other than supermarket if you would like to publish on tagged commits. Also pretty sure that azure and google now do per second billing.

Based on the platforms being tested on DO we can probably use docker in travis: https://github.com/dev-sec/chef-os-hardening/blob/v3.0.0/.kitchen.do.local.yml#L13-L37 if we needed something like AIX that would be another story but one azure would not liklely help with anyways.

[artem-sidorenko]

@majormoses We are using kitchen-dokken in chef-ssh-hardening and/or other cookbooks. I tried it already in the past for chef-os-hardening, the problem was: we are making many kernel related configurations which were not possible within container (I guess even in the privileged mode).

PR #184 might be a small improvement to this situation - we could run some tests with kitchen-dokken and all tests using cloud provider on the master branch

[majormoses]

OK makes sense, did you have a chance to look at my comments?

[artem-sidorenko]

Container support is implemented in #199 together with Travis CI configuration. Azure was implemented with puppet-os-hardening: it looks like usage of DO is still a better option:

• in all options the minimal amount I'm charged is 1 ct, so it does not make sense to go to azure because of costs
• Azure needs much-much more time to spin up the VM
• Cheap VMs are much less performant compared to DO cheap VMs
• In the end we have 4 mins in DO with 20 mins in Azure, difference of 5 times!

I'm closing this issue