"arvados" meta-state should complain if pre-requisites are not available

[mr-c]

No description provided.

[javierbertoli]

Hi, @mr-c You mean 'postgresql' and/or 'nginx' and/or 'letsencrypt'?

The point with those is that you have many choices available to satisfy these dependencies, and depend on how you setup your infrastructure, for example, you can have:

1. Database

• have a dedicated database server in another VM/host than the one you're setting up Arvados, or
• an AWS RDS postgresql server, with replication, or.
• a cluster of postgresql servers with a pool balancer in front of it, or
• other alternatives?

2. Web proxy

• use a separated nginx server to redirect to all the internal arvados components, or
• have multiple, individual nginx on each host, or
• use haproxy (didn't test it, but I'm almost sure it should work), or
• use AWS balancers (same as above)
• other alternatives?

3. Certificates

• use letsencrypt to get your certificates on each host, or
• use a central letsencrypt host to get wildcard certificates, or
• have some certificates from some other party, or
• use AWS acm to manage certificates, or
• other alternatives?

As those situations are more 'infrastructure orchestration' related than 'host/server' management (as this formula tries to achieve), I think it's outside of the scope of this formula to check/complain of those situations.

I added this section in the formula's readme file to make sure people know of these requirements beforehand, so the formula is suitable for a member of an Ops team managing a big infrastructure or to the person that wants to try Arvados in her basement infrastructure of two PCs or even in a vagrant or beefy docker instance. You just need to satisfy the deps as they suit your case.

There are also examples for these configurations that you people can use, if needed.

I'm guessing this is what you're asking for, right? Perhaps this all needs to be better documented?
Is there anything else you thought of?

[mr-c]

Hello @javierbertoli , thanks for the nice reply

The point with those is that you have many choices available to satisfy these dependencies, and depend on how you setup your infrastructure

Sure, but if I did any of that I would provide a pillar to specify those configurations.

If you're completely new to Salt like I am, it would be nice to get more feedback if no pillar was provided

[mr-c]

I think it would be super useful to see a working configuration for an "all in one" based upon Debian Buster or a recent Ubuntu LTS base.

[javierbertoli]

I can add an example file/script for that 😄

tl;rl is 'take a look in the kitchen.yml file, pick the states and pillars in there, put them all together and that would be it'.

Let me take a chance on it and I'll add it to the examples. I'll ping you when I have it, hopefully in a few hours.

[mr-c]

I can add an example file/script for that

Any updates on that, @javierbertoli ?

[javierbertoli]

Sorry for not responding before, but those 'few hours' turned to be 'A FEW' 😋

We added a provision script to make it easier to deploy a single-host Arvados cluster. The script uses the arvados-formula and sets some variables to make it easier to get a cluster up and running.

[mr-c]

@javierbertoli 😂 Thanks for the update! Can that script be linked to from https://github.com/saltstack-formulas/arvados-formula#id4 ?

[javierbertoli]

Sure! I'll add a reference to it under the Special Notes section

[mr-c]

@javierbertoli FYI, there is a broken link in the readme for the provision script: https://doc.arvados.org/v2.1/install/install-using-salt.html

[mr-c]

Does the single node install presume that one is using AWS? Or does it include SLURM as well?

[javierbertoli]

The single-node install does not include SLURM: it is intended for an all-in-one-host installation, so it uses crunch-dispatch-local to run containers in the same instance.

The provision script can be run anywhere, so you can run it in an AWS instance and you'll get a single-node Arvados cluster there.

The Arvados formula allows you to install any dispatcher available, provided you configure the pillars the way you need them.

Arvados currently has three dispatchers:

• crunch-dispatch-local (for single node installations),
• arvados-dispatch-cloud (for dynamic compute on AWS or Azure) and
• crunch-dispatch-slurm (for SLURM integration).

Adding support for other environments is possible. Pull requests are always welcome. Additionally, the Arvados team is happy to discuss adding support for specific environments.

[mr-c]

Thanks @javierbertoli ; can you also add that explanation to https://github.com/saltstack-formulas/arvados-formula ?

I hope @SasSwart and I can contribute a PR for for SLURM integration soon