tendrl-ansible playbook fails

[SaravanaStorageNetwork]

Hi,

Tried setting up tendrl-ansible with two nodes.

ansible-playbook of site.yml succeeded.

But, tendrl server accessed from browser, I can see the following errors after import.

error Failure in Job 74d8f944-53a9-4d14-84cb-bbf44c8d764a Flow tendrl.flows.ImportCluster with error: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/init.py", line 218, in process_job the_flow.run() File "/usr/lib/python2.7/site-packages/tendrl/commons/flows/import_cluster/init.py", line 103, in run raise ex AtomExecutionFailedError: Atom Execution failed. Error: Error executing post run function: tendrl.objects.Cluster.atoms.ConfigureMonitoring 20 Nov 2017 01:27:19

error Failed post-run: tendrl.objects.Cluster.atoms.ConfigureMonitoring for flow: Import existing Gluster Cluster 20 Nov 2017 01:27:17

error Could not find atom tendrl.objects.Cluster.atoms.ConfigureMonitoring 20 Nov 2017 01:27:15

Could you check?

[mbukatov]

Hi @SaravanaStorageNetwork ,

Tried setting up tendrl-ansible with two nodes.

Could you share the inventory file? You can anonymize hostnames, but I'm interested in number of machines in each group. I'm asking because 1) I want to understand your environment 2) at first sight, I see no way how could 2 node inventory be a valid setup for Tendrl.

tendrl server accessed from browser, I can see the following errors after import

What makes you think that this is a problem of tendrl-ansible?

@r0h4n This error message Could not find atom seems to come from tendrl-commons, what is likely broken for this error to appear?

[SaravanaStorageNetwork]

@mbukatov
inventory file
[root@centosmaster ~]# cat inventory

[gluster-servers]
centosmaster

[tendrl-server]
tmaster

Regarding, centosmaster - it is a simple single node gluster server.

What makes you think that this is a problem of tendrl-ansible?

There seem to be many components in tendrl. I am not sure whether the issue is specific to tendrl-ansible except that I deployed it using tendrl-ansible.

[mbukatov]

@SaravanaStorageNetwork your environment is not valid wrt both Gluster and Tendrl, you need at least 2 machines for gluster trusted storage pool to be created. With sheer single node, there is no cluster to be imported into Tendrl.

Retry with at least 2 nodes to check if the problem was caused by this.
If that is the case, report his problem for tendrl ui, which should not have allowed you to import single node. Tendrl dev team then creates other issues to fix this in backed if needed.

USM QE team tests with 6 gluster servers, where we deploy either beta or gama volume from https://github.com/usmqe/usmqe-setup/tree/master/gdeploy_config

[SaravanaStorageNetwork]

@SaravanaStorageNetwork your environment is not valid wrt both Gluster and Tendrl, you need at least 2 machines for gluster trusted storage pool to be created. With sheer single node, there is no cluster to be imported into Tendrl.

You can create a gluster volume with single node.
I have created a gluster volume a single node. I can import the cluster too.
I can see the following:

1. cluster
2. Node.
3. Brick.

But I cannot see the gluster volume. and few errors as mentioned above.

Is there any limitation in tendrl supporting volume with minimum of 2 nodes ?

Anyway, I will retry with two nodes and update.

[SaravanaStorageNetwork]

I have tried with two nodes. but it fails again with following messages.

ImportCluster: c6d98bea-b9aa-4cb1-9a01-55c3c320b658

Time Submitted:20 Nov 2017 19:04:17

Status:Failed

Events

error Failure in Job c6d98bea-b9aa-4cb1-9a01-55c3c320b658 Flow tendrl.flows.ImportCluster with error: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tendrl/commons/jobs/__init__.py", line 218, in process_job the_flow.run() File "/usr/lib/python2.7/site-packages/tendrl/commons/flows/import_cluster/__init__.py", line 103, in run raise ex AtomExecutionFailedError: Atom Execution failed. Error: Error executing atom: tendrl.objects.Cluster.atoms.ImportCluster on flow: Import existing Gluster Cluster 20 Nov 2017 07:04:31

error Failed atom: tendrl.objects.Cluster.atoms.ImportCluster on flow: Import existing Gluster Cluster 20 Nov 2017 07:04:29

info Released lock (c6d98bea-b9aa-4cb1-9a01-55c3c320b658) for Node (83eeaa57-151d-4d05-9110-2e623f628a61) 20 Nov 2017 07:04:29

error Could not install tendrl-gluster-integration on Node gluserver1Error: Error: Package: glusterfs-events-3.12.1-2.el7.x86_64 (centos-gluster312) Requires: glusterfs-server(x86-64) = 3.12.1-2.el7 Installed: glusterfs-server-3.12.2-2.el7.x86_64 (@centos-gluster312-test) glusterfs-server(x86-64) = 3.12.2-2.el7 Available: glusterfs-server-3.12.0-1.el7.x86_64 (centos-gluster312) glusterfs-server(x86-64) = 3.12.0-1.el7 Available: glusterfs-server-3.12.1-1.el7.x86_64 (centos-gluster312) glusterfs-server(x86-64) = 3.12.1-1.el7 Available: glusterfs-server-3.12.1-2.el7.x86_64 (centos-gluster312) glusterfs-server(x86-64) = 3.12.1-2.el7 20 Nov 2017 07:04:28

info Installing tendrl-gluster-integration on Node gluserver2 20 Nov 2017 07:04:26

info Check: Minimum required version (3.12.0) of Gluster Storage 20 Nov 2017 07:04:26

info Acquired lock (50568711-1260-4b70-af92-ba00765ada87) for Node (e95682bf-5649-4ea3-a747-
b27fe73f7d1c) 20 Nov 2017 07:04:26

info Running Flow tendrl.flows.ImportCluster 20 Nov 2017 07:04:26

info Processing Job 50568711-1260-4b70-af92-ba00765ada87 20 Nov 2017 07:04:26

info Installing tendrl-gluster-integration on Node gluserver1 20 Nov 2017 07:04:21

info Check: Minimum required version (3.12.0) of Gluster Storage 20 Nov 2017 07:04:21

info Importing (job: 50568711-1260-4b70-af92-ba00765ada87) Node e95682bf-5649-4ea3-a747-
b27fe73f7d1c to cluster 993dd13c-d189-44b1-9346-9c7f6d1bd7ba 20 Nov 2017 07:04:21

info Acquired lock (c6d98bea-b9aa-4cb1-9a01-55c3c320b658) for Node (83eeaa57-151d-4d05-9110-2e623f628a61) 20 Nov 2017 07:04:20

info Running Flow tendrl.flows.ImportCluster 20 Nov 2017 07:04:19
info Processing Job c6d98bea-b9aa-4cb1-9a01-55c3c320b658 20 Nov 2017 07:04:19

[mbukatov]

@SaravanaStorageNetwork looking at the log, the root cause is fortunately stated there, package tendrl-gluster-integration can't be installed on gluserver1 machine because of rpm package conflict:

error Could not install tendrl-gluster-integration on Node gluserver1Error: Error: Package: glusterfs-events-3.12.1-2.el7.x86_64 (centos-gluster312) Requires: glusterfs-server(x86-64) = 3.12.1-2.el7 Installed: glusterfs-server-3.12.2-2.el7.x86_64 (@centos-gluster312-test) glusterfs-server(x86-64) = 3.12.2-2.el7 Available: glusterfs-server-3.12.0-1.el7.x86_64 (centos-gluster312) glusterfs-server(x86-64) = 3.12.0-1.el7 Available: glusterfs-server-3.12.1-1.el7.x86_64 (centos-gluster312) glusterfs-server(x86-64) = 3.12.1-1.el7 Available: glusterfs-server-3.12.1-2.el7.x86_64 (centos-gluster312) glusterfs-server(x86-64) = 3.12.1-2.el7 20 Nov 2017 07:04:28

Note: this error message is brought to you by @shtripat who fixed Tendrl/node-agent#627 for us. Let's thank him here again :-)

Going back to your log output, based on this error, it seems you have some problem with gluster repositories. Have you enabled any additional repositories on your storage machines? What yum repolist shows?

Anyway, the log you provided when trying with 2 machines also shows that the original error is no longer shown, which means that you should report he issue as I suggested in #62 (comment)

[mbukatov]

@SaravanaStorageNetwork

You can create a gluster volume with single node.

This is news for me. Could you point me to upstream documentation for this single node gluster trusted storage pool feature?

[SaravanaStorageNetwork]

@mbukatov

This is news for me. Could you point me to upstream documentation for this single node gluster trusted storage pool feature?

Please check this - one scenario where single node glusterfs is possible -
http://lists.gluster.org/pipermail/gluster-users.old/2016-October/028821.html

Going back to your log output, based on this error, it seems you have some problem with gluster repositories. Have you enabled any additional repositories on your storage machines? What yum repolist shows?

I am using Centos 7.4 and then trying out tendrl-ansible.
I have followed Centos official docs. https://wiki.centos.org/SpecialInterestGroup/Storage/gluster-Quickstart to setup glusterfs, but concluded that it will not install glusterfs-events(which is a error).

Another error is, by default rubygem-json-2.0.2-2.el7.x86_64 gets installed and tendrl-ansible looks for rubygem-json-1.7.7-30.el7 and Fails.

[mbukatov]

That message about one node cluster looks interesting, thank you for the pointer. I don't have enough time to check this now, but will look into this later.

[mbukatov]

Transferring to Tendrl/gluster-integration#719