NutanixCE: Cluster Creation Error – Could not discover all nodes

This is just a quick post to detail something that I have come across today. I decided to rebuild my NutanixCE virtual cluster as there was a new version released on 16/07/2015 and my old cluster was failing to connect via Pulse. (A known bug apparently).

When I tried to create a cluster, as per my previous guide, I saw the error:

“Could not discover all nodes specified. Please make sure that the SVMs from which you wish to create the cluster are not already part of another cluster”

CLUSTERERROR1

I found this a little odd, as these were brand new nodes on a fresh install. All I’d done is follow my previous guide! So I checked each node and tried to see if I could create a single node cluster on each manually using the command:

This failed on two of my three CVM nodes with the following error:

“WARNING: genesis_utils.py:512 Failed to reach a node where Genesis is up. Retrying….”
CLUSTERERROR2

After some googling, I found that someone had suggested to restart genesis service and check the logs to find out what was happening.

CLUSTERERROR4

So it seems that the CVM is trying to contact the host KVM and being unsuccessful. Sure enough, if I tried to ssh to my KVM host from the CVM box, it was also failing with the error:

FIPS mode initialized
Read from socket failed: Connection reset by peer

CLUSTERERROR5

After some futher searching I found that someone on the forum that had fixed the issue by regenerating the SSH keys on the KVM host – they didn’t explain how to do this and I’m a bit of a novice with Linux/KVM but the process is easy on an ESXi host. After a quick dig I found “sshd-keygen” in the /bin folder which I ran.

I then tried to SSH from my CVM to KVM again, and this time it prompted me for a password! Success!

CLUSTERERROR6

I repeated this process (sshd-keygen and then SSH from CVM) on my other broken node. I then retried my cluster creation and it also worked!
CLUSTERERROR7

I thought it would be interesting to document as I came across the issue and a combination of forum posts and a little digging from myself paid off. I hope this helps anyone with this issue!

Leave a Reply

Your email address will not be published. Required fields are marked *