Replacing vCenter with vSAN Encryption Enabled

In my previous post, I talked about vSAN Encryption configuration, and key re-generation among other topics. On that post you can see that there is a trust relationship amongst the vCenter and KMS server/cluster. But what happens if my vCenter dies, gets corrupted, or I simply want to build a new vCenter and migrate my vSAN nodes to it with Encryption enabled???

One day, the lab that hosts my vCenter had a power issue and the VCSA appliance became corrupted. I was able to fix the issue, but then discovered that SSO was not working. I figured it was faster to deploy a new VCSA appliance rather than troubleshooting (yes, I’m impatient). I deleted the old vCenter and proceeded to deploy the a new VCSA.

As I was adding the host to the new vCenter, I remembered that vSAN encryption was enabled. Now what? Sure enough after all the hosts were moved, the drives from within the Disk Groups were showing unmounted. I went ahead and created a new relationship with the same KMS cluster, but the issue persisted.

If you run the command “vdq -q” from one of the host, you will see that your drives are not mounted and are ineligible for use by vSAN. In the UI you will see that your disks are encrypted and locked because the encryption key is not available.

The FIX:

In order to recover from this and similar scenarios, it is necessary to create a new cluster with the same exact configuration as before. Although I did establish a relationship with the same KMS cluster, I missed a very important step, the KMS cluster ID.

It is imperative that the same KMS cluster ID remains in order for the recovery feature to work. Let’s think about the behavior. Although the old vCenter is gone, the hosts still have the information and keys from the KMS cluster, if we connect to the same KMS cluster with the same cluster ID, the hosts will be able to retrieve the key (assuming the key still exists and was not deleted). The KMS credentials will be re-applied to all hosts so that hosts can connect to KMS to get the keys.

Remember that the old vCenter was removed, so I couldn’t retrieve the KMS cluster ID from the vCenter KMS config screen, and this environment was undocumented since it is one of my labs (it is now documented). Now what?

Glad you asked. Let’s take a look at the encryption operation.

In this diagram we can see how the keys are distributed to vCenter, hosts, etc. The KMS server settings are passed to hosts from vCenter by the KEK_id.

In order to obtain the kmip cluster ID, we need to look for it under the esx.conf file for the hosts.  You can use cat, vi, or grep (easier) to look at the conf file. You want to look for kmipClusterId, name(alias), etc. Make sure the KMS cluster on the new vCenter configured exactly as it was before.

cat /etc/vmware.esx.conf 

or something easier…

grep “/vsan/kmipServer/” /etc/vmware/esx.conf

After the KMS cluster has been added to new vCenter as it was configured in the old vCenter, there is no need for reboots. During reconfiguration the new credentials will be sent to all hosts and such hosts should reload keys for all disks in a few minutes.


vSAN Stats Object Out of Date

Several people asked me this question several times, so I figured I’ld write a quick post about it.

When the default vSAN policy was being changed, people started noticing that the Stats Objects (Health) will show as “Out of Date”, even though the policy was applied at the end of the wizard.

A few things to keep in mind:

  • The Stats Object is exactly that, an Object, just like a VM home folder, or VMDK.
    • That object is associated with a Policy, usually the default vSAN policy
  • If you change a policy, you can apply this immediately through the wizard
    •  However, this applies the policy to the VMs (objects)
    • Stats Object is not part of any VM
  • If you change the policy that the Stats Object is using or sharing with VMs, then you will need to manually re-apply that policy to the Stats Object.


  1. Policy change (Default in this case)
  2. Reapply Policy to VMs now
  3. Stats Object show “Out of Date”
  4. Edit the Storage Policy under Health and Performance and click OK
  5. This will bring the Object back to compliance













Quick Video about it

vSAN VCG Checks

One of the most important aspects of any storage solutions, involves utilizing hardware to its advantage. Many storage vendors have taken advantage of faster drives and other technologies to create fast storage solutions, and vSAN is no different. We will discuss why it is so important for vSAN to have compatible/supported hardware and how to check this.

One of the main requirements for VMware’s HCI solution is for hardware to be on its Hardware Compatibility List (HCL), also known as VMware Compatibility Guide (VCG). This compatibility guide will allow you to check existing hardware and/or hardware that you plan to purchase for vSAN. You can also check vSAN ready nodes against this guide.

Before you deploy vSAN, all hardware must have passed the compatibility test. This is to ensure that the best performance will be achieved, as well as reducing possible issues due to hardware. Hardware compatibility with vSAN includes but not limited to hard drives (MD), flash devices, storage controllers, etc. It is not only necessary for the hardware to be on the compatibility list, but also have the appropriate firmware and driver versions for the specific version of ESXi.

How to check hardware against VCG

You can check hardware, firmware and driver version by going to VMware’s VCG website here

You can also check compatibility of vSAN ready nodes at this site.

Once vSAN has been deployed, vSAN will check your hardware compatibility against the downloaded VCG version. You can also update the local VCG version from the Web UI. To make sure the HCL DB is up to date on your cluster go to Cluster>Manage>Settings>Health and Performance from the Web UI. You can update the list by clicking on the “Get latest version online”.



If your vCenter does not have access to the internet, you can download/Upload the file manually, as follows:

  • Log in to a workstation with access to the internet
  • Go to
  • Save the all.json file
  • From the same workstation connect to your vCenter, or you can copy the file to another workstation/server with access to vCenter
  • From Cluster>Manage>Settings>Health and Performance on the Web UI, select “update from file” and select the all.json file you downloaded


If your hardware/firmware/drivers are not compatible with the VCG, you will get a warning/error.



There is a “fling” tool that will also accomplish this, but in addition, it will provide more information as to why there is a warning or error. The tool is called “vsan hardware compatibility list checker”, very clever name, right?! It is an executable that runs from a Windows command prompt, and produces a nice html report. You can download the tool from here

Once downloaded extract it on a window system, open command prompt and navigate to the location of the folder. Launch hclCheck with the necessary flags (e.g. –hostname, –help, etc.). In my case, I did this on my home lab, I am using self-signed certs so I had to use –noSSLVerify flag. Notice that this tool will download the latest version of the HCL DB and check against it.


After a few seconds, the check is completed and a report is created on the current directory.


Double click on the file to open the report on your default browser.  One important piece to notice here is that the report also includes the PCI IDs for the device. So what? you may ask. Well, this can be used to check against the VCG, and get the correct firmware and driver info. If the VCG shows multiple instance of the same controller, SSD, etc., check the PCI IDs to pick the correct one and get the recommended driver and firmware version.

In this report, you can see that my home lab hardware is not supported for vSAN… it works, but not supported.



Example of multiple entries on VCG. Notice different SSID (Sub-device ID).




HCIBench null Results? But it worked last week?!

hcibench_logoLast week a new version of HCIBench was released (version If you are not familiar with HCIBench, this is a VMware Fling that gives you a nice web UI to conduct performance testing for vSAN environments. It leverages vdbench to create VMs and stress test vSAN. The reports generated in addition to realtime views of vSAN observer can give you a great look at what your vSAN cluster can do.

If you are running version, you may now encounter an issue where the test runs, but it doesn’t run as long as you told it to. It also displays zeros for results. You probably ran it prior to Nov. 1st as was fine, so what gives?

The issue is with the vm-template. HCIBench will spin up vdbench VMs during the test. The problem is, the password on those VMs has expired for version

In host-ESXi_IP-vm-deploy.log or vc-VC_IP-vm-deploy.log you will see the err msg: “no such mark “~pvscsi””
Or in all-in-one-testing.log, you will see err msg: “Too many authentication failures”
Or in io-test log you will see err msg: “Net::SCP::Error: SCP did not finish successfully (1)”
Your test will finish with 0s results.

What to do?

You have 2 choices. You can either upgrade HCIBench to version or replace the vm-template file within HCIBench. Ideally, you will upgrade, since there are more fixes on this new release.

Upgrade Path – Download HCIBench ova from and deploy.

Workaround – If you are not willing to upgrade, we are providing the vm-template file in the download, vm-template.tar.gz, please download this file, upload it to HCIBench:/root/, in HCIBench command line, run 

“tar -zxvf /root/vm-template.tar.gz ; mv -f vm-template/* /opt/output/vm-template/”


OR you can resolve the issue by yourself:

1. Deploy the perf-photon-vdbench vm from http://HCIBENCH_IP/vm-template/perf-photon-vdbench.ovf (KEEP THE VM NAME AS perf-photon-vdbench )
2. log into perf-photon-vdbench vm using root/vdbench and run “chage -I -1 -m 0 -M 99999 -E -1 root”
3. shutdown perf-photon-vdbench vm
4. Login into HCIBench and run “rvc ‘VC_USERNAME’@VCENTER_IP”
5. In RVC, go to /VC/DATACENTER/ and run “ /opt/output/vm-template vms/perf-photon-vdbench”, after downloading, exit rvc by typing “exit”
6. Run “mv /opt/output/vm-template/perf-photon-vdbench/* /opt/output/vm-template” and “chmod 755 /opt/output/vm-template/*”


List of Fixes on version

  • Increased Timeout value of client VM disk from 30 seconds to 180 seconds.
  • Disabled client VM password expiration.
  • Disabled client VM OS disk fsck.
  • Set Observer interval to 60 seconds to shrink the size of observer data.
  • Fixed PCPU calculation.
  • Created link directory of /opt/automation/logs, user will be able to review the testing logs in http://HCIBENCH/hcibench_logs/
  • Increased the RAM of HCIBench from 4GB to 8GB to avoid running out-of-resource issue.

VSAN Proactive Rebalance

balance1There has been a lot of questions as to what happens when a rebalance task is triggered in VSAN. By default, VSAN will try to do a proactive rebalance of the objects as the disks start hitting certain thresholds (80%). There are instances, during failures/rebuilds, or even when organic imbalance is discovered, where administrators may trigger a proactive rebalance task.

What happens

Once you click on the “balance disks” button. You are opening a 24-hr window where rebalance will take place. This means that the rebalance operation may take up to 24 hours, so be patient. Many people have voiced frustration because the UI shows a 5% progress (or lack there of) for a very long time, almost appearing as it is stuck. The rebalance is taking place on the background.

You may also not see any progress at all for the first 30 minutes. This is because VSAN wants to wait to make sure that the imbalance persists before it attempts to move any objects around. After all, the rebalance task is moving objects between disks/nodes, so copying data over the network will take resources, bandwidth and time; so plan accordingly if you must rebalance.

Background Tasks:

  • Task at 1 percent when created.
  • Task at 5 percent when rebalance command is triggered.
  • Then waits for the rebalance to complete before setting the percent done to 100.
    • During the waiting period, it will check to see if rebalance is done (clom-tool command).
    • If not done, it will sleep for 100 seconds and check again if rebalance is done.

By default when triggered from the VC UI, the task will run for 24 hours or whenever the rebalance effort is done, whichever comes first.

Notice that if your disks are balanced, the button is greyed out to avoid unnecessary object “shuffling”.