Automation

VUM Host upgrade fails – Another task is already in progress

dvdmorera Automation, Backup & Recovery, vCenter, Virtualization, vSphere ,

I came across this issue a few times already, so I figured it would be good to share the findings, especially since it took me down the rabbit hole.

First off, the error “Another task is already in progress” is kind of obvious but yet does not provide enough detail. I ran into this trying to upgrade 4 hosts in a cluster via VUM, no tasks were running at the time of the upgrade start. This was initiated from VMware Cloud Foundation. Thinking the issue was with VCF I tried to do a direct upgrade from VUM, skipping VCF… but received the same error. Also tested with/without vSAN. NFS, etc.

The issue appeared to be related to a race condition where both the VUM remediation AND this “other task” were trying to run at the same time on that host. From the task console we can see that there was another “install” operation running as soon as VUM was trying to remediate. The Initiator give me a hint, and we can see vcHms “cut in line” a did an install operation.

I started looking into the esxupdate log (/var/log/esxupdate.log), and noticed that the host was trying to install a vib. But this vib was already installed, so what gives?

The name of the initiator and one of the vibs gave me a clue (vr2c-firewall.vib). I was running vSphere Replication on that particular cluster, so I started digging in. To validate my suspicion, I shutdown the VR appliance and attempted to upgrade one host. The upgrade worked as expected with no errors, so I was pretty certain something within vSphere Replication was causing this.

First I needed to ssh into the VR appliance or use the console. You won’t be able to ssh into the appliance before enabling ssh. Log in to the console with the root password and run the script to enable ssh.

/usr/bin/enable-sshd.sh

Within the appliance I looked into the config settings for HMS and discovered that there were 2 vibs that were set to auto install at host reconnect. So it appears there is a race condition when VUM and VR and both trying to do an install task at the same time (at reconnect). Makes perfect sense now.

VR_vib_autoinstall

The ESXI update logs indicated that the vr2c-firewall.vib was the one trying to install. After re-checking the vibs (esxcli software vib list) on all the hosts, I did see this vib was already installed but the task from VR kept trying to install at reconnect.

As I workaround, I decided to disable the auto install of this particular vib by running the following command within the /opt/vmware/hms/bin directory and then restart the hms service:

./hms-configtool -cmd reconfig -property hms-auto-install-vr2c-vib=false

service hms restart

hms-configtool

This workaround worked as a charm and I was able to upgrade the rest of the cluster using VUM. I did not find an official KB about this, and this is by no means an official workaround/fix.

Disclaimer: If you plan to implement this fix, be aware that this is not an official VMware blog, and changes to products may or may not cause issues and/or affect support.

Golden Nuggets: #2 NetApp VSC – Provisioning

dvdmorera Automation, NetApp, NetApp&VMware integration , ,

Wizard_LegoI previously wrote a post or two related to NetApp’s Virtual Storage Console for VMware vCenter, including its uses and how to install it. In this post I would like to highlight its importance in a NetApp/VMware or FlexPod environment.

NetApp’s VSC is a very handy tool that will allow you to achieve many tasks in an automated fashion rather than doing them manually; therefore, it will save you time and eliminate the possibility of human error. During the configuration and provisioning process, the human error factor can result in a lot of frustration for the admin/engineer, as troubleshooting often ends with the finding of a simple step that was missed. VSC is not a new tool, but there is a new version (6.0) that introduces new features and fixes. VVols requires VSC 6.0 , by the way.

One of the coolest features of VSC (IMHO), is the provisioning of storage from the VMware Web client. If we were to create an NFS datastore for VMware, the manual process will include the creation of a volume, granting the correct permissions for the export, and then mounting the datastore to each host. This takes quite a bit of time and requires having to jump between UIs.

VSC allows you to do all the aforementioned steps from the VMware client (Web or C#) from one easy to use provisioning wizard. You can provision datastores, volumes, exports, and permissions by simply right-clicking the cluster or an individual host. If  you do this from the cluster level, VSC will create the volume and exports which is cool, but the coolest part is that it will also add the hosts’ IP addresses with the necessary permissions to the export, and it creates and mounts the datastores on all hosts within the cluster, NFS in this case. That alone is a good reason to have VSC, albeit there are many other tasks that VSC is capable of.

One of the frequently asked questions I see on both the VMware and NetApp communities, relates to errors and failure to mount an NFS datastore in vCenter. Often times it relates to the permissions within the exports, so VSC will do all this for you and prevent such issues.

Note: Please refer to NetApp’s Interoperability Matrix Tool (IMT) to determine which version you need. Specific versions are needed for VMware’s Web client and the same goes for the vSphere Client.

VSC_Cluster

 

 

 

 

 

 

 

VSC options for Cluster

 

VSC_Host

 

 

 

 

VSC options for Host

 

 

Get your NetApp – VVols while they are HOT

dvdmorera Automation, NetApp, NetApp&VMware integration, Storage , , ,

pistonToday, the long awaited NetApp VASA Provider (VP) and the new shiny VSC console have been released to general availability.

So what does VASA and VSC have to do with VVols? Everything. In previous posts I talked about both NetApp’s VSC and VASA provider for VMware here. These offerings along with VAAI provide a tight integration between VMware and NetApp. Given the transition from VMware’s C# Client (fat) to Web Client, it resulted in the need of updated versions, and this is how VSC 6.0 and VASA Provider 6.0 were born.

Now to VVols. In order to be able to deploy VVols with NetApp there are a few requirements.

  • vSphere 6.0 (or later)
  • NetApp Clustered Data ONTAP 8.2.1 or later (thanks Nick for the clarification)
  • VSC 6.0
  • NetApp VASA Provider 6.0

You can see now why this announcement is such a big deal, both VSC and VP make up the engine that powers up the VVols machine. Both vSphere and cDOT 8.2.1 have been out for a while, but those that wanted to test drive VVols with GA code could not do that until today except by using beta code.

VSC brings and additional enhancement with its new version and that is the addition of PowerShell cmdlets for most VSC features. These cmdlets along with PowerCLI and NetApp’s PowerShell Tool Kit can provide tighter integration and automation between NetApp and VMware.

 

You can download VSC and VP from the links below:

VSC_6.0_Download

NetApp_cDOT_VASA_Provider_6.0

vROps 6.0: Why you need it now

dvdmorera Automation, vCenter, vCOPS , , ,

One of the most time-consuming activities within an IT infrastructure is “keeping the lights on”. The operations of an IT environment requires many, many man hours, and this may also vary depending on the type of tool(s) utilized. There are many operations related tools that help administrators manage an environment, but VMware’s vRealize 6.0 brings great enhancement to its predecessors, and definitely sets itself apart from the competitors.

vRealize Operations Manager 6.0 introduces smart alerts. These types of alerts are the result of the aggregation of similar alerts into a single “smart” alert. This approach reduces the number of false-positive alerts, as well as decreasing the alert storms when issues appear. vROps, also provides a mitigation recommendation based on the smart alerts and allows for automatic problem remediation from vROps to the target vCenter, without the need to go to the vCenter Web Client. This integration provides management from one single console. The same applies within the vCenter Web Client, where information from vROps is visible for each entity.

The aforementioned features are key differentiators from older vCOps versions as well as from 3rd party competitors. vRealize Operations Management suite also allows to monitor other aspects outside the VMware environment such as storage, network, OS, and applications among others. This integration is done via management packs and plug-ins that allows businesses of all sizes to monitor their IT infrastructure from one single UI. It is also important to mention that vROps is capable of providing information for both security compliance as well as regulatory compliance such as HIPAA, SOX, and PCI among others.

For those that are currently running pre-vRealize version of Operations Manager (vCOps), I highly recommend that you make the move to vRealize 6.0. The additional features are worth the migration, which by the way is a breeze. I was able to deploy a new vROps instance and import from vCOps 5.8 in less than an hour. The new version provides many additional reports and dashboards that will reduce the time spent monitoring your IT Infrastructure.

By the way, to migrate to vROps 6.0 you need at least vCOps 5.8.1 version on the source system. I have a post regarding upgrade of vCOps here. You can either deploy vROps as a virtual appliance, on Windows OS or RHEL 6.5. Please refer to VMware’s documentation for details.

Here is a dashboard comparison between vCOps 5.8 and vROps 6.0.

Figure 1: vCOps 5.8.2

vCOps5.8.2

 

Figure 2: vROps 6.0

vROps6.0

 

vCenter Alarm Definition Migration

dvdmorera Automation, vCenter , , , ,

Alarms play a very important role in vCenter in order to keep the VMware admins aware of what’s going on with their environments. Even if you have vCOPS configured, I personally still like to have alarm definitions set up, specially in the event that the vCOPS vApp is shutdown, etc.

One of the use cases for alarm definition migration is the creation of a new vCenter, or even just wanting to have all vCenters configured with the same alarms. You can do all this manually, but If you have defined alarm definitions in the past, you already know this a very time consuming task and new versions of vCenter include more and more alarms as new releases are introduced. You can also have your own alarms, in which case, it make sense to have a method to copy such definition when needed.

VMware has a KB article with a script dating from 4.x version. I used this script, which works well, but it leaves out a couple of minor details that may be overlooked, specially with people with little to no experience with Power Shell and PowerCLI. In KB1032660 VMware has a script that can do this for you, all you have to do is edit the script and run it. However, the picture shows PowerShell ISE “run” button for the screen shot. In an “out of box” environment, this will fail. Why? Because this script requires PowerCLI cmdlets (connect-viserver, etc.) and those are not part of PowerShell ISE unless you import the snap-ins manually, which is totally feasible.

Error

To make the script work, you’ll have to either add the PSSnapins or run the script from PowerCLI. The easiest way is to just run the script from PowerCLI. Although this is self explanatory, I will show how to go about doing that.

Overview:

  • Download latest version of PowerCLI from VMware downloads that match your environment
  • Download the script from VMware’s site KB1032660
  • Open the script using PowerShell ISE, notepad, notepad++, or your editor of choice
  • Edit the variables for $vc1 and $vc2 and line 92 to (true), then save the script

variables

ifTrue

  • Open PowerCLI
  • Navigate to the location where you saved your script and run the script from there

navigate

  • You will get a few prompts, just hit Y [default]

At this point you should see all the vCenter alarms copied to your target vCenter server. As you add alerts to your source vCenter, you can use this script to keep all your vCenters in sync until VMware comes up with this tool.

Feature request!?!?