Backup & Recovery

VUM Host upgrade fails – Another task is already in progress

dvdmorera Automation, Backup & Recovery, vCenter, Virtualization, vSphere ,

I came across this issue a few times already, so I figured it would be good to share the findings, especially since it took me down the rabbit hole.

First off, the error “Another task is already in progress” is kind of obvious but yet does not provide enough detail. I ran into this trying to upgrade 4 hosts in a cluster via VUM, no tasks were running at the time of the upgrade start. This was initiated from VMware Cloud Foundation. Thinking the issue was with VCF I tried to do a direct upgrade from VUM, skipping VCF… but received the same error. Also tested with/without vSAN. NFS, etc.

The issue appeared to be related to a race condition where both the VUM remediation AND this “other task” were trying to run at the same time on that host. From the task console we can see that there was another “install” operation running as soon as VUM was trying to remediate. The Initiator give me a hint, and we can see vcHms “cut in line” a did an install operation.

I started looking into the esxupdate log (/var/log/esxupdate.log), and noticed that the host was trying to install a vib. But this vib was already installed, so what gives?

The name of the initiator and one of the vibs gave me a clue (vr2c-firewall.vib). I was running vSphere Replication on that particular cluster, so I started digging in. To validate my suspicion, I shutdown the VR appliance and attempted to upgrade one host. The upgrade worked as expected with no errors, so I was pretty certain something within vSphere Replication was causing this.

First I needed to ssh into the VR appliance or use the console. You won’t be able to ssh into the appliance before enabling ssh. Log in to the console with the root password and run the script to enable ssh.

/usr/bin/enable-sshd.sh

Within the appliance I looked into the config settings for HMS and discovered that there were 2 vibs that were set to auto install at host reconnect. So it appears there is a race condition when VUM and VR and both trying to do an install task at the same time (at reconnect). Makes perfect sense now.

VR_vib_autoinstall

The ESXI update logs indicated that the vr2c-firewall.vib was the one trying to install. After re-checking the vibs (esxcli software vib list) on all the hosts, I did see this vib was already installed but the task from VR kept trying to install at reconnect.

As I workaround, I decided to disable the auto install of this particular vib by running the following command within the /opt/vmware/hms/bin directory and then restart the hms service:

./hms-configtool -cmd reconfig -property hms-auto-install-vr2c-vib=false

service hms restart

hms-configtool

This workaround worked as a charm and I was able to upgrade the rest of the cluster using VUM. I did not find an official KB about this, and this is by no means an official workaround/fix.

Disclaimer: If you plan to implement this fix, be aware that this is not an official VMware blog, and changes to products may or may not cause issues and/or affect support.

Veeam Steps Up Game

dvdmorera Backup & Recovery, Veeam ,

Yesterday, Veeam made some announcements about their strategy as well as their product portfolio. Veeam is gaining market share rapidly given its plethora of features offered.

The announcements include:

  • Veeam Availability Console/Platform
  • Veeam Agents
  • Veeam Availability Suite 9.5 features
  • Sneak peek of v10
  • New Products

The Veeam Availability Platform will tie up all the great features and products that Veeam is introducing, including Orchestrator, Availability Console, Availability Suite, etc.

Veeam_AP

 

One of the great features introduced in my opinion, was in regards to ReFS support. This advanced ReFS integration will allow for faster and reliable backups, reducing the backup time dramatically. Synthetic full backups using ReFS, use pointers to the original blocks rather than copying the changed blocks, reducing both storage needed and time for backups to complete.

Veeam_ReFS

Veeam_vib_ReFS

 

 

 

 

 

Veeam Availability Orchestrator: Introduces DR orchestration via DR plans, automated testing, and reporting capabilities.

Veeam Availability Console: Gives Cloud Providers and Enterprises the ability to manage remote offices from a single console.

Veeam Agents: Wait, What? Yes, Veeam introduced agents in order to fulfill one of the most requested features. Being able to backup PHYSICAL Windows and Linux endpoints, workstations, and servers. You can backup directly to your cloud provider or your datacenter.

Veeam ONE 9.5: Now includes a chargeback capability for enterprises and service providers. This feature is available for VMware vSphere, VMware vCloud Director, and Microsoft Hyper-V.

Last, but not least. My favorite addition. Veeam can backup your O365 email to your datacenter. Nuff said…

Veeam_O365

Recover your VMs in a Snap with Veeam, NetApp & VMware

dvdmorera Backup & Recovery , , ,

recoveryWhen it comes to business continuity and disaster recovery, there are a plethora of options out there. From built-in vendor tools, third party tools and your typical backup/recovery strategy. Choosing a solution does not always result in the best solution, if we take into account the fact that no one person knows all the tools available, and no one has time to research them all. A lot of times tools are chosen based on the vendor’s size, reputation and word of mouth; and of course, a lot of marketing, which I am not a fan of. Every vendor will always say that their solution is the best out there, so don’t rely on marketing material, instead talk to colleagues and other IT individuals through social media, conferences, etc.

Recently I was tasked with providing a BC solution for a specific application. The business was overwhelmed when I presented six different approaches since they really didn’t know what they wanted. After some meetings, I was able to extract what they wanted to achieve, which was a cheap/free solution utilizing current infrastructure that provides granular, and fast recovery of VMs and file level. This sounded like a challenge, but luckily I had just attended a Veeam session at Cisco Live 2015.

The solution selected was Veeam Backup and Replication. This tool allowed for full visibility of existing NetApp volume snapshots of the VMware environment without having to run new backups/snapshots or any additional jobs. Veeam B&R, includes an Explorer tool that connects to both the VMware vCenter and the NetApp arrays. It is then capable of looking inside each volume snapshot, and present the actual VM instead of all individual files. At this point, you have the option to restore the entire VM or even individual files within that object. The great part about this is that if you have a snapshot for exchange server, you are actually able to restore files within that Exchange snap.

Veeam_Explorer

 

Veeam also supports other storage vendors such as HP Lefthand and 3PAR, and new storage vendors will soon be added to this product.

Give it a try, at some point this may be the right solution to a problem or business requirement.