One of my customers has recently implemented a new NetApp based Metro cluster, and their intention is to do as much deployment hands-off through tools such as Chef, Vagrant and Knife. This means that developers and application operations can deploy virtual machines without going through ops first. However, since NetApp is active/passive, this also means that virtual machines should be pinned to the hosts that are local to the storage controller they are located on.
Obviously, this can be achieved through the use of DRS "should-run" rules. However, adjusting these manually would require manual intervention from operations, and if for some reason a virtual machine slips through it might migrate to a non-local host, causing unneeded load on the Inter-site link. But to allow developers to set this automatically, it would require giving them permissions to adjust the DRS VM groups, which is far from ideal.
A easy solution to this is to use the new vSphere >= 5.1 feature of tagging. One of the reasons I am actually writing this blog is because I feel tagging can have a lot of value if properly implemented (and because Joep Piscaer was asking for it ;)), and this is a prime example where the power of tagging really shines.
First, what we do is create a Tag category called Virtual machine location. Set the cardinality to one tag per object (after all, virtual machines can only be located in one datacenter) and the Associable object types to Virtual Machines.
Then, we'll create two tags - one for each datacenter. Name these however you like, as long as they are descriptive and consistent.
In additon, we also create a tag category and tag for the clusters, this is to indicate that these clusters are actually configured as a metro cluster, which will come in handy later. This means that users can tag their virtual machines with a location tag, You could even integrate this into your deployment system by calling the powercli cmdlets for tagging (see http://blogs.vmware.com/PowerCLI/2013/12/using-tags-with-powercli.html for more information).
however at the moment this still doesn't do anything besides tagging the virtual machine. We're going to need some more scripting for that. Currently, this customer doesn't have vCenter orchestrator running or we'd create a workflow there, so for the moment we have the following powerCLI script that is being ran every hour:
# Define variables
$vcenter = "xxx"
$metrolefttag = "dc-loc01"
$metrorighttag = "dc-loc02"
$metroclustertag = "Cluster type/Metro"
$leftdrsgroupname = "spr-prod-dogo-vm"
$rightdrsgroupname = "spr-prod-rone-vm"
try {
Connect-VIServer $vcenter
}
catch {
}
First, we'll start with defining some variables and connecting to the vCenter.
function Load-PowerCLI {
Try {
if ( (Get-PSSnapin -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue) -eq $null ) {
Add-PSSnapin VMware.VimAutomation.Core
}
. 'C:Program Files (x86)VMwareInfrastructurevSphere PowerCLIScriptsInitialize-PowerCLIEnvironment.ps1'
}
Catch {
Throw "This script requires VMware PowerCLI to be installed. Please check configuration and try again. "
}
}
We load the PowerCLI modules..
Function Add-VMToDrsGroup {
param([parameter(Mandatory=$true)] $Cluster,
[parameter(Mandatory=$true)] $DrsGroup,
[parameter(Mandatory=$true, ValueFromPipeline=$true)] $VM)
begin {
$Cluster = Get-Cluster -Name $Cluster
}
process {
if ($Cluster) {
if ($DrsGroup.GetType().Name -eq "string") {
$DrsGroupName = $DrsGroup
$DrsGroup = Get-DrsGroup -Cluster $Cluster -Name $DrsGroup
}
if (-not $DrsGroup) {
Write-Error "The DrsGroup $DrsGroupName was not found on cluster $($Cluster.name)."
}
else {
if ($DrsGroup.GetType().Name -ne "ClusterVmGroup") {
Write-Error "The DrsGroup $DrsGroupName on cluster $($Cluster.Name) doesn't have the required type ClusterVmGroup."
}
else {
$VM = $Cluster | Get-VM -Name $VM
If ($VM) {
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$spec.groupSpec = New-Object VMware.Vim.ClusterGroupSpec[] (1)
$spec.groupSpec[0] = New-Object VMware.Vim.ClusterGroupSpec
$spec.groupSpec[0].operation = "edit"
$spec.groupSpec[0].info = $DrsGroup
$spec.groupSpec[0].info.vm += $VM.ExtensionData.MoRef$Cluster.ExtensionData.ReconfigureComputeResource_Task($spec, $true)
}
}
}
}
}
}
Function Remove-VMFromDrsGroup {
param([parameter(Mandatory=$true)] $Cluster,
[parameter(Mandatory=$true)] $DrsGroup,
[parameter(Mandatory=$true, ValueFromPipeline=$true)] $VM)
begin {
$Cluster = Get-Cluster -Name $Cluster
}
process {
if ($Cluster) {
if ($DrsGroup.GetType().Name -eq "string") {
$DrsGroupName = $DrsGroup
$DrsGroup = Get-DrsGroup -Cluster $Cluster -Name $DrsGroup
}
if (-not $DrsGroup) {
Write-Error "The DrsGroup $DrsGroupName was not found on cluster $($Cluster.name)."
}
else {
if ($DrsGroup.GetType().Name -ne "ClusterVmGroup") {
Write-Error "The DrsGroup $DrsGroupName on cluster $($Cluster.Name) doesn't have the required type ClusterVmGroup."
}
else {
$VM = $Cluster | Get-VM -Name $VM
If ($VM) {
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$spec.groupSpec = New-Object VMware.Vim.ClusterGroupSpec[] (1)
$spec.groupSpec[0] = New-Object VMware.Vim.ClusterGroupSpec
$spec.groupSpec[0].operation = "edit"
$spec.groupSpec[0].info = $DrsGroup
$spec.groupSpec[0].info.vm = ($cluster.ExtensionData.ConfigurationEx.Group | where {
$_.Name -eq $drsgroup.Name}).VM | where {$_ -ne $VM.ExtensionData.MoRef}
$Cluster.ExtensionData.ReconfigureComputeResource_Task($spec, $true)
}
}
}
}
}
}
Function Get-DrsGroup {
param([parameter(Mandatory=$true, ValueFromPipeline=$true)]$Cluster,
[string] $Name="*")
process {
$Cluster = Get-Cluster -Name $Cluster
if($Cluster) {
$Cluster.ExtensionData.ConfigurationEx.Group | `
Where-Object {$_.Name -like $Name}
}
}
}
The above functions were shamelessly stolen from Luc Dekens (https://communities.vmware.com/message/2336032, thanks Luc!) and are used to add and remove virtual machines from DRS groups, since there are no readily available built-in cmdlets available.
#Get all entities with the tag "Metro cluster"
$metroclusters = get-cluster (Get-TagAssignment -Category "Cluster Type" |? {$_.Tag.toString() -eq $metroclustertag }).Entity
$metroclusters |% {
$cluster = $_
$leftvms = Get-VM -Location $_ -Tag $metrolefttag
$rightvms = Get-VM -Location $_ -Tag $metrorighttagge
$leftdrsgroup = Get-DrsGroup -Cluster $_ -Name $leftdrsgroupname
$rightdrsgroup = Get-DrsGroup -Cluster $_ -Name $rightdrsgroupname
Here is where the magic happens. First, we get a list of all clusters that are tagged as a Metro Cluster. Then, we do a foreach-object over that and get all the virtual machines in that cluster that have the tag for the left or right datacenter respectively. We then get the DRS groups for that cluster for both the left and the right side of the cluster through the functions we defined earlier.
$leftvms |% {
if ($rightdrsgroup.Vm -contains $_.ExtensionData.MoRef) {
Remove-VMFromDrsGroup -Cluster $cluster -DrsGroup $rightdrsgroup -VM $_
}
if ($leftdrsgroup.Vm -notcontains $_.ExtensionData.MoRef) {
Add-VMToDrsGroup -Cluster $cluster -DrsGroup $leftdrsgroup -vm $_
}
}
$rightvms |% {
if ($leftdrsgroup.Vm -contains $_.ExtensionData.MoRef) {
Remove-VMFromDrsGroup -Cluster $cluster -DrsGroup $leftdrsgroup -VM $_
}
if ($rightdrsgroup.Vm -notcontains $_.ExtensionData.MoRef) {
Add-VMToDrsGroup -Cluster $cluster -DrsGroup $rightdrsgroup -vm $_
}
}
}
Then, finally, we're actually going to build the groups. First, we do a foreach-object over the vm's that are tagged as living in the left datacenter, and we check if the DRS VM group for the opposite datacenter contains the MoRef of the virtual machines. If it does, we'll remove it from that group since it shouldn't be there in the first place. Then, we check if the DRS VM group for the same datacenter does not contain the MoRef of the virtual machines. If it doesn't, we'll add them to it. Then, we repeat the whole procedure for the virtual machines in the other DRS VM group and reverse the datacenters to check.
After waiting for a short while, DRS should kick in automatically and start moving your virtual machines accross to where they belong.
In the future, I'd want to integrate this system into an orchestration tool. When you can automatically tag virtual machines during creation in your provisioning tool (such as vCenter Automation Center), it allows you to automatically run a workflow to add the virtual machine to a DRS group and run checks regularly instead of scheduling a powershell script on a windows machine. Unfortunately vSphere tags are not available through the API yet, so for the moment we're stuck with using powerCLI, but hopefully VMware will allow API access to tags in the near future.
For reference, the full script:
function Load-PowerCLI {
Try {
if ( (Get-PSSnapin -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue) -eq $null ) {
Add-PSSnapin VMware.VimAutomation.Core
}
. 'C:Program Files (x86)VMwareInfrastructurevSphere PowerCLIScriptsInitialize-PowerCLIEnvironment.ps1'
}
Catch {
Throw "This script requires VMware PowerCLI to be installed. Please check configuration and try again. "
}
}
Function Add-VMToDrsGroup {
param([parameter(Mandatory=$true)] $Cluster,
[parameter(Mandatory=$true)] $DrsGroup,
[parameter(Mandatory=$true, ValueFromPipeline=$true)] $VM)
begin {
$Cluster = Get-Cluster -Name $Cluster
}
process {
if ($Cluster) {
if ($DrsGroup.GetType().Name -eq "string") {
$DrsGroupName = $DrsGroup
$DrsGroup = Get-DrsGroup -Cluster $Cluster -Name $DrsGroup
}
if (-not $DrsGroup) {
Write-Error "The DrsGroup $DrsGroupName was not found on cluster $($Cluster.name)."
}
else {
if ($DrsGroup.GetType().Name -ne "ClusterVmGroup") {
Write-Error "The DrsGroup $DrsGroupName on cluster $($Cluster.Name) doesn't have the required type ClusterVmGroup."
}
else {
$VM = $Cluster | Get-VM -Name $VM
If ($VM) {
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$spec.groupSpec = New-Object VMware.Vim.ClusterGroupSpec[] (1)
$spec.groupSpec[0] = New-Object VMware.Vim.ClusterGroupSpec
$spec.groupSpec[0].operation = "edit"
$spec.groupSpec[0].info = $DrsGroup
$spec.groupSpec[0].info.vm += $VM.ExtensionData.MoRef
$Cluster.ExtensionData.ReconfigureComputeResource_Task($spec, $true)
}
}
}
}
}
}
Function Remove-VMFromDrsGroup {
param([parameter(Mandatory=$true)] $Cluster,
[parameter(Mandatory=$true)] $DrsGroup,
[parameter(Mandatory=$true, ValueFromPipeline=$true)] $VM)
begin {
$Cluster = Get-Cluster -Name $Cluster
}
process {
if ($Cluster) {
if ($DrsGroup.GetType().Name -eq "string") {
$DrsGroupName = $DrsGroup
$DrsGroup = Get-DrsGroup -Cluster $Cluster -Name $DrsGroup
}
if (-not $DrsGroup) {
Write-Error "The DrsGroup $DrsGroupName was not found on cluster $($Cluster.name)."
}
else {
if ($DrsGroup.GetType().Name -ne "ClusterVmGroup") {
Write-Error "The DrsGroup $DrsGroupName on cluster $($Cluster.Name) doesn't have the required type ClusterVmGroup."
}
else {
$VM = $Cluster | Get-VM -Name $VM
If ($VM) {
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$spec.groupSpec = New-Object VMware.Vim.ClusterGroupSpec[] (1)
$spec.groupSpec[0] = New-Object VMware.Vim.ClusterGroupSpec
$spec.groupSpec[0].operation = "edit"
$spec.groupSpec[0].info = $DrsGroup
$spec.groupSpec[0].info.vm = ($cluster.ExtensionData.ConfigurationEx.Group | where {
$_.Name -eq $drsgroup.Name}).VM | where {$_ -ne $VM.ExtensionData.MoRef}
$Cluster.ExtensionData.ReconfigureComputeResource_Task($spec, $true)
}
}
}
}
}
}
Function Get-DrsGroup {
param([parameter(Mandatory=$true, ValueFromPipeline=$true)]$Cluster,
[string] $Name="*")
process {
$Cluster = Get-Cluster -Name $Cluster
if($Cluster) {
$Cluster.ExtensionData.ConfigurationEx.Group | `
Where-Object {$_.Name -like $Name}
}
}
}
# Define variables
$vcenter = "spr-prod-vcent-01.springer-sbm.com"
$metrolefttag = "dogo"
$metrorighttag = "rone"
$metroclustertag = "Cluster type/Metro"
$leftdrsgroupname = "spr-prod-dogo-vm"
$rightdrsgroupname = "spr-prod-rone-vm"
try {
Connect-VIServer $vcenter
}
catch {
}
#Get all entities with the
$metroclusters = get-cluster (Get-TagAssignment -Category "Cluster Type" |? {$_.Tag.toString() -eq $metroclustertag }).Entity
$metroclusters |% {
$cluster = $_
$allvms = Get-VM -Location $_
$leftvms = Get-VM -Location $_ -Tag $metrolefttag
$rightvms = Get-VM -Location $_ -Tag $metrorighttag
$taggedvms = Get-VM -Location $_ -Tag $metrolefttag,$metrorighttag
if ($taggedvms -ne $null) {
Compare-Object $allvms $taggedvms |% {
Write-Host "ERROR" $_.InputObject.Name
#Send-MailMessage -To -From "[email protected]" -Subject "Alert: VM $_.Name is not tagged" -SmtpServer
}
}
$leftdrsgroup = Get-DrsGroup -Cluster $_ -Name $leftdrsgroupname
$rightdrsgroup = Get-DrsGroup -Cluster $_ -Name $rightdrsgroupname
$leftvms |% {
if ($rightdrsgroup.Vm -contains $_.ExtensionData.MoRef) {
Remove-VMFromDrsGroup -Cluster $cluster -DrsGroup $rightdrsgroup -VM $_
}
if ($leftdrsgroup.Vm -notcontains $_.ExtensionData.MoRef) {
Add-VMToDrsGroup -Cluster $cluster -DrsGroup $leftdrsgroup -vm $_
}
}
$rightvms |% {
if ($leftdrsgroup.Vm -contains $_.ExtensionData.MoRef) {
Remove-VMFromDrsGroup -Cluster $cluster -DrsGroup $leftdrsgroup -VM $_
}
if ($rightdrsgroup.Vm -notcontains $_.ExtensionData.MoRef) {
Add-VMToDrsGroup -Cluster $cluster -DrsGroup $rightdrsgroup -vm $_
}
}
}
