Detailed VM Storage Information in vCloud Director

I recently had a request from one of our customers who wanted an easy / scriptable method to determine the storage allocations on their hosted VMs in our vCloud platform, preferably from PowerShell. That should be easy I thought and set about my usual Google-based research. I initially found this post from Alan Renouf which I forwarded back to the client.

Unfortunately, while this achieved part of the answer, this particular customer had a number of VMs which had hard disks attached using multiple/different storage profiles and they wanted to get the details of these too. So I set about writing some code to see if I could get full storage information about the VM and all of its disks. I ended up having to access the vCloud REST API directly for this information but it wasn’t too bad.

First, I created a ‘worst-case’ test VM where the 3 attached hard disks which were created one each on our ‘Gold’, ‘Silver’ and ‘Bronze’ storage policies:

test02-hardware-properties

(Just to make sure everything would work I also created the 3 disks on 3 different storage Bus Types). I also set the VM storage policy to something different:

test02-general-properties

My first step was a function to access the vCloud REST API, I found this post from Matt Vogt’s blog which had some code for this which I shamelessly borrowed (hey, why reinvent the wheel unless you need to):

function Get-vCloudREST($href) {
  $request = [System.Net.HttpWebRequest]::Create($href)
  $request.Accept = "application/*+xml;version=20.0" # For vCloud Director 8.xx or later API
  $request.Headers.add("x-vcloud-authorization",$global:session.sessionID)
  $response = $request.GetResponse()
  $streamReader = new-object System.IO.StreamReader($response.getResponseStream())
  $xmldata = $streamReader.ReadToEnd()
  $streamReader.Close()
  $response.Close()
  return $xmldata
}

The return from the Get-CIVM cmdlet includes a reference to the VM object within the vCloud API:

PowerCLI C:\> $test02 = Get-CIVM -Name 'test02'
PowerCLI C:\> $test02 | fl
ExtensionData   : VMware.VimAutomation.Cloud.Views.Vm
Status          : PoweredOff
Deleted         : False
GuestOsFullName : Other Linux (64-bit)
CpuCount        : 1
MemoryMB        : 384
MemoryGB        : 0.375
VMVersion       : v11
Org             : MyOrg
OrgVdc          : MyOrgVDC
VApp            : test02
Description     : 
Href            : https://<cloud.com>/api/vApp/vm-ef4a4594-c631-421a-98f0-4d15670c98ad
Id              : urn:vcloud:vm:ef4a4594-c631-421a-98f0-4d15670c98ad
Name            : test02
Client          : /CIServer=<user>:<org>@<cloud.com>:443/
Uid             : /CIServer=<user>:<org>@<cloud.com>:443/CIVM=urn:vcloud:vm:ef4a4594-c631-421a-98f0-4d15670c98ad/

Using this we can obtain our disk information:

PowerCLI C:\> $queryHref = $test02.Href + "/virtualHardwareSection/disks"
PowerCLI C:\> [xml]$xml = Get-vCloudREST($queryHref)
PowerCLI C:\> $xml
xml                            RasdItemsList
---                            -------------
version="1.0" encoding="UTF-8" RasdItemsList

Filtering the returned RasdItemsList for a ResourceType of 17 (Hard Disk), we can get a list of attached hard disks:

PowerCLI C:\> $xml.RasdItemsList.Item | Where { $_.ResourceType -eq 17 }

AddressOnParent      : 0
Description          : Hard disk
ElementName          : Hard disk 1
HostResource         : HostResource
InstanceID           : 2000
Parent               : 2
ResourceType         : 17
VirtualQuantity      : 8589934592
VirtualQuantityUnits : byte

AddressOnParent      : 0
Description          : Hard disk
ElementName          : Hard disk 2
HostResource         : HostResource
InstanceID           : 2016
Parent               : 3
ResourceType         : 17
VirtualQuantity      : 8589934592
VirtualQuantityUnits : byte

AddressOnParent      : 0
Description          : Hard disk
ElementName          : Hard disk 3
HostResource         : HostResource
InstanceID           : 16060
Parent               : 4
ResourceType         : 17
VirtualQuantity      : 8589934592
VirtualQuantityUnits : byte

So this gets us to a point where we have all of the hard disk information, but how do we find the storage policy for each disk? It turns out that each disk has an attribute ‘HostResource’ which provides the URI to the storage policy from which the disk has been allocated:

PowerCLI C:\> $disks = $xml.RasdItemsList.Item | Where { $_.ResourceType -eq 17 }

PowerCLI C:\> $disks.Count
3

PowerCLI C:\> $disks[0].HostResource

vcloud                          : http://www.vmware.com/vcloud/v1.5
storageProfileHref              : https://<my cloud URI>/api/vdcStorageProfile/f89df73a-2fa5-40e4-9332-fdd6e29d36ac
busType                         : 6
busSubType                      : lsilogic
capacity                        : 8192
iops                            : 0
storageProfileOverrideVmDefault : true

PowerCLI C:\> $disks[1].HostResource

vcloud                          : http://www.vmware.com/vcloud/v1.5
storageProfileHref              : https:// <my cloud URI>/api/vdcStorageProfile/d14c6c2e-2cfb-4ffe-9f31-599c3de42150
busType                         : 6
busSubType                      : lsilogicsas
capacity                        : 8192
iops                            : 0
storageProfileOverrideVmDefault : true

PowerCLI C:\> $disks[2].HostResource

vcloud                          : http://www.vmware.com/vcloud/v1.5
storageProfileHref              : https:// <my cloud URI>/api/vdcStorageProfile/dc382284-bc65-4e61-b85f-ea7facba63e4
busType                         : 20
busSubType                      : vmware.sata.ahci
capacity                        : 8192
iops                            : 0
storageProfileOverrideVmDefault : true

So how can we convert the storageProfileHref values into meaningful (human readable) storage profile names? We can use another API call to establish the name of each vdcStorageProfile:

PowerCLI C:\> [xml]$sp = Get-vCloudREST($disks[0].HostResource.storageProfileHref)
PowerCLI C:\> $sp.VdcStorageProfile.name
Gold Storage Profile

PowerCLI C:\> [xml]$sp = Get-vCloudREST($disks[1].HostResource.storageProfileHref)
PowerCLI C:\> $sp.VdcStorageProfile.name
Silver Storage Profile

PowerCLI C:\> [xml]$sp = Get-vCloudREST($disks[2].HostResource.storageProfileHref)
PowerCLI C:\> $sp.VdcStorageProfile.name
Bronze Storage Profile

Querying the API for every vdcStorageProfile for every disk is going to generate a lot of calls for any significant number of VMs, so in the code below I’ve added a hash stored in a global variable which caches these results so that any storageProfileHref which has been seen before doesn’t need to generate an additional API call.

Putting it all together

So we now have a way of determining all of the information we need, using PowerShell custom objects allows us to write a function which returns all of our VM and storage details in a easily consumable form for further processing.

The script included at the bottom of this article produces the following output for my test environment containing 2 VMs of which the ‘pxetest01’ VM has no disks attached:

PowerCLI C:\> $vmobjs

VMName    : pxetest01
Id        : urn:vcloud:vm:45a86f24-1109-4ead-881d-f865c1f0692f
Status    : PoweredOff
vRAM      : 1
vCPU      : 1
HWVersion : v11
vAppName  : PXE Demo
GuestOS   : CentOS 4/5/6/7 (64-bit)
Disks     :

VMName    : test02
Id        : urn:vcloud:vm:ef4a4594-c631-421a-98f0-4d15670c98ad
Status    : PoweredOff
vRAM      : 0.375
vCPU      : 1
HWVersion : v11
vAppName  : test02
GuestOS   : Other Linux (64-bit)
Disks     : {@{Name=Hard disk 1; StorageProfile=Gold Storage Profile; Quantity=8589934592; QuantityUnits=byte}, @{Name=Hard disk 2; StorageProfile=Silver Storage Profile; Quantity=8589934592; QuantityUnits=byte}, @{Name=Hard disk 3; StorageProfile=BronzeStorage Profile; Quantity=8589934592; QuantityUnits=byte}}

It can also return just the disk information as another custom object:

PowerCLI C:\> $test02vm = $vmobjs | Where {$_.VMName -eq 'test02'}
PowerCLI C:\> $test02vm.Disks

Name StorageProfile Quantity QuantityUnits
---- -------------- -------- -------------
Hard disk 1 Gold Storage Profile 8589934592 byte
Hard disk 2 Silver Storage Profile 8589934592 byte
Hard disk 3 Bronze Storage Profile 8589934592 byte

And we can check the number of disks attached to any VM:

PowerCLI C:\> ($test02vm.Disks | Measure).Count
3

Finally because the output is a PowerShell object, we can easily turn this custom object into JSON for use in further processing:

PowerCLI C:\> $test02vm | ConvertTo-Json

{
	"VMName": "test02",
	"Id": "urn:vcloud:vm:ef4a4594-c631-421a-98f0-4d15670c98ad",
	"Status": "PoweredOff",
	"vRAM": 0.375,
	"vCPU": 1,
	"HWVersion": "v11",
	"vAppName": "test02",
	"GuestOS": "Other Linux (64-bit)",
	"Disks": {
		"value": [
			{
				"Name": "Hard disk 1",
				"StorageProfile": "Gold Storage Profile",
				"Quantity": 8589934592,
				"QuantityUnits": "byte"
			},
			{
				"Name": "Hard disk 2",
				"StorageProfile": "Silver Storage Profile",
				"Quantity": 8589934592,
				"QuantityUnits": "byte"
			},
			{
				"Name": "Hard disk 3",
				"StorageProfile": "Bronze Storage Profile",
				"Quantity": 8589934592,
				"QuantityUnits": "byte"
			}
		],
		"Count": 3
	}
}

Hopefully you’ve found this post useful, let me know in the comments if you have any issues or would like to see more examples like this.

Jon.

Full script to find storage policy information for vCloud VMs using the vCloud REST API:

$global:spolrefs = @{}
# Change to be appropriate for your scenario:
$cloudURL = "<cloud FQDN or IP Address>"
$cloudOrg = "<cloud Organization name>"

# Check if we are already logged-in (interactive session) and if not, prompt for login:
if (!$session.IsConnected) {
 $creds = Get-Credential -Message "Authenticate to $cloudOrg Cloud Service"
 $session = Connect-CIServer -Server $cloudURL -Org $cloudOrg -Credential $creds
}

# PS Function to query the vCloud Director REST API
# ('Borrowed' from Matt Vogt's blog at: http://blog.mattvogt.net/)
function Get-vCloudREST($href)
{
 $request = [System.Net.HttpWebRequest]::Create($href)
 $request.Accept = "application/*+xml;version=20.0" # For vCloud Director 8.xx or later API
 $request.Headers.add("x-vcloud-authorization",$global:session.sessionID)
 $response = $request.GetResponse()
 $streamReader = new-object System.IO.StreamReader($response.getResponseStream())
 $xmldata = $streamReader.ReadToEnd()
 $streamReader.Close()
 $response.Close()
 return $xmldata
}

# Function to get Storage Profile name from its Href (if not already known/cached in a global hash)
# if already known we don't need to call the API again and can just return the value from the hash
function Get-SPName($storage_href)
{
 if ($global:spolrefs.ContainsKey($storage_href)) {
 return $global:spolrefs.Get_Item($storage_href)
 } else {
 [xml]$sp = Get-vCloudREST($storage_href)
 $global:spolrefs.Add($storage_href, $sp.VdcStorageProfile.name)
 return $sp.VdcStorageProfile.name
 }
}

# Function to retrieve the disk information for a VM:
function Get-VMDisks($VM)
{
 $VMDisks = @() # Start with an empty array
 $queryHref = $VM.Href + "/virtualHardwareSection/disks" # API path for VM disk information
 [xml]$xml = Get-vCloudRESt($queryHref)
 $disks = $xml.RasdItemsList.Item | Where-Object {$_.ResourceType -eq 17} # Resource Type 17 = Hard disk drive
 foreach ($disk in $disks)
 {
 $sp = Get-SPName($disk.HostResource.storageProfileHref)
 $diskobj = New-Object -TypeName PSObject
 $diskobj | Add-Member -Type NoteProperty -Name "Name" -Value ([String]$disk.ElementName)
 $diskobj | Add-Member -Type NoteProperty -Name "StorageProfile" -Value ([String]$sp)
 $diskobj | Add-Member -Type NoteProperty -Name "Quantity" -Value ([Int64]$disk.VirtualQuantity)
 $diskobj | Add-Member -Type NoteProperty -Name "QuantityUnits" -Value ([String]$disk.VirtualQuantityUnits)
 $VMDisks += $diskobj
 }
 return $VMDisks
}

function Get-VMInfo($VM)
{
 $vmobj = New-Object -TypeName PSObject
 $vmobj | Add-Member -Type NoteProperty -Name "VMName" -Value ([String]$VM.Name)
 $vmobj | Add-Member -Type NoteProperty -Name "Id" -Value ([String]$VM.ExtensionData.Id)
 $vmobj | Add-Member -Type NoteProperty -Name "Status" -Value ([String]$VM.Status)
 $vmobj | Add-Member -Type NoteProperty -Name "vRAM" -Value ([Decimal]$VM.MemoryGB)
 $vmobj | Add-Member -Type NoteProperty -Name "vCPU" -Value ([Int]$VM.CpuCount)
 $vmobj | Add-Member -Type NoteProperty -Name "HWVersion" -Value ([String]$VM.VMVersion)
 $vmobj | Add-Member -Type NoteProperty -Name "vAppName" -Value ([String]$VM.VApp.Name)
 $vmobj | Add-Member -Type NoteProperty -Name "GuestOS" -Value ([String]$VM.GuestOsFullName)
 [PSCustomObject]$disks = Get-VMDisks($VM)
 $vmobj | Add-Member -Type NoteProperty -Name "Disks" -Value $disks
 return $vmobj
}

# Get all VMs in our Cloud Organization:
$vms = Get-CIVM

# Initialise an array for returned objects:
$vmobjs = @()

# For each VM, build our object:
foreach ($vm in $vms) {
 $vmobjs += Get-VMinfo($vm)
}

# Output our object to console:
$vmobjs

 

Uploading / running utilities directly on ESXi hosts

As part of planning our upgrade from VMware NSX-V from v6.2.2 to v6.2.4 we became aware of the VMware issue KB2146171 (link) which can cause VMs to lose network connectivity when vMotioned to other hosts following the upgrade. Obviously wishing to avoid this for our own (and customer) VMs, we raised a support case to obtain the VMware script to determine how many of our VMs (if any) were going to be affected. Unfortunately the VMware script we were supplied was configured to run *after* the upgrade had already been completed. Fortunately the VMware utility supplied (vsipioctl – a binary to be run directly on ESXi hosts) could tell us which VMs were affected prior to upgrading.

Since we have a reasonably large number of hosts and hosted VMs I set about writing some PowerShell to perform the following actions:

  • Connect to vCenter and enumerate all ESXi hosts.
  • Enable SSH access to each host in turn.
  • Upload the VMware vsipioctl utility to the host /tmp/ folder and make it executable.
  • Run vsipioctl and parse the return information.
  • Build a table / CSV of all VM network interfaces with the results of the vsipioctl utility.
  • Disable SSH on the hosts once done and move on to the next host.

At first I tried using PuTTY plink.exe and pscp.exe from PowerShell to perform the SSH and SCP file copy to the hosts, but had serious problems passing the right password & command line options due to the way PowerShell escapes quoted strings. In the end I found it easier to use the PoshSSH PowerShell library (https://github.com/darkoperator/Posh-SSH) for these functions rather than shelling out to PuTTY executables.

Note that we usually leave SSH access disabled on our ESXi hosts, so the script shown enables this and then re-disables SSH after running – adjust if necessary when using in your own environments.

If you need to run this check for your own environment you will still need to open a VMware support call to obtain the vsipioctl binary as far as I am aware as I don’t believe this is available any other way.

The script is shown below – hopefully this will be useful for some of you, just make sure you test properly before running against a production environment. Luckily in our case the script proved that none of our VMs are impacted by this issue and we can safely proceed with our NSX-V upgrade.

Jon.

# Script to run VMware vsipioctl utility against all VMs / all hosts
# NOTE: Requires darkoperator PoshSSH for PowerShell
#
# To install PoshSSH:
# iex (New-Object Net.WebClient).DownloadString("https://gist.github.com/darkoperator/6152630/raw/c67de4f7cd780ba367cccbc2593f38d18ce6df89/instposhsshdev")

# Specify our vCenter instance FQDN or IP:
$vCenter = "<vcenter FQDN or IP address>"

# Prompt user for vCenter credentials if not known:
if ($credVCenter -eq $null) {
    $credVCenter = Get-Credential -Message "Authenticate to vCenter Server"
}

# Prompt user for ESXi host credentials if not known:
if ($credESXi -eq $null) {
    $credESXi = Get-Credential -Message "Provide ESXi root password" -UserName root
}

# Directory containing this script:
$scriptDir = Split-Path -Path $($global:MyInvocation.MyCommand.Path)

# Location to find the VMware vsipioctl binary:
$source_file = $scriptDir+"\vsipioctl"

$target_path = "/tmp/"

# Output CSV File in current (script) directory:
$csvOut = $scriptDir+"\fwexport.csv"

# Location to copy vsipioctl to on ESXi hosts:
$target_file = "/tmp/vsipioctl"

Connect-VIServer -Server $vCenter -Credential $credVCenter

# Define an output table object with required columns:
$table = New-Object System.Data.DataTable
$col1 = New-Object System.Data.DataColumn Host,([string])
$col2 = New-Object System.Data.DataColumn VMName,([string])
$col3 = New-Object System.Data.DataColumn NIC,([string])
$col4 = New-Object System.Data.DataColumn ExportVersion,([int])
$table.Columns.Add($col1)
$table.Columns.Add($col2)
$table.Columns.Add($col3)
$table.Columns.Add($col4)

Get-VMHost | Foreach {

    Write-Host "Starting processing for Host: $_" -ForegroundColor Green

    Write-Host "- Enabling SSH on Host" -ForegroundColor Green
    $start = Start-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} )

    $ssh = New-SSHSession -ComputerName $_ -Credential $credESXi -Port 22 -AcceptKey:$true

    if ($ssh.Connected -eq $true) {

        $return = Invoke-SSHCommand -SSHSession $ssh -Command "uname -a"
        Write-Host "Host ID: " + $return.Output -ForegroundColor Yellow

        # Copy the VMware vsipioctl utility to the host:
        Write-Host "- Copying binary file" -ForegroundColor Green
        $scp = Set-SCPFile -ComputerName $_ -Credential $credESXi -Port 22 -LocalFile $source_file -RemotePath $target_path

        # And flag it as executable:
        Write-Host "- Changing execute mode" -ForegroundColor Green
        Invoke-SSHCommand -SSHSession $ssh -Command "chmod 'u+x' $target_file" | Out-Null

        Write-Host "- Retrieving directory list:" -ForegroundColor Green
        $dirlist = Invoke-SSHCommand -SSHSession $ssh -Command "ls -l $target_file"
        Write-Host $dirlist.Output
        
        # Retrieve summarize-dvfilter output:
        Write-Host "- Getting full list of dvfilters" -ForegroundColor Green 
        $dvflist = Invoke-SSHCommand -SSHSession $ssh -Command "summarize-dvfilter"

        # Parse each line in the summarize-dvfilter output:
        ForEach ($item in $dvflist.Output) {

            # If we see a new VM identifier, update our current VMName setting:
            if ($item -match '(?<=vmm0:)(.*)(?= vcUuid:)') {
                $VMName = $matches[0]
            }
            # If we see a line that looks like a fw export, grab that and run vsipioctl on it:
            if ($item -match '(?<=name: )(.*sfw.2)') {
                $VMNic = $matches[0]
                
                $fwexport = Invoke-SSHCommand -SSHSession $ssh -Command "/tmp/vsipioctl getexportversion -f $VMnic"
            
                # Parse the returned 'Current Export Version' from vsipioctl:
                $exmatch = $fwexport.Output[0] -match '(?<=version: )(.*)'
                $exportVersion = $matches[0]

                # Add a table row that includes the VM name, NIC ID and exportversion:
                $row = $table.NewRow()
                $row.Host = $_
                $row.VMName = $VMName
                $row.NIC = $VMNic
                $row.ExportVersion = $exportVersion
                $table.Rows.Add($row)
            } # Line that looks like a fw export

        } # Foreach line in summarize-dvfilter output

        Remove-SSHSession -SSHSession $ssh | Out-Null

    } else {
        Write-Host "Error connecting to host $_, skipping." -ForegroundColor Red
    }

    Write-Host "- Disabling SSH on Host" -ForegroundColor Green
    $stop = Stop-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} ) -Confirm:$false
 
} # Foreach VMHost

$table | Format-Table -AutoSize
$table | Export-Csv $csvOut

 

 

 

 

Live import VMs to vCloud Director

Tom Fojta wrote a great blog post about the new capability in vCloud Director 8.10 to import running VMs into vCloud Director. This is a huge asset in migration scenarios where customers can’t afford outages when being migrated into the vCD environment. Unfortunately the API syntax to actually initiate the import is a little convoluted and not the easiest process to manage.

I set about writing a PowerShell script to significantly simplify the process of initiating a live-import operation. The script itself is available from github at the following link: https://github.com/jondwaite/vcdliveimport.

The liveimport.ps1 script contained in this repository does the following:

  • Prompts for a credential to be used to connect to both vCloud Director (System context) and vCenter – if you have different usernames/passwords for each you’ll need to adjust this.
  • Enumerates the available vCenter instances registered as Provider Virtual Datacenters (PVDCs) in vCloud Director and allows one to be selected as the source vCenter for the migration.
  • Lists the available VMs in the selected vCenter instance, filters this list based on selectable criteria (e.g. don’t offer to import ‘Guest Introspection’ VMs) and allows the source VM to be selected.
  • Lists available destination Virtual Datacenters (VDCs) in the vCloud Director environment and allows the destination VDC to be selected.
  • Displays the appropriate POST request information to be submitted to vCloud Director to initiate the live-import of this VM.
  • Optionally – Submits the REST API request directly to the vCloud Director environment to actually initiate the import process.

An example transcript of this process is show below. Hopefully this helps someone else out and helps to make it easier for you to live-import running VMs into vCloud Director.

Jon.

Example Session Transcript:

PowerCLI C:\> liveimport.ps1
Connected to vCloud Director OK
vCenter(s) Found:
-----------------
vc01 (https://vcd01.dev.local/api/admin/extension/vimServer/158f73ec-a999-4332-8250-f4dd5e6c4971)

Selecting vc01 as only pVDC vCenter found.
vCenter vc01 selected.
Connected to vCenter Server vc01 OK

Evaluating vCenter VMs as migration candidates.............
Available candidate VMs:
------------------------
testvm01

Enter VM name to live-migrate to vCloud (or 'quit' to exit): testvm01
Selected VM testvm01 for live import to vCloud.

Available VDCs:
---------------
Lab VDC
Enter Destination VDC Name (or quit to exit): Lab VDC
VDC 'Lab VDC' selected.

URI for POST operation:
https://vcd01.dev.local/api/admin/extension/vimServer/158f73ec-a999-4332-8250-f4dd5e6c4971/importVmAsVApp

XML Document Body:
<?xml version="1.0" encoding="UTF-8"?>
<ImportVmAsVAppParams xmlns="http://www.vmware.com/vcloud/extension/v1.5" name="testvm01" sourceMove="true">
 <VmMoRef>vm-111</VmMoRef>
 <Vdc href="https://vcd01.dev.local/api/admin/vdc/b21c92fa-1a6f-42a9-8c1e-ef0947c7be76" />
</ImportVmAsVAppParams>

Content-Type: application/vnd.vmware.admin.importVmAsVAppParams+xml

Would you like to submit this API request to live import this VM? (y or n) (or quit to exit): y
Making request to live import VM testvm01...
Response was:
<?xml version="1.0" encoding="UTF-8"?>
<VApp xmlns="http://www.vmware.com/vcloud/v1.5" ovfDescriptorUploaded="true" deployed="false" status="0" name="testvm01" id="urn:vcloud:vapp:ff1a9021-2c21-4378-99c7-b77bdaf3e9e6" href="https://vcd01.ndev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6" type="application/vnd.vmware.vcloud.vApp+xml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.vmware.com/vcloud/v1.5 http://vcd01.dev.local/api/v1.5/schema/master.xsd">
 <Link rel="down" href="https://vcd01.dev.local/api/network/251f72a8-19b5-4e7d-94ba-0523ee919cd9" name="VM Network" type="application/vnd.vmware.vcloud.vAppNetwork+xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/controlAccess/" type="application/vnd.vmware.vcloud.controlAccess+xml" />
 <Link rel="up" href="https://vcd01.dev.local/api/vdc/b21c92fa-1a6f-42a9-8c1e-ef0947c7be76" type="application/vnd.vmware.vcloud.vdc+xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/owner" type="application/vnd.vmware.vcloud.owner+xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/metadata" type="application/vnd.vmware.vcloud.metadata+xml" />
 <Link rel="ovf" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/ovf" type="text/xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/productSections/" type="application/vnd.vmware.vcloud.productSections+xml" />
 <Tasks>
 <Task cancelRequested="false" expiryTime="2016-10-31T11:27:25.312+13:00" operation="Importing Virtual Application testvm01(ff1a9021-2c21-4378-99c7-b77bdaf3e9e6)" 
operationName="importSingletonVapp" serviceNamespace="com.vmware.vcloud" startTime="2016-08-02T11:27:25.312+12:00" status="queued" name="task" id="urn:vcloud:task:37576f8c-0d39-444c-8842-81436ddb421e" href="https://vcd01.dev.local/api/task/37576f8c-0d39-444c-8842-81436ddb421e" type="application/vnd.vmware.vcloud.task+xml">
 <Owner href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6" name="testvm01" type="application/vnd.vmware.vcloud.vApp+xml" />
 <User href="https://vcd01.dev.local/api/admin/user/fa7fb40e-648c-4787-ad0f-711dcc315261" name="system" type="application/vnd.vmware.admin.user+xml" />
<Organization href="https://vcd01.dev.local/api/org/3f46b162-f794-4d9e-8fa1-6ca1ee7b6377" name="Lab" type="application/vnd.vmware.vcloud.org+xml" />
 <Progress>1</Progress>
 <Details />
 </Task>
 </Tasks>
 <DateCreated>2016-08-02T11:27:25.079+12:00</DateCreated>
 <Owner type="application/vnd.vmware.vcloud.owner+xml">
 <User href="https://vcd01.dev.local/api/admin/user/1545cb33-9151-43e2-a156-9d20e3b966c0" name="system" type="application/vnd.vmware.admin.user+xml" />
 </Owner>
 <InMaintenanceMode>false</InMaintenanceMode>
</VApp>

 

Create an empty vApp in vCloud Director

Sometimes you just need to create a new vApp with no contents at all – maybe for testing, or maybe you want to populate it with VMs built ‘from scratch’ rather than cloned from templates. This is easy to do in the vCloud Director web UI – you just skip the addition of any VM templates or new VMs and can easily create empty vApps, but how about programatically?

The VMware documentation is remarkably slim in this regard – all the documented methods I could find for vApp creation require either cloning from existing vApp templates, from existing VMs or from uploaded OVF files.

So how do we create a brand-new empty vApp? Turns out it’s pretty simple – once you discover the ‘composeVApp’ method on an Organization VDC supports creation of empty vApps.

If using the REST API we can simply create an XML body document of type ‘composeVAppParams’ and submit it against the OrgVDC’s /action/composeVapp link.

An example XML document body could be:

<?xml version=”1.0″ encoding=”UTF-8″?>
<ComposeVAppParams
name=”MyEmptyVapp”
xmlns=”http://www.vmware.com/vcloud/v1.5″
xmlns:ovf=”http://schemas.dmtf.org/ovf/envelope/1″>
<Description>My vApp Description</Description>
<AllEULAsAccepted>true</AllEULAsAccepted>
</ComposeVAppParams>

We then ‘POST’ this document body to the link: ‘https://<Cloud Server DNS name or IP address>>/api/vdc/<ID of our VDC>/action/composeVApp’ not forgetting to add a header of ‘Content-Type: application/vnd.vmware.vcloud.composeVAppParams+xml’ to the POST request.

If we want to accomplish the same thing using PowerShell / PowerCLI it’s easy too (once connected to our cloud using Connect-CIServer):

$vapp = New-Object VMware.VimAutomation.Cloud.Views.ComposeVAppParams
$vapp.Name = “MyEmptyVapp”
$vapp.Description = “My vApp Description”
$myorgvdc = Get-OrgVdc -Name ‘My OrgVDC Name’
$myorgvdc.ExtensionData.ComposeVApp($vapp)

No idea if this is ‘officially’ supported or not – so use at your own risk and be aware that the implementation could change in a future release and break this (although I’d be surprised as this is almost certainly the action that the vCD web UI is submitting ‘behind the scenes’ when you manually create an empty vApp).

Jon.

Working with vCloud Metadata in PowerCLI – Part 1

Way back in 2012 Alan Renouf created a PowerCLI module to deal with manipulation of metadata entries for vCloud Director objects – this can be incredibly useful to track related information for these objects. The vCD metadata functionality was enhanced in v5.1 (and then later in 5.5, 5.6 and 8.0) – in particular typed values were added with functionality to use date/time, boolean and numeric values (as well as free-form string text). Also added were security levels so that metadata could be made read-only or hidden (from a tenant perspective) but still accessible/visible to system owners. I’ve taken the PowerShell module that Alan published here and updated it to cope with these enhancements. I’ve also updated the returned fields/views to include the extra attributes (where present) such as security levels of metadata entries.

Note that I am definitely not a professional developer (and most of my PowerShell knowledge comes from Google) so there’s probably significant room for improvement in the code – comment back if you have suggestions for improvement and I’ll update this post.

Use of the module requires a valid connection to a vCloud instance (using Connect-CIServer). This won’t work for versions prior to v5.1 (most of my testing has been with PowerCLI 6 against a v8 vCD deployment) so please use at your own risk and make sure you thoroughly test your own scenarios. I’ll write a follow-up post detailing some example code and usage scenarios which people may find useful in the next few days.

I’d suggest copy/pasting the code (below) into a PowerShell module (.psm1) file and including the module in your scripts as needed.

Function New-CIMetaData { 
    <# 
    .SYNOPSIS 
        Creates a Metadata Key/Value pair. 
    .DESCRIPTION 
        Creates a custom Metadata Key/Value pair on a specified vCloud object 
    .PARAMETER  Key 
        The name of the Metadata to be applied.
    .PARAMETER  Value
        The value of the Metadata to be applied, the string 'Now' can be used
        for the current date/time for values using the 'DateTime' type.
    .PARAMETER  Visibility
        The visibility of the Metadata entry (General, Private, ReadOnly)
    .PARAMETER  Type
        The type of the Metadata entry (String, Number, DateTime, Boolean)
        (these correspond to the types of: MetadataStringValue,
        MetadataNumberValue, MetadataDateTimeValue or MetadataBooleanValue
        respectively)
    .PARAMETER  CIObject
        The object on which to apply the Metadata.
    .EXAMPLE
        New-CIMetadata -Key "Owner" -Value "Alan Renouf" -CIObject (Get-Org Org1)
        Creates a new metadata value "Alan Renouf" in a key "Owner" on the Org1 object.
    .EXAMPLE
        New-CIMetadata -Key "Company" -Value "ABC Corp" -Visibility READONLY -CIObject (Get-CIVM 'client')
        Creates a new metadata value "ABC Corp" in a key "Company" on the 'client' VM object with the READONLY attribute set preventing changes by non-system users.
    .EXAMPLE
        New-CIMetadata -Key "Backup" -Value $false -Visibility Private -Type Boolean -CIObject (Get-CIVapp 'testvapp')
        Creates a new hidden metadata value $false in a key "Backup" on the vApp object with the 'Private' attribute set preventing visibility to non-system users.
    .NOTES
        NAME: Get-CIMetaData
        AUTHOR: Jon Waite based on code by Alan Renouf
        LASTEDIT: 2016-02-23
        KEYWORDS: metadata set vcloud director
    #Requires -Version 2.0
    #> 
     [CmdletBinding( 
         SupportsShouldProcess=$true, 
        ConfirmImpact="High" 
    )] 
    param( 
        [parameter(Mandatory=$true,ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)] 
            [PSObject[]]$CIObject, 
        [parameter(Mandatory=$true)]
            [String]$Key,
        [parameter(Mandatory=$true)]
            $Value,
        [ValidateSet('General','Private','ReadOnly')]
            [String]$Visibility = 'General',
        [ValidateSet('String','Number','DateTime','Boolean')]
            [String]$Type = "String"
        ) 
    Process { 
        Foreach ($Object in $CIObject) { 
            $Metadata = New-Object VMware.VimAutomation.Cloud.Views.Metadata 
            $Metadata.MetadataEntry = New-Object VMware.VimAutomation.Cloud.Views.MetadataEntry 
            
            $Metadata.MetadataEntry[0].Key = $Key

            switch($Type) {
              'String'   { $Metadata.MetadataEntry[0].TypedValue = New-Object VMware.VimAutomation.Cloud.Views.MetadataStringValue }
              'Number'   { $Metadata.MetadataEntry[0].TypedValue = New-Object VMware.VimAutomation.Cloud.Views.MetadataNumberValue }
              'DateTime' { $Metadata.MetadataEntry[0].TypedValue = New-Object VMware.VimAutomation.Cloud.Views.MetadataDateTimeValue }
              'Boolean'  { $Metadata.MetadataEntry[0].TypedValue = New-Object VMware.VimAutomation.Cloud.Views.MetadataBooleanValue }
            }

            if ($Type -eq 'DateTime' -and $Value -eq 'Now') {
                $Metadata.MetadataEntry[0].TypedValue.Value = [string](Get-Date).ToUniversalTime().GetDateTimeFormats('s')
            } else {
                $Metadata.MetadataEntry[0].TypedValue.Value = $Value
            }
            
            switch($Visibility) {
              'General'  { } #Default, don't need to change
              'Private'  { 
                $Metadata.MetadataEntry[0].Domain = New-Object VMware.VimAutomation.Cloud.Views.MetadataDomainTag
                $Metadata.MetadataEntry[0].Domain.Value = 'SYSTEM'
                $Metadata.MetadataEntry[0].Domain.Visibility = 'PRIVATE'
                }
              'ReadOnly' {
                $Metadata.MetadataEntry[0].Domain = New-Object VMware.VimAutomation.Cloud.Views.MetadataDomainTag
                $Metadata.MetadataEntry[0].Domain.Value = 'SYSTEM'
                $Metadata.MetadataEntry[0].Domain.Visibility = 'READONLY'
                }      
            }

            $Object.ExtensionData.CreateMetadata($Metadata) 
            ($Object.ExtensionData.GetMetadata()).MetadataEntry | Where {$_.Key -eq $key } | Select @{N="CIObject";E={$Object.Name}},
            @{N="Type";E={$_.TypedValue.GetType().Name}},
            @{N="Visibility";E={ if ($_.Domain.Visibility) { $_.Domain.Visibility } else { "General" }}},
            Key -ExpandProperty TypedValue
        } 
    } 
} 
Function Get-CIMetaData {
    <#
    .SYNOPSIS
        Retrieves all Metadata Key/Value pairs.
    .DESCRIPTION
        Retrieves all custom Metadata Key/Value pairs on a specified vCloud object
    .PARAMETER  CIObject
        The object on which to retrieve the Metadata.
    .PARAMETER  Key
        The key to retrieve.
    .EXAMPLE
        Get-CIMetadata -CIObject (Get-Org Org1)
    #>
    param(
        [parameter(Mandatory=$true,ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
            [PSObject[]]$CIObject,
            $Key
        )
    Process {
        Foreach ($Object in $CIObject) {
            If ($Key) {
                ($Object.ExtensionData.GetMetadata()).MetadataEntry | Where {$_.Key -eq $key } | Select @{N="CIObject";E={$Object.Name}},
                    @{N="Type";E={$_.TypedValue.GetType().Name}},
                    @{N="Visibility";E={ if ($_.Domain.Visibility) { $_.Domain.Visibility } else { "General" }}},
                    Key -ExpandProperty TypedValue
            } Else {
                ($Object.ExtensionData.GetMetadata()).MetadataEntry | Select @{N="CIObject";E={$Object.Name}},
                    @{N="Type";E={$_.TypedValue.GetType().Name}},
                    @{N="Visibility";E={ if ($_.Domain.Visibility) { $_.Domain.Visibility } else { "General" }}},
                    Key -ExpandProperty TypedValue
            }
        }
    }
}
Function Remove-CIMetaData {
    <#
    .SYNOPSIS
        Removes a Metadata Key/Value pair.
    .DESCRIPTION
        Removes a custom Metadata Key/Value pair on a specified vCloud object
    .PARAMETER  Key
        The name of the Metadata to be removed.
    .PARAMETER  CIObject
        The object on which to remove the Metadata.
    .EXAMPLE
        Remove-CIMetaData -CIObject (Get-Org Org1) -Key "Owner"
    #>
     [CmdletBinding(
         SupportsShouldProcess=$true,
        ConfirmImpact="High"
    )]
    param(
        [parameter(Mandatory=$true,ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
            [PSObject[]]$CIObject,
            $Key
        )
    Process {
        $CIObject | Foreach {
            $metadataValue = ($_.ExtensionData.GetMetadata()).GetMetaDataValue($Key)
            If($metadataValue) { $metadataValue.Delete() }
        }
    }
}