Uploading / running utilities directly on ESXi hosts

As part of planning our upgrade from VMware NSX-V from v6.2.2 to v6.2.4 we became aware of the VMware issue KB2146171 (link) which can cause VMs to lose network connectivity when vMotioned to other hosts following the upgrade. Obviously wishing to avoid this for our own (and customer) VMs, we raised a support case to obtain the VMware script to determine how many of our VMs (if any) were going to be affected. Unfortunately the VMware script we were supplied was configured to run *after* the upgrade had already been completed. Fortunately the VMware utility supplied (vsipioctl – a binary to be run directly on ESXi hosts) could tell us which VMs were affected prior to upgrading.

Since we have a reasonably large number of hosts and hosted VMs I set about writing some PowerShell to perform the following actions:

  • Connect to vCenter and enumerate all ESXi hosts.
  • Enable SSH access to each host in turn.
  • Upload the VMware vsipioctl utility to the host /tmp/ folder and make it executable.
  • Run vsipioctl and parse the return information.
  • Build a table / CSV of all VM network interfaces with the results of the vsipioctl utility.
  • Disable SSH on the hosts once done and move on to the next host.

At first I tried using PuTTY plink.exe and pscp.exe from PowerShell to perform the SSH and SCP file copy to the hosts, but had serious problems passing the right password & command line options due to the way PowerShell escapes quoted strings. In the end I found it easier to use the PoshSSH PowerShell library (https://github.com/darkoperator/Posh-SSH) for these functions rather than shelling out to PuTTY executables.

Note that we usually leave SSH access disabled on our ESXi hosts, so the script shown enables this and then re-disables SSH after running – adjust if necessary when using in your own environments.

If you need to run this check for your own environment you will still need to open a VMware support call to obtain the vsipioctl binary as far as I am aware as I don’t believe this is available any other way.

The script is shown below – hopefully this will be useful for some of you, just make sure you test properly before running against a production environment. Luckily in our case the script proved that none of our VMs are impacted by this issue and we can safely proceed with our NSX-V upgrade.


# Script to run VMware vsipioctl utility against all VMs / all hosts
# NOTE: Requires darkoperator PoshSSH for PowerShell
# To install PoshSSH:
# iex (New-Object Net.WebClient).DownloadString("https://gist.github.com/darkoperator/6152630/raw/c67de4f7cd780ba367cccbc2593f38d18ce6df89/instposhsshdev")

# Specify our vCenter instance FQDN or IP:
$vCenter = "<vcenter FQDN or IP address>"

# Prompt user for vCenter credentials if not known:
if ($credVCenter -eq $null) {
    $credVCenter = Get-Credential -Message "Authenticate to vCenter Server"

# Prompt user for ESXi host credentials if not known:
if ($credESXi -eq $null) {
    $credESXi = Get-Credential -Message "Provide ESXi root password" -UserName root

# Directory containing this script:
$scriptDir = Split-Path -Path $($global:MyInvocation.MyCommand.Path)

# Location to find the VMware vsipioctl binary:
$source_file = $scriptDir+"\vsipioctl"

$target_path = "/tmp/"

# Output CSV File in current (script) directory:
$csvOut = $scriptDir+"\fwexport.csv"

# Location to copy vsipioctl to on ESXi hosts:
$target_file = "/tmp/vsipioctl"

Connect-VIServer -Server $vCenter -Credential $credVCenter

# Define an output table object with required columns:
$table = New-Object System.Data.DataTable
$col1 = New-Object System.Data.DataColumn Host,([string])
$col2 = New-Object System.Data.DataColumn VMName,([string])
$col3 = New-Object System.Data.DataColumn NIC,([string])
$col4 = New-Object System.Data.DataColumn ExportVersion,([int])

Get-VMHost | Foreach {

    Write-Host "Starting processing for Host: $_" -ForegroundColor Green

    Write-Host "- Enabling SSH on Host" -ForegroundColor Green
    $start = Start-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} )

    $ssh = New-SSHSession -ComputerName $_ -Credential $credESXi -Port 22 -AcceptKey:$true

    if ($ssh.Connected -eq $true) {

        $return = Invoke-SSHCommand -SSHSession $ssh -Command "uname -a"
        Write-Host "Host ID: " + $return.Output -ForegroundColor Yellow

        # Copy the VMware vsipioctl utility to the host:
        Write-Host "- Copying binary file" -ForegroundColor Green
        $scp = Set-SCPFile -ComputerName $_ -Credential $credESXi -Port 22 -LocalFile $source_file -RemotePath $target_path

        # And flag it as executable:
        Write-Host "- Changing execute mode" -ForegroundColor Green
        Invoke-SSHCommand -SSHSession $ssh -Command "chmod 'u+x' $target_file" | Out-Null

        Write-Host "- Retrieving directory list:" -ForegroundColor Green
        $dirlist = Invoke-SSHCommand -SSHSession $ssh -Command "ls -l $target_file"
        Write-Host $dirlist.Output
        # Retrieve summarize-dvfilter output:
        Write-Host "- Getting full list of dvfilters" -ForegroundColor Green 
        $dvflist = Invoke-SSHCommand -SSHSession $ssh -Command "summarize-dvfilter"

        # Parse each line in the summarize-dvfilter output:
        ForEach ($item in $dvflist.Output) {

            # If we see a new VM identifier, update our current VMName setting:
            if ($item -match '(?<=vmm0:)(.*)(?= vcUuid:)') {
                $VMName = $matches[0]
            # If we see a line that looks like a fw export, grab that and run vsipioctl on it:
            if ($item -match '(?<=name: )(.*sfw.2)') {
                $VMNic = $matches[0]
                $fwexport = Invoke-SSHCommand -SSHSession $ssh -Command "/tmp/vsipioctl getexportversion -f $VMnic"
                # Parse the returned 'Current Export Version' from vsipioctl:
                $exmatch = $fwexport.Output[0] -match '(?<=version: )(.*)'
                $exportVersion = $matches[0]

                # Add a table row that includes the VM name, NIC ID and exportversion:
                $row = $table.NewRow()
                $row.Host = $_
                $row.VMName = $VMName
                $row.NIC = $VMNic
                $row.ExportVersion = $exportVersion
            } # Line that looks like a fw export

        } # Foreach line in summarize-dvfilter output

        Remove-SSHSession -SSHSession $ssh | Out-Null

    } else {
        Write-Host "Error connecting to host $_, skipping." -ForegroundColor Red

    Write-Host "- Disabling SSH on Host" -ForegroundColor Green
    $stop = Stop-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} ) -Confirm:$false
} # Foreach VMHost

$table | Format-Table -AutoSize
$table | Export-Csv $csvOut





Live import VMs to vCloud Director

Tom Fojta wrote a great blog post about the new capability in vCloud Director 8.10 to import running VMs into vCloud Director. This is a huge asset in migration scenarios where customers can’t afford outages when being migrated into the vCD environment. Unfortunately the API syntax to actually initiate the import is a little convoluted and not the easiest process to manage.

I set about writing a PowerShell script to significantly simplify the process of initiating a live-import operation. The script itself is available from github at the following link: https://github.com/jondwaite/vcdliveimport.

The liveimport.ps1 script contained in this repository does the following:

  • Prompts for a credential to be used to connect to both vCloud Director (System context) and vCenter – if you have different usernames/passwords for each you’ll need to adjust this.
  • Enumerates the available vCenter instances registered as Provider Virtual Datacenters (PVDCs) in vCloud Director and allows one to be selected as the source vCenter for the migration.
  • Lists the available VMs in the selected vCenter instance, filters this list based on selectable criteria (e.g. don’t offer to import ‘Guest Introspection’ VMs) and allows the source VM to be selected.
  • Lists available destination Virtual Datacenters (VDCs) in the vCloud Director environment and allows the destination VDC to be selected.
  • Displays the appropriate POST request information to be submitted to vCloud Director to initiate the live-import of this VM.
  • Optionally – Submits the REST API request directly to the vCloud Director environment to actually initiate the import process.

An example transcript of this process is show below. Hopefully this helps someone else out and helps to make it easier for you to live-import running VMs into vCloud Director.


Example Session Transcript:

PowerCLI C:\> liveimport.ps1
Connected to vCloud Director OK
vCenter(s) Found:
vc01 (https://vcd01.dev.local/api/admin/extension/vimServer/158f73ec-a999-4332-8250-f4dd5e6c4971)

Selecting vc01 as only pVDC vCenter found.
vCenter vc01 selected.
Connected to vCenter Server vc01 OK

Evaluating vCenter VMs as migration candidates.............
Available candidate VMs:

Enter VM name to live-migrate to vCloud (or 'quit' to exit): testvm01
Selected VM testvm01 for live import to vCloud.

Available VDCs:
Enter Destination VDC Name (or quit to exit): Lab VDC
VDC 'Lab VDC' selected.

URI for POST operation:

XML Document Body:
<?xml version="1.0" encoding="UTF-8"?>
<ImportVmAsVAppParams xmlns="http://www.vmware.com/vcloud/extension/v1.5" name="testvm01" sourceMove="true">
 <Vdc href="https://vcd01.dev.local/api/admin/vdc/b21c92fa-1a6f-42a9-8c1e-ef0947c7be76" />

Content-Type: application/vnd.vmware.admin.importVmAsVAppParams+xml

Would you like to submit this API request to live import this VM? (y or n) (or quit to exit): y
Making request to live import VM testvm01...
Response was:
<?xml version="1.0" encoding="UTF-8"?>
<VApp xmlns="http://www.vmware.com/vcloud/v1.5" ovfDescriptorUploaded="true" deployed="false" status="0" name="testvm01" id="urn:vcloud:vapp:ff1a9021-2c21-4378-99c7-b77bdaf3e9e6" href="https://vcd01.ndev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6" type="application/vnd.vmware.vcloud.vApp+xml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.vmware.com/vcloud/v1.5 http://vcd01.dev.local/api/v1.5/schema/master.xsd">
 <Link rel="down" href="https://vcd01.dev.local/api/network/251f72a8-19b5-4e7d-94ba-0523ee919cd9" name="VM Network" type="application/vnd.vmware.vcloud.vAppNetwork+xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/controlAccess/" type="application/vnd.vmware.vcloud.controlAccess+xml" />
 <Link rel="up" href="https://vcd01.dev.local/api/vdc/b21c92fa-1a6f-42a9-8c1e-ef0947c7be76" type="application/vnd.vmware.vcloud.vdc+xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/owner" type="application/vnd.vmware.vcloud.owner+xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/metadata" type="application/vnd.vmware.vcloud.metadata+xml" />
 <Link rel="ovf" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/ovf" type="text/xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/productSections/" type="application/vnd.vmware.vcloud.productSections+xml" />
 <Task cancelRequested="false" expiryTime="2016-10-31T11:27:25.312+13:00" operation="Importing Virtual Application testvm01(ff1a9021-2c21-4378-99c7-b77bdaf3e9e6)" 
operationName="importSingletonVapp" serviceNamespace="com.vmware.vcloud" startTime="2016-08-02T11:27:25.312+12:00" status="queued" name="task" id="urn:vcloud:task:37576f8c-0d39-444c-8842-81436ddb421e" href="https://vcd01.dev.local/api/task/37576f8c-0d39-444c-8842-81436ddb421e" type="application/vnd.vmware.vcloud.task+xml">
 <Owner href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6" name="testvm01" type="application/vnd.vmware.vcloud.vApp+xml" />
 <User href="https://vcd01.dev.local/api/admin/user/fa7fb40e-648c-4787-ad0f-711dcc315261" name="system" type="application/vnd.vmware.admin.user+xml" />
<Organization href="https://vcd01.dev.local/api/org/3f46b162-f794-4d9e-8fa1-6ca1ee7b6377" name="Lab" type="application/vnd.vmware.vcloud.org+xml" />
 <Details />
 <Owner type="application/vnd.vmware.vcloud.owner+xml">
 <User href="https://vcd01.dev.local/api/admin/user/1545cb33-9151-43e2-a156-9d20e3b966c0" name="system" type="application/vnd.vmware.admin.user+xml" />


Create an empty vApp in vCloud Director

Sometimes you just need to create a new vApp with no contents at all – maybe for testing, or maybe you want to populate it with VMs built ‘from scratch’ rather than cloned from templates. This is easy to do in the vCloud Director web UI – you just skip the addition of any VM templates or new VMs and can easily create empty vApps, but how about programatically?

The VMware documentation is remarkably slim in this regard – all the documented methods I could find for vApp creation require either cloning from existing vApp templates, from existing VMs or from uploaded OVF files.

So how do we create a brand-new empty vApp? Turns out it’s pretty simple – once you discover the ‘composeVApp’ method on an Organization VDC supports creation of empty vApps.

If using the REST API we can simply create an XML body document of type ‘composeVAppParams’ and submit it against the OrgVDC’s /action/composeVapp link.

An example XML document body could be:

<?xml version=”1.0″ encoding=”UTF-8″?>
<Description>My vApp Description</Description>

We then ‘POST’ this document body to the link: ‘https://<Cloud Server DNS name or IP address>>/api/vdc/<ID of our VDC>/action/composeVApp’ not forgetting to add a header of ‘Content-Type: application/vnd.vmware.vcloud.composeVAppParams+xml’ to the POST request.

If we want to accomplish the same thing using PowerShell / PowerCLI it’s easy too (once connected to our cloud using Connect-CIServer):

$vapp = New-Object VMware.VimAutomation.Cloud.Views.ComposeVAppParams
$vapp.Name = “MyEmptyVapp”
$vapp.Description = “My vApp Description”
$myorgvdc = Get-OrgVdc -Name ‘My OrgVDC Name’

No idea if this is ‘officially’ supported or not – so use at your own risk and be aware that the implementation could change in a future release and break this (although I’d be surprised as this is almost certainly the action that the vCD web UI is submitting ‘behind the scenes’ when you manually create an empty vApp).


Working with vCloud Metadata in PowerCLI – Part 1

Way back in 2012 Alan Renouf created a PowerCLI module to deal with manipulation of metadata entries for vCloud Director objects – this can be incredibly useful to track related information for these objects. The vCD metadata functionality was enhanced in v5.1 (and then later in 5.5, 5.6 and 8.0) – in particular typed values were added with functionality to use date/time, boolean and numeric values (as well as free-form string text). Also added were security levels so that metadata could be made read-only or hidden (from a tenant perspective) but still accessible/visible to system owners. I’ve taken the PowerShell module that Alan published here and updated it to cope with these enhancements. I’ve also updated the returned fields/views to include the extra attributes (where present) such as security levels of metadata entries.

Note that I am definitely not a professional developer (and most of my PowerShell knowledge comes from Google) so there’s probably significant room for improvement in the code – comment back if you have suggestions for improvement and I’ll update this post.

Use of the module requires a valid connection to a vCloud instance (using Connect-CIServer). This won’t work for versions prior to v5.1 (most of my testing has been with PowerCLI 6 against a v8 vCD deployment) so please use at your own risk and make sure you thoroughly test your own scenarios. I’ll write a follow-up post detailing some example code and usage scenarios which people may find useful in the next few days.

I’d suggest copy/pasting the code (below) into a PowerShell module (.psm1) file and including the module in your scripts as needed.

Function New-CIMetaData { 
        Creates a Metadata Key/Value pair. 
        Creates a custom Metadata Key/Value pair on a specified vCloud object 
    .PARAMETER  Key 
        The name of the Metadata to be applied.
    .PARAMETER  Value
        The value of the Metadata to be applied, the string 'Now' can be used
        for the current date/time for values using the 'DateTime' type.
    .PARAMETER  Visibility
        The visibility of the Metadata entry (General, Private, ReadOnly)
    .PARAMETER  Type
        The type of the Metadata entry (String, Number, DateTime, Boolean)
        (these correspond to the types of: MetadataStringValue,
        MetadataNumberValue, MetadataDateTimeValue or MetadataBooleanValue
    .PARAMETER  CIObject
        The object on which to apply the Metadata.
        New-CIMetadata -Key "Owner" -Value "Alan Renouf" -CIObject (Get-Org Org1)
        Creates a new metadata value "Alan Renouf" in a key "Owner" on the Org1 object.
        New-CIMetadata -Key "Company" -Value "ABC Corp" -Visibility READONLY -CIObject (Get-CIVM 'client')
        Creates a new metadata value "ABC Corp" in a key "Company" on the 'client' VM object with the READONLY attribute set preventing changes by non-system users.
        New-CIMetadata -Key "Backup" -Value $false -Visibility Private -Type Boolean -CIObject (Get-CIVapp 'testvapp')
        Creates a new hidden metadata value $false in a key "Backup" on the vApp object with the 'Private' attribute set preventing visibility to non-system users.
        NAME: Get-CIMetaData
        AUTHOR: Jon Waite based on code by Alan Renouf
        LASTEDIT: 2016-02-23
        KEYWORDS: metadata set vcloud director
    #Requires -Version 2.0
            [String]$Visibility = 'General',
            [String]$Type = "String"
    Process { 
        Foreach ($Object in $CIObject) { 
            $Metadata = New-Object VMware.VimAutomation.Cloud.Views.Metadata 
            $Metadata.MetadataEntry = New-Object VMware.VimAutomation.Cloud.Views.MetadataEntry 
            $Metadata.MetadataEntry[0].Key = $Key

            switch($Type) {
              'String'   { $Metadata.MetadataEntry[0].TypedValue = New-Object VMware.VimAutomation.Cloud.Views.MetadataStringValue }
              'Number'   { $Metadata.MetadataEntry[0].TypedValue = New-Object VMware.VimAutomation.Cloud.Views.MetadataNumberValue }
              'DateTime' { $Metadata.MetadataEntry[0].TypedValue = New-Object VMware.VimAutomation.Cloud.Views.MetadataDateTimeValue }
              'Boolean'  { $Metadata.MetadataEntry[0].TypedValue = New-Object VMware.VimAutomation.Cloud.Views.MetadataBooleanValue }

            if ($Type -eq 'DateTime' -and $Value -eq 'Now') {
                $Metadata.MetadataEntry[0].TypedValue.Value = [string](Get-Date).ToUniversalTime().GetDateTimeFormats('s')
            } else {
                $Metadata.MetadataEntry[0].TypedValue.Value = $Value
            switch($Visibility) {
              'General'  { } #Default, don't need to change
              'Private'  { 
                $Metadata.MetadataEntry[0].Domain = New-Object VMware.VimAutomation.Cloud.Views.MetadataDomainTag
                $Metadata.MetadataEntry[0].Domain.Value = 'SYSTEM'
                $Metadata.MetadataEntry[0].Domain.Visibility = 'PRIVATE'
              'ReadOnly' {
                $Metadata.MetadataEntry[0].Domain = New-Object VMware.VimAutomation.Cloud.Views.MetadataDomainTag
                $Metadata.MetadataEntry[0].Domain.Value = 'SYSTEM'
                $Metadata.MetadataEntry[0].Domain.Visibility = 'READONLY'

            ($Object.ExtensionData.GetMetadata()).MetadataEntry | Where {$_.Key -eq $key } | Select @{N="CIObject";E={$Object.Name}},
            @{N="Visibility";E={ if ($_.Domain.Visibility) { $_.Domain.Visibility } else { "General" }}},
            Key -ExpandProperty TypedValue
Function Get-CIMetaData {
        Retrieves all Metadata Key/Value pairs.
        Retrieves all custom Metadata Key/Value pairs on a specified vCloud object
    .PARAMETER  CIObject
        The object on which to retrieve the Metadata.
        The key to retrieve.
        Get-CIMetadata -CIObject (Get-Org Org1)
    Process {
        Foreach ($Object in $CIObject) {
            If ($Key) {
                ($Object.ExtensionData.GetMetadata()).MetadataEntry | Where {$_.Key -eq $key } | Select @{N="CIObject";E={$Object.Name}},
                    @{N="Visibility";E={ if ($_.Domain.Visibility) { $_.Domain.Visibility } else { "General" }}},
                    Key -ExpandProperty TypedValue
            } Else {
                ($Object.ExtensionData.GetMetadata()).MetadataEntry | Select @{N="CIObject";E={$Object.Name}},
                    @{N="Visibility";E={ if ($_.Domain.Visibility) { $_.Domain.Visibility } else { "General" }}},
                    Key -ExpandProperty TypedValue
Function Remove-CIMetaData {
        Removes a Metadata Key/Value pair.
        Removes a custom Metadata Key/Value pair on a specified vCloud object
        The name of the Metadata to be removed.
    .PARAMETER  CIObject
        The object on which to remove the Metadata.
        Remove-CIMetaData -CIObject (Get-Org Org1) -Key "Owner"
    Process {
        $CIObject | Foreach {
            $metadataValue = ($_.ExtensionData.GetMetadata()).GetMetaDataValue($Key)
            If($metadataValue) { $metadataValue.Delete() }