Uploading / running utilities directly on ESXi hosts
As part of planning our upgrade from VMware NSX-V from v6.2.2 to v6.2.4 we became aware of the VMware issue KB2146171 (link) which can cause VMs to lose network connectivity when vMotioned to other hosts following the upgrade. Obviously wishing to avoid this for our own (and customer) VMs, we raised a support case to obtain the VMware script to determine how many of our VMs (if any) were going to be affected. Unfortunately the VMware script we were supplied was configured to run *after* the upgrade had already been completed. Fortunately the VMware utility supplied (vsipioctl - a binary to be run directly on ESXi hosts) could tell us which VMs were affected prior to upgrading.
Since we have a reasonably large number of hosts and hosted VMs I set about writing some PowerShell to perform the following actions:
- Connect to vCenter and enumerate all ESXi hosts.
- Enable SSH access to each host in turn.
- Upload the VMware vsipioctl utility to the host /tmp/ folder and make it executable.
- Run vsipioctl and parse the return information.
- Build a table / CSV of all VM network interfaces with the results of the vsipioctl utility.
- Disable SSH on the hosts once done and move on to the next host.
At first I tried using PuTTY plink.exe and pscp.exe from PowerShell to perform the SSH and SCP file copy to the hosts, but had serious problems passing the right password & command line options due to the way PowerShell escapes quoted strings. In the end I found it easier to use the PoshSSH PowerShell library (https://github.com/darkoperator/Posh-SSH) for these functions rather than shelling out to PuTTY executables.
Note that we usually leave SSH access disabled on our ESXi hosts, so the script shown enables this and then re-disables SSH after running - adjust if necessary when using in your own environments.
If you need to run this check for your own environment you will still need to open a VMware support call to obtain the vsipioctl binary as far as I am aware as I don't believe this is available any other way.
The script is shown below - hopefully this will be useful for some of you, just make sure you test properly before running against a production environment. Luckily in our case the script proved that none of our VMs are impacted by this issue and we can safely proceed with our NSX-V upgrade.
Jon.
Upload and run ESXi CLI against each host in a vCenter
1# Script to run VMware vsipioctl utility against all VMs / all hosts
2# NOTE: Requires darkoperator PoshSSH for PowerShell
3#
4# To install PoshSSH:
5# iex (New-Object Net.WebClient).DownloadString("https://gist.github.com/darkoperator/6152630/raw/c67de4f7cd780ba367cccbc2593f38d18ce6df89/instposhsshdev")
6
7# Specify our vCenter instance FQDN or IP:
8$vCenter = "<vcenter FQDN or IP address>"
9
10# Prompt user for vCenter credentials if not known:
11if ($credVCenter -eq $null) {
12 $credVCenter = Get-Credential -Message "Authenticate to vCenter Server"
13}
14
15# Prompt user for ESXi host credentials if not known:
16if ($credESXi -eq $null) {
17 $credESXi = Get-Credential -Message "Provide ESXi root password" -UserName root
18}
19
20# Directory containing this script:
21$scriptDir = Split-Path -Path $($global:MyInvocation.MyCommand.Path)
22
23# Location to find the VMware vsipioctl binary:
24$source_file = $scriptDir+"\vsipioctl"
25
26$target_path = "/tmp/"
27
28# Output CSV File in current (script) directory:
29$csvOut = $scriptDir+"\fwexport.csv"
30
31# Location to copy vsipioctl to on ESXi hosts:
32$target_file = "/tmp/vsipioctl"
33
34Connect-VIServer -Server $vCenter -Credential $credVCenter
35
36# Define an output table object with required columns:
37$table = New-Object System.Data.DataTable
38$col1 = New-Object System.Data.DataColumn Host,([string])
39$col2 = New-Object System.Data.DataColumn VMName,([string])
40$col3 = New-Object System.Data.DataColumn NIC,([string])
41$col4 = New-Object System.Data.DataColumn ExportVersion,([int])
42$table.Columns.Add($col1)
43$table.Columns.Add($col2)
44$table.Columns.Add($col3)
45$table.Columns.Add($col4)
46
47Get-VMHost | Foreach {
48
49 Write-Host "Starting processing for Host: $_" -ForegroundColor Green
50
51 Write-Host "- Enabling SSH on Host" -ForegroundColor Green
52 $start = Start-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} )
53
54 $ssh = New-SSHSession -ComputerName $_ -Credential $credESXi -Port 22 -AcceptKey:$true
55
56 if ($ssh.Connected -eq $true) {
57
58 $return = Invoke-SSHCommand -SSHSession $ssh -Command "uname -a"
59 Write-Host "Host ID: " + $return.Output -ForegroundColor Yellow
60
61 # Copy the VMware vsipioctl utility to the host:
62 Write-Host "- Copying binary file" -ForegroundColor Green
63 $scp = Set-SCPFile -ComputerName $_ -Credential $credESXi -Port 22 -LocalFile $source_file -RemotePath $target_path
64
65 # And flag it as executable:
66 Write-Host "- Changing execute mode" -ForegroundColor Green
67 Invoke-SSHCommand -SSHSession $ssh -Command "chmod 'u+x' $target_file" | Out-Null
68
69 Write-Host "- Retrieving directory list:" -ForegroundColor Green
70 $dirlist = Invoke-SSHCommand -SSHSession $ssh -Command "ls -l $target_file"
71 Write-Host $dirlist.Output
72
73 # Retrieve summarize-dvfilter output:
74 Write-Host "- Getting full list of dvfilters" -ForegroundColor Green
75 $dvflist = Invoke-SSHCommand -SSHSession $ssh -Command "summarize-dvfilter"
76
77 # Parse each line in the summarize-dvfilter output:
78 ForEach ($item in $dvflist.Output) {
79
80 # If we see a new VM identifier, update our current VMName setting:
81 if ($item -match '(?<=vmm0:)(.*)(?= vcUuid:)') {
82 $VMName = $matches[0]
83 }
84 # If we see a line that looks like a fw export, grab that and run vsipioctl on it:
85 if ($item -match '(?<=name: )(.*sfw.2)') {
86 $VMNic = $matches[0]
87
88 $fwexport = Invoke-SSHCommand -SSHSession $ssh -Command "/tmp/vsipioctl getexportversion -f $VMnic"
89
90 # Parse the returned 'Current Export Version' from vsipioctl:
91 $exmatch = $fwexport.Output[0] -match '(?<=version: )(.*)'
92 $exportVersion = $matches[0]
93
94 # Add a table row that includes the VM name, NIC ID and exportversion:
95 $row = $table.NewRow()
96 $row.Host = $_
97 $row.VMName = $VMName
98 $row.NIC = $VMNic
99 $row.ExportVersion = $exportVersion
100 $table.Rows.Add($row)
101 } # Line that looks like a fw export
102
103 } # Foreach line in summarize-dvfilter output
104
105 Remove-SSHSession -SSHSession $ssh | Out-Null
106
107 } else {
108 Write-Host "Error connecting to host $_, skipping." -ForegroundColor Red
109 }
110
111 Write-Host "- Disabling SSH on Host" -ForegroundColor Green
112 $stop = Stop-VMHostService -HostService ($_ | Get-VMHostService | Where { $_.Key -eq "TSM-SSH"} ) -Confirm:$false
113
114} # Foreach VMHost
115
116$table | Format-Table -AutoSize
117$table | Export-Csv $csvOut