vCloud Director 9 HTML5 Portal Customization

One of the great features in vCloud Director 9 which has been further enhanced in the latest v9.5 release is the new HTML5 portal:

image

Even better, VMware has released a toolkit to allow Service Providers to fully customise the look and feel of the portal using CSS themes in their Clarity framework..

The toolkit itself is part of the VMware vcd-ext-sdk repository on github, available in the /ui/theme-generator folder.

The repository has good instructions on how to modify and build a custom theme, but actually uploading and configuring the theme in vCloud Director is only accessible via the vCD API and involves a reasonable amount of manual work.

To help speed up development and allow changes to be easily tested, in my usual mode I’ve written a small PowerShell module that allows quicker/easier theme configuration. The module is available on github at https://github.com/jondwaite/vcd-h5-themes. Hopefully this will help those of you who need to develop and test updated themes for your vCloud Director portals.

I’ve included documentation in the repository on each cmdlet, its function and arguments here.

To use the module you’ll need to be connected to a vCloud instance as a user with global ‘Administrator’ access in the ‘System’ organization since changes will affect all portal users. You’ll need to be connected to the vCD environment with PowerCLI (Connect-CIServer…) prior to using the cmdlets.

You can then download the vcd-ht-themes.psm1 file and add it to your session (‘Import-Module vcd-h5-themes.psm1’) to access the cmdlets.

As always, comments and feedback welcome – is there anything else you’d like to see added to this module?

Jon.

Getting detailed VM Disk Properties from the vCloud API

Since vCloud Director 8.10 VMware have allowed VMs to be created which have multiple disks using different storage policies. This can be very useful – for example, a database VM might have it’s database on fast storage but another disk containing backups or logs on slower/cheaper disk.

When trying to find out what storage is in use for a VM though this can create issues, the PowerCLI Get-CIVM cmdlet (and the Get-CIView cmdlet used to get extra information) aren’t able to properly report storage for VMs that consume multiple storage policies. This in turn can create problems for Service Providers when they need to report on overall VM disk usage divided by storage policy used.

As an example I’ve created a VM named ‘test01’ in a customer vDC which has 3 disks attached, the 2nd of these is on ‘Capacity’ tier storage while disks 1 and 3 are on ‘Performance’ storage. When we look at the VM details we see the following:

image

Digging into the ExtensionData shows

image

The StorageProfile element looks like it may contain what we need, but unfortunately this only shows the ‘home’ Storage for the VM and doesn’t indicate that at least one of the VMs disks is on a different storage profile:

image

After a lot of mucking around trying to find an easy way to discover the information, I ‘gave up’ and wrote a PowerShell module which accesses the vCD API directly to get the VM storage information (including storage tiers in use by each disk). The module isn’t overly efficient since it queries the storage profile reference for every disk on every VM (and so will result in a lot of calls if run for a large number of VMs), but otherwise works fine.

The module takes VM objects or a VM name as input and returns details on each disk attached to the VM including which storage profile they use. Save the script (e.g. as ‘Get-CIVMStorageProfile.psm1’) and then use ‘Import-Module .\Get-CIVMStorageProfile.psm1’ to import the function.

<#
  .Synopsis
   Gets detailed storage information from a vCloud VM.

.Description
   This function returns detailed disk information for a vCloud VM. Specifically
   it shows the number of disks attached and the storage policy assigned to each
   disk which is useful when VMs consume storage from multiple policies.

.Parameter CIVM
   The VM object (from Get-CIVM) to report storage for.

.Example
   Get-CIVM -Name 'test01' | Get-CIVMStorageDetail
#>

Function Get-CIVMStorageDetail
{
     [CmdletBinding()]
     Param(
         [Parameter(ValueFromPipeline)]
         $CIVM
     )
     begin {}
     process
     {

        # API version to use when communicating with vCloud Director - API 27.0 is vCloud Director 8.20:
         $vCDAPIVersion = "27.0"
                 
         # Check if we've been passed a VM name or an actual VM object and handle appropriately
         if ($CIVM.GetType() -eq [String]) {
             try {
                 $VMObj = Get-CIVM -Name $CIVM -ErrorAction Stop
             } catch {
                 Write-Host -ForegroundColor Red "Error: Could not find a VM with the name $CIVM."
                 Break
             }
         }
         else {
             $VMObj = $CIVM
         }

        # Find our vCloud SessionId that matches the URI of the VM object:
         $SessionId = $global:DefaultCIServers.SessionId | Where-Object { $VMObj.href -match $_.ServiceUri }

        try {
             $vmxml = Invoke-RestMethod -Method Get -Uri "$($VMObj.href)/virtualHardwareSection/disks" -Headers @{'x-vcloud-authorization'=$SessionId; 'Accept'="application/*+xml;version=$($vCDAPIVersion)"} -ErrorAction Stop
         } catch {
             Write-Host -ForegroundColor Red "Error attempting to get VM details from API:"
             Write-Host -ForegroundColor Red "Status Code: $($_.Exception.Response.StatusCode.value__)"
             Write-Host -ForegroundColor Red "Status Description: $($_.Exception.Response.StatusDescription)"
             Break
         }

        # Build an empty object for the VM disk details:
         $vmdisks = @()

        # RASD resource type 17 is a hard disk attached to a VM:
         foreach($disk in ($vmxml.RasdItemsList.Item | Where-Object -Property ResourceType -eq 17)) {
             
             # Dereference the StorageProfileHref for each disk to get the Storage Profile Name:
             try {
                 $sprof = Invoke-RestMethod -Method Get -Uri "$($disk.HostResource.storageProfileHref)" -Headers @{'x-vcloud-authorization'=$SessionId; 'Accept'="application/*+xml;version=$($vCDAPIVersion)"} -ErrorAction Stop
             } catch {
                 Write-Host -ForegroundColor Red "Error attempting to get Storage Profile Name:"
                 Write-Host -ForegroundColor Red "Status Code: $($_.Exception.Response.StatusCode.value__)"
                 Write-Host -ForegroundColor Red "Status Description: $($_.Exception.Response.StatusDescription)"
                 Break
             }
             
             $diskprops = @{
                 VMName         = [string]$VMObj.Name
                 InstanceID     = [string]$disk.InstanceID
                 StorageProfile = [string]$sprof.VdcStorageProfile.Name
                 CapacityGB     = [float][math]::Round(($disk.VirtualQuantity / 1024 / 1024 / 1024),3)
                 ElementName    = [string]$disk.ElementName
             }

            $diskobj = New-Object PSObject -Property $diskprops
             $vmdisks += $diskobj
         }

        return $vmdisks
     } # end process

}
Export-ModuleMember -Function Get-CIVMStorageDetail

And here is example output from the script for our test VM:

image

Hope this is useful to some of you and as always, appreciate any comments/feedback.

I’d also love to know if there’s an easier way of generating this information.

Jon.

vCloud Director v9 Multi-site

Since vCloud Director v9 was released last week (and previously as part of the closed beta), one of the new features I’m most excited about is support for multi-site deployments. This allows vCloud Director environments for the first time to properly span federated sites (e.g. a tenant who has resources in multiple datacenter locations for resiliency/redundancy can now manage these in the same place).

Configuring multi-site support in vCloud Director v9 is a 2-part process:

1) The service provider has to configure federation between their vCloud Director instances.
2) The tenant has to associate each of their Organizations in each vCloud Director instance.

This post will attempt to explain and show both processes, and there’s even a bonus of a PowerShell script I’ve written to help other service providers configure their site pairings. To help demonstrate the processes involved, I’ve built a test lab environment consisting of 4 separate vCloud Director instances (‘Auckland’, ‘Wellington’, ‘Christchurch’ and ‘Dunedin’ sites). Each of these has it’s own vCloud Director, vCenter, NSX and ESXi hosts. I’ve also created a tenant organization (‘Tenant X’) in all 4 instances and created and assigned a VDC to Tenant X in each location. Finally, I’ve federated Tenant X’s vCloud users with a directory service (Microsoft AD FS in this case) so that the same identity provider is available to all 4 vCloud instances.

If you’re not a service provider and just need to configure Organization pairing you’re probably safe to skip this section and proceed straight to the 2nd part of this post.

Part 1 – Service Provider Site Pairing

The basic process for a service provider to pair sites is:

– Check (and configure if necessary) the vCloud site name in each location. Note by default in the initial vCloud Director 9 release this is simply a GUID string so you’ll probably want to change it to something more meaningful.
– Download from each site the site association document (from /api/site/associations/localAssociationData)
– Upload this site association document to each other site that you want to pair with (to /api/site/associations)

In a scenario with only 2 sites you need to perform this process twice (once in each direction), but for our example with 4 sites we need to do this a total of 12 times to pair every site with every other site.

Being a bit of a pain to do manually against the REST API interface I ran true to form and wrote a PowerShell module to simplify the process of both administering the site names and also pairing sites together. The module is available on my github repository at https://github.com/jondwaite/vCDSitePair. The script uses my Invoke-vCloud module, so you’ll need that installed for it to run.

Once you’ve downloaded the vCDSitePair.psm1 file from github you can add it to your PowerShell session using ‘Import-Module ‘

There are a total of 4 functions provided by the module, and they are documented on the github repository but basically:

Get-vCloudSiteName Allows a service provider to check/confirm the ‘Site Name’ assigned to a vCloud Director instance.
Set-vCloudSiteName Allows a service provider to set/update the ‘Site Name’ assigned to a vCloud Director instance.
Get-vCloudSiteAssociations Shows the existing associations (if any) from a vCloud Director instance.
Invoke-vCDPairSites Performs the 2-way exchange of localAssociationData documents to pair two vCloud Director instances.

So to confirm/set the names of our ‘Auckland’ (akl.mycloud.local) and ‘Christchurch’ (chc.mycloud.local) sites we can use Get-vCloudSiteName and Set-vCloudSiteName:

PS C:\> Get-vCloudSiteName -siteDomain akl.mycloud.local
847f4785-f8a5-4e59-b0d7-5608723247dd

PS C:\> Set-vCloudSiteName -siteDomain akl.mycloud.local -siteName 'A03 - Auckland DC'
Task submitted successfully, waiting for result
q=queued, P=pre-running, .=Task Running:
q
Task completed successfully
True

PS C:\> Get-vCloudSiteName -siteDomain akl.mycloud.local
A03 - Auckland DC

Christchurch Site Name:

PS C:\> Get-vCloudSiteName -siteDomain chc.mycloud.local
b34798aa-f722-4c06-a8a2-392f591cf450

PS C:\> Set-vCloudSiteName -siteDomain chc.mycloud.local -siteName 'C00 - Christchurch DC'
Task submitted successfully, waiting for result
q=queued, P=pre-running, .=Task Running:
q 
Task completed successfully
True

PS C:\> Get-vCloudSiteName -siteDomain chc.mycloud.local
C00 - Christchurch DC

Now we have set the site names, we can check if they are already associated:

 

PS C:\> Get-vCloudSiteAssociations -siteDomain akl.mycloud.local
Displaying site associations for site Id: urn:vcloud:site:847f4785-f8a5-4e59-b0d7-5608723247dd with site Name: A03 - Auckland DC
No site associations found

PS C:\> Get-vCloudSiteAssociations -siteDomain chc.mycloud.local
Displaying site associations for site Id: urn:vcloud:site:b34798aa-f722-4c06-a8a2-392f591cf450 with site Name: C00 - Christchurch DC
No site associations found

Ok, so they’re not already associated, so we can run Invoke-vCDPairSites without the ‘WhatIf $false’ to see what would happen:

PS C:\> Invoke-vCDPairSites -siteAuri akl.mycloud.local -siteBuri chc.mycloud.local
Running in information mode only - no API changes will be made unless you run with -WhatIf $false
Site A returned site ID as: urn:vcloud:site:847f4785-f8a5-4e59-b0d7-5608723247dd
Site A returned site name as: A03 - Auckland DC
Site B returned site ID as: urn:vcloud:site:b34798aa-f722-4c06-a8a2-392f591cf450
Site B returned site name as: C00 - Christchurch DC
Not performing site association as running in information mode

That all looks good so now we can attempt the action pairing operation:

PS C:\> Invoke-vCDPairSites -siteAuri akl.mycloud.local -siteBuri chc.mycloud.local -WhatIf $false
Running in implementation mode, API changes will be committed
Site A returned site ID as: urn:vcloud:site:847f4785-f8a5-4e59-b0d7-5608723247dd
Site A returned site name as: A03 - Auckland DC
Site B returned site ID as: urn:vcloud:site:b34798aa-f722-4c06-a8a2-392f591cf450
Site B returned site name as: C00 - Christchurch DC
Associating A03 - Auckland DC (Site A) with C00 - Christchurch DC (Site B)
Task submitted successfully, waiting for result
q=queued, P=pre-running, .=Task Running:
q 
Task completed successfully
Returned Result = True
Associating C00 - Christchurch DC (Site B) with A03 - Auckland DC (Site A)
Task submitted successfully, waiting for result
q=queued, P=pre-running, .=Task Running:
q 
Task completed successfully
Returned Result = True

And confirm the associations using Get-vCloudSiteAssociation again:

PS C:\> Get-vCloudSiteAssociations -siteDomain akl.mycloud.local
Displaying site associations for site Id: urn:vcloud:site:847f4785-f8a5-4e59-b0d7-5608723247dd with site Name: A03 - Auckland DC
Associated sites:
https://chc.mycloud.local/api


href                    : https://akl.mycloud.local/api/site/associations/b34798aa-f722-4c06-a8a2-392f591cf450
type                    : application/vnd.vmware.admin.siteAssociation+xml
Link                    : {Link, Link, Link}
RestEndpoint            : https://chc.mycloud.local/api
TenantUiEndpoint        : https://chc.mycloud.local/tenant
RestEndpointCertificate : -----BEGIN CERTIFICATE-----
                          MII...
                          -----END CERTIFICATE-----
SiteId                  : urn:vcloud:site:b34798aa-f722-4c06-a8a2-392f591cf450
SiteName                : C00 - Christchurch DC
PublicKey               : -----BEGIN PUBLIC KEY-----
                          MII...
                          -----END PUBLIC KEY-----

(I’ve cut out the certificate dumps for brevity).

So now that our Auckland and Christchurch sites are paired we can move on with associating the Organization (‘Tenant X’) between these sites. I’ve also been through and associated all of the other sites to each other, so by this stage ‘Auckland’ is associated to ‘Wellington’,’Christchurch’ and ‘Dunedin’ etc.

Part 2 – Organization Site Pairing

Originally I was intending to write PowerShell functions for this too, and while this is certainly possible, VMware have been nice to us and created the capability in the new vCloud Director tenant UI. Logging in as a user with ‘Organizational Administrator’ access shows an ‘Administration’ tab:

Selecting the ‘Administration’ tab reveals the site pairing options:

When we select ‘Export Local Association Data’ a file is downloaded (named ‘Download’ weirdly enough) and this file can be uploaded to the ‘partner’ site using the other ‘Create New Organization Association’ button. Once completed, the association is shown in the panel – here is the Auckland site for ‘Tenant X’ once the Christchurch Local Association Data has been uploaded to it:

You can click on this panel to see the association details and even remove a site association if no longer required:

Here’s the view of the ‘Christchurch’ environment once I’ve paired the ‘other’ 3 sites to it for this Organization:

Once we’ve added all our associations (and logged out and back in) we can see the new multi-site drop-down menu item which allows us to select from any of our datacenter locations:

And selecting one (‘Auckland’ in this case) takes us to the Auckland resource view:

All in all, a little bit of a convoluted process, but at least it should only need to be done once and can then be left alone. Very excited to see what VMware do with this functionality in future – can definitely see a time when all of an Organization’s VMs are displayed / summarised in a single view regardless of which vCloud instance supports them.

I have several more thoughts generally on vCloud Director v9 which I’ll put into a separate post when I have time, but wanted to get this published for anyone else playing around with the new multi-site features.

As always, comments and feedback appreciated.

Jon.

Invoke-vCloud PowerShell Module

Several of the posts on here and many of the internal PowerShell projects I use at work require direct interaction with the vCloud Director REST API. Usually this is because features exposed in the API aren’t yet directly implemented as PowerCLI cmdlets. A good example would be the modules I wrote to allow manipulation of independent disks with vCD VMs here: Independent Disks in vCloud via PowerCLI.

As I wrote more and more scripts that require interaction with the vCD API I started to develop a generic function to allow easier interaction with it. I’m intending in future to use this as the basis for future scripts requiring this functionality which should be useful in keeping the script size smaller and provide an easier method of interacting with the API.

PowerShell has a built-in ‘Invoke-RestMethod’ call which submits a REST request and gathers any responses, but there are a variety of additional parameters required in a typical vCD API call which this method doesn’t know about and which need to be supplied with each call. Wrapping Invoke-RestMethod in a function allows us to much more easily consume the API and makes handling calls which return a long-running task much easier to handle by giving the option to return immediately or to wait for the complete interaction to complete before returning.

Typically to call the API we need to provide the following:

URI The URI (Uniform Resource Identifier) for the API call (typically of the form https://my.cloud.com/api/requestpath).
Method This is simply which HTML method we are invoking from the common HTML verbs (‘GET’,’PUT’,’POST’,’DELETE’) – the fifth verb (‘PATCH’) doesn’t appear to be used much (if at all) in the vCloud API, but could be specified if required. We can also assume a sensible/safe default value of ‘GET’ since that will only read from the API and not change anything.
Authorization Provided as an HTML Header using the x-vcloud-authorization tag and a previously existing session ID. We could supply the session ID as part of the call, but it’s usually much more convenient to attempt to match the request URI against PowerShell’s existing global view of connected vCD API endpoints and use the existing session ID available from here.
Accept Provided as part of the HTML header, this tag specifies the type of information we are prepared to accept back from the API, this will usually be set to ‘application/*+xml’ but we also need to specify which version of the API we wish to use – in vCloud Director 8.20 this is version 27.0 so the complete Accept token will be ‘application/*+xml;version=27.0’. We can provide a sensible default value for this in our module so that if it is ommitted we still get a sensible version specification.
ContentType When we are sending data to the API with a PUT or POST request, we need to specify the type of document we are sending using the ContentType header, for GET requests this isn’t required.
Body When sending data to the API, this will be the (usually XML formatted) document body.
Timeout Some API operations can take a while to complete, specifying a timeout value allows our script to carry on or raise an error if an API call is taking an excessively long time.

In addition to these, I’ve also included a flag ‘WaitforTask’ which can either be true or false. If set to ‘true’ and if the result of the API call is a ‘Task’ object reference then the script will monitor the task and wait until it has completed (or timeout exceeded) and then return the task status to our script. This can be useful for operations that can take considerable time when it doesn’t make sense for your script to continue until the previous action has been completed – e.g. cloning a new vApp and then modifying it’s network settings.

I was keen to explore using Microsoft’s PowerShell Gallery as a mechanism to distribute modules, this should make it easier to be able to use these modules and functions wherever needed by simply importing the module as required. The process to make a module available in the gallery is reasonably straightforward and documented on the site, but basically once you’ve signed up and got an API key you simply call Publish-Module and provide the path to your module code and manifest including your API key as an identifier.

So I’m pleased to announce that the initial release version of my Invoke-vCloud module is now available from the PowerShell Gallery and can be included in any scripts using:

Install-Module -Name Invoke-vCloud -Scope CurrentUser

It can also be downloaded/inspected using:

Save-Module -Name Invoke-vCloud -Path <path>

Of course if running an administrative PowerShell instance it can also be installed for all users on the system using:

Install-Module -Name Invoke-vCloud

Once installed, it can be used in your scripts by specifying the required URI parameter and any optional specifications:

PS C:\> Invoke-vCloud -URI https://my.cloud.com/api/org

xml OrgList
--- -------
version="1.0" encoding="UTF-8" OrgList

You will either need to have an existing PowerCLI vCloud session (Connect-CIServer), or have an x-vcloud-authorization token for a valid/authenticated session which you can pass to Invoke-vCloud using the -vCloudToken parameter.

I’ve included parameter descriptions in the module to assist which can be viewed using the Get-Help cmdlet:

PS C:\> Get-Help Invoke-vCloud -detailed

NAME
    Invoke-vCloud

SYNOPSIS
    Provides a wrapper for Invoke-RestMethod for vCloud Director API calls


SYNTAX
    Invoke-vCloud [-URI]  [[-ContentType] ] [[-Method] ] [[-ApiVersion] ] [[-Body] ] [[-Timeout] ] [[-WaitForTask] ] [[-vCloudToken] ] []


DESCRIPTION
    Invoke-vCloud provides an easy to use interface to the vCloud Director
    API and provides sensible defaults for many parameters. It wraps the native
    PowerShell Invoke-RestMethod cmdlet.


PARAMETERS
    -URI 
        A mandatory parameter which provides the API URI to be accessed.

    -ContentType 
        An optional parameter which provides the ContentType of any submitted data.

    -Method 
        An optional parameter to specify the HTML verb to be used (GET, PUT, POST or
        DELETE). Defaults to 'GET' if not specified.

    -ApiVersion 
        An optional parameter to specify the API version to be used for the call.
        Defaults to '27.0' if not specified which is the API version for vCloud
        Director v8.20.

    -Body 
        An optional parameter which specifies XML body to be submitted to the API
        (usually for a PUT or POST action).

    -Timeout 
        An optional parameter which specifies the time (in seconds) to wait for an API
        call to complete. Defaults to 40 seconds if not specified.

    -WaitForTask 
        If the API call we submit results in a Task object indicating an asynchronous
        vCloud task, should we wait for this to complete before returning? Defaults to
        $false.

    -vCloudToken 
        An alternative method of passing a session token to Invoke-vCloud if there is
        no current PowerCLI session established to a vCloud endpoint. The session must
        have already been established and be still valid (not timed-out). The value
        supplied is copied to the 'x-vcloud-authorization' header value in API calls.

I have some new scripts in development against some of the new features in the recently announced vCloud Director v9 which I’m intending to use this module for and will post as soon as v9 is released.

As always I welcome any comments and suggestions for improving this module.

Jon.

vCloud Director 8.20 Edge Gateway Roles

One of the key changes in vCloud Director 8.20 and 8.20.1 from 8.10 is the Advanced Networking for Edge Gateways, this allows customer control of several advanced networking features of the Edge Gateways which previously could not be made available to tenant administrators. vCloud Director 8.20 and later also change the Roles to be per-tenant organisation (rather than globally shared between all tenants). However, in order for tenant administrators to be able to take advantage of the new features, the new Edge Gateway roles need to be added to their organisation. The only way currently to achieve this is by the vCloud REST API and must be performed separately for each organisation in the vCloud infrastructure.

Here is what the available rights looks like prior to the change being made – note there is no ‘Gateway Advanced Services’ section at all:

Since manually modifying the OrgRights XML is time-consuming and a bit prone to error, I set about writing a PowerCLI script to make the change automatically for a given organisation. Note that this change does not alter the defined roles for an organisation, it simply adds the new Edge Gateway permissions as available entities which can then be selectively added to roles.

Once the script has been run for an organisation, editing the properties of a role allows the new Gateway Advanced Services entities to be selected for that role:

The script is included below, as always I welcome any thoughts/comments/feedback.

Jon

# Script to add new OrgRights options for administering advanced Edge Gateway to a vCloud Director organisation.
# Note that Organisation roles (e.g. Organizational Administrator) still need to be edited to add these rights once
# this script has been run against their org.
# NOTE: You must be connected to the vCloud API (Connect-CIServer) with a System administrative user prior to running the script for this to work.

$OrgToUpdate = 'Organisation'
$APIendpoint = 'API Endpoint - api.my.cloud.com'

Function vCloud-REST(
[Parameter(Mandatory=$true)][string]$URI,
[string]$ContentType,
[string]$Method = 'Get',
[string]$ApiVersion = '27.0',
[string]$Body,
[int]$Timeout = 40
)
{
$mysessionid = ($global:DefaultCIServers | Where { $_.Name -eq $APIendpoint }).SessionId
$Headers = @{"x-vcloud-authorization" = $mysessionid; "Accept" = 'application/*+xml;version=' + $ApiVersion}
if (!$ContentType) { Remove-Variable ContentType }
if (!$Body) { Remove-Variable Body }
Try
{
[xml]$response = Invoke-RestMethod -Method $Method -Uri $URI -Headers $headers -Body $Body -ContentType $ContentType -TimeoutSec $Timeout
}
Catch
{
Write-Host "Exception: " $_.Exception.Message
if ( $_.Exception.ItemName ) { Write-Host "Failed Item: " $_.Exception.ItemName }
Write-Host "Exiting."
Return
}
return $response
} # Function vCloud-REST End

# The new vCloud Director API v27.0 OrgRights for Advanced Networking:
$newrights = @{}
$newrights.Add("Organization vDC Distributed Firewall: Enable/Disable", "a100f6a0-2c81-3b61-90c3-c4dbd721b3a8")
$newrights.Add("Organization vDC Gateway: Configure BGP Routing", "2c4eb5ac-15f5-33f0-8b4a-680b3a1d3707")
$newrights.Add("Organization vDC Gateway: Configure DHCP", "be1abe9a-7ddc-38f6-bdf3-94affb01e46b")
$newrights.Add("Organization vDC Gateway: Configure Firewall", "b755b050-772e-3c9c-9197-111c286f563d")
$newrights.Add("Organization vDC Gateway: Configure IPSec VPN", "209cde55-55db-33f1-8357-b27bba6898ed")
$newrights.Add("Organization vDC Gateway: Configure L2 VPN", "eeb2b2a0-33a1-36d4-a121-6547ad992d59")
$newrights.Add("Organization vDC Gateway: Configure Load Balancer", "27be9828-4ce4-353e-8f68-5cd69260d94c")
$newrights.Add("Organization vDC Gateway: Configure NAT", "c9e19573-3d54-3d4a-98f2-f56e446a8ef9")
$newrights.Add("Organization vDC Gateway: Configure OSPF Routing", "3b337aef-42a8-3ed1-8616-341152bc5790")
$newrights.Add("Organization vDC Gateway: Configure Remote Access", "72c5e652-c8d7-3f19-ab83-283d30cb679f")
$newrights.Add("Organization vDC Gateway: Configure SSL VPN", "92b7d500-6bb6-3176-b9eb-d1fda4ce444d")
$newrights.Add("Organization vDC Gateway: Configure Static Routing", "f72af304-97b0-379e-9d6d-68eb89bdc6cf")
$newrights.Add("Organization vDC Gateway: View BGP Routing", "d9dabcab-579e-33c5-807b-dc9232bf7eff")
$newrights.Add("Organization vDC Gateway: View DHCP", "8e16d30d-1ae3-3fff-8d4b-64c342b186a9")
$newrights.Add("Organization vDC Gateway: View Firewall", "7fee6646-ec0c-34c9-9585-aff6f4d92473")
$newrights.Add("Organization vDC Gateway: View IPSec VPN", "82beb471-ab7f-3e2b-a615-136ba6645525")
$newrights.Add("Organization vDC Gateway: View L2 VPN", "105191de-9e29-3495-a917-05fcb5ec1ad0")
$newrights.Add("Organization vDC Gateway: View Load Balancer", "2a097e48-f4c4-3714-8b24-552b2d573754")
$newrights.Add("Organization vDC Gateway: View NAT", "fb860afe-2e15-3ca9-96d8-4435d1447732")
$newrights.Add("Organization vDC Gateway: View OSPF Routing", "eb525145-08e5-3934-91ef-ec80837c9177")
$newrights.Add("Organization vDC Gateway: View Remote Access", "65439584-6aad-3c2c-916f-794099ee85bf")
$newrights.Add("Organization vDC Gateway: View SSL VPN", "cdb0edb0-9623-30a8-89de-b133db7cfeab")
$newrights.Add("Organization vDC Gateway: View Static Routing", "9740be24-4dd7-373c-9237-91896338c11e")

$myendpoint = $global:DefaultCIServers | Where { $_.Name -eq $APIendpoint }

if (!$myendpoint.IsConnected) {
Write-Host "Not connected to this vCloud endpoint, use 'Connect-CIServer' before running this script."
Exit
}

$org = Get-Org -Name $OrgToUpdate -Server $APIendpoint

if (!$org) {
Write-Host "Couldn't match organization with name $OrgToUpdate, exiting."
Exit
}

$rightsuri = 'https://' + $APIendpoint + "/api/admin/org/" + $org.Id.Substring($org.Id.LastIndexOf(':')+1) + "/rights"

[xml]$rights = vCloud-REST -URI $rightsuri -ContentType 'application/vnd.vmware.admin.org.rights+xml' -Method 'Get' -ApiVersion '27.0'

# Add the new API v27 'RightsReference' elements to the XML returned:
foreach($newrule in $newrights.Keys) {
$newright = $rights.CreateElement("RightReference", "http://www.vmware.com/vcloud/v1.5")
$newright.SetAttribute("href","https://$APIEndpoint/api/admin/right/$($newrights.Item($newrule))")
$newright.SetAttribute("name",$newrule)
$newright.SetAttribute("type","application/vnd.vmware.admin.right+xml")
$rights.OrgRights.AppendChild($newright)
}

# Update the Organization with the ammended rights:
vCloud-REST -URI $rightsuri -ContentType 'application/vnd.vmware.admin.org.rights+xml' -Body $rights.InnerXml -Method 'Put' -ApiVersion '27.0'

Using Independent Disks in vCloud

Yesterday I wrote about the PowerShell module I’ve written (CIDisk.psm1) to allow manipulation of independent disks in a vCloud environment. This post shows some usage options and details some of the caveats to be aware of when using disks in this manner.

My test environment has two VMs (named imaginatively ‘vm01’ and ‘vm02’), and the VDC they are in has access to four different storage profiles (‘Platinum’, ‘Gold’, ‘Silver’ and ‘Bronze’ storage). The default storage policy for the VDC is ‘Bronze’, but what if we want to create independent disks on other profiles? The -StorageProfileHref parameter to New-CIDisk lets us do this. Once connected to our cloud (Connect-CIServer) we can find the Hrefs of the available storage profiles we can use:

C:\> $vdc = Get-OrgVdc -Name '<My VDC Name>'

C:\> $vdc.ExtensionData.VdcStorageProfiles.VdcStorageProfile | Select Name, Href

Name                     Href                                                                                       
----                     ----                                                                                       
Platinum Storage Profile https://my.cloud.com/api/vdcStorageProfile/4777f1a3-1f71-4e47-831e-fc7ed80376c3
Silver Storage Profile   https://my.cloud.com/api/vdcStorageProfile/d14c6c2e-2cfb-4ffe-9f31-599c3de42150
Bronze Storage Profile   https://my.cloud.com/api/vdcStorageProfile/dc382284-bc65-4e61-b85f-ea7facba63e4
Gold Storage Profile     https://my.cloud.com/api/vdcStorageProfile/f89df73a-2fa5-40e4-9332-fdd6e29d36ac

Let’s create 2 independent disks, a 10G disk on ‘Platinum’ storage and a 100G disk on ‘Silver’ storage:

C:\> New-CIDisk -DiskName 'disk01-plat' -DiskSize 10G -StorageProfileHref https://my.cloud.com/api/vdcStorageProfile/4777f1a3-1f71-4e47-831e-fc7ed80376c3 -DiskDescription 'Platinum test disk'
Request submitted, waiting for task to complete...
Task completed successfully.

Name        : disk01-plat
Href        : https://my.cloud.com/api/disk/a1b00fa4-3d84-4c75-bc76-7604b96cdcab
Description : Platinum test disk
Size        : 10 GB
BusType     : lsilogicsas
Storage     : Platinum Storage Profile
AttachedTo  : Not Attached

C:\> New-CIDisk -DiskName 'disk02-silv' -DiskSize 100G -StorageProfileHref https://my.cloud.com/api/vdcStorageProfile/d14c6c2e-2cfb-4ffe-9f31-599c3de42150 -DiskDescription 'Silver test disk'
Request submitted, waiting for task to complete...
Task completed successfully.

Name        : disk02-silv
Href        : https://my.cloud.com/api/disk/b02b50fd-1305-454d-923c-26ef9874a3fd
Description : Silver test disk
Size        : 100 GB
BusType     : lsilogicsas
Storage     : Silver Storage Profile
AttachedTo  : Not Attached

We can see in the vCloud interface that these disks now exist in our VDC (Note: you may have to completely refresh your vCloud session using your browser’s refresh before the ‘Independent Disks’ tab appears):

There are no context actions for these disks though and we can’t attach/detach them to VMs in the vCloud interface.

Our VM01 virtual machine currently has a 40GB base disk attached and no other storage:

 

We can mount both our new independent disks to this VM using the following:

C:\> $vm01 = Get-CIVM -Name 'vm01'

C:\> $disk01 = Get-CIDisk -DiskName 'disk01-plat'

C:\> $disk02 = Get-CIDisk -DiskName 'disk02-silv'

C:\> Mount-CIDisk -VMHref $vm01.Href -DiskHref $disk01.Href
Request submitted, waiting for task to complete...
Task completed successfully.

C:\> Mount-CIDisk -VMHref $vm01.Href -DiskHref $disk02.Href
Request submitted, waiting for task to complete...
Task completed successfully.

Looking at the VM01 Hardware tab following this shows both disks mounted:

Note again that no manipulation options are available in the vCloud UI, but at least it’s obvious that independent disks have been attached to VM01.

After rescanning storage in the guest, we can see the new storage devices on VM01:

And once these are brought online, initialized, storage volumes created and drive letters assigned, we can use the disks inside the guest (the volume names don’t get automatically mapped – I’ve just named the volumes the same as the independent disk objects for consistency):

At this point everything appears to be working fine, but there can be a catch here – if you restart the virtual machine you may find that the server attempts to boot from one of the newly mounted independent disks. Luckily vCloud Director 8.10 allows us to get into the VM BIOS and change the boot order settings:

Once restarted into BIOS we can select the correct boot order:

With the server restarted, we can create some test content in ‘disk01-plat’ to prove that the data moves when we reattach this disk to VM02:

And to dismount ‘disk01-plat’ from VM01 and mount it to VM02 we can:

C:\> Dismount-CIDisk -VMHref $vm01.Href -DiskHref $disk01.Href
Request submitted, waiting for task to complete...
Task completed successfully.

C:\> $vm02 = Get-CIVM -Name 'vm02'

C:\> Mount-CIDisk -VMHref $vm02.Href -DiskHref $disk01.Href
Request submitted, waiting for task to complete...
Task completed successfully.

Looking at the available storage in VM02 after a disk rescan shows our disk has transfered across:

Finally, checking the contents of the ‘E:\’ drive shows our test folder & file have made it across:

And Get-CIDisk can be used to verify the disk attachments after moving disk01 to VM02:

C:\> Get-CIDisk | ft -AutoSize

Name        Href                                                               Description        Size   BusType     Storage                  AttachedTo
----        ----                                                               -----------        ----   -------     -------                  ----------
disk01-plat https://my.cloud.com/api/disk/a1b00fa4-3d84-4c75-bc76-7604b96cdcab Platinum test disk 10 GB  lsilogicsas Platinum Storage Profile vm02      
disk02-silv https://my.cloud.com/api/disk/b02b50fd-1305-454d-923c-26ef9874a3fd Silver test disk   100 GB lsilogicsas Silver Storage Profile   vm01

Hopefully this gives a better idea of how CIDisk can be used to manage independent disks in a vCloud environment, it would be nice if VMware included the management functions in the UI, but for now at least you can use PowerShell to easily achieve the same results without having to write against the API directly.

As always, any comments / feedback greatly appreciated.

Jon

Independent Disks in vCloud via PowerCLI

Another day, another customer requirement which I figured ‘this will be easy’ and turned out not to be quite so easy…

The customer in question is a tenant on our cloud platform and has built a VM to be their offline root Certificate Authority (CA). In line with their security practice, this VM has no network connectivity and is usually powered-off in their environment unless specifically required to issue or renew certificates.

They asked if there was an easy way to transfer certificate files issued by this VM to other servers in their infrastructure. In their (old) vSphere environment they would simply attach a new temporary virtual disk to the VM, copy the certificate files over and then attach the disk to the destination VM. Surely there had to be some similar functionality in vCloud Director?

Well, there’s a bit of good and bad news on that…

By default disks in vCloud Director are assigned (permanently) to a VM, they can’t be moved to different VMs. (That’s the bad news). The good news is that vCD supports ‘independent disks’ which can be moved between VMs. The bad news is that this is an API-only operation (nothing in the web UI allows creation or manipulation of Independent disks, although you can see them if they exist). The worst news is that VMware PowerCLI even in the latest 6.5R1 version doesn’t have any cmdlets to manipulate independent disks attached to vCloud VMs either.

So while I could have hacked something together to run directly against the vCloud Director REST API for this customer, I figured it would be better to have some reusable PowerShell cmdlets for this. So I set about writing some and I’m pleased to announce the first release of ‘CIDisk’, a collection of PowerShell cmdlets to manipulate independent disks in vCloud Director environments.

The module code, documentation and examples are now available on my github at https://github.com/jondwaite/cidisk

I’ll do a followup post detailing some more advanced options and scenarios in the next day or two.

Edit – Followup post is now available here.

As always I appreciate any/all feedback and hope someone else finds these useful.

Jon

Detailed VM Storage Information in vCloud Director

I recently had a request from one of our customers who wanted an easy / scriptable method to determine the storage allocations on their hosted VMs in our vCloud platform, preferably from PowerShell. That should be easy I thought and set about my usual Google-based research. I initially found this post from Alan Renouf which I forwarded back to the client.

Unfortunately, while this achieved part of the answer, this particular customer had a number of VMs which had hard disks attached using multiple/different storage profiles and they wanted to get the details of these too. So I set about writing some code to see if I could get full storage information about the VM and all of its disks. I ended up having to access the vCloud REST API directly for this information but it wasn’t too bad.

First, I created a ‘worst-case’ test VM where the 3 attached hard disks which were created one each on our ‘Gold’, ‘Silver’ and ‘Bronze’ storage policies:

test02-hardware-properties

(Just to make sure everything would work I also created the 3 disks on 3 different storage Bus Types). I also set the VM storage policy to something different:

test02-general-properties

My first step was a function to access the vCloud REST API, I found this post from Matt Vogt’s blog which had some code for this which I shamelessly borrowed (hey, why reinvent the wheel unless you need to):

function Get-vCloudREST($href) {
  $request = [System.Net.HttpWebRequest]::Create($href)
  $request.Accept = "application/*+xml;version=20.0" # For vCloud Director 8.xx or later API
  $request.Headers.add("x-vcloud-authorization",$global:session.sessionID)
  $response = $request.GetResponse()
  $streamReader = new-object System.IO.StreamReader($response.getResponseStream())
  $xmldata = $streamReader.ReadToEnd()
  $streamReader.Close()
  $response.Close()
  return $xmldata
}

The return from the Get-CIVM cmdlet includes a reference to the VM object within the vCloud API:

PowerCLI C:\> $test02 = Get-CIVM -Name 'test02'
PowerCLI C:\> $test02 | fl
ExtensionData   : VMware.VimAutomation.Cloud.Views.Vm
Status          : PoweredOff
Deleted         : False
GuestOsFullName : Other Linux (64-bit)
CpuCount        : 1
MemoryMB        : 384
MemoryGB        : 0.375
VMVersion       : v11
Org             : MyOrg
OrgVdc          : MyOrgVDC
VApp            : test02
Description     : 
Href            : https://<cloud.com>/api/vApp/vm-ef4a4594-c631-421a-98f0-4d15670c98ad
Id              : urn:vcloud:vm:ef4a4594-c631-421a-98f0-4d15670c98ad
Name            : test02
Client          : /CIServer=<user>:<org>@<cloud.com>:443/
Uid             : /CIServer=<user>:<org>@<cloud.com>:443/CIVM=urn:vcloud:vm:ef4a4594-c631-421a-98f0-4d15670c98ad/

Using this we can obtain our disk information:

PowerCLI C:\> $queryHref = $test02.Href + "/virtualHardwareSection/disks"
PowerCLI C:\> [xml]$xml = Get-vCloudREST($queryHref)
PowerCLI C:\> $xml
xml                            RasdItemsList
---                            -------------
version="1.0" encoding="UTF-8" RasdItemsList

Filtering the returned RasdItemsList for a ResourceType of 17 (Hard Disk), we can get a list of attached hard disks:

PowerCLI C:\> $xml.RasdItemsList.Item | Where { $_.ResourceType -eq 17 }

AddressOnParent      : 0
Description          : Hard disk
ElementName          : Hard disk 1
HostResource         : HostResource
InstanceID           : 2000
Parent               : 2
ResourceType         : 17
VirtualQuantity      : 8589934592
VirtualQuantityUnits : byte

AddressOnParent      : 0
Description          : Hard disk
ElementName          : Hard disk 2
HostResource         : HostResource
InstanceID           : 2016
Parent               : 3
ResourceType         : 17
VirtualQuantity      : 8589934592
VirtualQuantityUnits : byte

AddressOnParent      : 0
Description          : Hard disk
ElementName          : Hard disk 3
HostResource         : HostResource
InstanceID           : 16060
Parent               : 4
ResourceType         : 17
VirtualQuantity      : 8589934592
VirtualQuantityUnits : byte

So this gets us to a point where we have all of the hard disk information, but how do we find the storage policy for each disk? It turns out that each disk has an attribute ‘HostResource’ which provides the URI to the storage policy from which the disk has been allocated:

PowerCLI C:\> $disks = $xml.RasdItemsList.Item | Where { $_.ResourceType -eq 17 }

PowerCLI C:\> $disks.Count
3

PowerCLI C:\> $disks[0].HostResource

vcloud                          : http://www.vmware.com/vcloud/v1.5
storageProfileHref              : https://<my cloud URI>/api/vdcStorageProfile/f89df73a-2fa5-40e4-9332-fdd6e29d36ac
busType                         : 6
busSubType                      : lsilogic
capacity                        : 8192
iops                            : 0
storageProfileOverrideVmDefault : true

PowerCLI C:\> $disks[1].HostResource

vcloud                          : http://www.vmware.com/vcloud/v1.5
storageProfileHref              : https:// <my cloud URI>/api/vdcStorageProfile/d14c6c2e-2cfb-4ffe-9f31-599c3de42150
busType                         : 6
busSubType                      : lsilogicsas
capacity                        : 8192
iops                            : 0
storageProfileOverrideVmDefault : true

PowerCLI C:\> $disks[2].HostResource

vcloud                          : http://www.vmware.com/vcloud/v1.5
storageProfileHref              : https:// <my cloud URI>/api/vdcStorageProfile/dc382284-bc65-4e61-b85f-ea7facba63e4
busType                         : 20
busSubType                      : vmware.sata.ahci
capacity                        : 8192
iops                            : 0
storageProfileOverrideVmDefault : true

So how can we convert the storageProfileHref values into meaningful (human readable) storage profile names? We can use another API call to establish the name of each vdcStorageProfile:

PowerCLI C:\> [xml]$sp = Get-vCloudREST($disks[0].HostResource.storageProfileHref)
PowerCLI C:\> $sp.VdcStorageProfile.name
Gold Storage Profile

PowerCLI C:\> [xml]$sp = Get-vCloudREST($disks[1].HostResource.storageProfileHref)
PowerCLI C:\> $sp.VdcStorageProfile.name
Silver Storage Profile

PowerCLI C:\> [xml]$sp = Get-vCloudREST($disks[2].HostResource.storageProfileHref)
PowerCLI C:\> $sp.VdcStorageProfile.name
Bronze Storage Profile

Querying the API for every vdcStorageProfile for every disk is going to generate a lot of calls for any significant number of VMs, so in the code below I’ve added a hash stored in a global variable which caches these results so that any storageProfileHref which has been seen before doesn’t need to generate an additional API call.

Putting it all together

So we now have a way of determining all of the information we need, using PowerShell custom objects allows us to write a function which returns all of our VM and storage details in a easily consumable form for further processing.

The script included at the bottom of this article produces the following output for my test environment containing 2 VMs of which the ‘pxetest01’ VM has no disks attached:

PowerCLI C:\> $vmobjs

VMName    : pxetest01
Id        : urn:vcloud:vm:45a86f24-1109-4ead-881d-f865c1f0692f
Status    : PoweredOff
vRAM      : 1
vCPU      : 1
HWVersion : v11
vAppName  : PXE Demo
GuestOS   : CentOS 4/5/6/7 (64-bit)
Disks     :

VMName    : test02
Id        : urn:vcloud:vm:ef4a4594-c631-421a-98f0-4d15670c98ad
Status    : PoweredOff
vRAM      : 0.375
vCPU      : 1
HWVersion : v11
vAppName  : test02
GuestOS   : Other Linux (64-bit)
Disks     : {@{Name=Hard disk 1; StorageProfile=Gold Storage Profile; Quantity=8589934592; QuantityUnits=byte}, @{Name=Hard disk 2; StorageProfile=Silver Storage Profile; Quantity=8589934592; QuantityUnits=byte}, @{Name=Hard disk 3; StorageProfile=BronzeStorage Profile; Quantity=8589934592; QuantityUnits=byte}}

It can also return just the disk information as another custom object:

PowerCLI C:\> $test02vm = $vmobjs | Where {$_.VMName -eq 'test02'}
PowerCLI C:\> $test02vm.Disks

Name StorageProfile Quantity QuantityUnits
---- -------------- -------- -------------
Hard disk 1 Gold Storage Profile 8589934592 byte
Hard disk 2 Silver Storage Profile 8589934592 byte
Hard disk 3 Bronze Storage Profile 8589934592 byte

And we can check the number of disks attached to any VM:

PowerCLI C:\> ($test02vm.Disks | Measure).Count
3

Finally because the output is a PowerShell object, we can easily turn this custom object into JSON for use in further processing:

PowerCLI C:\> $test02vm | ConvertTo-Json

{
	"VMName": "test02",
	"Id": "urn:vcloud:vm:ef4a4594-c631-421a-98f0-4d15670c98ad",
	"Status": "PoweredOff",
	"vRAM": 0.375,
	"vCPU": 1,
	"HWVersion": "v11",
	"vAppName": "test02",
	"GuestOS": "Other Linux (64-bit)",
	"Disks": {
		"value": [
			{
				"Name": "Hard disk 1",
				"StorageProfile": "Gold Storage Profile",
				"Quantity": 8589934592,
				"QuantityUnits": "byte"
			},
			{
				"Name": "Hard disk 2",
				"StorageProfile": "Silver Storage Profile",
				"Quantity": 8589934592,
				"QuantityUnits": "byte"
			},
			{
				"Name": "Hard disk 3",
				"StorageProfile": "Bronze Storage Profile",
				"Quantity": 8589934592,
				"QuantityUnits": "byte"
			}
		],
		"Count": 3
	}
}

Hopefully you’ve found this post useful, let me know in the comments if you have any issues or would like to see more examples like this.

Jon.

Full script to find storage policy information for vCloud VMs using the vCloud REST API:

$global:spolrefs = @{}
# Change to be appropriate for your scenario:
$cloudURL = "<cloud FQDN or IP Address>"
$cloudOrg = "<cloud Organization name>"

# Check if we are already logged-in (interactive session) and if not, prompt for login:
if (!$session.IsConnected) {
 $creds = Get-Credential -Message "Authenticate to $cloudOrg Cloud Service"
 $session = Connect-CIServer -Server $cloudURL -Org $cloudOrg -Credential $creds
}

# PS Function to query the vCloud Director REST API
# ('Borrowed' from Matt Vogt's blog at: http://blog.mattvogt.net/)
function Get-vCloudREST($href)
{
 $request = [System.Net.HttpWebRequest]::Create($href)
 $request.Accept = "application/*+xml;version=20.0" # For vCloud Director 8.xx or later API
 $request.Headers.add("x-vcloud-authorization",$global:session.sessionID)
 $response = $request.GetResponse()
 $streamReader = new-object System.IO.StreamReader($response.getResponseStream())
 $xmldata = $streamReader.ReadToEnd()
 $streamReader.Close()
 $response.Close()
 return $xmldata
}

# Function to get Storage Profile name from its Href (if not already known/cached in a global hash)
# if already known we don't need to call the API again and can just return the value from the hash
function Get-SPName($storage_href)
{
 if ($global:spolrefs.ContainsKey($storage_href)) {
 return $global:spolrefs.Get_Item($storage_href)
 } else {
 [xml]$sp = Get-vCloudREST($storage_href)
 $global:spolrefs.Add($storage_href, $sp.VdcStorageProfile.name)
 return $sp.VdcStorageProfile.name
 }
}

# Function to retrieve the disk information for a VM:
function Get-VMDisks($VM)
{
 $VMDisks = @() # Start with an empty array
 $queryHref = $VM.Href + "/virtualHardwareSection/disks" # API path for VM disk information
 [xml]$xml = Get-vCloudRESt($queryHref)
 $disks = $xml.RasdItemsList.Item | Where-Object {$_.ResourceType -eq 17} # Resource Type 17 = Hard disk drive
 foreach ($disk in $disks)
 {
 $sp = Get-SPName($disk.HostResource.storageProfileHref)
 $diskobj = New-Object -TypeName PSObject
 $diskobj | Add-Member -Type NoteProperty -Name "Name" -Value ([String]$disk.ElementName)
 $diskobj | Add-Member -Type NoteProperty -Name "StorageProfile" -Value ([String]$sp)
 $diskobj | Add-Member -Type NoteProperty -Name "Quantity" -Value ([Int64]$disk.VirtualQuantity)
 $diskobj | Add-Member -Type NoteProperty -Name "QuantityUnits" -Value ([String]$disk.VirtualQuantityUnits)
 $VMDisks += $diskobj
 }
 return $VMDisks
}

function Get-VMInfo($VM)
{
 $vmobj = New-Object -TypeName PSObject
 $vmobj | Add-Member -Type NoteProperty -Name "VMName" -Value ([String]$VM.Name)
 $vmobj | Add-Member -Type NoteProperty -Name "Id" -Value ([String]$VM.ExtensionData.Id)
 $vmobj | Add-Member -Type NoteProperty -Name "Status" -Value ([String]$VM.Status)
 $vmobj | Add-Member -Type NoteProperty -Name "vRAM" -Value ([Decimal]$VM.MemoryGB)
 $vmobj | Add-Member -Type NoteProperty -Name "vCPU" -Value ([Int]$VM.CpuCount)
 $vmobj | Add-Member -Type NoteProperty -Name "HWVersion" -Value ([String]$VM.VMVersion)
 $vmobj | Add-Member -Type NoteProperty -Name "vAppName" -Value ([String]$VM.VApp.Name)
 $vmobj | Add-Member -Type NoteProperty -Name "GuestOS" -Value ([String]$VM.GuestOsFullName)
 [PSCustomObject]$disks = Get-VMDisks($VM)
 $vmobj | Add-Member -Type NoteProperty -Name "Disks" -Value $disks
 return $vmobj
}

# Get all VMs in our Cloud Organization:
$vms = Get-CIVM

# Initialise an array for returned objects:
$vmobjs = @()

# For each VM, build our object:
foreach ($vm in $vms) {
 $vmobjs += Get-VMinfo($vm)
}

# Output our object to console:
$vmobjs

 

Live import VMs to vCloud Director

Tom Fojta wrote a great blog post about the new capability in vCloud Director 8.10 to import running VMs into vCloud Director. This is a huge asset in migration scenarios where customers can’t afford outages when being migrated into the vCD environment. Unfortunately the API syntax to actually initiate the import is a little convoluted and not the easiest process to manage.

I set about writing a PowerShell script to significantly simplify the process of initiating a live-import operation. The script itself is available from github at the following link: https://github.com/jondwaite/vcdliveimport.

The liveimport.ps1 script contained in this repository does the following:

  • Prompts for a credential to be used to connect to both vCloud Director (System context) and vCenter – if you have different usernames/passwords for each you’ll need to adjust this.
  • Enumerates the available vCenter instances registered as Provider Virtual Datacenters (PVDCs) in vCloud Director and allows one to be selected as the source vCenter for the migration.
  • Lists the available VMs in the selected vCenter instance, filters this list based on selectable criteria (e.g. don’t offer to import ‘Guest Introspection’ VMs) and allows the source VM to be selected.
  • Lists available destination Virtual Datacenters (VDCs) in the vCloud Director environment and allows the destination VDC to be selected.
  • Displays the appropriate POST request information to be submitted to vCloud Director to initiate the live-import of this VM.
  • Optionally – Submits the REST API request directly to the vCloud Director environment to actually initiate the import process.

An example transcript of this process is show below. Hopefully this helps someone else out and helps to make it easier for you to live-import running VMs into vCloud Director.

Jon.

Example Session Transcript:

PowerCLI C:\> liveimport.ps1
Connected to vCloud Director OK
vCenter(s) Found:
-----------------
vc01 (https://vcd01.dev.local/api/admin/extension/vimServer/158f73ec-a999-4332-8250-f4dd5e6c4971)

Selecting vc01 as only pVDC vCenter found.
vCenter vc01 selected.
Connected to vCenter Server vc01 OK

Evaluating vCenter VMs as migration candidates.............
Available candidate VMs:
------------------------
testvm01

Enter VM name to live-migrate to vCloud (or 'quit' to exit): testvm01
Selected VM testvm01 for live import to vCloud.

Available VDCs:
---------------
Lab VDC
Enter Destination VDC Name (or quit to exit): Lab VDC
VDC 'Lab VDC' selected.

URI for POST operation:
https://vcd01.dev.local/api/admin/extension/vimServer/158f73ec-a999-4332-8250-f4dd5e6c4971/importVmAsVApp

XML Document Body:
<?xml version="1.0" encoding="UTF-8"?>
<ImportVmAsVAppParams xmlns="http://www.vmware.com/vcloud/extension/v1.5" name="testvm01" sourceMove="true">
 <VmMoRef>vm-111</VmMoRef>
 <Vdc href="https://vcd01.dev.local/api/admin/vdc/b21c92fa-1a6f-42a9-8c1e-ef0947c7be76" />
</ImportVmAsVAppParams>

Content-Type: application/vnd.vmware.admin.importVmAsVAppParams+xml

Would you like to submit this API request to live import this VM? (y or n) (or quit to exit): y
Making request to live import VM testvm01...
Response was:
<?xml version="1.0" encoding="UTF-8"?>
<VApp xmlns="http://www.vmware.com/vcloud/v1.5" ovfDescriptorUploaded="true" deployed="false" status="0" name="testvm01" id="urn:vcloud:vapp:ff1a9021-2c21-4378-99c7-b77bdaf3e9e6" href="https://vcd01.ndev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6" type="application/vnd.vmware.vcloud.vApp+xml" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.vmware.com/vcloud/v1.5 http://vcd01.dev.local/api/v1.5/schema/master.xsd">
 <Link rel="down" href="https://vcd01.dev.local/api/network/251f72a8-19b5-4e7d-94ba-0523ee919cd9" name="VM Network" type="application/vnd.vmware.vcloud.vAppNetwork+xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/controlAccess/" type="application/vnd.vmware.vcloud.controlAccess+xml" />
 <Link rel="up" href="https://vcd01.dev.local/api/vdc/b21c92fa-1a6f-42a9-8c1e-ef0947c7be76" type="application/vnd.vmware.vcloud.vdc+xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/owner" type="application/vnd.vmware.vcloud.owner+xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/metadata" type="application/vnd.vmware.vcloud.metadata+xml" />
 <Link rel="ovf" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/ovf" type="text/xml" />
 <Link rel="down" href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6/productSections/" type="application/vnd.vmware.vcloud.productSections+xml" />
 <Tasks>
 <Task cancelRequested="false" expiryTime="2016-10-31T11:27:25.312+13:00" operation="Importing Virtual Application testvm01(ff1a9021-2c21-4378-99c7-b77bdaf3e9e6)" 
operationName="importSingletonVapp" serviceNamespace="com.vmware.vcloud" startTime="2016-08-02T11:27:25.312+12:00" status="queued" name="task" id="urn:vcloud:task:37576f8c-0d39-444c-8842-81436ddb421e" href="https://vcd01.dev.local/api/task/37576f8c-0d39-444c-8842-81436ddb421e" type="application/vnd.vmware.vcloud.task+xml">
 <Owner href="https://vcd01.dev.local/api/vApp/vapp-ff1a9021-2c21-4378-99c7-b77bdaf3e9e6" name="testvm01" type="application/vnd.vmware.vcloud.vApp+xml" />
 <User href="https://vcd01.dev.local/api/admin/user/fa7fb40e-648c-4787-ad0f-711dcc315261" name="system" type="application/vnd.vmware.admin.user+xml" />
<Organization href="https://vcd01.dev.local/api/org/3f46b162-f794-4d9e-8fa1-6ca1ee7b6377" name="Lab" type="application/vnd.vmware.vcloud.org+xml" />
 <Progress>1</Progress>
 <Details />
 </Task>
 </Tasks>
 <DateCreated>2016-08-02T11:27:25.079+12:00</DateCreated>
 <Owner type="application/vnd.vmware.vcloud.owner+xml">
 <User href="https://vcd01.dev.local/api/admin/user/1545cb33-9151-43e2-a156-9d20e3b966c0" name="system" type="application/vnd.vmware.admin.user+xml" />
 </Owner>
 <InMaintenanceMode>false</InMaintenanceMode>
</VApp>

 

Working with the vCloud API and PHP SDK – Part 1

Introduction

While I love using the PowerCLI tools for manipulating vCloud Director, sometimes you need to perform actions that require hitting the API directly. Tools such as the RESTClient plugin for Firefox, cURL and HTTPie from the command line are good for interactive manipulation of the API, but what if you want to automate these API interactions?

Fortunately, rather than having to reinvent the wheel, VMware publish a variety of SDK’s (currently for PHP, Java and Microsoft .NET) which make this (relatively) straightforward, although sadly the documentation for these are lacking in basic configuration information which makes actually using them more problematic than it should be. They are still a better alternative than writing code directly against the HTTP API where you have to deal with decoding and re-encoding the XML objects used by the vCloud API itself directly.

In this series I’ll concentrate on the VMware PHP SDK for vCloud Director (PHP SDK). This first post will cover how to install and configure it. Once we have a working environment configured the following articles in this series will detail some (hopefully) useful scripts which use the PHP SDK to perform automation tasks against the vCloud API.

Quick note on VMware PHP SDK versions

The link to download the PHP SDK for vCloud is on VMware’s site here, however if you work for a VMware Service Provider and have access to the vCloud Director for Service Providers code you’ll find a more recent version as follows:

Log in to your ‘My VMware‘ account and select the ‘View & Download Products’ section. From the list select the ‘Download’ link against the ‘vCloud Director for Service Providers’ product.

Next select the ‘Drivers & Tools’ tab and expand the ‘Automation Tools and SDK(s)’ section, then select the Download link against the ‘VMware vCloud SDKs for Service Providers’ item. This will take you to a page where you can download a later (v8.0.0 build 3010704 currently) version of the PHP SDK. (The direct link for this is here, but I believe will only work if you have a valid Service Provider login for My VMware).

Download the .zip or .tar.gz version of the PHP SDK that suits your development environment, as far as I can tell the contents is identical between both versions so ease of unarchiving is the only difference. For the purposes of this series of posts it shouldn’t matter which version of the SDK (5.5 or 8.0) you use.

None of this makes sense to me – I know the vCloud Director product past version 5.5 is a service provider only offering, but since clients of vCloud Powered service providers are just as entitled to use the API as service providers then surely VMware should make the latest SDK available to everyone?

Configuring your devlopment environment

There are 5 components involved in setting up a functioning development environment with the vCloud PHP SDK:

  • A working PHP installation (No web server required).
  • The PEAR modules ‘HTTP2_Request’ and ‘Log’.
  • The PHP extensions ‘openssl’ and ‘mbstring’ added and enabled.
  • The VMware downloaded PHP SDK files.
  • A Configuration.ini file to control SDK logging.

Unfortunately only the first 2 of these are (partially) covered in the VMware documentation. The following sections detail how to configure a working development environment for all of these.

The documentation included on VMware’s site for the PHP SDK download mentions that the ‘Pear HTTP_Request2’ package is required to use the PHP SDK, unfortunately it doesn’t mention that 2 additional extensions are also required (PEAR Log and the PHP mbstring). Without these additional packages you will either not be able to use the SDK at all, or receive strange error messages.

PHP installation on Linux

If you are using a Linux platform with a package manager you will usually find a packaged PHP distribution available. On most CentOS systems this can be installed with:

$ sudo yum install php

On other Linux distributions the commands will vary so check the documentation for your particular environment, the PHP Documentation has good installation instructions for a variety of platforms. You will also require Pear (PHP Extension and Application Repository framework) in order to be able to install the support packages required by the SDK. Again on CentOS this can be installed with:

$ sudo yum install php-pear

Adding the required PHP extensions on Linux

Your Linux distribution should have an available package to install the mbstring (Multibyte Character support) extension for PHP, e.g. for CentOS Linux:

$ sudo yum install php-mbstring

To install the PEAR modules required for the vCloud PHP SDK:

$sudo pear install HTTP_Request2 Log

PHP Installation on Windows

On Windows systems I would strongly advise using 64-bit Windows and the PHP version 7 releases from http://windows.php.net/download as this will support native PHP 64-bit integers on 64-bit platforms. This can be useful dealing with disk capacities (usually expressed in bytes) within the SDK which could overflow a 32-bit integer. (Note that even the 64-bit versions of PHP v5.x do not support 64-bit integers). For v7 on Windows you will also need to install the Visual C++ Redistributable for Visual Studio 2015 from Microsoft if you don’t already have it installed – make sure you install the version appropriate to your PHP environment (32-bit/64-bit).

Extract the downloaded PHP .zip file into a new folder (e.g. C:\PHP) and install the Visual C++ Redistributable appropriate to your version if needed.

Copy the ‘php.ini-development’ file included in the download in this folder to ‘php.ini’ – this will serve as the configuration point for your installation and will be used to enable extensions.

To install the PHP Pear extension on Windows, go to https://pear.php.net/manual/en/installation.getting.php and download the ‘go-pear.phar’ into the same folder you extracted PHP into. Install it from a command prompt opened into the same folder you extracted PHP into using:

php go-pear.phar

Note: if you are not using an ‘administrator’ command prompt you will need to change the default path for ‘pear.ini’ to something other than C:\Windows – I just placed it in the PHP directory (C:\PHP\pear.ini).

You will need to double-click the ‘PEAR_ENV.reg’ file to add appropriate environment variables for PEAR to your user account to the Windows registry. You should also add the directory you extracted PHP into to your Windows PATH – this is under Windows ‘Advanced System Settings’. Note that you will need to re-open a command prompt for PATH changes to take effect.

Adding required PHP extensions on Windows

To install the ‘HTTP_Request2’ and ‘Log’ modules from a Windows Command prompt type:

pear install HTTP_Request2 Log

The mbstring (multibyte) extension is usually present (in the ‘ext’ subdirectory of your PHP installation) as ‘php_mbstring.dll’ on Windows, but not enabled by default. The same is true for the openssl extension. To enable these extensions edit your php.ini file and find the lines:

;extension=php_mbstring.dll
:::
;extension=php_openssl.dll

And remove the leading semi-colons (;) then save the file. You can test whether the modules are loaded correctly by typing ‘php -m’ and checking that ‘mbstring’ and ‘openssl’ are listed in the output in the [PHP Modules] section.

VMware PHP SDK Files

The PHP SDK itself can be extracted to any convenient folder – on Linux I’d suggest under your home directory and on Windows pick somewhere easy to locate (e.g. C:\PHPSDK). Ensure that the folder structure remains intact – you should have 3 subdirectories in the extracted SDK named ‘docs’, ‘library’ and ‘samples’. The essential one (and only one required to actually use the SDK) is the ‘library’ folder.

Configuration.ini (pear/Log configuration file)

The Vmware documentation makes no mention of it, but the ‘ServiceAbstract.php’ file included in the library file (/library/VMware/VCloud/ServiceAbstract.php) relies on both the Pear Log module (which we installed) and a text file ‘Configuration.ini’ which is read at various points in the file to determine logging options for API interactions.

This is very useful (once you know about it), but the file itself is not supplied with the PHP SDK download and must be manually created. To do this, create a new text file in the folder where you will be working with the SDK (e.g. C:\PHPSDK) named ‘Configuration.ini’ and specify the contents as:

[log_section]
log_handler_name=file
log_file_location=phpsdk.log
log_level=PEAR_LOG_DEBUG

This will log all API interactions to a text file (phpsdk.log) in the current directory and is incredibly useful for troubleshooting – you can control the verbosity of the logging by changing the log_level parameter (as well as the log filename and type using the other options). For full documentation on available options and values see the Pear Log documentation.

config.ini (VMware PHP SDK configuration file)

The last piece of configuration required is to copy the file ‘config.ini’ from the PHP SDK ‘samples’ folder into the location where you will be developing your PHP code. You should edit the copied file and ensure that the section that reads:

set_include_path(implode(PATH_SEPARATOR, array('.','../library',get_include_path(),)));

Is updated to refer to the location where your PHP SDK ‘library’ folder exists. For example, on Windows if the PHP SDK is extracted to C:\PHPSDK then this line should be updated to read:

set_include_path(implode(PATH_SEPARATOR, array('.','C:\PHPSDK\library',get_include_path(),)));

To check whether everything is working, try creating a file ‘test.php’ in the root of your working directory (where you’ve edited config.php and saved Configuration.ini)

<?php
  include './config.php';
?>

If you’ve configured everything correctly then running this from a command prompt php test.php should return with no output or errors.

Note: If you are using PHP 7 on Windows you may get a warning from the Log.php file included by the SDK which looks similar to:

PHP Deprecated: Methods with the same name as their class will not be constructors in a future version of PHP; Log has a deprecated constructor in C:\PHP\pear\Log.php on line 38

This is harmless and safe to ignore, but if it annoys you and you want to prevent it being displayed you can add a new line to the Log.php file specified in the warning message as below just prior to the ‘public static function factory($handler, $name = ”, $ident = ”,…) line:

public function __construct(){}

You will proabbly also need to add this line after the ‘class’ definition in the pear/Log/file.php file.

In the next parts of this series I’ll be using this environment to show you how to achieve some useful interactions with a vCloud platform using the Perl SDK, I’ll update this post with links once these are posted. As always, please leave any feedback in the comments, I try to answer as much as I can.

Jon.