vCloud Director 9 HTML5 Portal Customization

One of the great features in vCloud Director 9 which has been further enhanced in the latest v9.5 release is the new HTML5 portal:

image

Even better, VMware has released a toolkit to allow Service Providers to fully customise the look and feel of the portal using CSS themes in their Clarity framework..

The toolkit itself is part of the VMware vcd-ext-sdk repository on github, available in the /ui/theme-generator folder.

The repository has good instructions on how to modify and build a custom theme, but actually uploading and configuring the theme in vCloud Director is only accessible via the vCD API and involves a reasonable amount of manual work.

To help speed up development and allow changes to be easily tested, in my usual mode I’ve written a small PowerShell module that allows quicker/easier theme configuration. The module is available on github at https://github.com/jondwaite/vcd-h5-themes. Hopefully this will help those of you who need to develop and test updated themes for your vCloud Director portals.

I’ve included documentation in the repository on each cmdlet, its function and arguments here.

To use the module you’ll need to be connected to a vCloud instance as a user with global ‘Administrator’ access in the ‘System’ organization since changes will affect all portal users. You’ll need to be connected to the vCD environment with PowerCLI (Connect-CIServer…) prior to using the cmdlets.

You can then download the vcd-ht-themes.psm1 file and add it to your session (‘Import-Module vcd-h5-themes.psm1’) to access the cmdlets.

As always, comments and feedback welcome – is there anything else you’d like to see added to this module?

Jon.

Writing vRealize Orchestrator Workflows for vCloud Director v9.1

One of the great new features in vCloud Director 9.1 is the ability to publish Orchestrator workflows to tenants which can be consumed from within their vCloud portal. Markus Kraus has written an excellent post showing the configuration process for linking Orchestrator and vCloud Director. This post shows the process for creating and deploying a workflow into this environment and shows the user experience when invoking the finished workflow. I can see a large range of possible use-cases for this capability – hopefully this post will give you an idea of what is possible.

Our demonstration workflow will be reasonably simple and cover a scenario where a tenant consuming an Allocated VDC needs more resources assigned to it. While this could be fully automated (directly make the changes to the vCloud Director VDC), it is probably more likely in these scenarios that the service provider would want to check and implement the change themselves. Therefore the following actions are required. The workflow tasks are therefore:

  • Extract the environment details (so we know which tenant user in which vCloud Organisation initiated the request.
  • Allow the tenant to enter the parameters for the extra resources they require.
  • Send an email to the service provider that contains the request details.
  • Send an email to the tenant user confirming the request details.

This example assumes that you have a functioning vRealize Orchestrator instance in the vCloud Director environment, and that you’ve registered the vCloud Director endpoint in Orchestrator. It also requires that you’ve configured the Orchestrator integration in the vCD provider portal and granted the new Service Library permissions to the tenant Organization Administrator role.

From the Orchestrator client, we first create a new workflow in Design view (I’ve created a folder to contain this called ‘vCD Workflows’ too), I’ve called my workflow ‘Request VDC Resources’:

image

The new (empty) workflow is created and displayed in the vRO editor:

image

Our first task is to create some workflow inputs to capture required inputs, these can be created in the ‘Inputs’ tab using the ‘Add parameter’ button:

image

We need the following information from the user to be able to process this workflow, so I’ve created the parameters, given them the correct Type and set a description for each one. In my lab environment there are two classes of storage profiles (performance and capacity) so I’ll ask the user to provide values for both if requesting a storage increase. To change the parameter names, types and descriptions just click in the fields. The final ‘Inputs’ tab now looks like this:

image

Now we can alter the Presentation tab to group the input fields appropriately. It can be a bit fiddly to add groups (Orchestrator adds steps each time which need to be removed), but with a bit of fiddling about you can get to something like this:

image

Next we need to add some attributes to our workflow to contain the subject and content of the email message together with some parameters to be passed to the send mail workflow, this can be done on the ‘General’ tab using the ‘add attribute’ button:

image

So that should be all of the information we require, next step is to go to the Schema tab and add a Scriptable Task by dragging the element between our start and end markers:

image

Double clicking on the title ‘Scriptable task’ allows us to set a friendly script name (‘Build Resource Email’ in this example). We can then select the ‘In’ tab for the script and select our input attributes and parameters:

image

We select the in-parameters (‘tenantEmail’, ‘addCPU’, ‘addRAM’, ‘addStoragePerf’ and ‘addStorageCap’) from this dialog and get the following listed under our script’s ‘IN’ tab:

image

We do the same to bind the ‘emailSubject’ and ‘emailContent’ attributes to our script’s ‘OUT’ tab:

image

Switching to the ‘Visual Binding’ tab, (sliding the edit pane up a bit for clarity) should now show something like the following:

image

We can now switch to the Scripting tab to actually write the code that will generate our email subject and body:

image

The code from this script is included below in case copy/paste is useful:

There’s nothing too complicated going on in the script, just building some HTML strings based on the input values we’ve received from the workflow. We query the _vdc_orgName, _vdc_orgId and _vdc_userName from our script environment to retrieve these values which are provided by the vCloud Director Service Library so we know which user in which tenant organisation has initiated the workflow.

Next we need to add the ‘send notification’ workflow to actually send an email, this can be dragged from the Mail folder under ‘All Workflows’ and placed after our script in the workflow Schema:

image

This first ‘Send notification’ will be the email sent to the service provider so I’ve renamed it as ‘Mail Provider’ and selected the ‘Source parameter’ items on the ‘IN’ tab as follows:

image

(To set each source parameter simply click on the ‘not set’ value and select the appropriate workflow attribute from the pop-up, ensure that unused parameters are set as NULL).

We can add a 2nd ‘Send Notification’ task to our workflow and configure it to send the same email content back to our tenant’s email address (the only change here is that the ‘toAddress’ parameter is set to the provided workflow tenant email address rather than the provider email address):

image

Our workflow is now complete and should be functional, we can ‘Validate’ and then Save and Close it in the Orchestrator client.

Our next step is to publish the workflow from the the provider interface of vCloud Director at https://<my vcloud IP>/provider

One logged in we see the following:

image

Selecting the 3-bars (highlighted) option and selecting ‘Content Libraries’ shows the Service Library:

image

First we need to select ‘Service Mangement’ and then ‘Service Categories’ tab to create a new Service Category (group) into which our workflow will be imported:

image

Clicking the ‘+’ sign allows us to define a new category and provide an icon for it:

image

Once saved we can return to the ‘Service Library’:

image

The ‘Import’ option allows us to add our workflow to the library, first we select the category we just created:

image

Next we select the source Orchestrator instance where our workflow lives:

image

Now we can browse the workflow tree and select our new workflow:

image

Finally we review and select ‘Done’ to create the library entry:

image

Using the ‘Manage’ button we can chose who the workflow should be published to:

image

If we don’t select ‘Publish to All Tenants’ then we can chose individual tenancies with the check boxes:

image

Clicking Save completes the process and publishes our workflow.

Note: I’ve had several instances where changes to publishing are not saved correctly, I’d suggest going back into the settings and checking these after making any changes.

Now if we log in to vCD as a tenant we can see the published workflow in the ‘Libraries’ option:

image

image

Clicking ‘Execute’ initialises our workflow and requests the input parameters we defined for the workflow:

image

Here I’ve asked for 16GB more RAM, 10GHz more CPU, 2TB more Capacity storage and 1TB more Performance storage.

Clicking ‘Finish’ submits the request and we see in our email that they have arrived populated with the details from our workflow:

image

This is of course a fairly basic example, and not designed to be useful ‘as is’, but hopefully has given you a good idea of the power and flexibility that Service Libraries introduce in vCloud Director v9.1 and given you some ideas of how they can be used.

Generating emails is a trivial example, but much more complicated workflows could easily be built (for example, to directly submit API requests to other systems as well as directly provisioning resources in vCloud Director on request).

As always, comments and feedback appreciated.

Jon

Cassandra SSL for vCloud Director metrics database

In vCloud Director v9 the requirement for both Apache Cassandra and KairosDB for storing metrics has been reduced to just Apache Cassandra. In addition, the ability to view VM metrics directly in the new  vCD 9 HTML5 tenant UI makes it much more important to have a reliable Cassandra infrastructure.

As I found when researching this post, configuring SSL in Cassandra is a bit of a pain, since Cassandra runs as a Java application it has some issues with the types of CA certificates and trusts it can use, which is further complicated by the options of both node to node encryption as well as cluster to clients.

I set out to produce an ‘easy’ way to configure a Cassandra cluster with SSL for both node to node and node to client communication in a way which could be reasonably easily implemented and reproduced for future installs.

My test environment consists of the minimum supported configuration of 4 Cassandra nodes (of which 2 are ‘seed’ nodes) running on CentOS Linux 7 (since that’s what we tend to use most for our backend infrastructure services). This configuration can almost certainly be adapted for other Linux distributions and I’ve tried to document the certificate generation process sufficiently that this will be straightforward.

Inspiration for this post was from Antoni Spiteri’s blog and script to configure a Cassandra cluster for vCloud Director metrics which I found extremely useful background.

Initial setup of my node servers used the following pattern to install and configure Cassandra:

Ensure all appropriate updates have been applied to our new CentOS installation if appropriate:

# yum update -y

Install Java (currently on my systems this installs java-1.8.0-openjdk.x86_64 1:1.8.0.161-b14.el7_4 which appears to work fine):

# yum install -y java

Create a file /etc/yum.repos.d/cassandra.repo (with vi or your favourite text editor) to include the Cassandra 3.0.x repository with the following contents. Note that I’m using the Cassandra 3.0.x repository (30X) and not the latest 311x release repository as this is not yet supported by VMware for vCloud Director:

[cassandra]
name=Apache Cassandra
baseurl=https://www.apache.org/dist/cassandra/redhat/30x/
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://www.apache.org/dist/cassandra/KEYS

Install Cassandra software itself (currently on my test nodes this pulls in the cassandra.noarch 0:3.0.15-1 package):

# yum install -y cassandra

The main configuration file for Cassandra is (on CentOS installed from this repository) in /etc/cassandra/conf/cassandra.yaml.

At least the following options in this file will need to be changed before we can get a cluster up and running:

Original cassandra.yaml  Edited cassandra.yaml Notes
cluster_name: ‘Test Cluster’ cluster_name: ‘My vCD Cluster’ Doesn’t absolutely have to be changed, but you probably should do. Note that this setting must exactly match on each of your node servers or they won’t be able to join the cluster.
authenticator: AllowAllAuthenticator authenticator: PasswordAuthenticator We’ll want to use password security from vCloud Director to the cluster.
authorizer: AllowAllAuthorizer authorizor: CassandraAuthorizer Required to enforce password security.
 – seeds: “127.0.0.1” – seeds: “Seed Node 1 IP address,Seed Node 2 IP address” Set the seeds for the cluster (minimum 2 nodes must be configured as seeds).
listen_address: localhost listen_address: This node IP address Must be configured for the node to listen for non-local traffic (from other nodes and clients).
rpc_address: localhost rpc_address: This node IP address Must be configured for the node to listen for non-local traffic (from other nodes and clients).

We’ll need to change some additional settings later to implement SSL security, but these settings should be enough to get the cluster functioning.

You’ll also need to permit the Cassandra traffic through the default CentOS 7 firewall, the following commands will open the appropriate ports (as root):

# firewall-cmd --zone public --add-port 7000/tcp --add-port 7001/tcp --add-port 7199/tcp --add-port 9042/tcp --add-port 9160/tcp --add-port 9142/tcp --permanent
# firewall-cmd --reload

Once you’ve performed these steps on each of the 4 nodes, you should be able to bring up a (non-encrypted) cassandra cluster by running:

# service cassandra start

on each node, you should probably also enable the service to auto-start on server reboot:

# chkconfig cassandra on

Note that you should start the nodes 1 by 1 and allow a minimum of 30 seconds between each one to allow the cluster to reconfigure as each node is added before adding the next one, check /var/log/cassandra/cassandra.log and /var/log/cassandra/system.log if you have issues with the cluster forming.

My test cluster has 4 nodes (named node01, node02, node03 and node04 imaginatively enough), and node01 and node02 are the seeds. The IP addresses are 10.0.0.101,102,103 and 104 (/24 netmask).

If everything has worked ok running ‘nodetool status’ on any node should show the cluster members all in a state of ‘UN’ (Up/Normal):

You should also be able to login to the cluster via any of the nodes using the cqlsh command (the default password is ‘cassandra’ for the builtin cassandra user)

[root@node01 ~]# cqlsh 10.0.0.101 -u cassandra -p cassandra
Connected to My vCD Cluster at 10.0.0.101:9042.
[cqlsh 5.0.1 | Cassandra 3.0.15 | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh>

To reconfigure the cluster for SSL encrypted communication we need to complete a number of tasks:

  • Create a new CA certificate authority (we could use an existing / external CA authority, but I’m trying to keep this simple).
  • Export the public key for our new CA so we can tell vCloud Director later to trust certificates it has issued.
  • Create a new truststore for Cassandra so Cassandra trusts our CA.
  • Create a new keystore for Cassandra for each node which includes the public key of our CA.
  • Create a certificate request from each node and submit this to our CA for signing.
  • Import the certificate response from the CA into the keystore for each node.
  • Move the generated truststore and keystore into appropriate locations and configure security on the files.
  • Reconfigure Cassandra to enable encryption and use our certificates.

That’s a lot of steps to go through and extremely tedious to get right, so I wrote a script to do most of these steps, create a file (I called my ‘gencasscerts.sh’ on one of the node servers and copy/paste the script contents from below):

Update 2018/01/26: I realised shortly after publishing that the original version of this script included both the public and private keys for the CA in the .truststore keystore. While this isn’t a huge issue in a private environment it’s definitely not ‘best practice’ to distribute the private key of the CA to each node so I’ve refined the script and removed the private keys from the generated .truststore files in this version. If you need to regenerate your environment keys remember that if you re-run the script both the .truststore and .keystore files will need to be updated on each node.

Make the script executable using:

chmod u+x gencasscerts.sh

Edit the settings and passwords for the certificates (we’ll need these later) in the variables at the top of the file to be appropriate for your environment (in particular the node names in the 3rd line) and any other options you want to change. The default validity period of the generated certificates is set to 10 years (3650 days). Obviously you should also change the passwords in your copy of the script for the CASTOREPASS and NODESTOREPASS variables.

When you run the script you should get output similar to the following:
[root@node01 ~]# ./gencasscerts.sh
MAC verified OK
MAC verified OK
Processing node 0: node01
Certificate was added to keystore
Signature ok
subject=/C=US/ST=MyState/L=MyCity/O=MyOrg/OU=MyDept/CN=node01
Getting CA Private Key
Certificate reply was installed in keystore
Processing node 1: node02
Certificate was added to keystore
Signature ok
subject=/C=US/ST=MyState/L=MyCity/O=MyOrg/OU=MyDept/CN=node02
Getting CA Private Key
Certificate reply was installed in keystore
Processing node 2: node03
Certificate was added to keystore
Signature ok
subject=/C=US/ST=MyState/L=MyCity/O=MyOrg/OU=MyDept/CN=node03
Getting CA Private Key
Certificate reply was installed in keystore
Processing node 3: node04
Certificate was added to keystore
Signature ok
subject=/C=US/ST=MyState/L=MyCity/O=MyOrg/OU=MyDept/CN=node04
Getting CA Private Key
Certificate reply was installed in keystore

Looking at the directory from where the script was run you should see the certificate files:

In each node directory there will be 3 files (.keystore, .truststore and chain.pem):

The .truststore files will all be identical between the node directories, the .keystore and client.pem files will be unique.

Next we need to move the generated certificate stores to an appropriate location, easiest way to do this is to use scp from the directory where the script was run. We’ll place the files in the Cassandra configuration directory (/etc/cassandra/conf).

On the node where the files have been generated we can just copy them (node01 in our case):

# cp node01/.keystore node01/.truststore /etc/cassandra/conf

To copy the appropriate files to the other nodes we can use scp:

# scp node02/.keystore node02/.truststore root@10.0.0.102:/etc/cassandra/conf
# scp node03/.keystore node03/.truststore root@10.0.0.103:/etc/cassandra/conf
# scp node04/.keystore node04/.truststore root@10.0.0.104:/etc/cassandra/conf

On each node we now run the following to set appropriate permissions and ownership on the files:

# chown cassandra:cassandra /etc/cassandra/conf/.keystore /etc/cassandra/conf/.truststore
# chmod 400 /etc/cassandra/conf/.keystore /etc/cassandra/conf/.truststore

We can now reconfigure the cassandra.yaml configuration file on each node to use our new certificates and enable encrypted communication.

The original settings from cassandra.yaml and the required changes are:

In the server_encryption_options: section to encrypt node-to-node communications:

Original Value New Value
internode_encryption: none internode_encryption: all
keystore: conf/.keystore keystore: <location of copied .keystore file>
keystore_password: cassandra keystore_password: <value you set for the NODESTOREPASS variable in the script above>
truststore: conf/.truststore truststore: <location of copied .truststore file>
truststore_password: cassandra truststore_password: <value you set for the CASTOREPASS variable in the script above>

 

Optionally you can also change the cipher_suites: setting to restrict available ciphers to the more secure versions (e.g. cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA]').

In the client_encryption_options: section to encrypt node-to-client communications:

Original Value New Value
enabled: false enabled: true
keystore: conf/.keystore keystore: <location of copied .keystore file>
keystore_password: cassandra keystore_password: <value you set for the NODESTOREPASS variable in the script above>

Again you can change the cipher_suites: setting if desired to use more secure ciphers.

Now we need to stop the cassandra service on ALL nodes:

service cassandra stop

Starting the cassandra service back up (service cassandra start – remember to wait between each node to give the cluster time to settle) you should now see the following in the /var/log/cassandra/system.log file:

INFO [main] 2018-01-25 20:52:38,290 MessagingService.java:541 - Starting Encrypted Messaging Service on SSL port 7001

Once the nodes are all back up and running nodetool status should show them all as status of ‘UN’ (Up/Normal).

If we attempt to use cqlsh to connect to the cluster now, we should get an error as we’re not using an encrypted connection:

cqlsh 10.0.0.101 -u cassandra -p cassandra
Connection error: ('Unable to connect to any servers', {'10.0.0.101': ConnectionShutdown('Connection <AsyncoreConnection(24713424) 10.0.0.101:9042 (closed)> is already closed',)})

If we specify the ‘--ssl‘ flag to cqlsh, we still get an error as we haven’t provided a client certificate for the connection:

cqlsh 10.0.0.101 -u cassandra -p cassandra --ssl
Validation is enabled; SSL transport factory requires a valid certfile to be specified. Please provide path to the certfile in [ssl] section as 'certfile' option in /root/.cassandra/cqlshrc (or use [certfiles] section) or set SSL_CERTFILE environment variable.

This is where the client.pem file is used as generated by the script, copy this file into the .cassandra folder in your user home path and then create/edit a file in this .cassandra folder called cqlshrc with the following content:

[connection]
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = ~/.cassandra/client.pem
validate = false

Save the file and now we should be able to establish an encrypted session:

cqlsh 10.0.0.101 -u cassandra -p cassandra --ssl
Connected to My vCD Cluster at 10.0.0.101:9042.
[cqlsh 5.0.1 | Cassandra 3.0.15 | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh>

When configuring vCloud Director to use our new metrics cluster, we must first tell vCloud Director that it can trust the CA we’ve created for our Cassandra cluster by importing the public key of our CA (the myca.pem file generated by the script) into the vCloud Director cell server cacerts repository. Ludovic Rivallain has a great post written up at https://vuptime.io/2017/08/30/VMware-Patch-vCloudDirector-cacerts-file/ showing how to do this. Note that this must be performed on each vCloud Director cell server as the cacerts repository is not shared between them.

You should also add a new admin user to Cassandra with a complex password and disable the builtin ‘cassandra’ user account before using the cluster.

Finally, you can follow the VMware documentation (link) to configure your vCloud Director cell to use this Cassandra cluster for metrics storage.

It’s reasonably easy to adjust this process to use an external CA rather than generating a new self-signing one, but this post is long enough already so let me know in the comments if you’d like to see this and I’ll write up a separate post detailing the changes to do this.

As always, comments/corrections/feedback welcome.

Jon.

vCloud Director Extender – Part 5 – Stretch Networking (L2VPN)

In this 5th part of my look into vCloud Director Extender (CX), I deal with the extension of a customer vCenter network into a cloud provider network using the L2VPN network extension functionality. Apologies that this post has been a bit delayed, turned out that I needed a VMware support request and a code update to vCloud Director 9.0.0.1 before I could get this functionality working. (I also had an issue with my lab environment which runs as a nested platform inside a vCloud Director environment and it turned out that the networking environment I had wasn’t quite flexible enough to get this working).

Update: an earlier version of this article didn’t include the steps to configure the L2 appliance settings in the vCloud Director Extender web interface – I’ve now added these to provide a more complete guide.

Links to the other parts of this series:
Part 1 – Overview
Part 2 – Cloud Provider / Service Provider installation and configuration (MyCloud)
Part 3 – Customer / Tenant installation and configuration (Tyrell)
Part 4 – Customer / Tenant connecting to a Cloud Provider and Virtual Machine migration (Tyrell)

I won’t deal with the use-case here that the customer already has NSX networking installed and configured, since in most cases you can simply create L2VPN networks directly between the customer and provider NSX Edge appliances and don’t really need to use the CX L2VPN functionality.

In order to be able to use the standalone L2VPN connectivity, the following pre-requisites are required:

  • A tenant vSphere environment with the vCloud Director Extender appliance deployed (it does not appear to be necessary to deploy the replication appliance if you only wish to use the L2VPN functionality, but obviously if you are intending to migrate VMs too you will need this deployed and configured as described in Part 3 of this series. In either case you will still need to register the cloud provider in the CX interface.
  • A configured vCloud Director VDC for the tenant to connect to. This environment must also have an Advanced Edge Gateway deployed with at least one uplink having a publicly accessible (internet) IP address. Note that you do not need to configure the L2VPN service on this gateway – the CX wizard completes this for you.
  • At least one OrgVDC network created as a subinterface on this edge gateway. The steps to create a suitable new OrgVDC network are detailed below.
  • Outbound internet connectivity to allow the standalone edge deployed in the tenant vCenter to communicate with the cloud-hosted edge gateway – only port 443/tcp is required for this.
  • Administrative credentials to connect to both the tenant vCenter and the cloud tenancy/VDC (Organization Administrator role is required).

Opening the tenant vCenter environment and selecting the ‘Home’ page shows the following:

Selecting the vCloud Director Extender icon opens the CX interface:

If you have not yet configured the L2 appliance settings, selecting the ‘DC Extensions’ tab will show the following error:

To fix this, open the vCloud Director Extender web interface in a browser by opening https://<ip address of deployed cx appliance>/ and log in, select the ‘DC Extensions’ tab:

Select the ‘Add Appliance Configuration’ option and complete the form to provide the deployment parameters where the standalone NSX edge appliance will be deployed:

The ‘Uplink Network Pool IP’ setting is a bit strange – it appears to be asking for a network pool or IP range, but the ‘help text’ in the field is asking for a single IP address. I found that the validation on this field is a bit odd – it will basically accept any input at all (even random strings) without complaining, but obviously deployment won’t work. What you need to do is add individual IPv4 addresses and click the ‘Add’ button for each. You will need 1 address for each stretched network you will be extending to your cloud platform. In this example I am only extending a single network so have added a single IPv4 address (192.168.0.201).

Once you click the ‘Create’ button you will be returned to the ‘DC Extensions’ tab and shown a summary of the L2 appliance configuration:

Note that there doesn’t appear to be any way to edit an existing L2 Appliance configuration, so if you need to change settings (e.g. to add additional uplink IP pool addresses) you will likely need to delete and recreate the entire entry.

 

Next we need to add a new ‘subinterface’ network to our hosted Edge gateway appliance, logging in to our cloud provider portal we can select the ‘Administration’ tab and the ‘Org VDC Networks’ sub-option, clicking the ‘Add’ button shows the dialog to create a new Org VDC Network. We need to select ‘Create a routed network by connecting to an existing edge gateway’ and then check the ‘Create as subinterface’ check box:

Next we configure the standard network information (Gateway, Network mask, DNS etc.) Since this network will be bridged to our on-premises network we can use the same details. Optionally a new Static IP pool can also be created so that new VMs provisioned in the cloud service can use this pool for their IP addresses. This won’t be an issue for VMs being migrated as they will carry across whatever IP addresses are already assigned to them. Note that the gateway address is set to be the same address as the existing (on-premises) gateway – this means that re-configuring the default gateway setting in the guest OS isn’t required either:

Now we supply a name for the new Org VDC network and optionally a description. The check box can also be used if the customer has multiple VDCs and wishes to share the new network across them:

Finally the summary screen allows us to check the information provided and go back and make any changes required if not correct. The most important setting is to make sure the network is attached to the edge gateway as a subinterface:

Once finished creating, the Org VDC network will be shown in the list with a type of ‘Routed’ and an interface type of ‘Subinterface’:

Next we access the vCloud Extender interface from within the customer vCenter plugin, selecting the ‘DC Extensions’ tab takes us to the following dialog:

Selecting ‘New Extension’ shows the dialog to create a new L2 extension, the fields are mostly populated for you. The ‘Enable egress’ allows you to select which gateway(s) will be allowed to forward traffic outside of the extended network. In this example I’ve only configured egress on the Source (on-premises) side through the existing gateway:

When you click ‘Start’, the status will go to ‘Connecting’ and a number of activities will take place in the customer vCenter:

Reading from the bottom (oldest) upwards, a new port group is created, an NSX Edge Standalone appliance is deployed and powered-on and the new port group is reconfigured once this has completed (ignore the VM migration task, that just happened to occur during the same time window in my lab). In this case the new NSX standalone edge was named ‘mcloudext-edge-4’ and the port group ‘mcxt-tpg-l2vpn-vlan-Tyrell-VDC15’.

Once deployment has completed (takes a few minutes) the vCloud Extender client interface shows the new DC extension network with a status of ‘Connected’:

In the tenant vCloud Director portal you can also see the status of the tunnel under ‘Statistics’ and ‘L2 VPN’ within the edge gateway interface:

You will now find that any VMs connected to the stretched network (OrgVDC network) in your cloud environment have L2 connectivity with the on-premises network and will continue to function as if they were still located in the customer’s own datacenter.

As I mentioned at the start of this post, I hit a number of issues when configuring this environment and getting it working took several attempts and a couple of rebuilds of my lab. The main issue was that in the initial release of vCloud Director v9.0.0.0 there is an issue that prevents the details required for the standalone NSX edge being deployed from being returned by the API. This prevents the deployment of the customer edge at all and resulted in my VMware support call. The specific issue is referenced in the vCloud Director 9.0.0.1 release notes  as ‘Resolves an issue where the vCloud Director API does not return a tunnelID parameter in response to a GET /vdcnetworks request sent against a routed Organization VCD network that has a subinterface enabled.’ As far as I can work out, it will be impossible to successfully use L2VPN in CX without upgrading the provider to vCloud Director 9.0.0.1 to resolve this issue.

The other issue I hit in my lab was that my hosted ‘Tenant Edge’ was NAT’d behind another NSX Edge gateway which was also performing NAT translation (Double-NAT). This was due to the way my lab is built in a nested environment inside vCloud Director. Unfortunately this meant the external interface of my hosted ‘Tenant Edge’ was actually an internal network address, so when the customer/on-premise edge tried to establish contact it was using an internal network address which obviously wasn’t going to work. I solved this by connecting a ‘real’ external internet network to my hosted Tenant Edge.

As always, comments and feedback always appreciated.

Jon.

vCloud Director Extender – Part 4 – Connect to Provider & VM Migration

In the first 3 parts of this series I covered an overview of vCloud Director Extender (CX), the installation and configuration of CX at the Cloud Provider site and the installation and configuration of CX at the customer/tenant site. In this 4th part I will be covering the configuration of the tenant environment to connect to the provider cloud and then migrate VM workloads to the provider.

This part follows on from the configuration completed in part 3 of this series and assumes that Tyrell (the customer site) have an existing virtual datacenter (VDC) environment available from MyCloud (the provider) and an appropriate Organization Administrator login to this environment. I’ve also created local DNS entries in the Tyrell network for the ‘chc.mycloud.local’ and ‘vcde.mycloud.local’ DNS names which resolve to the public IP addresses for the MyCloud vCloud Director instance and the provider CX endpoint respectively. Obviously in the real world these would be registered Internet DNS names.

In the Tyrell vCenter server when we select the ‘vCloud Director Extender’ icon we are shown an initial view of the CX plugin interface:

Selecting the ‘New Provider Cloud’ button opens a wizard to configure the connection to the Cloud Provider endpoints:

The ‘Provider Cloud URL’ needs to be set to include the appropriate path for the vCloud Director Organisation which is being connected to (the /cloud/org/Tyrell part in this example). The user details hold the Organization Administrator role within this cloud organisation.

When clicking ‘Add’ you will be presented with a certificate warning if the cloud provider is not using trusted/signed certificates, you can optionally select to trust these certificates if this is the case (very handy for a lab environment).

You can use the ‘Test’ button to confirm the settings are valid – you will see a status update at the bottom of the dialog showing the status of this test:

Note that even if the ‘Test’ succeeds, there are still some circumstances to do with network connectivity that can result in the enablement process failing – this is shown in the following capture from the ‘Provider Clouds’ tab where you can see the ‘Status’ shows ‘Enable Failed’:

This is usually caused by incorrect firewall rules, NAT rules or Public Endpoint URL’s set incorrectly when the CX appliances are deployed, I’m intending to cover this in a future ‘Troubleshooting’ part to this series of posts.

Once the networking and URLs are configured correctly you will see the new provider cloud registered under the ‘Provider Clouds’ tab with a status of ‘Running’, you will also see any accessible virtual datacenters (vDC) to which you have access:

Now that our provider cloud is properly registered, we can submit a migration request using the ‘Migrations’ tab in the CX interface, first we will be asked if we wish to perform a ‘Cold’ or ‘Warm’ migration – the differences between these are well explained in the dialog. Note that ‘Warm’ migration is not a vMotion, but does involve a period of network disconnection as the VM is cutover to the Cloud Provider. For this example we’ll select a ‘Warm’ migration:

Clicking ‘Next’ takes us to an inventory view where we can select the source VM(s) to be migrated. The grey panel below the ‘Inventory Browser’ dynamically expands to show candidate VMs from the vCenter environment. When a VM is selected the status and disk sizes are update in the right-side panel. For this example we’ve selected the ‘deckard’ VM:

Clicking ‘Next’ takes us on to the Target selection – here we can select the Cloud Provider, vDC, VM storage profile for the remote copy and the network to be connected to the VM in the Cloud Provider. Note that we are not L2-extending our on-premises network in this example and relying on our Cloud Provider (MyCloud) having already defined an Org vDC network for us (in this case called ‘Tyrell Servers’). All of the values are populated automatically from the vCloud Director environment and drop-downs allow easy select of other options. Finally we have the option when migrating multiple VMs together to group these into a single vApp rather than creating a new vApp for each VM:

In the final migration configuration step we can specify when the VM synchronisation should start, what our target Recovery Point Objective (RPO) is in minutes and whether to provision the destination disks as ‘Thin’ provisioned or ‘Thick’ provisioned. Finally we can add an optional tag to reference against this job later:

If everything has worked, you’ll now see a progress indicator against the VM in the Migrations tab. Initially the status will be ‘Created’:

Once data synchronisation begins this status will be updated to show the synchronised percentage for the migration. If you get an ‘Error’ prior to the sync percentage moving from 0% this is almost certainly a network configuration issue (and one which I encountered frequently when first building my lab environment). I’ll cover the common causes and remedies for this more in my vCloud Extender Troubleshooting post.

Once the initial synchronisation process has completed you will see the VM listed as ‘Cutover ready’ which means it’s staged and ready to be migrated:

Logging in to the Tyrell vCloud Director portal at this point shows that nothing actually has been provisioned into the Tyrell VDC:

Looking at the ‘Home’ page for the CX environment in vCenter shows our VM as in a ‘Transition’ state:

In the Migrations tab we can now select the ‘Start Cutover’ button to actually cutover the VM to the Cloud Provider environment which opens the Cutover dialog:

Clicking ‘Start’ asks for confirmation and then performs the actual cutover to running the VM in the Cloud Provider datacenter, progress is updated during the cutover procedure:

When the cutover process is complete you will see the Status update:

Looking in vCenter at this point shows the original VM still in place, but now powered off, you should probably take steps to ensure that this VM cannot be accidentally started at this point or risk having two running instances of the same VM (potentially on the same network if your network is extended to the Cloud Provider):

Refreshing the Tyrell vCloud Director portal shows the migrated VM now running in the Tyrell Cloud Provider VDC:

The status in the vCloud Extender vCenter plugin also now shows the completed migration total:

In the next part of this series of articles I look at the options to extend L2 networking directly from a customer site into vCloud Director using CX and the changes this introduces into the migration workflow.

Link back to Part 3 || Link to Part 5

As always, corrections, comments and feedback are always appreciated.

Jon.

vCloud Director Extender – Part 3 – Tenant Setup

In part 1 and part 2 of this series I detailed an overview of VMware vCloud Director Extender (CX) and the configuration from a Service Provider perspective to configure their platform to support CX.

This third article in the series details the configuration steps required for a tenant/customer environment to deploy and configure CX into their environment.

Once a service provider configuration is complete, any customers of that provider with sufficient allocated resources in a Virtual Datacenter (VDC) can configure the tenant CX environment and connect this to their vCenter environment. Once complete they will be able to migrate and replicate vSphere VMs between their own vCenter and the service provider datacenter extremely easily. Optionally they can use L2VPN functionality to stretch their networks into the Cloud Provider’s datacenter removing the requirement to have a pre-configured network in place. Of course many customers will wish to move to dedicated networking later, but having the initial ability to quickly provision their networks into a Cloud provider can dramatically shorten migration timeframes.

The initial deployment steps for customers deploying CX are exactly the same as for a Service Provider – download (or have provided to them by their Cloud Provider) the ova appliance for vCloud Director Extender and deploy this into their vCenter environment.

Right-clicking on the desired location and selecting ‘Deploy OVF Template…’ allows the local CX .ova file to be selected

The appliance name and folder are selected next:

Followed by the vCenter Cluster which will run the deployed appliance:

Check the template details and then click ‘Next’ to continue:

Read and accept the VMware license agreement:

Next select the Datastore storage on which the appliance will be deployed:

Select the required network for the appliance:

Make sure that ‘cx-connector’ (default) is selected for the ‘Deployment Type’ and fill out the IP addressing information for the appliance:

Check the summary information carefully and click ‘Finish’ to begin the deployment operation:

Once the appliance deployment task has configured, power-on the deployed VM in vCenter and wait for it to initialise. When it is running you can open a web browser to the IP address you configured for the appliance and login using the password configured. Note that you have to add ‘/ui/mgmt’ to the login URL for the appliance, so the full URL will be ‘https://<IP address of appliance>/ui/mgmt’:

The initial CX dialog when logged in allows you to start the Setup Wizard, note that in contrast to the Service Provider UI, there is no ‘Replication Managers’ tab in the cx-connector configuration:

The first step of the wizard is to link to the existing on-premise vCenter environment, note that if you are using an external Platform Services Controller (PSC) you will need to specify the PSC URL for the Lookup Service URL (although this is optional). The user specified needs to have administrative permissions within the vCenter environment:

Once the vCenter details and credentials are accepted, CX will provide a success notification, click ‘Next’ to continue:

The next page asks you to register the CX plugin with vCenter, this will likely become important in future as CX is updated, but for now leave the Version as 1.0.0 and click ‘Next’:

Once the plugin has registered into vCenter you will see a success notification. In testing I found that if the CX plugin had previously been registered with the vCenter (and not manually removed), this step would generate an error notification, but it was still possible to continue with the wizard and everything appeared to function fine afterwards:

Next you need to provide the configuration for the ‘Replicator’ appliance that will be deployed into the on-premise vCenter. The VMware documentation advises not to use DHCP for this and to manually specify a static IP configuration:

The ‘Replicator’ appliance is now deployed into vCenter and powered on. Once it has established network communication with the CX environment you will see a success notification:

The next step is to activate the Replicator appliance by providing a root password and authentication details for the on-premise vCenter environment. Note that you will need to set the Public Endpoint URL correctly in order for the appliance to be reachable by your cloud provider. If the on-premise Replicator appliance is behind a corporate firewall (as most will be), you will need to configure inbound firewall and translation rules and make sure this field is set correctly.

In my lab setup I configured the replicator public URL to be on port 443 on the public (Internet) address of the outside of the Tyrell firewall and used NAT port translation (see the networking configuration information below).

If everything is accepted you’ll receive a success notification in the wizard (note that I blanked the Public Endpoint URL field in this capture which is why it doesn’t show in the grab below):

The wizard is now complete, click ‘Finish’ to return to the UX interface:

The ‘vCenter Management’ tab should now show the on-premise vCenter details

The ‘Replicators’ tab should show the details for the replicator appliance deployed in the wizard:

Once vCenter has been closed and restarted you should now see a new ‘vCloud Director Extender’ item in the UI:

The networking configuration for a customer environment is a little simpler than for the cloud provider side, you will need to permit 2 inbound ports through the firewall, both of which need to communicate directly with the ‘Replicator’ appliance.

Assuming that you configured the ‘Public Endpoint URL’ with port 443, you will need to use NAT translation to divert this to port 8043 on the appliance:

Source Address Destination Destination Port/Protocol Translated Port/Protocol Translated Internal Address
External (Internet) Public IP Address 443/tcp 8043/tcp Replicator appliance internal address
External (Internet) Public IP Address 44045/tcp 44045/tcp Replication appliance internal address

You can (and should) limit the public/external addresses permitted to communicate with your Replicator appliance to just those public IP addresses used by your Cloud Provider – they should be able to provide you with this information.

Also note that if you restrict outbound internet traffic from your CX network you will also need to permit the following traffic in an Outbound direction:

Source Destination Source Port/Protocol Destination Port/Protocol Description
CX Server Network Cloud Provider Public CX Address Any 443/tcp Required for communications with the provider CX appliance
CX Server Network Cloud Provider Public CX Address Any 8044/tcp Required for communications with the provider Replication Manager appliance
CX Server Network Cloud Provider Public CX Address Any 44045/tcp Required for communications with the provider Replicator appliance

Of course if your provider has configured different ports for these components you will need to allow access to these instead of the defaults listed.

In the next part of this series I’ll continue with configuring the customer environment to connect to a cloud provider CX environment and to migrate some VMs.

Link back to Part 2 || Link to Part 4

As always, corrections, comments and feedback are always appreciated.

Jon.

vCloud Director Extender – Part 2 – Cloud Provider Setup

In the first part of this series of articles I described the new vCloud Director Extender (CX) software released by VMware. In this article I will show the steps required to install and configure the software from a Cloud Provider perspective. Included in this will be the necessary network and firewall configuration required.

vCloud Director Extender is supplied as a single .ova appliance from the VMware download site (login required). The download is located in the ‘Drivers & Tools’ section of the vCloud Director for Service Providers v9.0 page:

The ova file will generate the 3 different server components required to create a functional deployment:

CX Cloud Service The main vCloud Director Extender appliance, this is used to provide the UI for setup/configuration. This is the appliance initially deployed from the vCloud Director Extender appliance download package.
Cloud Continuity Manager (CCM) This component (also known as the ‘Replicator Manager’) is the operational manager of the deployment. CCM only runs in provider deployments and manages the replicator (CCE) appliances. CCM appliances are deployed and managed by the CX appliance (no additional download is required).
Cloud Continuity Engine (CCE) This component (also known as the ‘Replicator’) is the transfer engine that deals with data transfers between the customer and provider environments. CCE runs in both the provider and client environments. CCE appliances are deployed and managed by the CX appliance (no additional download is required).

The downloaded CX appliance is deployed from vCenter, the first selection allows you to specify the VM name and datacenter/folder location to deploy. In most service providers this would likely be the management cluster for their environment (as opposed to resource vcenters used for customer workloads)

Next you select which cluster/resource pool to deploy the CX appliance into:

A Review screen is presented which allows you to confirm the ova details:

And of course we have to read/accept the license agreement:

Next we select the datastore location for deployment:

And the internal network which the appliance will be connected to:

Make sure in the ‘Customize template’ screen (below) you change the ‘Deployment Type’ to ‘cx-cloud-service’ and don’t leave the default selection (cx-connector) selected as this will install the customer/tenant environment instead of the service provider configuration! The rest of the configuration options on this page are straightforward:

A summary screen is displayed showing a summary of the customization options selected, check these carefully as if they are wrong you’ll probably have to re-deploy from scratch:

Once the appliance is deployed, you will need to manually power it on from the vSphere client (or I did anyway – not sure if this is by design or not). Once it has booted and configured itself it will show the browser link to access to begin the environment configuration:

Note that if you open a page to just the hostname/IP address you’ll get an error, you must include the ‘/ui/mgmt’ suffix to the URL. You can now login with the ‘initial root login’ password you configured during the ova deployment. As you can see from the screen grab below I pre-configured DNS entries for the 3 provider components and used these wherever possible to avoid IP address confusion:

The main screen opens to the Setup Wizard, the tabs at the top of the screen allow you to easily navigate between sections, but these won’t show much until you complete the wizard:

Clicking on the ‘Setup Wizard’ opens a series of dialogs to provide the initial system configuration, first we have to specify the management vCenter authentication details. Note that the ‘Lookup Service URL’ as well as being optional also requires the path to the Platform Services Controller (PSC) if you are using external PSCs. The full path is truncated in this grab but should be https://<psc or vcenter with embedded psc address>/lookupservice/sdk:

The wizard includes very useful feedback at each step to show you if the previous actions have been successful or not, just click ‘Next’ through if everything is ok, or go back and fix the issue if not:

Now we need to provide a ‘system’ (administrator) level login to vCloud Director, you don’t need to specify the @system part of the user name here:

Again we get confirmation that we’ve successfully linked to vCloud Director and can continue with ‘Next’:

Next we can add the resource vCenters (where customer workloads actually run). In my lab environment this is the same vCenter that supports the management environment so the details are the same, but in production environments this will almost certainly be different. The setup wizard is intelligent enough to retrieve the names of any vCenter servers being used in Provider VDCs (pVDCs) in vCloud Director so for these you only need to ‘Update’.

When you click update you’ll be asked to provide administrator credentials to the resource vCenter environment. Be careful here as the default ‘Lookup Service URL’ will be set to the vCenter name, even if the vCenter is using an external Platform Services Controller (PSC) as mine was and will need to be manually edited to point to the PSC. This caught me out initially and I couldn’t work out why authentication to the resource vCenter was failing.

Once the resource vCenter(s) are authenticated they’ll show as ‘Registered’ in the wizard:

Next we need to configure the 2nd appliance configuration – this will be the ‘Replication Manager’ (also called the Cloud Continuity Manager / CCM in the documentation). We need to specify the parameters shown (the dialog scrolls down and also asks for default gateway address, DNS server address and netmask).

The wizard will now deploy and start up the replication manager appliance on the vCenter specified. If the networking information is incorrect the process will stall at this point as the wizard relies on establishing network connectivity with the replication manager before continuing. A status update is given at the top of the dialog as the appliance is deployed and started up. Once the replication manager appliance is running and seen on the network you’ll see the success message:

Next the replication manager appliance must be ‘activated’ by setting the password for the root user and the ‘Public Endpoint URL’. Make sure you set this to the correct external (public) IP address that your customers will be using to connect to your CX environment. I haven’t found any way yet to alter this setting after deployment if specified incorrectly without deleting the entire CX environment and starting over (the xx’s in this grab are simply to hide the real internet addressing I was using – I’m also pretty sure I eventually used the default port of 8044 for this public URL):

If everything has gone ok, you’ll get the screen below showing that the replication manager deployment has succeeded and you can move on to the replicator configuration:

The deployment details for the Replicator are specified next – the wizard helpfully copies across some of the settings from the Replication Manager deployment, but you still need to specify the (unique) IP and Netmask details:

The Replicator appliance will now be deployed in vCenter in exactly the same way as the Replication Manager was previously. Once it becomes available on the network the wizard will detect this and show the screen below:

Next we have to ‘Activate’ the Replicator appliance by completing the settings shown below to authenticate to the resource vCenter which this Replicator will be responsible for.

If everything worked ok you’ll get a ‘Successfully Activated’ message:

Clicking ‘Next’ takes you to the ‘Complete’ screen and shows that if you have additional Resource vCenters you’ll need to deploy additional Replicator appliances for these (1 per vCenter):

Clicking through the tabs in the management UI should now show that all the required CX components are now deployed and registered. The ‘Cloud Resoures’ tab shows linked vCloud Director instances and resource vCenters:

The ‘Replication Manager’ tab shows the deployed Replication Manager appliance:

Th ‘Replicators’ tab shows the deployed Replicator appliance(s) – 1 per resource vCenter if you have multiples of these.

That completes the appliance installation and initial configuration, next you will need to configure appropriate NAT/firewall rules so that customers on the internet can connect to your new CX service!

Assuming that you wish to use a single external (public) Internet IP address for the entire CX service, the configuration is a little tricky since traffic will need to be directed to either the CX, Replication Manager or Replicator appliance depending on what port it is attempting to aceess. The NAT/Firewall rules that I worked out from the documentation and found that worked are:

Source Address Destination Destination Port/Protocol Translated Port/Protocol Translated Internal Address
External (Internet) CX Service Public IP Address 443/tcp 443/tcp CX (vCD Extender) appliance internal address
External (Internet) CX Service Public IP Address 8044/tcp 8044/tcp Replication Manager appliance internal address
External (Internet) CX Service Public IP Address 44045/tcp 44045/tcp Replicator appliance internal address

Also note that if you restrict outbound internet traffic from your CX network you will also need to permit the following traffic in an Outbound direction:

Source Destination Source Port/Protocol Destination Port/Protocol Description
CX Server Network External (Internet) Any 443/tcp Required for CX to be able to communicate with customer Replicator management interface
CX Server Network External (Internet) Any 44045/tcp Required for CX to be able to communicate with customer Replicator data interface

In the next part of this series of articles I’ll continue with the installation and configuration of the CX components required on the customer / tenant site.

Link back to Part 1 || Link to Part 3

As always, corrections, comments and feedback are always appreciated.

Jon.

vCloud Director Extender – Part 1 Overview

Last week VMware released version 1.0 of the new vCloud Director Extender (CX) (link to documentation set). This provides some extremely flexible options for customers to migrate servers to/from a vCloud service provider cloud platform, including the use of L2VPN to transparently stretch their on-premise networks to the cloud provider. Together with a ‘warm’ cutover feature, this enables any customer with an appropriately configured vCloud tenancy and resources to safely and easily move their virtual servers to the most suitable hosting location with minimal application downtime.

As always, there are a few pre-requisites:

– The customer site must be running vSphere 6 Update 3 or later (6.5.0 and 6.5 Update 1 are also both supported).
– If the customer wishes to use L2VPN network extension and is already running VMware NSX, this must be v6.2.8 or v6.3.2.
– The cloud provider must be running vCloud Director v8.20 or v9.0.

Deployment of the replication environment is different for the Cloud Provider and tenant (as you would expect) and firewall rules and address translation need to be appropriately configured to permit the required traffic flows at both the provider and customer end.

This series of articles will detail the installation and configuration of vCloud Director Extender and is intended to be useful for both Cloud Providers needing to configure their own environments to support CX and for customers wishing to configure their environments to allow migration to/from a CX-enabled provider.

The environment that I will be describing and building through this series is shown in the graphic below, Tyrell Corporation is the client organisation and MyCloud is the Cloud Provider which Tyrell wish to use to host 3 of their production VMs (‘Deckard’, ‘Rachael’ and ‘Roy’). In this example Tyrell and MyCloud happen to use different internal IP network ranges, but that is not a requirement to use CX since NAT firewalls are in place at both organisations.

Since I built this environment using ‘real’ public Internet addresses and VMware NSX edge gateways as the firewalls for both Tyrell and MyCloud, I have stripped the public IP addresses from the configurations shown in these articles, but it should be easy to see where these are substituted.

I’m expecting this series to consist of 6 parts eventually including this introduction:

Part 1 – This overview
Part 2 – Cloud Provider / Service Provider installation and configuration (MyCloud)
Part 3 – Customer / Tenant installation and configuration (Tyrell)
Part 4 – Customer / Tenant connecting to a Cloud Provider and Virtual Machine migration (Tyrell)
Part 5 – Stretched networking (L2VPN) configurations
Part 6 – Troubleshooting

I’m still working on the later parts of this series so check back if I haven’t published all of them yet.

Link to Part 2

As always, corrections, comments and feedback are always appreciated.

Jon.

vCloud Director v9 Multi-site

Since vCloud Director v9 was released last week (and previously as part of the closed beta), one of the new features I’m most excited about is support for multi-site deployments. This allows vCloud Director environments for the first time to properly span federated sites (e.g. a tenant who has resources in multiple datacenter locations for resiliency/redundancy can now manage these in the same place).

Configuring multi-site support in vCloud Director v9 is a 2-part process:

1) The service provider has to configure federation between their vCloud Director instances.
2) The tenant has to associate each of their Organizations in each vCloud Director instance.

This post will attempt to explain and show both processes, and there’s even a bonus of a PowerShell script I’ve written to help other service providers configure their site pairings. To help demonstrate the processes involved, I’ve built a test lab environment consisting of 4 separate vCloud Director instances (‘Auckland’, ‘Wellington’, ‘Christchurch’ and ‘Dunedin’ sites). Each of these has it’s own vCloud Director, vCenter, NSX and ESXi hosts. I’ve also created a tenant organization (‘Tenant X’) in all 4 instances and created and assigned a VDC to Tenant X in each location. Finally, I’ve federated Tenant X’s vCloud users with a directory service (Microsoft AD FS in this case) so that the same identity provider is available to all 4 vCloud instances.

If you’re not a service provider and just need to configure Organization pairing you’re probably safe to skip this section and proceed straight to the 2nd part of this post.

Part 1 – Service Provider Site Pairing

The basic process for a service provider to pair sites is:

– Check (and configure if necessary) the vCloud site name in each location. Note by default in the initial vCloud Director 9 release this is simply a GUID string so you’ll probably want to change it to something more meaningful.
– Download from each site the site association document (from /api/site/associations/localAssociationData)
– Upload this site association document to each other site that you want to pair with (to /api/site/associations)

In a scenario with only 2 sites you need to perform this process twice (once in each direction), but for our example with 4 sites we need to do this a total of 12 times to pair every site with every other site.

Being a bit of a pain to do manually against the REST API interface I ran true to form and wrote a PowerShell module to simplify the process of both administering the site names and also pairing sites together. The module is available on my github repository at https://github.com/jondwaite/vCDSitePair. The script uses my Invoke-vCloud module, so you’ll need that installed for it to run.

Once you’ve downloaded the vCDSitePair.psm1 file from github you can add it to your PowerShell session using ‘Import-Module ‘

There are a total of 4 functions provided by the module, and they are documented on the github repository but basically:

Get-vCloudSiteName Allows a service provider to check/confirm the ‘Site Name’ assigned to a vCloud Director instance.
Set-vCloudSiteName Allows a service provider to set/update the ‘Site Name’ assigned to a vCloud Director instance.
Get-vCloudSiteAssociations Shows the existing associations (if any) from a vCloud Director instance.
Invoke-vCDPairSites Performs the 2-way exchange of localAssociationData documents to pair two vCloud Director instances.

So to confirm/set the names of our ‘Auckland’ (akl.mycloud.local) and ‘Christchurch’ (chc.mycloud.local) sites we can use Get-vCloudSiteName and Set-vCloudSiteName:

Now we have set the site names, we can check if they are already associated:

 

Ok, so they’re not already associated, so we can run Invoke-vCDPairSites without the ‘WhatIf $false’ to see what would happen:

That all looks good so now we can attempt the action pairing operation:

And confirm the associations using Get-vCloudSiteAssociation again:

(I’ve cut out the certificate dumps for brevity).

So now that our Auckland and Christchurch sites are paired we can move on with associating the Organization (‘Tenant X’) between these sites. I’ve also been through and associated all of the other sites to each other, so by this stage ‘Auckland’ is associated to ‘Wellington’,’Christchurch’ and ‘Dunedin’ etc.

Part 2 – Organization Site Pairing

Originally I was intending to write PowerShell functions for this too, and while this is certainly possible, VMware have been nice to us and created the capability in the new vCloud Director tenant UI. Logging in as a user with ‘Organizational Administrator’ access shows an ‘Administration’ tab:

Selecting the ‘Administration’ tab reveals the site pairing options:

When we select ‘Export Local Association Data’ a file is downloaded (named ‘Download’ weirdly enough) and this file can be uploaded to the ‘partner’ site using the other ‘Create New Organization Association’ button. Once completed, the association is shown in the panel – here is the Auckland site for ‘Tenant X’ once the Christchurch Local Association Data has been uploaded to it:

You can click on this panel to see the association details and even remove a site association if no longer required:

Here’s the view of the ‘Christchurch’ environment once I’ve paired the ‘other’ 3 sites to it for this Organization:

Once we’ve added all our associations (and logged out and back in) we can see the new multi-site drop-down menu item which allows us to select from any of our datacenter locations:

And selecting one (‘Auckland’ in this case) takes us to the Auckland resource view:

All in all, a little bit of a convoluted process, but at least it should only need to be done once and can then be left alone. Very excited to see what VMware do with this functionality in future – can definitely see a time when all of an Organization’s VMs are displayed / summarised in a single view regardless of which vCloud instance supports them.

I have several more thoughts generally on vCloud Director v9 which I’ll put into a separate post when I have time, but wanted to get this published for anyone else playing around with the new multi-site features.

As always, comments and feedback appreciated.

Jon.