vCloud Director 9 HTML5 Portal Customization

One of the great features in vCloud Director 9 which has been further enhanced in the latest v9.5 release is the new HTML5 portal:

image

Even better, VMware has released a toolkit to allow Service Providers to fully customise the look and feel of the portal using CSS themes in their Clarity framework..

The toolkit itself is part of the VMware vcd-ext-sdk repository on github, available in the /ui/theme-generator folder.

The repository has good instructions on how to modify and build a custom theme, but actually uploading and configuring the theme in vCloud Director is only accessible via the vCD API and involves a reasonable amount of manual work.

To help speed up development and allow changes to be easily tested, in my usual mode I’ve written a small PowerShell module that allows quicker/easier theme configuration. The module is available on github at https://github.com/jondwaite/vcd-h5-themes. Hopefully this will help those of you who need to develop and test updated themes for your vCloud Director portals.

I’ve included documentation in the repository on each cmdlet, its function and arguments here.

To use the module you’ll need to be connected to a vCloud instance as a user with global ‘Administrator’ access in the ‘System’ organization since changes will affect all portal users. You’ll need to be connected to the vCD environment with PowerCLI (Connect-CIServer…) prior to using the cmdlets.

You can then download the vcd-ht-themes.psm1 file and add it to your session (‘Import-Module vcd-h5-themes.psm1’) to access the cmdlets.

As always, comments and feedback welcome – is there anything else you’d like to see added to this module?

Jon.

VM Guest Customization in vCloud Director via PowerCLI

Bit of a quick post this, but hopefully useful to others.

I got asked recently if there was an easy way to set Guest Customization options for VMs hosted in vCloud Director via Powershell/PowerCLI. It turns out there is an extremely simple way, but the syntax is a bit awkward so figured it would make a good/quick blog post.

The Guest Customization settings are available as one of the ‘Section’ entries returned by accessing the ExtensionData properties on a CIVM object. Once connected (Connect-CIServer) you can see this from PowerCLI:

image

The ‘trick’ is that there are typically 5 sections (one each for OvfVSSD, OvfMsg, network connections, guest Customization and VMware tools). I’ve seen some approaches that rely on the ‘guest Customization’ setting always being found at the Section[3] index in the ExtensionData collection, but this could easily change in future and break any functionality relying on this. A much more reliable way of finding the guest Customization section values is:

image

But how about if you need to change/update a setting, luckily there is a method provided (UpdateServerData) which does exactly this. So if we want to (for example) change the ‘CustomizationScript’ setting to “echo “Hello World!” we can:

image

You can change other settings using the same method (e.g. ComputerName or Domain join settings).

Note that for many changes the VM must be powered off, and you may need to ‘Power On and Force Recustomization’ too.

As always, comments & feedback appreciated.

Jon.

Getting detailed VM Disk Properties from the vCloud API

Since vCloud Director 8.10 VMware have allowed VMs to be created which have multiple disks using different storage policies. This can be very useful – for example, a database VM might have it’s database on fast storage but another disk containing backups or logs on slower/cheaper disk.

When trying to find out what storage is in use for a VM though this can create issues, the PowerCLI Get-CIVM cmdlet (and the Get-CIView cmdlet used to get extra information) aren’t able to properly report storage for VMs that consume multiple storage policies. This in turn can create problems for Service Providers when they need to report on overall VM disk usage divided by storage policy used.

As an example I’ve created a VM named ‘test01’ in a customer vDC which has 3 disks attached, the 2nd of these is on ‘Capacity’ tier storage while disks 1 and 3 are on ‘Performance’ storage. When we look at the VM details we see the following:

image

Digging into the ExtensionData shows

image

The StorageProfile element looks like it may contain what we need, but unfortunately this only shows the ‘home’ Storage for the VM and doesn’t indicate that at least one of the VMs disks is on a different storage profile:

image

After a lot of mucking around trying to find an easy way to discover the information, I ‘gave up’ and wrote a PowerShell module which accesses the vCD API directly to get the VM storage information (including storage tiers in use by each disk). The module isn’t overly efficient since it queries the storage profile reference for every disk on every VM (and so will result in a lot of calls if run for a large number of VMs), but otherwise works fine.

The module takes VM objects or a VM name as input and returns details on each disk attached to the VM including which storage profile they use. Save the script (e.g. as ‘Get-CIVMStorageProfile.psm1’) and then use ‘Import-Module .\Get-CIVMStorageProfile.psm1’ to import the function.

And here is example output from the script for our test VM:

image

Hope this is useful to some of you and as always, appreciate any comments/feedback.

I’d also love to know if there’s an easier way of generating this information.

Jon.

Tenant Portal Displays ‘No Datacenters are available’ in vCloud Director 9.1

We had an issue recently when updating our vCloud Director environment to v9.1 where the new tenant portal would show ‘No Datacenters are available’ for every tenant even though the remainder of the site worked correctly (and other tabbed options like the Service Library & catalogs worked fine). Initially we suspected that our SSL certificate chain or public URI’s were set incorrectly.

Adrian Begg has a great blog post here: http://www.pigeonnuggets.com/2018/03/vcloud-director-9-1-tenant-portal-displays-no-datacenters-available-after-upgrade/ which details this issue and how to ensure the correct settings are applied, however in our case this didn’t resolve our issue.

Eventually an offhand remark in a slack channel by Tom Fojta put me on the right track to solving the issue, I’ve written this post up in case anyone else comes across the same issue. If you’re impatient and want to know the solution – it’s DNS (isn’t it always DNS?), but that’s jumping ahead a bit.

In our environment we have 3 vCloud Director cell servers behind a load balancer, we also load-balance internally so that our management environment can talk to the vCD API and we can conduct testing of the environment without necessarily having it open to the public internet. The arrangement looks logically like this:

 

vCloud Director Load Balancing

Users from the internet accessing ‘portal.cloud.com’ get redirected to one of the vCD cell servers (and if one of them is unavailable the monitoring on the Load Balancer doesn’t direct requests there). The same happens for internal users, but in this case the ‘portal.cloud.com’ DNS entry has been overridden to point at the internal (192.168.0.10) address to allow connectivity to the cells even if the external LB or internet link is unavailable.

The issue in our environment was that the cell servers themselves use DNS to access the vCloud API – and they use the public URL specified in the vCloud Director configuration.

The cell servers were configured with our internal DNS servers, so when they attempted to access the public URL (‘portal.cloud.com’) were being given the internal Load Balancer address (192.168.0.10). For reasons we’re still exploring, this didn’t allow them to get a response from the vCD API and resulted in the ‘No Datacenters are available’ error in the tenant portal.

The fix turned out to be reasonably simple – on each cell server we added an entry to the /etc/hosts file to resolve the public URL to the cell’s own IP address, so on cell 01:

192.168.0.11    portal.cloud.com

On cell02:

192.168.0.12   portal.cloud.com

And on cell03:

192.168.0.13    portal.cloud.com

Once we’d made this change the tenant portals began functioning correctly (note that no restart of the cell servers or vCloud Director services was required).

What I assume is happening is that when the internal load balancer responds the the request it gives out a different cell server’s address (since the ‘source’ of the request will be a cell server) and that cell server has no knowledge of the session being used by the original cell and so responds incorrectly (either with nothing, or with an error). Not sure if this is actually a bug, or just something to be aware of, but either way overriding name resolution in this way fixes the issue. Note that simply using ‘localhost’ or 127.0.0.1 for the hosts file entry doesn’t work since the vCloud web server doesn’t respond on the loopback interface in the default configuration.

Just posting this here in the hope it will save someone else any frustration caused by this issue.

Jon.

Using VMware Container Service Extension (CSE)

Yesterday I wrote showing the currently available container hosting options from VMware. As we’ve recently deployed one of these options – CSE in our environment I thought it would be useful to show a sample workflow on how the service functions and how customers can use this to deploy and manage both CSE clusters, and also micro-service applications onto those clusters.

There are a few requirements on the tenant side which must be completed prior to any of this working:

  • An Organizational Administrator login to the vCloud platform where CSE is deployed.
  • Access to a virtual datacenter (VDC) with sufficient CPU, Memory and Storage resources for the cluster to be deployed into.
  • An Org VDC network which can be used by the cluster and has sufficient free IP addresses in a Static Pool to allocate to the cluster nodes (clusters take 1 IP address for the ‘master’ node and an additional address for each ‘worker’ node deployed).
  • A client prepared with Python v3 installed and the vcd-cli and container-service-extension packages installed on it.
  • The {$HOMEDIR}\.vcd-cli\profiles.yaml file edited to add the CSE extension to vcd-cli.
  • The kubectl utility installed to administer the Kubernetes cluster once deployed and working. kubectl can be obtained most easily from here.

Detailed instructions for the client setup can be found in the CSE documentation at https://vmware.github.io/container-service-extension/#tenant-installation. Note that on a Windows platform the .vcd-cli folder and profiles.yaml file will not be automatically created, but you can do this manually by

from a DOS prompt and then using vcd-cli to log in and out of your cloud provider. This will cause profiles.yaml to be generated in the .vcd-cli folder. The profiles.yaml file can then be edited in your favourite text editor to add the required CSE extension lines.

Deploying a Cluster with CSE

When deploying a cluster, you will need to know the storage profile and network names which the cluster will use, the easiest way of obtaining these is either from the vCloud portal, or using the vcd vdc info command when logged in to your environment:

image

If you have multiple VDCs available to you, the ‘vcd vdc use <VDC Name>’ command to set which one to work with.

In this example we will be using the highlighted entries (the ‘Tyrell-Servers’ network and the ‘CHC Performance’ storage profile).

To retrieve a list of available cluster deployment templates that the Service Provider has made available to us we can use the vcd cse template list command:

image

In this example only the Photon OS template is available and is also the default template. CSE actually comes with 2 profiles (Photon OS v2 and Ubuntu Linux 16-04, but I’ve only installed the Photon OS v2 template in my lab environment). The default template will be used if you do not specify the ‘–template’ switch when creating a cluster.

The cluster create command takes a number of parameters which are documented in the CSE page:

image

Be careful with the memory specification is it is in MB and not GB.

I chose to generate a public/private key to access the cluster nodes without needing a password, but this is optional. If you want to use key authentication you will need to generate a key pair and specify the public key filename in the cluster creation command using the –ssh-key switch.

To deploy a cluster with 3 worker nodes into our VDC where each node has 4GB of RAM and 2 CPUs using my public key and the network and storage profile identified above:

image

The deployment process will take several minutes to complete as the cluster VMs are deployed and started.

In to the vCloud Director portal, we can see the new vApp that has been deployed with our master and worker nodes inside it, we can also see that all 4 VMs are connected to the network we specified:

image

To see the details of the nodes deployed we can use ‘vcd cse node list <cluster name>’:

image

To manage the cluster with kubectl, we need a configuration file for Kubernetes containing our authentication certificates. kubectl by default looks for a file named ‘config’ in a folder called ‘.kube’ under the current user’s home directory. The config file itself can be downloaded using CSE. To create the folder and write the config file:

image

If you have multiple deployed clusters you can create separate config files for each one (with different file names) and use the –kubeconfig= switch to kubectl to select which one to use.

To test kubectl we can ask for a list of all containers (‘pods’ in Kubernetes) from the cluster, the ‘–all-namespaces’ switch shows system pods as well as any user created pods (which we don’t have yet). This must be run from a machine that has network connectivity with the deployed nodes (the ‘Tyrell-Servers’ network in this example):

image

 

Cluster Scaling

Adding Nodes to Clusters

If we need to add worker nodes to a cluster this is accomplished with the ‘vcd cse node create’ command. For example, we can add a 4th worker node to our ‘myCluster’ cluster as follows:

image

The node list now shows our cluster with 4 worker nodes including our new one:

image

Removing Nodes from Clusters

To remove a cluster member is just as easy using the ‘vcd cse node delete’ command:

image

You will be prompted to confirm the node deletion, and if you have deployed container applications you should ensure that the node is properly drained and/or replica sets and deployments configured correctly so that the node deletion will not impact your applications.

 

Cluster Host Affinity

One item that CSE does not deal with yet is creating vCloud Anti-Affinity rules to ensure that your worker nodes are spread across different physical hosts. This means that with appropriately configured applications a host failure will not impact on the availability of your deployed services. It is reasonably straightforward to add anti-affinity rules in the vCloud portal though.

Our test cluster is back to 3 nodes following the deletion example:

image

In the vCloud portal we can go to ‘Administration’ and select our virtual datacenter in the left pane, we will then see an ‘Affinity Rules’ tab:

image

Clicking the ‘+’ icon under Anti-Affinity Rules allows us to create a new rule to keep our worker nodes on separate hosts:

image

Provided the VDC has sufficient backing physical hosts, the screen will update to show the new rule and that it has successfully been applied and separated the worker nodes to different hosts:

image

Of course if the host running the master node experiences a failure then this will be unavailable until the VMware platform restarts the VM on another host.

 

Application Deployment using kubectl

Of course now that our cluster is up and running, it would be nice to actually deploy a workload to it. The ‘sock shop’ example mentioned in the CSE documentation is a good example application to try as it consists of several pods running in a separate namespace.

First we use kubectl to create the namespace:

image

Now we can deploy the application into our name space from the microservices-demo project on github. You can read more about the sock-shop demo app at https://github.com/microservices-demo/microservices-demo.

C:\Users\jon>kubectl apply -n sock-shop -f "https://github.com/microservices-demo/microservices-demo/blob/master/deploy/kubernetes/complete-demo.yaml?raw=true"
deployment "carts-db" created
service "carts-db" created
deployment "carts" created
service "carts" created
deployment "catalogue-db" created
service "catalogue-db" created
deployment "catalogue" created
service "catalogue" created
deployment "front-end" created
service "front-end" created
deployment "orders-db" created
service "orders-db" created
deployment "orders" created
service "orders" created
deployment "payment" created
service "payment" created
deployment "queue-master" created
service "queue-master" created
deployment "rabbitmq" created
service "rabbitmq" created
deployment "shipping" created
service "shipping" created
deployment "user-db" created
service "user-db" created
deployment "user" created
service "user" created

We can see deployment status by getting the pod status in our namespace:

image

After a short while all the pods should have been created and show a status of ‘Running’:

image

The ‘sock-shop’ demo creates a service which listens on port 30001 on all nodes (including the master node) for http traffic, so we can get our master node IP address from ‘vcd cse node list myCluster’ and open this page in a browser:

image

And here’s our deployed application running!

image

Summary / Further Reading

Of course there’s much more that can be done with Docker and Kubernetes, but hopefully I’ve been able to demonstrate how easily a cluster can be deployed using CSE and how micro-services applications can be run in this platform.

For further reading on kubectl and all the available functionality I can recommend the Kubernetes kubectl documentation at https://kubernetes.io/docs/reference/kubectl/overview/. In fact the entire Kubernetes site is well worth a read for those considering deployment of these architectures.

As always, comments, feedback, suggestions and corrections always welcome.

Jon.

VMware Container Solutions

VMware appears to have gone a little ‘mad’ with regards to containerisation (or containerization for any American readers) lately. Last week saw the release of Pivotal Container Service (PKS) as launched at VMworld 2017 US back in August. With this there are now a total of three VMware technologies all enabling customers to run micro-service applications in their environments. So why three different products to do the same thing? Well, they are targeted at different environments and use-cases, and actually it makes a lot of sense for VMware to have solutions for all 3 scenarios. Of course there’s always the 4th option of building your own container hosting platform from scratch on a VMware platform, but lets concentrate for now on those provided by VMware.

So what are the available solutions?

Pivotal Container Service (PKS)

This was announced at VMworld 2017 and recently became available for download. PKS a full stack solution to manage both initial formation of clusters to support containerised applications and manage their ‘day 2’ operations once deployed. While PKS could be deployed in an Enterprise environment (and may be for organisations using containerised applications at significant scale) it appears to be more targetted towards cloud service providers wishing to offer a managed/hosted platform for multiple tenants.

vSphere Integrated Containers (VIC)

VIC has been around for a while now (this was based on VMware’s Project Bonneville which started back in 2015), recently VIC has been updated to v1.3.1 and gained the capability to use Docker hosts natively at version 1.2 (prior to this VMware Host Containers had to be used). VIC supports vSphere version 6.0 and upwards and is primarily targeted at Enterprise customers wishing to provide a managed container hosting environment within their own infrastructure.

Container Service Extension (CSE) for vCloud Director

Sitting somewhat in between the other offerings, VMware has also released CSE via an open source Github repository. CSE is targeted at Service Providers using VMware’s vCloud Director platform who wish to make delivering container hosting to tenants much easier. It provides an extension to vCloud Director which allows the creation and maintenance of clusters of VMs providing Docker in Kubernetes clusters.

Comparing the solutions

The table below shows a summary of the options

 

Solution Pivotal Container Service (PKS) vSphere Integrated Containers (VIC) Container Service Extension (CSE)
Current release / link v1.0.0 GA v1.3.1 v0.4.2
Container Runtime Docker Docker & Virtual Container Host (VCH) Docker[1]
Container Management Kubernetes VMware Admiral Kubernetes[1]
Container OS BOSH Virtual Container Host (Photon OS based) Any (Ubuntu & Photon provided)
Container Registry VMware Harbor VMware Harbor Any (None provided)
Deployed to Bare metal / VM Bare metal / vSphere VM vCloud Director Virtual Datacenter (VDC)
Multi-tenant Supported Yes Yes Yes
Network Support VMware NSX-T vSphere & VMware NSX-V Org VDC Networks (vCloud Director) / VMware NSX-V
Licensing / Support Open Source, Paid Support available from Pivotal Open Source, vSphere S&S Support covers VIC Open Source, Service Provider Support
Primarily Targeted At Service Providers & Enterprise using containers at scale Enterprise Service Provider / vCloud Tenants

[1] CSE allows service providers to provide any versions of Docker and Kubernetes in their templates. This can allow much more up-to-date versions than those supported in PKS or VIC.

CSE deployment for a Service Provider

I’ve recently been involved with deploying CSE to our own vCloud Director hosting platform, the VMware github.io page is extremely useful and well documented to help get up and running with CSE so I won’t repeat this here.

The main advantages it offered us as a service provider:

No new billing / Integration required
This is a huge deal for most service providers, it can be time-consuming (and therefore expensive) to integrate any new platform offering, not just in the time taken to deploy the components and get them all working correctly (including alerting, monitoring etc.) but what is often overlooked is the additional effort required to correctly meter platform consumption and ensure that customer bills are correctly prepared and reflect the resources their environments have consumed. Taking the ‘full stack’ of PKS and offering this as a service would involve considerable work, but with CSE this workload is effectively neutralised since the clusters deployed are directly into tenant virtual datacenters (VDCs) and service providers will already be metering and billing customers for resources consumed in tenant VDCs.

No new licensing
As there is no additional licensing for CSE this makes it extremely easy to deploy in a service provider platform.

No new security model
Since all tenant interaction with CSE is via the vCloud Director API, there is very little work required (if any) to publish the service to customers since most Service Providers will already be making the vCD API accessible to their tenants. Additionally, since the CSE service itself integrates directly into vCloud Director’s RabbitMQ backend it is likely that very few security or firewall changes are required either.

Flexible environment
One of the really nice aspects of CSE is that the templates made available to tenants to deploy into clusters are fully customisable. This means that service providers can chose to offer additional templates beyond the 2 examples provided ‘out of the box’ with CSE. For example, if a Service Provider wishes to offer a ‘bleeding edge’ template which has the absolute latest releases of Docker and Kubernetes (and maybe add additional packages to the deployed images to include Harbor and maybe a ceph or glusterfs client) this is reasonably straightforward and easy to do. The downside of this is that maintenance and updating these templates has to be performed regularly to ensure that they include all appropriate bug-fixes and security patches and updates.

Note that at this time the VMware documentation doesn’t yet include instructions for modifying or adding additional CSE templates, I’ll write up a separate post on how I did this in our environment which may prove useful for others deploying CSE into their own environments.

Other CSE Considerations

Of course no platform is ever perfect, and the following should be noted too as potential pitfalls or things to be aware of when considering CSE:

No registry service by default
Both PKS and VIC provide container registry services (to deal with storing, securing, scanning and replicating container images) based on VMware’s Project Harbor which is a very nice registry system. While Harbor can be added to clusters deployed with CSE, it isn’t there by default in the templates currently provided with CSE.

No persistent or dynamic Kubernetes volumes
Containers by design are meant to be ephemeral and stateless, so they shouldn’t be storing any persistent data or require backup protection. Of course most business applications (including those provided by containerised images) generally need some form of permanent/persistent storage behind them. In Kubernetes environments this is generally accomplished by the concept of persistent volumes which are mapped into containers at runtime and allow data to be retained. In CSE currently there is no provider for persistent volumes which means that external storage is required. This can however be delivered from a variety of sources – other databases running in the environment, file or object storage services etc. I’m currently looking into easy ways to add dynamically provisioned persistent volumes to a CSE cluster and will write this up as a separate post when done.

Template maintenance
As mentioned previously, the templates deployed by CSE are completely flexible and can be easily customised by editing their deployment scripts, the process of maintaining the templates is reasonably manual though and requires stopping the CSE service, patching and updating the templates in a vCloud Director shared catalog and then re-enabling the CSE service. It would be nice to have a way to automate the rebuild of templates and to allow the CSE services to remain online while this is happening.

Relative immaturity
The CSE service is a very ‘early’ release and a number of bugs are still being fixed. There’s nothing too serious that I’ve encountered yet, but occasionally templates will fail to build correctly (generally due to failures in 3rd party repositories) and it can take time to identify and resolve these issues. Fortunately the VMware developers have been extremely fast and active in responding to issues raised in the CSE github repository and every issue I’ve found has been very quickly fixed.

Summary

Hopefully this post has given you an idea of the capabilities and features available in the 3 current VMware container hosting solutions and given you a better idea of what the Cloud Service Extension for vCloud Director does. I’m aiming to write some follow-up posts on CSE including how we have deployed it into our environment, how new templates can be created (and existing templates customised) and how to address some of the current missing features such as integrating Harbor as a registry service in future posts. Let me know in the comments if there are any areas you are particularly interested in and I’ll see what I can do. I’ve also written a session abstract proposal to present a Service Provider view of CSE at VMworld US 2018, so hoping that that will be accepted too.

References / Links

Some of the components mentioned may not be familiar so I’ve provided links to each one below:

BOSH: https://bosh.io/
Docker: https://www.docker.com/
Harbor: https://vmware.github.io/harbor/
Kubernetes: https://kubernetes.io/
Ubuntu Linux: https://www.ubuntu.com/
VMware Photon OS: https://vmware.github.io/photon/

As always, comments & corrections welcome, I’m reasonably new to the whole ‘containerised applications’ scene so there may well be inaccuracies in this post(!)

Jon.

vCloud Director Extender – Part 5 – Stretch Networking (L2VPN)

In this 5th part of my look into vCloud Director Extender (CX), I deal with the extension of a customer vCenter network into a cloud provider network using the L2VPN network extension functionality. Apologies that this post has been a bit delayed, turned out that I needed a VMware support request and a code update to vCloud Director 9.0.0.1 before I could get this functionality working. (I also had an issue with my lab environment which runs as a nested platform inside a vCloud Director environment and it turned out that the networking environment I had wasn’t quite flexible enough to get this working).

Update: an earlier version of this article didn’t include the steps to configure the L2 appliance settings in the vCloud Director Extender web interface – I’ve now added these to provide a more complete guide.

Links to the other parts of this series:
Part 1 – Overview
Part 2 – Cloud Provider / Service Provider installation and configuration (MyCloud)
Part 3 – Customer / Tenant installation and configuration (Tyrell)
Part 4 – Customer / Tenant connecting to a Cloud Provider and Virtual Machine migration (Tyrell)

I won’t deal with the use-case here that the customer already has NSX networking installed and configured, since in most cases you can simply create L2VPN networks directly between the customer and provider NSX Edge appliances and don’t really need to use the CX L2VPN functionality.

In order to be able to use the standalone L2VPN connectivity, the following pre-requisites are required:

  • A tenant vSphere environment with the vCloud Director Extender appliance deployed (it does not appear to be necessary to deploy the replication appliance if you only wish to use the L2VPN functionality, but obviously if you are intending to migrate VMs too you will need this deployed and configured as described in Part 3 of this series. In either case you will still need to register the cloud provider in the CX interface.
  • A configured vCloud Director VDC for the tenant to connect to. This environment must also have an Advanced Edge Gateway deployed with at least one uplink having a publicly accessible (internet) IP address. Note that you do not need to configure the L2VPN service on this gateway – the CX wizard completes this for you.
  • At least one OrgVDC network created as a subinterface on this edge gateway. The steps to create a suitable new OrgVDC network are detailed below.
  • Outbound internet connectivity to allow the standalone edge deployed in the tenant vCenter to communicate with the cloud-hosted edge gateway – only port 443/tcp is required for this.
  • Administrative credentials to connect to both the tenant vCenter and the cloud tenancy/VDC (Organization Administrator role is required).

Opening the tenant vCenter environment and selecting the ‘Home’ page shows the following:

Selecting the vCloud Director Extender icon opens the CX interface:

If you have not yet configured the L2 appliance settings, selecting the ‘DC Extensions’ tab will show the following error:

To fix this, open the vCloud Director Extender web interface in a browser by opening https://<ip address of deployed cx appliance>/ and log in, select the ‘DC Extensions’ tab:

Select the ‘Add Appliance Configuration’ option and complete the form to provide the deployment parameters where the standalone NSX edge appliance will be deployed:

The ‘Uplink Network Pool IP’ setting is a bit strange – it appears to be asking for a network pool or IP range, but the ‘help text’ in the field is asking for a single IP address. I found that the validation on this field is a bit odd – it will basically accept any input at all (even random strings) without complaining, but obviously deployment won’t work. What you need to do is add individual IPv4 addresses and click the ‘Add’ button for each. You will need 1 address for each stretched network you will be extending to your cloud platform. In this example I am only extending a single network so have added a single IPv4 address (192.168.0.201).

Once you click the ‘Create’ button you will be returned to the ‘DC Extensions’ tab and shown a summary of the L2 appliance configuration:

Note that there doesn’t appear to be any way to edit an existing L2 Appliance configuration, so if you need to change settings (e.g. to add additional uplink IP pool addresses) you will likely need to delete and recreate the entire entry.

 

Next we need to add a new ‘subinterface’ network to our hosted Edge gateway appliance, logging in to our cloud provider portal we can select the ‘Administration’ tab and the ‘Org VDC Networks’ sub-option, clicking the ‘Add’ button shows the dialog to create a new Org VDC Network. We need to select ‘Create a routed network by connecting to an existing edge gateway’ and then check the ‘Create as subinterface’ check box:

Next we configure the standard network information (Gateway, Network mask, DNS etc.) Since this network will be bridged to our on-premises network we can use the same details. Optionally a new Static IP pool can also be created so that new VMs provisioned in the cloud service can use this pool for their IP addresses. This won’t be an issue for VMs being migrated as they will carry across whatever IP addresses are already assigned to them. Note that the gateway address is set to be the same address as the existing (on-premises) gateway – this means that re-configuring the default gateway setting in the guest OS isn’t required either:

Now we supply a name for the new Org VDC network and optionally a description. The check box can also be used if the customer has multiple VDCs and wishes to share the new network across them:

Finally the summary screen allows us to check the information provided and go back and make any changes required if not correct. The most important setting is to make sure the network is attached to the edge gateway as a subinterface:

Once finished creating, the Org VDC network will be shown in the list with a type of ‘Routed’ and an interface type of ‘Subinterface’:

Next we access the vCloud Extender interface from within the customer vCenter plugin, selecting the ‘DC Extensions’ tab takes us to the following dialog:

Selecting ‘New Extension’ shows the dialog to create a new L2 extension, the fields are mostly populated for you. The ‘Enable egress’ allows you to select which gateway(s) will be allowed to forward traffic outside of the extended network. In this example I’ve only configured egress on the Source (on-premises) side through the existing gateway:

When you click ‘Start’, the status will go to ‘Connecting’ and a number of activities will take place in the customer vCenter:

Reading from the bottom (oldest) upwards, a new port group is created, an NSX Edge Standalone appliance is deployed and powered-on and the new port group is reconfigured once this has completed (ignore the VM migration task, that just happened to occur during the same time window in my lab). In this case the new NSX standalone edge was named ‘mcloudext-edge-4’ and the port group ‘mcxt-tpg-l2vpn-vlan-Tyrell-VDC15’.

Once deployment has completed (takes a few minutes) the vCloud Extender client interface shows the new DC extension network with a status of ‘Connected’:

In the tenant vCloud Director portal you can also see the status of the tunnel under ‘Statistics’ and ‘L2 VPN’ within the edge gateway interface:

You will now find that any VMs connected to the stretched network (OrgVDC network) in your cloud environment have L2 connectivity with the on-premises network and will continue to function as if they were still located in the customer’s own datacenter.

As I mentioned at the start of this post, I hit a number of issues when configuring this environment and getting it working took several attempts and a couple of rebuilds of my lab. The main issue was that in the initial release of vCloud Director v9.0.0.0 there is an issue that prevents the details required for the standalone NSX edge being deployed from being returned by the API. This prevents the deployment of the customer edge at all and resulted in my VMware support call. The specific issue is referenced in the vCloud Director 9.0.0.1 release notes  as ‘Resolves an issue where the vCloud Director API does not return a tunnelID parameter in response to a GET /vdcnetworks request sent against a routed Organization VCD network that has a subinterface enabled.’ As far as I can work out, it will be impossible to successfully use L2VPN in CX without upgrading the provider to vCloud Director 9.0.0.1 to resolve this issue.

The other issue I hit in my lab was that my hosted ‘Tenant Edge’ was NAT’d behind another NSX Edge gateway which was also performing NAT translation (Double-NAT). This was due to the way my lab is built in a nested environment inside vCloud Director. Unfortunately this meant the external interface of my hosted ‘Tenant Edge’ was actually an internal network address, so when the customer/on-premise edge tried to establish contact it was using an internal network address which obviously wasn’t going to work. I solved this by connecting a ‘real’ external internet network to my hosted Tenant Edge.

As always, comments and feedback always appreciated.

Jon.

vCloud Director Extender – Network Ports

One of the things which appears to be missing from the published documentation on vCloud Director Extender (CX) is any mention of the communications internally between the deployed appliances and other VMware infrastructure components (vCenter, vCloud Director etc.) In a service provider context it is unlikely that the appliances will be deployed into the same network/security zone as these components so it is important to know what these communication requirements are.

Using the Flow Monitoring functionality in VMware NSX I was able to capture all traffic flows during vCloud Extender migrations and produce the drawing below detailing these traffic flows.

Network Traffic Flows for vCloud Extender (Provider Side)

 

Note that the http (tcp/80) access from the replicator appliance to the ESXi hosts appears anomolous – I would have expected this to be on https (tcp/443) at the very least and this probably needs further investigation.

The 8044/tcp port to the replication manager can be NAT’d from a different external (public) port if necessary – this can be configured using the ”Public Endpoint URL” field when activating the replication manager appliance during vCloud Extender deployment (see my post: http://kiwicloud.ninja/2017/10/vcloud-director-extender-part-2-cloud-provider-setup/).

The 44045/tcp port to the replicator appliance can also be NAT’d from a different external (public) port if necessary – this can be configured using the “Public Endpoint URL” field when activating the replicator appliance during vCloud Extender deployment  (see my post: http://kiwicloud.ninja/2017/10/vcloud-director-extender-part-2-cloud-provider-setup/).

Be careful when activating the “Replication Manager” and “Replicator” appliances – the configuration screens look very similar and it is reasonably easy to get them mixed up and enter incorrect parameters.

Also note that this diagram only depicts traffic flows for migration activity and doesn’t capture additional flows involved in L2 network extensions (which typically will be from a hosted NSX edge to either the tenant NSX edge or standalone NSX appliance in the tenant site).

At least the information presented should allow other service providers to configure appropriate network security to protect their internal vCloud and vSphere environments when deploying vCloud Extender components into a DMZ network (for example).

As always, comments and feedback appreciated.

Jon

vCloud Director Extender – Part 4 – Connect to Provider & VM Migration

In the first 3 parts of this series I covered an overview of vCloud Director Extender (CX), the installation and configuration of CX at the Cloud Provider site and the installation and configuration of CX at the customer/tenant site. In this 4th part I will be covering the configuration of the tenant environment to connect to the provider cloud and then migrate VM workloads to the provider.

This part follows on from the configuration completed in part 3 of this series and assumes that Tyrell (the customer site) have an existing virtual datacenter (VDC) environment available from MyCloud (the provider) and an appropriate Organization Administrator login to this environment. I’ve also created local DNS entries in the Tyrell network for the ‘chc.mycloud.local’ and ‘vcde.mycloud.local’ DNS names which resolve to the public IP addresses for the MyCloud vCloud Director instance and the provider CX endpoint respectively. Obviously in the real world these would be registered Internet DNS names.

In the Tyrell vCenter server when we select the ‘vCloud Director Extender’ icon we are shown an initial view of the CX plugin interface:

Selecting the ‘New Provider Cloud’ button opens a wizard to configure the connection to the Cloud Provider endpoints:

The ‘Provider Cloud URL’ needs to be set to include the appropriate path for the vCloud Director Organisation which is being connected to (the /cloud/org/Tyrell part in this example). The user details hold the Organization Administrator role within this cloud organisation.

When clicking ‘Add’ you will be presented with a certificate warning if the cloud provider is not using trusted/signed certificates, you can optionally select to trust these certificates if this is the case (very handy for a lab environment).

You can use the ‘Test’ button to confirm the settings are valid – you will see a status update at the bottom of the dialog showing the status of this test:

Note that even if the ‘Test’ succeeds, there are still some circumstances to do with network connectivity that can result in the enablement process failing – this is shown in the following capture from the ‘Provider Clouds’ tab where you can see the ‘Status’ shows ‘Enable Failed’:

This is usually caused by incorrect firewall rules, NAT rules or Public Endpoint URL’s set incorrectly when the CX appliances are deployed, I’m intending to cover this in a future ‘Troubleshooting’ part to this series of posts.

Once the networking and URLs are configured correctly you will see the new provider cloud registered under the ‘Provider Clouds’ tab with a status of ‘Running’, you will also see any accessible virtual datacenters (vDC) to which you have access:

Now that our provider cloud is properly registered, we can submit a migration request using the ‘Migrations’ tab in the CX interface, first we will be asked if we wish to perform a ‘Cold’ or ‘Warm’ migration – the differences between these are well explained in the dialog. Note that ‘Warm’ migration is not a vMotion, but does involve a period of network disconnection as the VM is cutover to the Cloud Provider. For this example we’ll select a ‘Warm’ migration:

Clicking ‘Next’ takes us to an inventory view where we can select the source VM(s) to be migrated. The grey panel below the ‘Inventory Browser’ dynamically expands to show candidate VMs from the vCenter environment. When a VM is selected the status and disk sizes are update in the right-side panel. For this example we’ve selected the ‘deckard’ VM:

Clicking ‘Next’ takes us on to the Target selection – here we can select the Cloud Provider, vDC, VM storage profile for the remote copy and the network to be connected to the VM in the Cloud Provider. Note that we are not L2-extending our on-premises network in this example and relying on our Cloud Provider (MyCloud) having already defined an Org vDC network for us (in this case called ‘Tyrell Servers’). All of the values are populated automatically from the vCloud Director environment and drop-downs allow easy select of other options. Finally we have the option when migrating multiple VMs together to group these into a single vApp rather than creating a new vApp for each VM:

In the final migration configuration step we can specify when the VM synchronisation should start, what our target Recovery Point Objective (RPO) is in minutes and whether to provision the destination disks as ‘Thin’ provisioned or ‘Thick’ provisioned. Finally we can add an optional tag to reference against this job later:

If everything has worked, you’ll now see a progress indicator against the VM in the Migrations tab. Initially the status will be ‘Created’:

Once data synchronisation begins this status will be updated to show the synchronised percentage for the migration. If you get an ‘Error’ prior to the sync percentage moving from 0% this is almost certainly a network configuration issue (and one which I encountered frequently when first building my lab environment). I’ll cover the common causes and remedies for this more in my vCloud Extender Troubleshooting post.

Once the initial synchronisation process has completed you will see the VM listed as ‘Cutover ready’ which means it’s staged and ready to be migrated:

Logging in to the Tyrell vCloud Director portal at this point shows that nothing actually has been provisioned into the Tyrell VDC:

Looking at the ‘Home’ page for the CX environment in vCenter shows our VM as in a ‘Transition’ state:

In the Migrations tab we can now select the ‘Start Cutover’ button to actually cutover the VM to the Cloud Provider environment which opens the Cutover dialog:

Clicking ‘Start’ asks for confirmation and then performs the actual cutover to running the VM in the Cloud Provider datacenter, progress is updated during the cutover procedure:

When the cutover process is complete you will see the Status update:

Looking in vCenter at this point shows the original VM still in place, but now powered off, you should probably take steps to ensure that this VM cannot be accidentally started at this point or risk having two running instances of the same VM (potentially on the same network if your network is extended to the Cloud Provider):

Refreshing the Tyrell vCloud Director portal shows the migrated VM now running in the Tyrell Cloud Provider VDC:

The status in the vCloud Extender vCenter plugin also now shows the completed migration total:

In the next part of this series of articles I look at the options to extend L2 networking directly from a customer site into vCloud Director using CX and the changes this introduces into the migration workflow.

Link back to Part 3 || Link to Part 5

As always, corrections, comments and feedback are always appreciated.

Jon.

vCloud Director Extender – Part 3 – Tenant Setup

In part 1 and part 2 of this series I detailed an overview of VMware vCloud Director Extender (CX) and the configuration from a Service Provider perspective to configure their platform to support CX.

This third article in the series details the configuration steps required for a tenant/customer environment to deploy and configure CX into their environment.

Once a service provider configuration is complete, any customers of that provider with sufficient allocated resources in a Virtual Datacenter (VDC) can configure the tenant CX environment and connect this to their vCenter environment. Once complete they will be able to migrate and replicate vSphere VMs between their own vCenter and the service provider datacenter extremely easily. Optionally they can use L2VPN functionality to stretch their networks into the Cloud Provider’s datacenter removing the requirement to have a pre-configured network in place. Of course many customers will wish to move to dedicated networking later, but having the initial ability to quickly provision their networks into a Cloud provider can dramatically shorten migration timeframes.

The initial deployment steps for customers deploying CX are exactly the same as for a Service Provider – download (or have provided to them by their Cloud Provider) the ova appliance for vCloud Director Extender and deploy this into their vCenter environment.

Right-clicking on the desired location and selecting ‘Deploy OVF Template…’ allows the local CX .ova file to be selected

The appliance name and folder are selected next:

Followed by the vCenter Cluster which will run the deployed appliance:

Check the template details and then click ‘Next’ to continue:

Read and accept the VMware license agreement:

Next select the Datastore storage on which the appliance will be deployed:

Select the required network for the appliance:

Make sure that ‘cx-connector’ (default) is selected for the ‘Deployment Type’ and fill out the IP addressing information for the appliance:

Check the summary information carefully and click ‘Finish’ to begin the deployment operation:

Once the appliance deployment task has configured, power-on the deployed VM in vCenter and wait for it to initialise. When it is running you can open a web browser to the IP address you configured for the appliance and login using the password configured. Note that you have to add ‘/ui/mgmt’ to the login URL for the appliance, so the full URL will be ‘https://<IP address of appliance>/ui/mgmt’:

The initial CX dialog when logged in allows you to start the Setup Wizard, note that in contrast to the Service Provider UI, there is no ‘Replication Managers’ tab in the cx-connector configuration:

The first step of the wizard is to link to the existing on-premise vCenter environment, note that if you are using an external Platform Services Controller (PSC) you will need to specify the PSC URL for the Lookup Service URL (although this is optional). The user specified needs to have administrative permissions within the vCenter environment:

Once the vCenter details and credentials are accepted, CX will provide a success notification, click ‘Next’ to continue:

The next page asks you to register the CX plugin with vCenter, this will likely become important in future as CX is updated, but for now leave the Version as 1.0.0 and click ‘Next’:

Once the plugin has registered into vCenter you will see a success notification. In testing I found that if the CX plugin had previously been registered with the vCenter (and not manually removed), this step would generate an error notification, but it was still possible to continue with the wizard and everything appeared to function fine afterwards:

Next you need to provide the configuration for the ‘Replicator’ appliance that will be deployed into the on-premise vCenter. The VMware documentation advises not to use DHCP for this and to manually specify a static IP configuration:

The ‘Replicator’ appliance is now deployed into vCenter and powered on. Once it has established network communication with the CX environment you will see a success notification:

The next step is to activate the Replicator appliance by providing a root password and authentication details for the on-premise vCenter environment. Note that you will need to set the Public Endpoint URL correctly in order for the appliance to be reachable by your cloud provider. If the on-premise Replicator appliance is behind a corporate firewall (as most will be), you will need to configure inbound firewall and translation rules and make sure this field is set correctly.

In my lab setup I configured the replicator public URL to be on port 443 on the public (Internet) address of the outside of the Tyrell firewall and used NAT port translation (see the networking configuration information below).

If everything is accepted you’ll receive a success notification in the wizard (note that I blanked the Public Endpoint URL field in this capture which is why it doesn’t show in the grab below):

The wizard is now complete, click ‘Finish’ to return to the UX interface:

The ‘vCenter Management’ tab should now show the on-premise vCenter details

The ‘Replicators’ tab should show the details for the replicator appliance deployed in the wizard:

Once vCenter has been closed and restarted you should now see a new ‘vCloud Director Extender’ item in the UI:

The networking configuration for a customer environment is a little simpler than for the cloud provider side, you will need to permit 2 inbound ports through the firewall, both of which need to communicate directly with the ‘Replicator’ appliance.

Assuming that you configured the ‘Public Endpoint URL’ with port 443, you will need to use NAT translation to divert this to port 8043 on the appliance:

Source Address Destination Destination Port/Protocol Translated Port/Protocol Translated Internal Address
External (Internet) Public IP Address 443/tcp 8043/tcp Replicator appliance internal address
External (Internet) Public IP Address 44045/tcp 44045/tcp Replication appliance internal address

You can (and should) limit the public/external addresses permitted to communicate with your Replicator appliance to just those public IP addresses used by your Cloud Provider – they should be able to provide you with this information.

Also note that if you restrict outbound internet traffic from your CX network you will also need to permit the following traffic in an Outbound direction:

Source Destination Source Port/Protocol Destination Port/Protocol Description
CX Server Network Cloud Provider Public CX Address Any 443/tcp Required for communications with the provider CX appliance
CX Server Network Cloud Provider Public CX Address Any 8044/tcp Required for communications with the provider Replication Manager appliance
CX Server Network Cloud Provider Public CX Address Any 44045/tcp Required for communications with the provider Replicator appliance

Of course if your provider has configured different ports for these components you will need to allow access to these instead of the defaults listed.

In the next part of this series I’ll continue with configuring the customer environment to connect to a cloud provider CX environment and to migrate some VMs.

Link back to Part 2 || Link to Part 4

As always, corrections, comments and feedback are always appreciated.

Jon.