First thoughts on AWS Re:Invent 2018 and where VMware is missing the point

Firstly I’ll start this off by saying that I don’t usually write opinion pieces – I’d much rather share some cool technology or tips that I’ve come across in my day job (or even playing with technology in my own time). Secondly, my perspective is likely a bit skewed – I work in one of the most virtualized and cloud-adopting countries in the world (New Zealand) where the vast majority of customers I speak to have no on-premises server environment any more or are planning to retire the ones they have. These customers have either shifted entirely to public cloud, or are using the services of providers such as my employer to provide local private cloud platforms for them.

In particular in Christchurch where I’m based we had a series of devastating earthquakes in 2010 and 2011 which accelerated this shift – many customers simply felt no need to rebuild datacenters when rebuilding their offices given available local provider and public cloud options open to them. Having a high-speed urban fibre network across the entire country has almost certainly helped to accelerate this trend.

For those not familiar, hosting providers such as ourselves who run a largely VMware software stack use the usual components (vSphere, vCenter, ESXi, NSX networking and sometimes VSAN storage) as the foundation of our platforms, but VMware provide an additional software layer ‘vCloud Director’ (vCD) which sits on top of all of these to provide a secure multi-tenant platform. It also provides a feature-rich public-facing portal and API to allow orchestration and automation as well as being extensible by a plugin architecture so that Operations Management, Backup/Recovery, Replication, Container hosting (to name 4) and other services can be easily integrated and published to our customers. The way this architecture has been implemented also makes it reasonably easy for 3rd parties to write their own vCD extensions and have these seamlessly published into the same environment.

Recent improvements in vCD 9.5 have included the move to a native HTML5 portal, the addition of many more customization and configuration options, support for multi-site deployments where customers consume resources in multiple physical locations, as well as new networking functionality allowing seamless networking across multi-site deployments. In addition, the new tenant portals for vCloud Availability Cloud to Cloud DR and vRealize Operations allow customers to completely manage their DR replication and failover as well as gain operational insights and management of their deployed workloads. This blog post covers the most recently enabled functionality in vCloud Director for those that wish to find more information.

Many of our customers are also leveraging public cloud infrastructure platforms for a variety of reasons including advanced features, hyper-scale elasticity, ease of operations and management and (often) to decrease their overall IT infrastructure spend. Often overlooked though is that the main driver for many customers is to have their internal IT teams concentrating on business applications and data – looking for ways to add business value to their organizations rather than in ‘feeding and watering’ infrastructure in their own datacenters. In fact many of the reasons customers chose a provider such as ourselves are very similar. The determining factors are often older applications that can’t survive at the reasonably significant latencies which are inevitable from New Zealand to our closest public cloud platforms in Australia, and some concerns around data sovereignty (although these are largely diminishing).

Of the AWS announcements made this week at their annual Re:Invent , AWS Outposts is a fascinating platform proposition which, if done well, will be a great option for a local AWS consistent platform in local (NZ) datacenters. I’m also  impressed with the announcements for new and enhanced AWS services – S3 Glacier Deep Archive could well spell the ‘final’ end of tape as a data archival technology for example. AWS Control Tower as a simplified way to easily deploy a landing zone into AWS is another service which I think will resonate well with many of our customers who find it challenging to deploy their initial AWS footprints with appropriate security, controls and governance. AWS TimeStream finally provides a ‘proper’ way of dealing with time-series data in huge volumes without trying to squish it into a relational database with all the issues that creates. Perhaps the most interesting is AWS RDS on VMware which was announced back in August at VMworld and allows AWS RDS services to run in a vSphere environment in a local datacenter with support for data replication and DR.

So with all that said, why do I think VMware is missing the point? – after all there are some great technologies and services being made available from both vendors.

Let’s take a look at some new/recent VMware products and services and see:

1) VMware Cloud Assembly

A great new technology to allow easy construction of templates and blueprints to speed deployment of application environments to multiple cloud endpoints. Supports all the major public cloud endpoints (AWS, Azure, GCP) as well as vCenter as the deployment endpoint.

2) VMware HCX (Hybrid Cloud Extension)

Awesome technology which allows live-migration of running business applications between vSphere sites (and even between vSphere on-premises and VMware cloud on AWS).

3) AWS RDS on vSphere

Mentioned above, but provides capability to run AWS consistent database services from all major RDS providers (MariaDB, MySQL, PostgreSQL, Oracle and SQL Server) in an on-premises vSphere environment. Can even allow these databases to span both an on-premises and AWS environment to provide scale, high availability and DR options.

4) VMware Cloud on AWS

Fantastic option that gives customers the option of deploying vSphere environments directly into public cloud and run their VMs with no changes whatsoever in that platform. Also with HCX (above) can live-migrate workloads in and out of public cloud. Provides consistent management, operations and security options across both platforms.

What do all of these have in common? You’ve probably guessed it – not a single one of them works with vCloud Director. If a customer wants to use HCX (for example) to seamlessly move workloads from a VCPP provider to AWS and back – not supported. Deploy Cloud Assembly blueprints to their VCPP provider? Nope, doesn’t work either, no support for the vCD API as an endpoint. Allow them to use AWS RDS services alongside their VCPP provider hosted VMs? No there also.

Basically it comes down to this: If you build a tool or service that only talks to vCenter (or vCenter APIs) and not to vCloud Director, you are missing out on making your products and services available to a large number (around 4,200 I believe right now) of VMware Cloud Provider Partners (VCPP) such as ourselves that offer vCloud Director as the primary interface and API for customers to manage their workloads. What’s more, from figures mentioned by VMware themselves, the number of workloads in VCPP provider datacenters managed through vCD is increasing massively ahead of vSphere and vCenter on-premises solutions.

One of the likely comments I’ll get to this post is ‘Well, you could just provided dedicated vSphere environments for each customer that needs these functions’. This is accurate – we definitely could do this, but the overhead of managing and maintaining a large number of discreet vSphere instances (including all of the management and operations tooling that these require) doesn’t scale well and would result in a huge amount of extra work. In addition, because we can’t securely multi-tenant vSphere environments there would be a huge amount of wasted capacity on hosts which aren’t heavily loaded or only exist to provide cluster hardware redundancy. This would make the solutions incredibly expensive by comparison to a true multi-tenanted platform.

So… if anyone from VMware is still reading by now… you’ve given your VCPP partners and providers an awesome platform in vCloud Director to allow a true multi-tenanted cloud platform, uptake and usage of this platform is in massive growth right now. Now please make sure the rest of your technologies can work with it.

As always, comments and feedback appreciated, how do other VCPP providers feel about this?

Jon

Update:

A few days after I first posted this, I saw this tweet from Steve Dockar pop up in my news feed linking to this youtube video which shows a preview version of Cloud Assembly using vCloud Director as the endpoint. This is awesome, and something which the VMware people I spoke to on the show floor at re:Invent obviously knew nothing about.

vCloud Director 9 HTML5 Portal Customization

One of the great features in vCloud Director 9 which has been further enhanced in the latest v9.5 release is the new HTML5 portal:

image

Even better, VMware has released a toolkit to allow Service Providers to fully customise the look and feel of the portal using CSS themes in their Clarity framework..

The toolkit itself is part of the VMware vcd-ext-sdk repository on github, available in the /ui/theme-generator folder.

The repository has good instructions on how to modify and build a custom theme, but actually uploading and configuring the theme in vCloud Director is only accessible via the vCD API and involves a reasonable amount of manual work.

To help speed up development and allow changes to be easily tested, in my usual mode I’ve written a small PowerShell module that allows quicker/easier theme configuration. The module is available on github at https://github.com/jondwaite/vcd-h5-themes. Hopefully this will help those of you who need to develop and test updated themes for your vCloud Director portals.

I’ve included documentation in the repository on each cmdlet, its function and arguments here.

To use the module you’ll need to be connected to a vCloud instance as a user with global ‘Administrator’ access in the ‘System’ organization since changes will affect all portal users. You’ll need to be connected to the vCD environment with PowerCLI (Connect-CIServer…) prior to using the cmdlets.

You can then download the vcd-ht-themes.psm1 file and add it to your session (‘Import-Module vcd-h5-themes.psm1’) to access the cmdlets.

As always, comments and feedback welcome – is there anything else you’d like to see added to this module?

Jon.

VM Guest Customization in vCloud Director via PowerCLI

Bit of a quick post this, but hopefully useful to others.

I got asked recently if there was an easy way to set Guest Customization options for VMs hosted in vCloud Director via Powershell/PowerCLI. It turns out there is an extremely simple way, but the syntax is a bit awkward so figured it would make a good/quick blog post.

The Guest Customization settings are available as one of the ‘Section’ entries returned by accessing the ExtensionData properties on a CIVM object. Once connected (Connect-CIServer) you can see this from PowerCLI:

image

The ‘trick’ is that there are typically 5 sections (one each for OvfVSSD, OvfMsg, network connections, guest Customization and VMware tools). I’ve seen some approaches that rely on the ‘guest Customization’ setting always being found at the Section[3] index in the ExtensionData collection, but this could easily change in future and break any functionality relying on this. A much more reliable way of finding the guest Customization section values is:

image

But how about if you need to change/update a setting, luckily there is a method provided (UpdateServerData) which does exactly this. So if we want to (for example) change the ‘CustomizationScript’ setting to “echo “Hello World!” we can:

image

You can change other settings using the same method (e.g. ComputerName or Domain join settings).

Note that for many changes the VM must be powered off, and you may need to ‘Power On and Force Recustomization’ too.

As always, comments & feedback appreciated.

Jon.

Getting detailed VM Disk Properties from the vCloud API

Since vCloud Director 8.10 VMware have allowed VMs to be created which have multiple disks using different storage policies. This can be very useful – for example, a database VM might have it’s database on fast storage but another disk containing backups or logs on slower/cheaper disk.

When trying to find out what storage is in use for a VM though this can create issues, the PowerCLI Get-CIVM cmdlet (and the Get-CIView cmdlet used to get extra information) aren’t able to properly report storage for VMs that consume multiple storage policies. This in turn can create problems for Service Providers when they need to report on overall VM disk usage divided by storage policy used.

As an example I’ve created a VM named ‘test01’ in a customer vDC which has 3 disks attached, the 2nd of these is on ‘Capacity’ tier storage while disks 1 and 3 are on ‘Performance’ storage. When we look at the VM details we see the following:

image

Digging into the ExtensionData shows

image

The StorageProfile element looks like it may contain what we need, but unfortunately this only shows the ‘home’ Storage for the VM and doesn’t indicate that at least one of the VMs disks is on a different storage profile:

image

After a lot of mucking around trying to find an easy way to discover the information, I ‘gave up’ and wrote a PowerShell module which accesses the vCD API directly to get the VM storage information (including storage tiers in use by each disk). The module isn’t overly efficient since it queries the storage profile reference for every disk on every VM (and so will result in a lot of calls if run for a large number of VMs), but otherwise works fine.

The module takes VM objects or a VM name as input and returns details on each disk attached to the VM including which storage profile they use. Save the script (e.g. as ‘Get-CIVMStorageProfile.psm1’) and then use ‘Import-Module .\Get-CIVMStorageProfile.psm1’ to import the function.

And here is example output from the script for our test VM:

image

Hope this is useful to some of you and as always, appreciate any comments/feedback.

I’d also love to know if there’s an easier way of generating this information.

Jon.

Tenant Portal Displays ‘No Datacenters are available’ in vCloud Director 9.1

We had an issue recently when updating our vCloud Director environment to v9.1 where the new tenant portal would show ‘No Datacenters are available’ for every tenant even though the remainder of the site worked correctly (and other tabbed options like the Service Library & catalogs worked fine). Initially we suspected that our SSL certificate chain or public URI’s were set incorrectly.

Adrian Begg has a great blog post here: http://www.pigeonnuggets.com/2018/03/vcloud-director-9-1-tenant-portal-displays-no-datacenters-available-after-upgrade/ which details this issue and how to ensure the correct settings are applied, however in our case this didn’t resolve our issue.

Eventually an offhand remark in a slack channel by Tom Fojta put me on the right track to solving the issue, I’ve written this post up in case anyone else comes across the same issue. If you’re impatient and want to know the solution – it’s DNS (isn’t it always DNS?), but that’s jumping ahead a bit.

In our environment we have 3 vCloud Director cell servers behind a load balancer, we also load-balance internally so that our management environment can talk to the vCD API and we can conduct testing of the environment without necessarily having it open to the public internet. The arrangement looks logically like this:

 

vCloud Director Load Balancing

Users from the internet accessing ‘portal.cloud.com’ get redirected to one of the vCD cell servers (and if one of them is unavailable the monitoring on the Load Balancer doesn’t direct requests there). The same happens for internal users, but in this case the ‘portal.cloud.com’ DNS entry has been overridden to point at the internal (192.168.0.10) address to allow connectivity to the cells even if the external LB or internet link is unavailable.

The issue in our environment was that the cell servers themselves use DNS to access the vCloud API – and they use the public URL specified in the vCloud Director configuration.

The cell servers were configured with our internal DNS servers, so when they attempted to access the public URL (‘portal.cloud.com’) were being given the internal Load Balancer address (192.168.0.10). For reasons we’re still exploring, this didn’t allow them to get a response from the vCD API and resulted in the ‘No Datacenters are available’ error in the tenant portal.

The fix turned out to be reasonably simple – on each cell server we added an entry to the /etc/hosts file to resolve the public URL to the cell’s own IP address, so on cell 01:

192.168.0.11    portal.cloud.com

On cell02:

192.168.0.12   portal.cloud.com

And on cell03:

192.168.0.13    portal.cloud.com

Once we’d made this change the tenant portals began functioning correctly (note that no restart of the cell servers or vCloud Director services was required).

What I assume is happening is that when the internal load balancer responds the the request it gives out a different cell server’s address (since the ‘source’ of the request will be a cell server) and that cell server has no knowledge of the session being used by the original cell and so responds incorrectly (either with nothing, or with an error). Not sure if this is actually a bug, or just something to be aware of, but either way overriding name resolution in this way fixes the issue. Note that simply using ‘localhost’ or 127.0.0.1 for the hosts file entry doesn’t work since the vCloud web server doesn’t respond on the loopback interface in the default configuration.

Just posting this here in the hope it will save someone else any frustration caused by this issue.

Jon.

Writing vRealize Orchestrator Workflows for vCloud Director v9.1

One of the great new features in vCloud Director 9.1 is the ability to publish Orchestrator workflows to tenants which can be consumed from within their vCloud portal. Markus Kraus has written an excellent post showing the configuration process for linking Orchestrator and vCloud Director. This post shows the process for creating and deploying a workflow into this environment and shows the user experience when invoking the finished workflow. I can see a large range of possible use-cases for this capability – hopefully this post will give you an idea of what is possible.

Our demonstration workflow will be reasonably simple and cover a scenario where a tenant consuming an Allocated VDC needs more resources assigned to it. While this could be fully automated (directly make the changes to the vCloud Director VDC), it is probably more likely in these scenarios that the service provider would want to check and implement the change themselves. Therefore the following actions are required. The workflow tasks are therefore:

  • Extract the environment details (so we know which tenant user in which vCloud Organisation initiated the request.
  • Allow the tenant to enter the parameters for the extra resources they require.
  • Send an email to the service provider that contains the request details.
  • Send an email to the tenant user confirming the request details.

This example assumes that you have a functioning vRealize Orchestrator instance in the vCloud Director environment, and that you’ve registered the vCloud Director endpoint in Orchestrator. It also requires that you’ve configured the Orchestrator integration in the vCD provider portal and granted the new Service Library permissions to the tenant Organization Administrator role.

From the Orchestrator client, we first create a new workflow in Design view (I’ve created a folder to contain this called ‘vCD Workflows’ too), I’ve called my workflow ‘Request VDC Resources’:

image

The new (empty) workflow is created and displayed in the vRO editor:

image

Our first task is to create some workflow inputs to capture required inputs, these can be created in the ‘Inputs’ tab using the ‘Add parameter’ button:

image

We need the following information from the user to be able to process this workflow, so I’ve created the parameters, given them the correct Type and set a description for each one. In my lab environment there are two classes of storage profiles (performance and capacity) so I’ll ask the user to provide values for both if requesting a storage increase. To change the parameter names, types and descriptions just click in the fields. The final ‘Inputs’ tab now looks like this:

image

Now we can alter the Presentation tab to group the input fields appropriately. It can be a bit fiddly to add groups (Orchestrator adds steps each time which need to be removed), but with a bit of fiddling about you can get to something like this:

image

Next we need to add some attributes to our workflow to contain the subject and content of the email message together with some parameters to be passed to the send mail workflow, this can be done on the ‘General’ tab using the ‘add attribute’ button:

image

So that should be all of the information we require, next step is to go to the Schema tab and add a Scriptable Task by dragging the element between our start and end markers:

image

Double clicking on the title ‘Scriptable task’ allows us to set a friendly script name (‘Build Resource Email’ in this example). We can then select the ‘In’ tab for the script and select our input attributes and parameters:

image

We select the in-parameters (‘tenantEmail’, ‘addCPU’, ‘addRAM’, ‘addStoragePerf’ and ‘addStorageCap’) from this dialog and get the following listed under our script’s ‘IN’ tab:

image

We do the same to bind the ‘emailSubject’ and ‘emailContent’ attributes to our script’s ‘OUT’ tab:

image

Switching to the ‘Visual Binding’ tab, (sliding the edit pane up a bit for clarity) should now show something like the following:

image

We can now switch to the Scripting tab to actually write the code that will generate our email subject and body:

image

The code from this script is included below in case copy/paste is useful:

There’s nothing too complicated going on in the script, just building some HTML strings based on the input values we’ve received from the workflow. We query the _vdc_orgName, _vdc_orgId and _vdc_userName from our script environment to retrieve these values which are provided by the vCloud Director Service Library so we know which user in which tenant organisation has initiated the workflow.

Next we need to add the ‘send notification’ workflow to actually send an email, this can be dragged from the Mail folder under ‘All Workflows’ and placed after our script in the workflow Schema:

image

This first ‘Send notification’ will be the email sent to the service provider so I’ve renamed it as ‘Mail Provider’ and selected the ‘Source parameter’ items on the ‘IN’ tab as follows:

image

(To set each source parameter simply click on the ‘not set’ value and select the appropriate workflow attribute from the pop-up, ensure that unused parameters are set as NULL).

We can add a 2nd ‘Send Notification’ task to our workflow and configure it to send the same email content back to our tenant’s email address (the only change here is that the ‘toAddress’ parameter is set to the provided workflow tenant email address rather than the provider email address):

image

Our workflow is now complete and should be functional, we can ‘Validate’ and then Save and Close it in the Orchestrator client.

Our next step is to publish the workflow from the the provider interface of vCloud Director at https://<my vcloud IP>/provider

One logged in we see the following:

image

Selecting the 3-bars (highlighted) option and selecting ‘Content Libraries’ shows the Service Library:

image

First we need to select ‘Service Mangement’ and then ‘Service Categories’ tab to create a new Service Category (group) into which our workflow will be imported:

image

Clicking the ‘+’ sign allows us to define a new category and provide an icon for it:

image

Once saved we can return to the ‘Service Library’:

image

The ‘Import’ option allows us to add our workflow to the library, first we select the category we just created:

image

Next we select the source Orchestrator instance where our workflow lives:

image

Now we can browse the workflow tree and select our new workflow:

image

Finally we review and select ‘Done’ to create the library entry:

image

Using the ‘Manage’ button we can chose who the workflow should be published to:

image

If we don’t select ‘Publish to All Tenants’ then we can chose individual tenancies with the check boxes:

image

Clicking Save completes the process and publishes our workflow.

Note: I’ve had several instances where changes to publishing are not saved correctly, I’d suggest going back into the settings and checking these after making any changes.

Now if we log in to vCD as a tenant we can see the published workflow in the ‘Libraries’ option:

image

image

Clicking ‘Execute’ initialises our workflow and requests the input parameters we defined for the workflow:

image

Here I’ve asked for 16GB more RAM, 10GHz more CPU, 2TB more Capacity storage and 1TB more Performance storage.

Clicking ‘Finish’ submits the request and we see in our email that they have arrived populated with the details from our workflow:

image

This is of course a fairly basic example, and not designed to be useful ‘as is’, but hopefully has given you a good idea of the power and flexibility that Service Libraries introduce in vCloud Director v9.1 and given you some ideas of how they can be used.

Generating emails is a trivial example, but much more complicated workflows could easily be built (for example, to directly submit API requests to other systems as well as directly provisioning resources in vCloud Director on request).

As always, comments and feedback appreciated.

Jon

Using vCloud Director PowerCLI and vcd-cli with Federated User Accounts

One of the issues that vCloud Director user can run into is user authentication when using the PowerCLI and vcd-cli tools to manage their cloud deployments. For ‘Local’ user accounts defined in the vCloud Director portal this isn’t an issue as username/password are stored in the vCD database and can be directly authenticated. However, many customers want to federate their vCloud users with an external directory service (often Microsoft AD FS or other similar service). Typically this is done so that security groups in the external directory can be used to control access levels, and so that additional authentication mechanisms like 2-Factor Authentication (2FA) can be applied to accounts.

If you attempt to use CLI tools like vcd-cli or PowerCLI to authenticate with a federated user account you will get a ‘Login Failed’ or ‘Unauthorized’ failure and won’t be able to connect to the service.

Fortunately, both vcd-cli and PowerCLI allow you to use an existing browser vCloud session ID to connect to the vCD API. To use this you connect to your vCloud portal in a web browser and then then use your browser’s tools to find the session ID for your connection. Once you have the session ID you can create a PowerCLI or vcd-cli session using that token.

It can sometimes be easier to use a browser plugin or extension to help find the session ID, ones which show session cookies and/or HTTP headers work best, but even without these it is possible.

In Google Chrome for example, use <ctrl + shift + I> (or Menu / More Tools / Developer Tools) to open the developer interface. Next click on the ‘Network’ heading at the top of the developer panel and refresh the vCloud Director portal. Scroll down to one of the ‘amfsecure’ document lines and select the ‘Headers’ tab, you should see a panel similar to this:

image

You can simply copy the value from the highlighted entry (87489f6a17044d66bc36704ce5c4e45c in this example) and use that to establish a vcd-cli or PowerCLI session:

For vcd-cli:

vcd login <cloud endpoint> <org name> <user name> –d <session ID string>

e.g.

vcd login mycloudprovider.com myorg joebloggs –d 87489f6a17044d66bc36704ce5c4e45c

For PowerCLI:

Connect-CIServer –Server <cloud endpoint> –SessionID <session ID string>

e.g.

Connect-CIServer –Server mycloudprovider.com –SessionID 87489f6a17044d66bc36704ce5c4e45c

You will then be connected as the same user from your browser session and able to run all the PowerCLI or vcd-cli commands with that user account.

An easier way?

Rather than digging around for HTTP headers and cookies in a browser, vcd-cli has a built-in module which is meant to retrieve the sessionID from a browser session automatically and use this to authenticate, the syntax is:

vcd login session list chrome

Which should return the session ID from an instance of Chrome, but in my initial testing this was not returning any output at all.

Reading through the vcd-cli sources it appears that this option relies on a Python extension ‘browsercookie’ which can be installed using pip install --user browsercookie. Browsercookie has a dependency on the ‘pycrypto’ module which must also be installed. However, even with both pycrypto and browsercookie installed I couldn’t get this option to work.

I did manage to get this working by installing the browser_cookie3 module from https://pypi.python.org/pypi/browser-cookie3/0.6.0 by using pip install --user browser-cookie3 and then making the following changes in the vcd-cli\login.py file:

Line 24: Change:

from vcd_cli import browsercookie to: import browser_cookie3

On both lines 126 and 148: Change:

cookies = browsercookie.chrome() to: cookies = browser_cookie3.chrome()

Once these changes are complete the ‘vcd login session list chrome’ command can be used to obtain the current session ID from Chrome automatically:

image

And this can be used directly to login automatically once a Chrome session exists using the --use-browser-session switch.

Also note that you can obtain the session ID like this from vcd-cli and use it to authenticate a PowerCLI session with no issues at all.

Jon.

Using VMware Container Service Extension (CSE)

Yesterday I wrote showing the currently available container hosting options from VMware. As we’ve recently deployed one of these options – CSE in our environment I thought it would be useful to show a sample workflow on how the service functions and how customers can use this to deploy and manage both CSE clusters, and also micro-service applications onto those clusters.

There are a few requirements on the tenant side which must be completed prior to any of this working:

  • An Organizational Administrator login to the vCloud platform where CSE is deployed.
  • Access to a virtual datacenter (VDC) with sufficient CPU, Memory and Storage resources for the cluster to be deployed into.
  • An Org VDC network which can be used by the cluster and has sufficient free IP addresses in a Static Pool to allocate to the cluster nodes (clusters take 1 IP address for the ‘master’ node and an additional address for each ‘worker’ node deployed).
  • A client prepared with Python v3 installed and the vcd-cli and container-service-extension packages installed on it.
  • The {$HOMEDIR}\.vcd-cli\profiles.yaml file edited to add the CSE extension to vcd-cli.
  • The kubectl utility installed to administer the Kubernetes cluster once deployed and working. kubectl can be obtained most easily from here.

Detailed instructions for the client setup can be found in the CSE documentation at https://vmware.github.io/container-service-extension/#tenant-installation. Note that on a Windows platform the .vcd-cli folder and profiles.yaml file will not be automatically created, but you can do this manually by

from a DOS prompt and then using vcd-cli to log in and out of your cloud provider. This will cause profiles.yaml to be generated in the .vcd-cli folder. The profiles.yaml file can then be edited in your favourite text editor to add the required CSE extension lines.

Deploying a Cluster with CSE

When deploying a cluster, you will need to know the storage profile and network names which the cluster will use, the easiest way of obtaining these is either from the vCloud portal, or using the vcd vdc info command when logged in to your environment:

image

If you have multiple VDCs available to you, the ‘vcd vdc use <VDC Name>’ command to set which one to work with.

In this example we will be using the highlighted entries (the ‘Tyrell-Servers’ network and the ‘CHC Performance’ storage profile).

To retrieve a list of available cluster deployment templates that the Service Provider has made available to us we can use the vcd cse template list command:

image

In this example only the Photon OS template is available and is also the default template. CSE actually comes with 2 profiles (Photon OS v2 and Ubuntu Linux 16-04, but I’ve only installed the Photon OS v2 template in my lab environment). The default template will be used if you do not specify the ‘–template’ switch when creating a cluster.

The cluster create command takes a number of parameters which are documented in the CSE page:

image

Be careful with the memory specification is it is in MB and not GB.

I chose to generate a public/private key to access the cluster nodes without needing a password, but this is optional. If you want to use key authentication you will need to generate a key pair and specify the public key filename in the cluster creation command using the –ssh-key switch.

To deploy a cluster with 3 worker nodes into our VDC where each node has 4GB of RAM and 2 CPUs using my public key and the network and storage profile identified above:

image

The deployment process will take several minutes to complete as the cluster VMs are deployed and started.

In to the vCloud Director portal, we can see the new vApp that has been deployed with our master and worker nodes inside it, we can also see that all 4 VMs are connected to the network we specified:

image

To see the details of the nodes deployed we can use ‘vcd cse node list <cluster name>’:

image

To manage the cluster with kubectl, we need a configuration file for Kubernetes containing our authentication certificates. kubectl by default looks for a file named ‘config’ in a folder called ‘.kube’ under the current user’s home directory. The config file itself can be downloaded using CSE. To create the folder and write the config file:

image

If you have multiple deployed clusters you can create separate config files for each one (with different file names) and use the –kubeconfig= switch to kubectl to select which one to use.

To test kubectl we can ask for a list of all containers (‘pods’ in Kubernetes) from the cluster, the ‘–all-namespaces’ switch shows system pods as well as any user created pods (which we don’t have yet). This must be run from a machine that has network connectivity with the deployed nodes (the ‘Tyrell-Servers’ network in this example):

image

 

Cluster Scaling

Adding Nodes to Clusters

If we need to add worker nodes to a cluster this is accomplished with the ‘vcd cse node create’ command. For example, we can add a 4th worker node to our ‘myCluster’ cluster as follows:

image

The node list now shows our cluster with 4 worker nodes including our new one:

image

Removing Nodes from Clusters

To remove a cluster member is just as easy using the ‘vcd cse node delete’ command:

image

You will be prompted to confirm the node deletion, and if you have deployed container applications you should ensure that the node is properly drained and/or replica sets and deployments configured correctly so that the node deletion will not impact your applications.

 

Cluster Host Affinity

One item that CSE does not deal with yet is creating vCloud Anti-Affinity rules to ensure that your worker nodes are spread across different physical hosts. This means that with appropriately configured applications a host failure will not impact on the availability of your deployed services. It is reasonably straightforward to add anti-affinity rules in the vCloud portal though.

Our test cluster is back to 3 nodes following the deletion example:

image

In the vCloud portal we can go to ‘Administration’ and select our virtual datacenter in the left pane, we will then see an ‘Affinity Rules’ tab:

image

Clicking the ‘+’ icon under Anti-Affinity Rules allows us to create a new rule to keep our worker nodes on separate hosts:

image

Provided the VDC has sufficient backing physical hosts, the screen will update to show the new rule and that it has successfully been applied and separated the worker nodes to different hosts:

image

Of course if the host running the master node experiences a failure then this will be unavailable until the VMware platform restarts the VM on another host.

 

Application Deployment using kubectl

Of course now that our cluster is up and running, it would be nice to actually deploy a workload to it. The ‘sock shop’ example mentioned in the CSE documentation is a good example application to try as it consists of several pods running in a separate namespace.

First we use kubectl to create the namespace:

image

Now we can deploy the application into our name space from the microservices-demo project on github. You can read more about the sock-shop demo app at https://github.com/microservices-demo/microservices-demo.

C:\Users\jon>kubectl apply -n sock-shop -f "https://github.com/microservices-demo/microservices-demo/blob/master/deploy/kubernetes/complete-demo.yaml?raw=true"
deployment "carts-db" created
service "carts-db" created
deployment "carts" created
service "carts" created
deployment "catalogue-db" created
service "catalogue-db" created
deployment "catalogue" created
service "catalogue" created
deployment "front-end" created
service "front-end" created
deployment "orders-db" created
service "orders-db" created
deployment "orders" created
service "orders" created
deployment "payment" created
service "payment" created
deployment "queue-master" created
service "queue-master" created
deployment "rabbitmq" created
service "rabbitmq" created
deployment "shipping" created
service "shipping" created
deployment "user-db" created
service "user-db" created
deployment "user" created
service "user" created

We can see deployment status by getting the pod status in our namespace:

image

After a short while all the pods should have been created and show a status of ‘Running’:

image

The ‘sock-shop’ demo creates a service which listens on port 30001 on all nodes (including the master node) for http traffic, so we can get our master node IP address from ‘vcd cse node list myCluster’ and open this page in a browser:

image

And here’s our deployed application running!

image

Summary / Further Reading

Of course there’s much more that can be done with Docker and Kubernetes, but hopefully I’ve been able to demonstrate how easily a cluster can be deployed using CSE and how micro-services applications can be run in this platform.

For further reading on kubectl and all the available functionality I can recommend the Kubernetes kubectl documentation at https://kubernetes.io/docs/reference/kubectl/overview/. In fact the entire Kubernetes site is well worth a read for those considering deployment of these architectures.

As always, comments, feedback, suggestions and corrections always welcome.

Jon.

VMware Container Solutions

VMware appears to have gone a little ‘mad’ with regards to containerisation (or containerization for any American readers) lately. Last week saw the release of Pivotal Container Service (PKS) as launched at VMworld 2017 US back in August. With this there are now a total of three VMware technologies all enabling customers to run micro-service applications in their environments. So why three different products to do the same thing? Well, they are targeted at different environments and use-cases, and actually it makes a lot of sense for VMware to have solutions for all 3 scenarios. Of course there’s always the 4th option of building your own container hosting platform from scratch on a VMware platform, but lets concentrate for now on those provided by VMware.

So what are the available solutions?

Pivotal Container Service (PKS)

This was announced at VMworld 2017 and recently became available for download. PKS a full stack solution to manage both initial formation of clusters to support containerised applications and manage their ‘day 2’ operations once deployed. While PKS could be deployed in an Enterprise environment (and may be for organisations using containerised applications at significant scale) it appears to be more targetted towards cloud service providers wishing to offer a managed/hosted platform for multiple tenants.

vSphere Integrated Containers (VIC)

VIC has been around for a while now (this was based on VMware’s Project Bonneville which started back in 2015), recently VIC has been updated to v1.3.1 and gained the capability to use Docker hosts natively at version 1.2 (prior to this VMware Host Containers had to be used). VIC supports vSphere version 6.0 and upwards and is primarily targeted at Enterprise customers wishing to provide a managed container hosting environment within their own infrastructure.

Container Service Extension (CSE) for vCloud Director

Sitting somewhat in between the other offerings, VMware has also released CSE via an open source Github repository. CSE is targeted at Service Providers using VMware’s vCloud Director platform who wish to make delivering container hosting to tenants much easier. It provides an extension to vCloud Director which allows the creation and maintenance of clusters of VMs providing Docker in Kubernetes clusters.

Comparing the solutions

The table below shows a summary of the options

 

Solution Pivotal Container Service (PKS) vSphere Integrated Containers (VIC) Container Service Extension (CSE)
Current release / link v1.0.0 GA v1.3.1 v0.4.2
Container Runtime Docker Docker & Virtual Container Host (VCH) Docker[1]
Container Management Kubernetes VMware Admiral Kubernetes[1]
Container OS BOSH Virtual Container Host (Photon OS based) Any (Ubuntu & Photon provided)
Container Registry VMware Harbor VMware Harbor Any (None provided)
Deployed to Bare metal / VM Bare metal / vSphere VM vCloud Director Virtual Datacenter (VDC)
Multi-tenant Supported Yes Yes Yes
Network Support VMware NSX-T vSphere & VMware NSX-V Org VDC Networks (vCloud Director) / VMware NSX-V
Licensing / Support Open Source, Paid Support available from Pivotal Open Source, vSphere S&S Support covers VIC Open Source, Service Provider Support
Primarily Targeted At Service Providers & Enterprise using containers at scale Enterprise Service Provider / vCloud Tenants

[1] CSE allows service providers to provide any versions of Docker and Kubernetes in their templates. This can allow much more up-to-date versions than those supported in PKS or VIC.

CSE deployment for a Service Provider

I’ve recently been involved with deploying CSE to our own vCloud Director hosting platform, the VMware github.io page is extremely useful and well documented to help get up and running with CSE so I won’t repeat this here.

The main advantages it offered us as a service provider:

No new billing / Integration required
This is a huge deal for most service providers, it can be time-consuming (and therefore expensive) to integrate any new platform offering, not just in the time taken to deploy the components and get them all working correctly (including alerting, monitoring etc.) but what is often overlooked is the additional effort required to correctly meter platform consumption and ensure that customer bills are correctly prepared and reflect the resources their environments have consumed. Taking the ‘full stack’ of PKS and offering this as a service would involve considerable work, but with CSE this workload is effectively neutralised since the clusters deployed are directly into tenant virtual datacenters (VDCs) and service providers will already be metering and billing customers for resources consumed in tenant VDCs.

No new licensing
As there is no additional licensing for CSE this makes it extremely easy to deploy in a service provider platform.

No new security model
Since all tenant interaction with CSE is via the vCloud Director API, there is very little work required (if any) to publish the service to customers since most Service Providers will already be making the vCD API accessible to their tenants. Additionally, since the CSE service itself integrates directly into vCloud Director’s RabbitMQ backend it is likely that very few security or firewall changes are required either.

Flexible environment
One of the really nice aspects of CSE is that the templates made available to tenants to deploy into clusters are fully customisable. This means that service providers can chose to offer additional templates beyond the 2 examples provided ‘out of the box’ with CSE. For example, if a Service Provider wishes to offer a ‘bleeding edge’ template which has the absolute latest releases of Docker and Kubernetes (and maybe add additional packages to the deployed images to include Harbor and maybe a ceph or glusterfs client) this is reasonably straightforward and easy to do. The downside of this is that maintenance and updating these templates has to be performed regularly to ensure that they include all appropriate bug-fixes and security patches and updates.

Note that at this time the VMware documentation doesn’t yet include instructions for modifying or adding additional CSE templates, I’ll write up a separate post on how I did this in our environment which may prove useful for others deploying CSE into their own environments.

Other CSE Considerations

Of course no platform is ever perfect, and the following should be noted too as potential pitfalls or things to be aware of when considering CSE:

No registry service by default
Both PKS and VIC provide container registry services (to deal with storing, securing, scanning and replicating container images) based on VMware’s Project Harbor which is a very nice registry system. While Harbor can be added to clusters deployed with CSE, it isn’t there by default in the templates currently provided with CSE.

No persistent or dynamic Kubernetes volumes
Containers by design are meant to be ephemeral and stateless, so they shouldn’t be storing any persistent data or require backup protection. Of course most business applications (including those provided by containerised images) generally need some form of permanent/persistent storage behind them. In Kubernetes environments this is generally accomplished by the concept of persistent volumes which are mapped into containers at runtime and allow data to be retained. In CSE currently there is no provider for persistent volumes which means that external storage is required. This can however be delivered from a variety of sources – other databases running in the environment, file or object storage services etc. I’m currently looking into easy ways to add dynamically provisioned persistent volumes to a CSE cluster and will write this up as a separate post when done.

Template maintenance
As mentioned previously, the templates deployed by CSE are completely flexible and can be easily customised by editing their deployment scripts, the process of maintaining the templates is reasonably manual though and requires stopping the CSE service, patching and updating the templates in a vCloud Director shared catalog and then re-enabling the CSE service. It would be nice to have a way to automate the rebuild of templates and to allow the CSE services to remain online while this is happening.

Relative immaturity
The CSE service is a very ‘early’ release and a number of bugs are still being fixed. There’s nothing too serious that I’ve encountered yet, but occasionally templates will fail to build correctly (generally due to failures in 3rd party repositories) and it can take time to identify and resolve these issues. Fortunately the VMware developers have been extremely fast and active in responding to issues raised in the CSE github repository and every issue I’ve found has been very quickly fixed.

Summary

Hopefully this post has given you an idea of the capabilities and features available in the 3 current VMware container hosting solutions and given you a better idea of what the Cloud Service Extension for vCloud Director does. I’m aiming to write some follow-up posts on CSE including how we have deployed it into our environment, how new templates can be created (and existing templates customised) and how to address some of the current missing features such as integrating Harbor as a registry service in future posts. Let me know in the comments if there are any areas you are particularly interested in and I’ll see what I can do. I’ve also written a session abstract proposal to present a Service Provider view of CSE at VMworld US 2018, so hoping that that will be accepted too.

References / Links

Some of the components mentioned may not be familiar so I’ve provided links to each one below:

BOSH: https://bosh.io/
Docker: https://www.docker.com/
Harbor: https://vmware.github.io/harbor/
Kubernetes: https://kubernetes.io/
Ubuntu Linux: https://www.ubuntu.com/
VMware Photon OS: https://vmware.github.io/photon/

As always, comments & corrections welcome, I’m reasonably new to the whole ‘containerised applications’ scene so there may well be inaccuracies in this post(!)

Jon.

Cassandra SSL for vCloud Director metrics database

In vCloud Director v9 the requirement for both Apache Cassandra and KairosDB for storing metrics has been reduced to just Apache Cassandra. In addition, the ability to view VM metrics directly in the new  vCD 9 HTML5 tenant UI makes it much more important to have a reliable Cassandra infrastructure.

As I found when researching this post, configuring SSL in Cassandra is a bit of a pain, since Cassandra runs as a Java application it has some issues with the types of CA certificates and trusts it can use, which is further complicated by the options of both node to node encryption as well as cluster to clients.

I set out to produce an ‘easy’ way to configure a Cassandra cluster with SSL for both node to node and node to client communication in a way which could be reasonably easily implemented and reproduced for future installs.

My test environment consists of the minimum supported configuration of 4 Cassandra nodes (of which 2 are ‘seed’ nodes) running on CentOS Linux 7 (since that’s what we tend to use most for our backend infrastructure services). This configuration can almost certainly be adapted for other Linux distributions and I’ve tried to document the certificate generation process sufficiently that this will be straightforward.

Inspiration for this post was from Antoni Spiteri’s blog and script to configure a Cassandra cluster for vCloud Director metrics which I found extremely useful background.

Initial setup of my node servers used the following pattern to install and configure Cassandra:

Ensure all appropriate updates have been applied to our new CentOS installation if appropriate:

# yum update -y

Install Java (currently on my systems this installs java-1.8.0-openjdk.x86_64 1:1.8.0.161-b14.el7_4 which appears to work fine):

# yum install -y java

Create a file /etc/yum.repos.d/cassandra.repo (with vi or your favourite text editor) to include the Cassandra 3.0.x repository with the following contents. Note that I’m using the Cassandra 3.0.x repository (30X) and not the latest 311x release repository as this is not yet supported by VMware for vCloud Director:

[cassandra]
name=Apache Cassandra
baseurl=https://www.apache.org/dist/cassandra/redhat/30x/
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://www.apache.org/dist/cassandra/KEYS

Install Cassandra software itself (currently on my test nodes this pulls in the cassandra.noarch 0:3.0.15-1 package):

# yum install -y cassandra

The main configuration file for Cassandra is (on CentOS installed from this repository) in /etc/cassandra/conf/cassandra.yaml.

At least the following options in this file will need to be changed before we can get a cluster up and running:

Original cassandra.yaml  Edited cassandra.yaml Notes
cluster_name: ‘Test Cluster’ cluster_name: ‘My vCD Cluster’ Doesn’t absolutely have to be changed, but you probably should do. Note that this setting must exactly match on each of your node servers or they won’t be able to join the cluster.
authenticator: AllowAllAuthenticator authenticator: PasswordAuthenticator We’ll want to use password security from vCloud Director to the cluster.
authorizer: AllowAllAuthorizer authorizor: CassandraAuthorizer Required to enforce password security.
 – seeds: “127.0.0.1” – seeds: “Seed Node 1 IP address,Seed Node 2 IP address” Set the seeds for the cluster (minimum 2 nodes must be configured as seeds).
listen_address: localhost listen_address: This node IP address Must be configured for the node to listen for non-local traffic (from other nodes and clients).
rpc_address: localhost rpc_address: This node IP address Must be configured for the node to listen for non-local traffic (from other nodes and clients).

We’ll need to change some additional settings later to implement SSL security, but these settings should be enough to get the cluster functioning.

You’ll also need to permit the Cassandra traffic through the default CentOS 7 firewall, the following commands will open the appropriate ports (as root):

# firewall-cmd --zone public --add-port 7000/tcp --add-port 7001/tcp --add-port 7199/tcp --add-port 9042/tcp --add-port 9160/tcp --add-port 9142/tcp --permanent
# firewall-cmd --reload

Once you’ve performed these steps on each of the 4 nodes, you should be able to bring up a (non-encrypted) cassandra cluster by running:

# service cassandra start

on each node, you should probably also enable the service to auto-start on server reboot:

# chkconfig cassandra on

Note that you should start the nodes 1 by 1 and allow a minimum of 30 seconds between each one to allow the cluster to reconfigure as each node is added before adding the next one, check /var/log/cassandra/cassandra.log and /var/log/cassandra/system.log if you have issues with the cluster forming.

My test cluster has 4 nodes (named node01, node02, node03 and node04 imaginatively enough), and node01 and node02 are the seeds. The IP addresses are 10.0.0.101,102,103 and 104 (/24 netmask).

If everything has worked ok running ‘nodetool status’ on any node should show the cluster members all in a state of ‘UN’ (Up/Normal):

You should also be able to login to the cluster via any of the nodes using the cqlsh command (the default password is ‘cassandra’ for the builtin cassandra user)

[root@node01 ~]# cqlsh 10.0.0.101 -u cassandra -p cassandra
Connected to My vCD Cluster at 10.0.0.101:9042.
[cqlsh 5.0.1 | Cassandra 3.0.15 | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh>

To reconfigure the cluster for SSL encrypted communication we need to complete a number of tasks:

  • Create a new CA certificate authority (we could use an existing / external CA authority, but I’m trying to keep this simple).
  • Export the public key for our new CA so we can tell vCloud Director later to trust certificates it has issued.
  • Create a new truststore for Cassandra so Cassandra trusts our CA.
  • Create a new keystore for Cassandra for each node which includes the public key of our CA.
  • Create a certificate request from each node and submit this to our CA for signing.
  • Import the certificate response from the CA into the keystore for each node.
  • Move the generated truststore and keystore into appropriate locations and configure security on the files.
  • Reconfigure Cassandra to enable encryption and use our certificates.

That’s a lot of steps to go through and extremely tedious to get right, so I wrote a script to do most of these steps, create a file (I called my ‘gencasscerts.sh’ on one of the node servers and copy/paste the script contents from below):

Update 2018/01/26: I realised shortly after publishing that the original version of this script included both the public and private keys for the CA in the .truststore keystore. While this isn’t a huge issue in a private environment it’s definitely not ‘best practice’ to distribute the private key of the CA to each node so I’ve refined the script and removed the private keys from the generated .truststore files in this version. If you need to regenerate your environment keys remember that if you re-run the script both the .truststore and .keystore files will need to be updated on each node.

Make the script executable using:

chmod u+x gencasscerts.sh

Edit the settings and passwords for the certificates (we’ll need these later) in the variables at the top of the file to be appropriate for your environment (in particular the node names in the 3rd line) and any other options you want to change. The default validity period of the generated certificates is set to 10 years (3650 days). Obviously you should also change the passwords in your copy of the script for the CASTOREPASS and NODESTOREPASS variables.

When you run the script you should get output similar to the following:
[root@node01 ~]# ./gencasscerts.sh
MAC verified OK
MAC verified OK
Processing node 0: node01
Certificate was added to keystore
Signature ok
subject=/C=US/ST=MyState/L=MyCity/O=MyOrg/OU=MyDept/CN=node01
Getting CA Private Key
Certificate reply was installed in keystore
Processing node 1: node02
Certificate was added to keystore
Signature ok
subject=/C=US/ST=MyState/L=MyCity/O=MyOrg/OU=MyDept/CN=node02
Getting CA Private Key
Certificate reply was installed in keystore
Processing node 2: node03
Certificate was added to keystore
Signature ok
subject=/C=US/ST=MyState/L=MyCity/O=MyOrg/OU=MyDept/CN=node03
Getting CA Private Key
Certificate reply was installed in keystore
Processing node 3: node04
Certificate was added to keystore
Signature ok
subject=/C=US/ST=MyState/L=MyCity/O=MyOrg/OU=MyDept/CN=node04
Getting CA Private Key
Certificate reply was installed in keystore

Looking at the directory from where the script was run you should see the certificate files:

In each node directory there will be 3 files (.keystore, .truststore and chain.pem):

The .truststore files will all be identical between the node directories, the .keystore and client.pem files will be unique.

Next we need to move the generated certificate stores to an appropriate location, easiest way to do this is to use scp from the directory where the script was run. We’ll place the files in the Cassandra configuration directory (/etc/cassandra/conf).

On the node where the files have been generated we can just copy them (node01 in our case):

# cp node01/.keystore node01/.truststore /etc/cassandra/conf

To copy the appropriate files to the other nodes we can use scp:

# scp node02/.keystore node02/.truststore root@10.0.0.102:/etc/cassandra/conf
# scp node03/.keystore node03/.truststore root@10.0.0.103:/etc/cassandra/conf
# scp node04/.keystore node04/.truststore root@10.0.0.104:/etc/cassandra/conf

On each node we now run the following to set appropriate permissions and ownership on the files:

# chown cassandra:cassandra /etc/cassandra/conf/.keystore /etc/cassandra/conf/.truststore
# chmod 400 /etc/cassandra/conf/.keystore /etc/cassandra/conf/.truststore

We can now reconfigure the cassandra.yaml configuration file on each node to use our new certificates and enable encrypted communication.

The original settings from cassandra.yaml and the required changes are:

In the server_encryption_options: section to encrypt node-to-node communications:

Original Value New Value
internode_encryption: none internode_encryption: all
keystore: conf/.keystore keystore: <location of copied .keystore file>
keystore_password: cassandra keystore_password: <value you set for the NODESTOREPASS variable in the script above>
truststore: conf/.truststore truststore: <location of copied .truststore file>
truststore_password: cassandra truststore_password: <value you set for the CASTOREPASS variable in the script above>

 

Optionally you can also change the cipher_suites: setting to restrict available ciphers to the more secure versions (e.g. cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA]').

In the client_encryption_options: section to encrypt node-to-client communications:

Original Value New Value
enabled: false enabled: true
keystore: conf/.keystore keystore: <location of copied .keystore file>
keystore_password: cassandra keystore_password: <value you set for the NODESTOREPASS variable in the script above>

Again you can change the cipher_suites: setting if desired to use more secure ciphers.

Now we need to stop the cassandra service on ALL nodes:

service cassandra stop

Starting the cassandra service back up (service cassandra start – remember to wait between each node to give the cluster time to settle) you should now see the following in the /var/log/cassandra/system.log file:

INFO [main] 2018-01-25 20:52:38,290 MessagingService.java:541 - Starting Encrypted Messaging Service on SSL port 7001

Once the nodes are all back up and running nodetool status should show them all as status of ‘UN’ (Up/Normal).

If we attempt to use cqlsh to connect to the cluster now, we should get an error as we’re not using an encrypted connection:

cqlsh 10.0.0.101 -u cassandra -p cassandra
Connection error: ('Unable to connect to any servers', {'10.0.0.101': ConnectionShutdown('Connection <AsyncoreConnection(24713424) 10.0.0.101:9042 (closed)> is already closed',)})

If we specify the ‘--ssl‘ flag to cqlsh, we still get an error as we haven’t provided a client certificate for the connection:

cqlsh 10.0.0.101 -u cassandra -p cassandra --ssl
Validation is enabled; SSL transport factory requires a valid certfile to be specified. Please provide path to the certfile in [ssl] section as 'certfile' option in /root/.cassandra/cqlshrc (or use [certfiles] section) or set SSL_CERTFILE environment variable.

This is where the client.pem file is used as generated by the script, copy this file into the .cassandra folder in your user home path and then create/edit a file in this .cassandra folder called cqlshrc with the following content:

[connection]
factory = cqlshlib.ssl.ssl_transport_factory
[ssl]
certfile = ~/.cassandra/client.pem
validate = false

Save the file and now we should be able to establish an encrypted session:

cqlsh 10.0.0.101 -u cassandra -p cassandra --ssl
Connected to My vCD Cluster at 10.0.0.101:9042.
[cqlsh 5.0.1 | Cassandra 3.0.15 | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh>

When configuring vCloud Director to use our new metrics cluster, we must first tell vCloud Director that it can trust the CA we’ve created for our Cassandra cluster by importing the public key of our CA (the myca.pem file generated by the script) into the vCloud Director cell server cacerts repository. Ludovic Rivallain has a great post written up at https://vuptime.io/2017/08/30/VMware-Patch-vCloudDirector-cacerts-file/ showing how to do this. Note that this must be performed on each vCloud Director cell server as the cacerts repository is not shared between them.

You should also add a new admin user to Cassandra with a complex password and disable the builtin ‘cassandra’ user account before using the cluster.

Finally, you can follow the VMware documentation (link) to configure your vCloud Director cell to use this Cassandra cluster for metrics storage.

It’s reasonably easy to adjust this process to use an external CA rather than generating a new self-signing one, but this post is long enough already so let me know in the comments if you’d like to see this and I’ll write up a separate post detailing the changes to do this.

As always, comments/corrections/feedback welcome.

Jon.