VMworld 2013 – Self Healing Datacenter Example 1 – Automating HA/DRS Configurations

First of all, thank you if you attended the session I presented with Dan Mitchell on the Self Healing Datacenter. Also a big thanks to Kim Jahnz and Dan for giving me the opportunity to go up there and talk about some of the things I’ve been working on.

Here is the detailed walkthrough of the 1st example I gave.


In this example I demonstrated how you could effectively use vCO as a Configuration Management tool for vSphere settings. Now this does not mean that there aren’t more advanced tools out there for configuration management. Puppet, Chef etc. are your big knife tools for more serious configuration management, but vCO can fill some very easy gaps without needing to go to more complex tools.


Always start with a goal in mind and then work your way through all the components required.

  • HA Settings
    • HA should be turned on
    • HA admission control should be enabled
    • HA Admission Control policy should be set to percentage based and allow for 1 host in the cluster to be out of service. (e.g. for a 4 host cluster, I would set this to 25%)
    • DRS
      • DRS should be turned on
      • DRS should be set to fully automated

Break it down:

The very first time I created this workflow, I used workflows containing scriptable tasks. Later I improved it ahead of VMWorld and turned that workflow of scriptable tasks into an action item.

I don’t think there is a right or wrong answer to this, but using the action item seems cleaner and more reusable, with less chance of error.

The main workflow:


Now for the action item code…

Part 1: Calculating your HA % for the amount of hosts in your cluster.

var Hosts = System.getModule("com.vmware.library.vc.cluster").getAllHostSystemsOfCluster(cluster);

System.log("Number of Hosts in Cluster: " + Hosts.length);

var HApercent = ((1/Hosts.length)*100);
HApercent = HApercent.toFixed(0);

System.log("HA Percent which will be used for cluster is: " + HApercent);

Part 2: The cluster specifications and task to reconfigure the cluster

//Create variables for DRS/HA config

var clusterConfigSpec = new VcClusterConfigSpecEx();
clusterConfigSpec.drsConfig = new VcClusterDrsConfigInfo();
clusterConfigSpec.dasConfig = new VcClusterDasConfigInfo();

//Enable DRS/HA

clusterConfigSpec.dasConfig.enabled = true;
clusterConfigSpec.drsConfig.enabled = true;

//Set DRS to INPUT (Passed to the Action)

System.log("Setting DRS to Fully Automated");

clusterConfigSpec.drsConfig.defaultVmBehavior = drsBehaviour;

//Fix Admissions control policy

System.log("Updating HA Admission Control policy for " + cluster.name);

clusterConfigSpec.dasConfig.admissionControlPolicy = new VcClusterFailoverResourcesAdmissionControlPolicy();
clusterConfigSpec.dasConfig.admissionControlEnabled = true;

//Set host monitoring to the setting passed to the action

clusterConfigSpec.dasConfig.hostMonitoring = haHostMonitoring;
clusterConfigSpec.dasConfig.admissionControlPolicy.cpuFailoverResourcesPercent = HApercent;
clusterConfigSpec.dasConfig.admissionControlPolicy.memoryFailoverResourcesPercent = HApercent;

//Reconfigure the cluster, by adding the True parameter this ensures any previous settings remain

System.log("Executing Cluster Reconfiguration for " + cluster.name);
task = cluster.reconfigureComputeResource_Task(clusterConfigSpec, true);

Putting it all together:

So now we have an action item which can take in the following inputs:

1. Cluster (Type: VCCluster)
2. DRS Behaviour (Type: DRS Behaviour)
3. HA Host Monitoring (Type: Boolean)

Now comes the easy part, you can just create a workflow where the 3 inputs required above are inputs, or general attributes. If you use Inputs then you will be prompted each time. If you choose general attributes then they are set permanently in the vCO workflow unless you change them.
I chose to set the DRS Behaviour/HA Host Monitoring settings as general attributes inside the workflow which I called “Configure HA/DRS Settings for Cluster”. Then I would just select the cluster I wanted to apply the settings to when I ran the workflow.

Now once I decided this worked great, I wanted to push the settings out to ALL of my clusters, so I created the main operational workflow “Configure HA/DRS for ALL Clusters”. I then made an array of clusters as the general attribute, which I could add in, or take out clusters as I wanted to.

Finally, schedule the workflow in vCO to run every night at midnight, and you know that all your clusters are exactly as they should be. If you are working on a cluster, just take it out of the array and put it back in when you are done.

vCenter Orchestrator Appliance – Guest File Operations (Copying a file to guest VM)

One of the things you will often find you need to do with vCO is to get a file to a guest VM, or just run a file from inside the VM. Now for Windows you can use Powershell remote features in many cases, but what if your server isn’t on the network yet? Until version 5.1 we had to rely on VIX as a way to do this, but now VMware has added a number of new workflows under “Guest Operations” which are much more reliable.

vCO Guest Operations

vCO Guest Operations

“Copy file from vCO to guest” is the one I’m going to be using in this example.

First of all copy the workflow into a sandbox area. This way you can move a bunch of the inputs to attributes and not have to key them in each time (e.g. The local administrator username, password, and test VM).

In my example, I’m going to create a text file called test.txt in a new folder under /opt called “vcofiles”.

My target machine is a Windows 2008 R2 server, where I will copy the file and place it in the C:\temp\ folder with the name “testcopy.txt”

If you run the workflow then these are my input parameters:



The problem is that if you run this you will get an error similar to this:

“No permissions on the file for the attempted operation (Workflow: Copying files from vCO appliance to guest/Scriptable task…”



In order to fix this you first need to give the correct rights to the folder and file on your vCO Appliance.

1. Login as root onto the appliance
2. Give Read/Write/Execution rights to the new folder


3. Give Read/Write rights to the Text file you made



Unfortunately we aren’t quite done yet. You also need to tell orchestrator which locations it can read/write/execute from. This involves editing the “js-io-rights.conf” file located in “/opt/vmo/app-server/server/vmo/conf”


Add the line “+rwx /opt/vcofiles/” as shown above.

If anyone isn’t too sure on the linux commands to do this:

  • Type “cd /opt/vmo/app-server/server/vmo/conf” and press enter.
  • Type “vi js-io.rights.conf” and press enter.
  • Use the arrow keys to move the cursor where you want and press the insert key
  • Press Enter and type in the line “+rwx /opt/vcofiles”
  • Press ESC
  • Type “:wq” and press enter.

4. Now, there’s one more thing. You need to restart the vCO service for this to take effect.

Login to the vCO configuration manager, go to startup, and click restart service.


5. Now run your workflow and see if your text file copied across.


You can see a quick video demo of this on youtube. (apologies for the mouse pointer issue..)


Thanks for reading. Let me know if you have any questions.





VCAP-ICD – Objective 1.1

Objective 1.1 – Create a Conceptual Design Based on Business Requirements

Skills and Abilities 

  • Distinguish between virtualization, automation and cloud computing.
  • Distinguish between private, public, hybrid and community cloud computing.
  • Analyze a customer use case to determine how cloud computing can satisfy customerrequirements.
  • Given a customer use case, determine the appropriate cloud computing model.

Distinguish between virtualization, automation and cloud computing.

First paragraph taken from the VCAT 3.1 Toolkit (Service Definitions PDF): http://www.vmware.com/cloud-computing/cloud-architecture/vcat-toolkit3.html

Virtualization has reduced costs and increased server efficiency, often dramatically, but it does not, by itself, deliver the level of automation and control required to achieve the efficiencies or agility associated with cloud computing. Cloud computing offers the opportunity to further improve cost efficiency, quality of service, and business agility. It enables IT to support a wide range of changing business objectives, from deployment of new tools, products, and services to expansion into new markets. Cloud computing transforms IT from a cost center into a service provider.


Essentially allows us to consolidate and become much more efficient with the resources we have. Instead of multiple physical servers we can now use 1 physical server to host multiple workloads. I still think of virtualization as a technology which helps to enable the Cloud framework. At the end of the day, if you virtualized 100% of your apps, this does not mean you have a Cloud, only that you are 100% virtualized, but no doubt have saved your company a lot of money.


Automation helps us to do things better by making us more efficient. Here the emphasis is on scripting, workflow design, configuration management tools, and any other methods to reduce the amount of human touch to make things work. It is an essential skill required to enable any serious Cloud delivery.

Cloud Computing: 

Key Terms: Business Agility, Quality, Cost efficiency

Automation and Virtualization help us get there. Business Agility is key, as by fully embracing a cloud model it enables IT to respond quickly to changing business demands. As VMware states above, this transforms IT from a cost center, into a service provider. The way our customers consume resources is radically changed when we use a cloud model.

Again keep in mind, if you just virtualized your environments and then put in automation, you are efficient, but you haven’t changed the entire process.

Distinguish between private, public, hybrid and community cloud computing.  

Taken from the VCAT 3.1 Toolkit (Service Definitions PDF): http://www.vmware.com/cloud-computing/cloud-architecture/vcat-toolkit3.html 

The following are the commonly accepted definitions for cloud computing deployment models:

  • Private vCloud – The vCloud infrastructure is operated solely for an organization and can be managed by the organization or a third party. The infrastructure can be located on-premises or off- premises.
  • Public vCloud – The vCloud infrastructure is made available to the general public or to a large industry group and is owned by an organization that sells vCloud services.
  • Hybrid vCloud – The vCloud infrastructure is a composite of two or more vCloud instances (private and public) that remain unique entities but are bound together by standardized technology. This enables data and application portability, for example, cloud bursting for load balancing between vCloud instances. With a hybrid vCloud, an organization gets the advantages of both, with the ability to burst into the public vCloud when needed while maintaining critical assets on-premises.
  • Community vCloud – The vCloud infrastructure is shared by several organizations and supports a specific community that has shared concerns, such as mission, security requirements, policy, and compliance considerations. It can be managed by the organizations or a third party, and can be located on-premises or off-premises.

At this time I think it is also worth mentioning the 3 service models.

  • Software as a Service (SaaS) – Business-focused services are presented directly to the consumer from a service catalog.
  • Platform as a Service (PaaS) – Technology-focused services are presented for application development and deployment to application developers from a service catalog.
  • Infrastructure as a Service (IaaS) – Infrastructure containers are presented to consumers to provide agility, automation, and delivery of components.

Analyze a customer use case to determine how cloud computing can satisfy customer requirements.

Given a customer use case, determine the appropriate cloud computing model.

These last 2 objectives are covered very well in the Service Definitions PDF which lists a number of use case examples. I’m going to revisit this section after I finish going through all the objectives to see what examples I have found.


Cloud thoughts…and Operations 101

Cloud/VCAP-ICD Studying

My usual mentality is to fully understand the technology before trying to get others to buy in on the ideas. This way I know if it is vendor’s lying, or whether or not a suite like vCloud Director can actually make a lasting impact on the business.

So my focus over the next few months is going to be ensuring I know enough about how something like vCloud Director can work in practice. I will be developing sample policies and procedures to further enhance my understanding of what this might actually look like in practice. I will start this by going through the VCAP-ICD Exam Blueprint and working through every topic. You will see a new menu bar for VCAP-ICD very soon.

VMware is also releasing a new exam around governance, so it will be very interesting to see what extra material they bring to the table there.

For more information on the VCAP-ICD check out: http://mylearn.vmware.com/mgrReg/plan.cfm?plan=32565&ui=www_cert 

Operations 101

One thing has become clear to me as I’ve been reading books on the subject…If you don’t have basic blocking and tackling right in a static environment, your cloud ideas are bound to fail.

So in addition to learning about Cloud and studying for my VCAP-ICD I will be documenting all the core essentials I believe need to be in place before going to Cloud, or in some cases may be solved by going to a Cloud type model.

So what then are the core essentials BEFORE starting on a cloud journey.

My experience working in an enterprise has taught me that attitudes to operations matter far more than the technologies which enable them.  I’ve had the pleasure of working with some people that really care about their environments, and others that just come to do the bare minimum.

Operations must work. We can go and read ITIL books etc. but what is the reality, and how does operations work for the day to day VMware Admin…How do we get to that well oiled machine, and what does it take to keep it there…?

As I begin this section I am going to be covering the following solutions as they relate to VMware Environments:

1. VM Monitoring

I have a lot of experience now with nWorks, vKernel, SCOM, vCOPs…I will be discussing the very quick and easy way I was able to reduce noise, get relevant tickets, and get lower level engineers able to take over support for our day to day activities. Keep it simple and don’t over complicate your monitoring.

2. Performance Troubleshooting/VM Environment Recovery

If we can’t free ourselves up from bridge calls where infrastructure has to be checked and rechecked for performance we can never work on the things we need to work on. How do we quickly and effectively deal with these situations.

3. Capacity Planning

Divided up between Compute and Storage…how do you ensure you have the capacity you need all the time? Again I have created very simple ideas which can be modified to work in any environment.

4. Inventory/Tagging

This is an area I’m working on right now…would love to hear how other people are doing this in their virtual environments.

5. Automation/Configuration Management

Automation skills, be it scripting, config management with puppet/sccm, orchestration tools, are essential not just to Cloud but I believe Operations.

Standards are so important here…you can’t automate or maintain configurations, if you don’t even have a set of documented standards for your environment.

The 3 main areas I will cover here are ESXi Host standards, vCenter Standards, and VM Standards.

6. Disaster Recovery/Bacups

This journey is only beginning…I’m looking forward to thinking through all the challenges here.

7. More to be determined..

In closing, I believe this still comes back to standards. Someone once told me “There’s a standard for everything, and everything has a standard” and this in itself really helps guide the thoughts for the above.

Welcome to v(NIC)k…a virtualization blog about Automation, Orchestration and VMWare Certifications

Hello VMware Community and welcome to my first Virtualization Blog.

I will be primarily covering these 3 topics:

  • VCAP  – Study notes for the VCAP exam which I am studying for
  • vCenter Orchestrator Workflows including full Windows System Provisioning
  • Datacenter Automation with PowerCLI and vCenter Orchestrator – The goal is to create a self healing VMware Datacenter

Please feel free to comment on any of my topics and please let me know if you find any of my work useful.

Nick Colyer