PowerCLI Mastery, Volume 3

Phew, Volume 1 and Volume 2 down, lets wrap this monster up. See, I told you I wasn't going to leave you hanging for weeks to nail through all this.

So, picking up right from where we left off. We need to perform a configuration operation on an ESX host. Of the available properties we're looking at now, which is most likely to tell us what we can configure? That's right, the configManager property! If you guessed "config", you were close - the config property tells us _what_ the current configuration values are. The configManager object tells _how_ we can do the configuring (is that even a word?). Lets click on that.


Now we're cookin! AS you can see at the top of the page, the configManager object is a Data Object of Type 'HostConfigManager', the properties of which contain a whole bunch of Managed Objects. The task at hand is to modify a portgroup configuration, so let’s check out the 'networkSystem' property.



Here we see all the available properties and methods available for the HostNetworkSystem Managed Object. And low and behold, down the bottom we see the method by which we can modify a portgroup, UpdatePortGroup. We can see that 'networkSystem' is actually a Managed object, but we have to traverse the configManager _Data_ object to get there (because we need to provide the correct context for the managed object reference). Recalling that Get-View can only return _Managed_ Object references, in order to get the HostNetworkSystem Managed Object reference for our particular host, we have to again use Get-View but this time passing it in the networkSystem property of the configManager Data object. Which is a lot easier in PowerShell than it is in English.

$esxNetSys = $esxMoRef.configManager.networkSystem
$esxNetSysMoRef = Get-View $esxNetSys

Finally click on the UpdatePortGroup method, and hopefully we're done


OK, finally we see that in order to use the UpdatePortGroup method of a HostNetworkSystem Managed Object, we need to pass in a portgroup name as a string, and a portgroup as a... HostPortGroupSpec object??? WTF!!! Stu, you lead us all this way to a dead fucking end!!! There won't be anything left for Boche to mash at VMworld Europe 2009 by the time I'm done with you!!!

Breathe. Stay Calm. Focus. I have only given you 2 of the 3 tools needed for PowerCLI mastery. Now, we turn to the dark side... err I mean we turn to the SDK documentation, version 2.5 of which is online here.

The online raw SDK documentation is the equivalent of that goo they eat for breakfast in The Matrix - it contains all the things a PowerCLI Master's body needs. You can have a read of the text in the main window there if you want to get a much better explanation of the different object types that what I've given, or you can just jump in, find what you need, then get the fuck out. Yes, while the raw SDK documentation contains everything the body needs, reading it is also a bit like staring at the Sun. Do it for too long, and you'll go insane like that guy in Sunshine.

Open up the online SDK documentation site. Recalling our conversation about the types of object in the VI API, you can reasonably deduce that a 'HostPortGroupSpec' object is likely a Data object. So click on the "Data Object Types" link in the top left hand corner, wait for the list underneath to update then scroll down to find "HostPortGroupSpec", and click on it. This is the equivalent of the part of our PowerShell code where we do:

$pgspec = New-Object VMware.Vim.HostPortGroupSpec

Now where did I pull that “VMware.Vim.” from? We are using a native .net assembly given to us by the VI Toolkit, Vmware.Vim.dll. You probably haven't realised this up until now, I certainly didn’t until about a year ago, but the VI Toolkit is not just PowerShell - it's actually also a native .net wrapper for the VI API, that you can use with C# instead of compiling your own proxy dll from the VI web service. But you don't need to know the nuts and bolts of that - this is something I'm going to ask you to take on faith. You will eat your greens, you will look both ways when crossing the road, and you will know that the VMware.Vim.dll contains a reference for _every_ object available in the underlying API and then some. And for those sceptics out there who don't believe me, execute the following in any old PowerShell window on a box that you have PowerCLI installed on (and make sure u have a large row buffer ;)

$vimDll = [Reflection.Assembly]::LoadWithPartialName("VMware.Vim")
foreach ($type in $vimDll.GetTypes()){write-host $type.name}

Back to the SDK documentation and the code. Now we see what the properties of a HostPortGroupSpec are. There are 4, and all are mandatory. But what’s this, one of those properties is _another_ object, a "HostNetworkPolicy". So what do we need to populate one of those? Click on it and we'll see!

$pgspec.vswitchName = vSwitch0
$pgspec.Name = "VM Network"
$pgspec.vlanId = "0"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy

OMG!!! We just opened up a Pandora's box. HostNetworkPolicy has 4 properties, _all of them other objects_! But before you bury your head in your hands, check out those red asterisks - anything with one of those is an optional parameter, which is everything in our case!. Since we're only interested in changing the nic order, which has to do with nic teaming, we're going to leave all properties except the nicTeaming property as empty. Which basically means that this portgroup will inherit the offload policy, security policy and shaping policy of the virtual switch that it is created on. We can see that the nicTeaming property requires a "HostNicTeamingPolicy" object. Click through to that.

$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy

Once again we see a whole bunch of properties, all optional as denoted by the red asterisk. We are interested in changing what vmnics are active, standby and unavailable for this portgroup, so we'll need to populate the nicOrder property. Which as you can tell by now (see, you get the hang of this pretty fast), is another object named 'HostNicOrderPolicy'. So click on through...

$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy

Finally! We're at the end of the line. No objects here, just data, and again all optional. As you can see, there are only 2 properties - activeNic and standbyNic. Any nics present on the parent vSwitch but not part of either of these properties is by definition "unused". So in our example, I'll tell you we have 4 vmnics on the parent virtual switch. On the parent virtual switch, all 4 are active (again, this is all in the imaginary pre-existing host this script will be run against). But in this portgroup, we only want vmnic1 and vmnic2 to be active, and vmnic0 and vmnic3 standby. We can see in the SDK documentation that activeNic and standbyNic need values of type 'string array', so we simply do the following to make a valid assignment:

$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic1","vmnic2")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic0","vmnic3")

OK, now we've finally populated our HostPortGroupSpec object with all the required properties, objects, data types and data, it's time to finish off with a call to the UpdatePortGroup() method, via the HostNetworkSystem Managed Object reference we setup in the 3rd line of the script, with the required parameters we got from the MOB. Those being the name of portgroup we want to update, and a HostPortGroupSpec object containing the information we want to apply.

$esxNetSysMoRef.UpdatePortGroup($pgspec.Name, $pgspec)

Job done.

Now you understand why I'm such a huge fan of PowerCLI. I've had the displeasure of coding against the VI API directly in C#, and Shyam did most of the heavy lifting in Java when we developed statelesx. It often seems convoluted and contradictory, you often find that what you thought would be a simple operation based on another seemingly similar task, is actually anything but. As you've seen in this example, there are layers upon layers upon layers, you can get lost pretty easy. But thanks to the foresight of the VI Toolkit team to expose the underlying VI API via Get-View and provide us with VMware.Vim.dll assembly, anything is possible with the Power of Shanklin. Ooops I mean the Power of Shell ;-). But the Power of Shanklin does compel you.

Hopefully now you can not only do whatever you want to with PowerCLI, but also look at any script that the forum wizards may post, and understand exactly what they’ve done and how you might tweak them to your needs.

Now go forth and conquer. You are Jedi.

PowerCLI Mastery, Volume 2

Thought I was going to drag this out in some scandalous attempt to keep the viewers coming back week after week? Actually, wish I had thought of that before just now. Anyways, let's continue. If you haven't read Volume 1 in the series, go do so now. I'll wait for you.

So continuing on, let’s use the MOB to find a portgroup, and see if that helps us to know what we need to do. The MOB displays the internal object names for things rather than the display name you see in the VI client (the display name is still in there, but it’s usually a property of an object rather than the name of the object itself). If it makes navigation easier, open up the VI Client as well so you can see what internal object names map to which display names. To access the MOB:

1. Browse to https://vchostname/mob

2. Authenticate with whatever credentials you would use for vCenter

You are now for all intents and purposes at the landing page of where 'connect-viserver' takes you.


Click on the "Content" link in the value field of the content property. You can now see the top level of all the available managed objects of the vCenter box you connected to.


We know that networking is configured at the host level, so we need to find a host. The rootFolder property is the start of the inventory, let’s go there.


We are now effectively at the top level "Hosts & Clusters" folder in the VI Client Hosts and Clusters inventory view. As you can see here, in my case there is only 1 Datacenter defined, with an internal object name of 'datacenter-2'. Let’s see what’s in there.


OK, so by looking at the 'name' property, we can see that indeed we are looking at a Datacenter object (name removed to protect the innocent). Pretty easy to guess where we might find hosts... in the hostFolder object! Click through...


Because we have clusters defined in this Datacenter (in this case 2 clusters, 'domain-c28' and 'domain-c297' - again, internal object names), we are now actually looking at the cluster container. Clicking into one of the cluster objects will give us our host list.


OK, here's some useful info. We can see up the top of the screen that we are looking at a "Managed Object" Type. In the VI API there are Managed Objects (higher level objects), Data Objects (lower level objects), and then the data primitives like strings and integers. To illustrate, in this screen the host property contains an array of Managed Objects, the configurationEx property contains a single value that is a Data Object, and the name property contains a single value that is a data primitive. We can also see that the name of this Managed Object Type is ManagedObjectReference:ClusterComputeResource - logical for a cluster. Looking at the properties, we see one named 'host', which is also a Managed Object Type and named 'HostSystem'. Listed in the value field are our 2 host objects. Clicking on one of them, we come to the top level of a HostSystem Managed Object.


Cast your mind back to Volume 1, where I said that 'Get-View' was the window to the underlying objects of the VI API. Now that you know there are 2 different types of objects in the VI API, Managed Objects and Data Objects, I can be more specific and say that Get-View only returns _Managed_ objects. We used Get-View to get a Managed Object (hence why it's common to see variables of this nature have 'mor' or 'moref' in the name) to an ESX host. In order to do so, we had to pass the Get-View cmdlet the Id property of the object returned from Get-VMHost. The reason we had to use the Id property is just the way that the VI Toolkit team designed the Get-View cmdlet to work at this particular level.

$esxMoRef = Get-VMHost esxbox.corp.com | % {Get-View $_.Id}

What Managed Object Type did we get a reference to? The exact one we are looking at now - a ManagedObjectReference:HostSystem. And to prove I'm not lying, try this for yourself. Create an $esxHostMoRef variable by issuing the command above in your own environment, and then type $esxHostMoRef.overallStatus and hit enter - it will return whatever value is in the overallStatus row of the webpage in front of you.

And that's where we'll leave it for this installment, Padawans (cue Yoda laugh). In Volume 3 we'll start to disect the script from Volume 1, and follow the rabbit hole a bit deeper.

PowerCLI Mastery, Volume 1

OK, this one has been a looooooooooong time coming. I've been meaning to take a shot at conveying the advanced features of the VI SDK as applied to PowerShell for quite some time now. When Carter corrected me some months ago with regards to what is and isn't possible with PowerCLI (in the first statelesx video, about 1 minute in I mistakenly suggested there was something it _couldn't_ do :-), I've thought I really should get this info out there. Whether I will succeed in this post and the ones that follow is of course for you to decide (this is way to long for a single post), but I'm going to take you through what I think is the fastest way to get your head around the _entire_ SDK. That's right, the whole fucking thing. For as you will see, when you conquer a small piece of the SDK, the rest falls like dominoes.

Now the usual caveats. Yes there are other, possibly more efficient ways to find this info, or write the example code in this article. And some of the things I say will make people who work for Carter cringe. But I've done all this intentionally for the sake of simplicity. And about the the term 'PowerCLI' - Carter dropped it once on Twitter, and all bloggers being opportunists at heart, I'm going to seize on that one with both hands. Besides, it sounds a lot cooler than the VI Toolkit (or at least it more specifically refers to a part of the VI Toolkit). But to avoid any confusion, yes I am talking about this.

It's going to be difficult to go further without the help of an example, so let’s use an example of an operation that there is no native cmdlet for (yet) - modifying the nic order of a portgroup. I don't know how often other people do this, but every ESX blade built where I work gets this treatment. And there are hundreds of those.

Here's the code:

$esxMoRef = Get-VMHost esxbox.corp.com | % {Get-View $_.Id}
$esxNetSys = $esxMoRef.configManager.networkSystem
$esxNetSysMoRef = Get-View $esxNetSys

$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch0"
$pgspec.Name = "VM Network"
$pgspec.vlanId = "0"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic1","vmnic2")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic0","vmnic3")

$esxNetSysMoRef UpdatePortGroup($pgspec.Name,$pgspec)

But how did I know those methods and properties are available, what are required and optional values, and what type of values to use? It's not by remote chi, the force, and I didn't pull them out of my arse. I used a combination of Get-View, the MOB and the SDK documentation to do this. No it's not as scary as it sounds, anyone can be on their way to PowerCLI mastery with these tools.

The Get-View cmdlet is part of PowerCLI, and is the window to the underlying VI API. When you need to do something that isn't available via the usual PowerCLI cmdlets, you can use Get-View to acquire an object reference that the underlying API exposes. And of course with PowerShell natively supporting .net objects, you can then access all the associated methods and properties of those underlying objects. That's all well and good, but how do we know what underlying objects, methods and properties we need to use to get our desired result? Continue reading in our next exciting episode, when you will be introduced to the MOB. No, not the sleeps with the fishes type mob, the Managed Object Browser.

ESXi 3 Update 3 - Free Version Unshackled, hoo-rah!

I'm not going to try and claim credit for this development, but Rich Brambley (the only blog with a description that I feel outshines my own :-) broke the news today that the free version ESXi 3 Update 3 appears to have a fully functional API. I plan on testing this out tomorrow, and will report back as I'm sure others will too!

UPDATE The great Mike D has beaten me to it. Looks like all systems are go for the evolution of statelesx... ;-)

UPDATE 2 Oh man... who the !#$% is running QA in VMware? What a shocker!

Transcript of Best VMTN Community Roundtable Yet!

Hooo-rah! to Rod Haywood for transcribing VMTN Community Roundtable #22, which was in my mind the best roundtable yet (you got Krusty sized shoes to fill next week Chad :-). Bill Shelton was bang on point with just about everything he said. If you've spent any time with the guts of the SDK you'll know just how inconsistent it can be. In fact a while back I asked Carter and Steve how come the API was so quirky with guys like them working at VMware who clearly know better (otherwise they wouldn't have felt the need to write their respective wrappers). Their answer? "We've only been here for about a year." Bill Shelton falls into that category as well. Knowing how people think in those kinds of positions at VMware is very comforting indeed. I wonder what the Windows Azure API will look like. Actually no, I don't even care.

VM Template for Citrix Provisioning Server

A rather odd bug with VirtualCenter is the inability to deploy VM's with a SCSI controller but without a disk... even creating a template from a VM that has a SCSI controller but no disk results in a template with no SCSI controller (you then have to convert it to a VM, re-add the SCSI controller, then convert back to template. Only to find the SCSI controller stripped during the deploy from template. There's 10 minutes of my life I'll never get back).

If you've been reading this blog for any length of time you'll know I'm not exactly a fan of Citrix XenServer, but Citrix Provisioning Server on the hand is _very_ cool (albeit prohibitively expensive and not without it's drawbacks... another post maybe). For the uninitiated, Provisioning Server streams a disk via the network. But obviously the streamed disk needs to be access via a disk controller (duh!). Which is why I want to create a diskless template in the first place.

Before firing up the trusty PowerShellified version of Notepad2, i did a cursory search of the VI Toolkit forum to see if someone had done this already... and surprise surprise, Cool Hand LucD had done my work for me.

So yeh, now all I need to do is deploy my scsiless and diskless template with new-vm and call the function from LucD, and all is well in the world - big ups to LucD.

/me makes W shape with fingers on one hand while simultaneously pounding fist on chest with the other

VM Reconfiguration via the VI Toolkit for Windows

Reconfiguring VM's via the API is something I've had to dabble in lately, due to some.... err.. "interesting" behaviour with deploying VM's from templates. Such as the inability to deploy a VM from a template that has a SCSI controller but no disk - useful if using LUN based snapshots, where the .vmdk already exists and you want to move said vmdk into the same directory as the VM after it's been created (and thus can't use the -Diskpath option of New-VM as the path doesn't exist yet). Or if you want to create VM's for use with Citrix Provisioning server, which don't require a local disk but do require a SCSI controller.

While this operation is uglier than I'd like it (most things that involve Get-View and specifications are uglier than I'd like them :-), I found this very good paper which explains the VI object model as applied to VM hardware reconfiguration _much_ better than the SDK documentation does.

Sure the examples are in Perl, but the theory is the same. Combine the document with a listing of all posts by Carter, LucD and halr9000 in the VI Toolkit for Windows community and there's nothing you can't do!

First Official VI Client Plug-in Document Released - round of applause for VC architects / devs!

OK, so I've been giving the VC devs a _bit_ of a hard time lately. So just to show I'm as quick to dish out praise as I am to dish out flames, I wanted to say damn good job on the VI plugin architecture!

I have been eagerly awaiting some official documentation surrounding VI plug-in development (both client and server), ever since Andrew Kutz's amazing job of reverse engineering the framework. I guess the document release probably got lost in the publicity surrounding the Virtual Disk Development Kit released a few days prior to the release of this.

Peronsally I think the whole VI plugin idea is nothing short of a stroke of genius. What better way to combat the potential for the likes of Microsoft to relegate VirtualCenter to the background by building full VMware infrastructure management into SCVMM, than to open up the VI management framework in such a way. And the architecture... as much as I might accuse VMware's tools of lacking enterprise scalability from time to time, they've absolutely nailed it this time. I mean, look at this (from the document)



Outstanding! On the coding side, it looks so straight forward that even a hacker like me should be able to put something together fairly easily... running off the example in the document, you could pretty easily add a context menu item to ESX hosts that fires up the out of band management web interface (ie the iLO page for HP kit) of that particular host.

Hopefully it won't be too long before the framework moves out of experimental status, and we start seeing 3rd party tools leverage the functionality. I've already seen the Xsigo plugin and it looks great.

Download it here

Use Passthrough Authentication in VMware SDK Apps

It turns out that it was staring me in the face all along... there's a new method mentioned right inderneath the standard Login() method in the online VMware API reference, called LoginBySSPI().

But using it isn't quite as straight forward as replacing Login() with LoginBySSPI() on it's own... you first need to grab a security context via C++ and pass that as an argument to LoginBySSPI(). I can barely read C++, let alone write it, so I need to get my head around the easiest way to use it. When I work that out, I'll post back a few samples from the VI SDK using LoginBySSPI() instead of Login(). Unless someone like Andrew Kutz happens to read this and expend 0.00001% brain power to provide the answer :-)

I'll also be asking VMware when SSPI will be moved out of 'experimental support' status... things like VCB could really benefit from it, as could a raft of other 3rd party apps.