ESXi Health Status Inequality - HP vs Dell

Ok, so I'm pretty sure most of you know I've been haxing around with ESXi for quite some time, but I only recently noticed the difference in what you see in vCenter Health Status depending on which hardware vendor you use. I'm sure you also all know that I'm a massive fan of HP, which is why I'm a so unimpressed with their integration with vCenter. For example, lets have a look at recent generation HP BL685c blade running the HP OEM version of ESXi 3.5 (installable), downloaded from the VMware website.


Notice anything strange? Or rather, notice the distinct _lack_ of anything out of the ordinary. That's right, when comparing this blade to a non-OEM ESXi installable, it looks exactly the same in vCenter! At first I thought there must have been a corruption in the CIM providers or something, but after investigating with my VMware SE, we could find nothing out of order.

Now compare the image above to what you see in vCenter on an old Dell rackmount (note the bios date - nearly 2 years old!).



Wow.

Now I can only hazard a guess that HP want you to use Insight Manager to perform health monitoring exclusively, but why force me to deploy more monitoring infrastructure than I need to? They could at least create a VI client plugin for Insight Manager so I can use a single pane of glass! But even with that, it doesn't seem like you get as much information as what you do with Dell - looking at Appendix B of this HP whitepaper, storage subsystem information is strangely absent.

If anyone else out there has seen this, seen different, or can compare with other hardware platforms like Sun or IBM, please share your experiences in the comments!

ESXi 3 Update 3 - Free Version Unshackled, hoo-rah!

I'm not going to try and claim credit for this development, but Rich Brambley (the only blog with a description that I feel outshines my own :-) broke the news today that the free version ESXi 3 Update 3 appears to have a fully functional API. I plan on testing this out tomorrow, and will report back as I'm sure others will too!

UPDATE The great Mike D has beaten me to it. Looks like all systems are go for the evolution of statelesx... ;-)

UPDATE 2 Oh man... who the !#$% is running QA in VMware? What a shocker!

Symantec Does _NOT_ Support Vmotion... WTF!?!?!

Make sure you're sitting down before reading this Symantec KB article which is only 1 month old and clearly states they do _not_ support any current version of their product on ESX if Vmotion is enabled. No, it's not a joke.

It's a _fucking_ joke. Symantec have essentially pulled together a list of random issues that are almost certainly intermittent in nature and could have a near infinite number of causes, and somehow slapped the blame on Vmotion without as much as a single word of how they arrived at this conclusion. When Microsoft shut the AV vendors out of the Vista kernel, I actually felt a little sympathy for them (the AV vendors). But after reading this, I can't imagine the bullshit Microsoft must have had to put up with over the years. It's no wonder Microsoft tried to do their own thing with AV.

I urge every enterprise on the planet who are customers of both VMware and Symantec to rain fire and brimstone upon Symantec (I've already started), because your entire server and VDI infrastructure is at this time officially unsupported. The VMware vendor relationship team have already kicked into gear, we need to raise some serious hell as customers to drive this point home, and hard.

This is absolutely disgraceful and must not stand.

UPDATE The original KB article as been updated, support has been restored to normal!

Microsoft Azure Infrastructure Deployment Video

I've been a bit quiet of late, mainly due to the digestion of Microsoft's PDC 2008 content (what a great idea to provide the content for free after the conference - VMware could take a leaf out of that book). I meant to post this up yesterday as another SAAAP installment but missed the deadline... but what the hell, I'll tag this post anyway!

One of the more infrastructure oriented PDC sessions was "Under the Hood - Inside the Windows Azure Hosting Environment". Skip forward to around 43 minutes into the presentation, where Chuck Lenzmeier goes into the deployment model used within the Azure cloud (you can stop watching at around 48 minutes).

Conceptually, this is _exactly_ the deployment model I and ppl like Lance Berc envisage for ESXi. Rather than put that base VHD onto local USB devices ala ESXi, Microsoft PXE boot a Windows PE "maintenance os", drop a common base image onto the endpoint, dynamically build a personality (offline) as a differencing disk (ie linked clone), drop that down to the endpoint, and then boot straight off the differencing disk VHD (booting directly off VHD's is a _very_ cool feature of Win7 / Server 2008 R2). I'm glad even Microsoft recognise the massive benefits of this approach - no installation, rapid rollback, etc.

Now ESXi of course has one *massive* advantage over Windows in this scenario - it is weighs in at around the same size as a typical Windows PE build, much smaller than a Hyper-V enabled Server Core build.

And if only VMware would support PXE booting ESXi, you could couple it with Lance's birth client and midwife or our statelesx, and you don't even need the 'maintenance OS'. You get an environment that is almost identical conceptually, but can be deployed much much faster due to the ~150MB or so total size of ESXi (including Windows, Linux and Solaris vmtools iso's and customised configuration to enable kernel mode options like swap or LVM.EnableResignature within ESXi that would otherwise require a reboot (chicken, meet egg :-)) versus near a GB of Windows PE and Hyper-V with Server Core. Of course I don't even need to go into the superiority of ESXi over Hyper-V ;-)

With Windows Server 2008 R2 earmarked for a 2010 release, it will be sometime before the sweet deployment scenario outlined in that video is available to Microsoft customers (gotta love how Microsoft get the good stuff to themselves first - jeez, what a convenient way to ensure your cloud service is much more efficient than anything your customer could build with your own technology). But by that time they will have a helluva set of best practice deployment knowledge for liquid infrastructure like this, and you can bet aspects of their "fabric controller" will find their way into the System Center management suite.

The liquid compute deployment gauntlet has been thrown VMware, time to step up to the plate.

ESX 3.5 Update 3 Release Imminent!

It seems the ESX 3.5 Update 3 release is imminent, as per some recent updates to the VMware website (screeny below)



Wonder if we'll get any timebomb action this time :D

Future ESXi Version Call to Arms!

While there wasn't a lot of concrete details that came out of VMworld, one thing is for certain - VMware are working on a _lot_ of new functionality for their next major release. Of particular relevance to us at vinternals is the host profiles feature, as it provides the same functionality as Statelesx, but is done in a much more user friendly way. Which is fine - it's not like we threw in out jobs and tried to position statelesx as a commercial product, and in some ways it's a validation that our idea was a good one.

But VMware need to ensure they implement the other half of the solution, ala the client initiator. Without this, there is still an unnecessary manual task of joining a host to a cluster in the first place. You may think this isn't such a big deal, but if your hosts are PXE booted then a client initiator is an absolute requirement.

Fortunately for us, there is at least one senior guy at VMware who gets all this - Lance Berc (I get a laugh eveytime I see his 'novice' status in the VMware forums - for fucks sake, this is the guy who wrote the original esxtop). So far most attention has been on his post-build configuration scripts, but Lance put his C skills to work and released the crucial client initiator piece as well as a 'midwife' which essentially performs the same duties as statelesx. Put all this together and you have a truly automated, scalable, liquid environment - the likes of which is usually only seen in compute clusters. Except this time it's a _virtual_ environment.

VMware seem to be taking notice of the need for cluster wide configuration automation with host profiles and the distributed virtual switch, we as customers need to make sure they understand the need for the client initiator functionality as well. The 'midwife' functionality would also be nice if it was wrapped into VirtualCenter, but if it isn't then maybe Statelesx will have some longevity after all, even if it is somewhat more limited to automating the application of a host-profile that is defined elsewhere.

So here's the call to arms. We as customers have the strongest sway with regards to product features. And with a new version of the platform in development, now is the time to start hitting our account managers inboxes with requests for this functionality. Hell, just send 'em a link to this post if it makes it easier - the important thing is to push this through the account channels, not Lance (you'd just be preaching to the converted), and the time to act is now!

ESXi Free by End of July!

Yes I've been quiet of late, but for a very good reason... which I'll divulge maybe on the weekend ;-)

But todays news (Alessandro - you are a machine aren't you? When do you sleep!) was just too big to let go by - ESXi is finally free. Well... finally officially free - technically it's been free for ages, all you had to was download a patch for it, as ESXi patches are essentially entire new images (although of course it would've run in eval mode for 60 days... by which time another patch would have come out ;-)

This is a great way for VMware to hit back after the raft of troubles lately. All the arguments from Microsoft, Xen, etc have always been about price, and they no longer have a leg to stand on.

This actually makes Hyper-V more expensive than ESXi, as you still have to pay for the Windows OS in the parent partition. And pay for no less than Enterprise Edition if you want to cluster Hyper-V. Lets not forget the storage add-ons if you want a clustered file system.

This puts the ball squarely back in the competition's court, and might as well be the final nail in the coffin for Citrix XenServer as it is now the only major commercial hypervisor on the market you have to pay for.

VI 3.5 Update 1 released - ESXi finally has full HA support!

I thought VMware were going to make a liar of me for a minute, but as reported here VirtualCenter 2.5 Update 1, ESX 3.5 Update 1 and ESXi 3.5 Update 1 have been released on April 10th. Several other plugins (Update Manager, Converter) have also received Update 1's.

Doesn't look to be anything mind blowing in the release notes, but one thing that stands out is that 3i finally has full HA support (albeit a homogenous cluster is required). Well done VMware, I'm glad we didn't have to wait for the next major release for that!

I'll go through an upgrade over the weekend and report back any findings, unless Mike beats me to it of course :-)

Download the bits here

VMware Workstation 6.5 - ESX / ESXi will not run as a Guest :-(

Thank god the beta has gone public, because for the past 6 months that I have been on the private release program, I have been dying to get this into the community at large.

Workstation has undergone significant changes in 6.5, one of which is that it is no longer possible to run ESX as a VM. See this thread in the VMware forums for details, and I urge anyone else who cares to request the feature be brought back like I did in the 7th post of that thread :-)

VirtualCenter 2.5 & ESX / ESXi 3.5 Update 1 imminent - what caveats will there be?

I must admit, I'm a bit sceptical that VMware will really deliver on it's new naming convention. My understanding is that they want to move away from dot-dot releases and go to 'updates' in order to imply a common code base or something, kind of like Microsoft are doing with their 'no new functionality in service packs' mantra. But like Windows Service Packs, I have no doubt 'updates' will still have all the hardware vendor re-certification requirements of dot releases. Here's hoping they can at least do away with forking patches based on 'update' level... otherwise, what is the bl__dy point?

Dot-dot releases have long been called 'maintenance' releases by VMware, but somehow they are considered significant enough to have serious interop requirements. For example, managing ESX 3.0.2 hosts with VC 2.0.1 is an unsupported configuration. If updating VirtualCenter to a new 'maintenance release' level wasn't tantamount to a full uninstall and reinstall of VC (kissing goodbye to your database in the process if you're not careful), it wouldn't be such a big deal... but we all know that is not the case! I really hope the same upgrade pain and support requirements don't carry through with the new 'update' nomenclature, guess we'll know in a couple of days ;-)

VI 3.5 Update 1 Release Date - April 10th

Someone slipped up earlier today and posted links to ESX 3.5 Update 1, ESX 3i Update 1 and VirtualCenter 2.5 Update 1 on the VMware downloads page (there was nothing actually available for download though). Presumably that person thought the world runs on California time or something, as the links disappeared a short time later.



Anyway, April 10th looks to be the release date for Update 1 so if you're planning anything significant in the near future, might be worth holding off to see exactly what is included in the update.

I just noticed that 3i has a name change to ESXi in that screenshot too... good to see the marketers getting their share of this release (assuming it's not a typo)!

ESX 3i - Almost, but not quite there yet

What a contentious title for a post, however I couldn't help but notice how many times I saw / heard "not supported in 3i" during VMworld Europe 2008, despite the significant keynote time given to IBM, HP and Dell who all duly announced you could get 3i on a USB stick with their hardware now!

Sure many of the things are relatively minor, but the biggest concerns for me are:

- HA is only experimental (no I'm not kidding)
- No advanced networking support that is found in 3.5 (TSO, Jumbo Frames)
- No Infiniband support that is found in 3.5

Ordinarily I wouldnt really care, but the marketing ppl at VMware would have us believe that the 2 products are identical except for the lack of a service console. As they say, the 2 kernels are identical, except when they're not. One could argue that the benefits of no service console outweigh the limitations above, but as bullish as I was on 3i when I first saw it in the beta program, I just can't justify deploying it in it's current state.

Mind you, I believe absolutely that 3i is the way of the future, and will continue to work on that stateless infrastructure I mentioned a few weeks back... which reminds me, I should get to work on that stateless infrastructure I mentioned a few weeks back!

PXE Boot ESX 3i Installable Edition

My quest for statelessness went one step further today, when I seemingly PXE booted ESX 3i. I say seemingly because I didn't actually run up any VM's on it - I did however join it to VirtualCenter, create a vSwitch and browsed it's datastore, so it's all looking pretty promising.

I'm not suggesting that such a thing should be done in production environments using the 3i installable version, however taking the files from an OEM'ed 3i and ensuring only that specific hardware model can boot that particular image could be a workable solution. I'll post the steps here not because they're especially difficult, but in the hope that others will join the quest and help to shape where VMware is going with all this. So here goes:

1. unzip install.tgz from ESX 3i Installable media

2. unrar install.rar (ie. the contents of install.tgz)

3. Go to \usr\lib\vmware\installer

4. Unzip VMware-VMvisor-big-3.5.0-XXXXX.i386.dd.bz2

You now have the main image transferred to disk when installing ESX 3i

5. Open up the VMware-VMvisor-big-3.5.0-XXXXX.i386.dd image with your tool of choice (I like the free ftk imager lite from AccessData )

6. Extract the mboot.c32 file from the [root] directory of the 'Hypervisor0' partition

7. Extract the entire [root] directory from the 'Hypervisor1' partition

You now have the requisite boot files and image files for a functioning 3i environment. To PXE boot it, just copy all the extracted files to your PXE server, and create a menu option for PXElinux that looks something like:

label 3i
kernel mboot.c32
append vmkernel.gz --- binmod.tgz --- environ.tgz --- cim.tgz --- oem.tgz --- license.tgz

You're now done!

You can obviously go in a few different directions from here... from something as simple as creating DHCP reservations for your hosts and setting the relevant options for hostname and domain name, to modifying the 3i startup script to do something more elaborate.

And again - there could easily be something entirely missing... there's a small vmfs partition on the dd image and I'm not sure what that's for. I'm sure 3i would need some kind of swap partition too which possibly isnt there when it's PXE booted in this manner. Further investigation is warranted (and will be done, at least by me :-)

I'll be following the rabbit hole a little deeper on this one, stay tuned...