SAAAP!? VDI - Why is Storage Such an Issue?

A rather contentious title for a post, but there are a few things on the storage front that have been bugging me lately. The body of this post will probably have very little to do with storage, just bear with me on this little derailed train of thought.

The most common thing we see around the interwebs lately on the storage front all seem to be targeted at this issue of the storage cost with VDI. Single instancing, de-duplication, linked cloning, disk streaming... you all know what I'm talking about. But is this storage issue best solved by the storage vendors?

Maintaining state on the endpoints is the reason why we need so much storage for VDI. Even with some jedi mind trickery on the storage side to reduce duplicate OS files, there's still the applications. Which could be addressed to a similar degree with the same tricks and some kind of de-duplication, but I honestly don't think the technology is there yet. Not when Microsoft is releasing security patches on a monthly basis. The de-dup functionality from the likes of NetApp is way cool no doubt, but it's a post-processing operation. And while it may be pure philosophy on my part, I just can't see that approach scaling too well - prevention is better than cure, after all. And while de-dup via post processing is certainly a more viable option than what is currently available in a pre-processing engine, all I can say is keep an eye on that space.

Another kind of de-dup I'm sure we'll see bandied about more is Citrix Provisioning Server, the rebranded Ardence product. But again, while very, very cool, it's not very practical currently due to the non-persistence of the cache. One reboot and you're back to square 1 - which is great for a grid compute farm, but not so great for VDI.

Application virtualisation obviously goes a long way towards solving this problem. VMware knows this, otherwise the Thinstall acquisition wouldn't make much sense. With app virtualisation and presentation virtualisation and we're finally starting to address the storage problem. But a major piece is still missing - environment virtualisation.

Environment virtualisation is something that people in the presentation virtualisation (read:Citrix folk) space are very familiar with. I'm not talking about simple profile redirection or mandatory profiles - I'm talking about real environment virtualisation. The kind of thing that allows you to logon to a desktop, fire up Excel, then logon to a VDI machine on the other side of the world and open Excel on that, then fire up another Excel session from a presentation server in New York, and then another one from a presentation server in Sydney, and have all your application preferences individually delivered and saved accordingly. The kind of technology that streams your profile to wherever it needs to go. Mark it for replication around the globe or don't - the choice is yours.

Which leads me inexorably back to the title of this post. Why is storage such an issue for VDI? If the endpoint was stateless, would this issue still exist? In such a world, could we use, dare I say it, local storage on a large scale? Does this have greater implications - with a fully virtualised (machine, app, and environment) stack, could Microsoft finally have a leg to stand on with the "Quick Migration vs VMotion" debate? Comments are open - what do you think?

NOTE: I changed the title of this post after finishing it and realising the original title was just plain wrong. So apologies if you came here via an RSS feed!

SAAAP!? //Sunday Afternoon Architecture And Philosophy

I really wanted to keep this blog strictly technical, but alas I'm going to have to indulge in some ranting philosophy on at most a weekly basis. Hence a new post category, which is a rather poor wordplay on the abbreviated form of "what's up". I couldn't think of anything classier unfortunately.

But with a GTA IV PS3 pack on the way to my place next week and GT5 and MGS4 only a few months away, I'm sure this little excursion won't be too painful for you all - I'll probably be lucky to get something out once a month!

First Official VI Client Plug-in Document Released - round of applause for VC architects / devs!

OK, so I've been giving the VC devs a _bit_ of a hard time lately. So just to show I'm as quick to dish out praise as I am to dish out flames, I wanted to say damn good job on the VI plugin architecture!

I have been eagerly awaiting some official documentation surrounding VI plug-in development (both client and server), ever since Andrew Kutz's amazing job of reverse engineering the framework. I guess the document release probably got lost in the publicity surrounding the Virtual Disk Development Kit released a few days prior to the release of this.

Peronsally I think the whole VI plugin idea is nothing short of a stroke of genius. What better way to combat the potential for the likes of Microsoft to relegate VirtualCenter to the background by building full VMware infrastructure management into SCVMM, than to open up the VI management framework in such a way. And the architecture... as much as I might accuse VMware's tools of lacking enterprise scalability from time to time, they've absolutely nailed it this time. I mean, look at this (from the document)



Outstanding! On the coding side, it looks so straight forward that even a hacker like me should be able to put something together fairly easily... running off the example in the document, you could pretty easily add a context menu item to ESX hosts that fires up the out of band management web interface (ie the iLO page for HP kit) of that particular host.

Hopefully it won't be too long before the framework moves out of experimental status, and we start seeing 3rd party tools leverage the functionality. I've already seen the Xsigo plugin and it looks great.

Download it here

VirtualCenter 2.5 Update 1 Upgrade Process - Not So Sucky After All!

EEEK! In my original post on the VC 2.5 Update 1 upgrade process, I incorrectly stated that the VC database user needs full sysadmin on the database server for the upgrade, and pondered why that was the case. Well it turns out that is entirely NOT the case at all!

Had I RTFM, I would have seen that the VC database user just needs the db_owner role on the MSDB system database, not sysadmin on the entire server. This is apparently a requirement for the SQL Agent job creation. Curiously, although the documentation states this is a requirement for a clean install as well, you only get the permissions error if you run the interactive installer. If you permission the VC database user with only the db_datareader, public and SQLAgentUser roles on the MSDB system database (I work in the finance industry - least privilege is important!) then run the installer silently, you don't get the permissions error and the SQL Agent jobs are still created!

Thanks to John over at the VMTN blog and the VMware support guys for sorting me out on that one!

Veeam Backup Install with Remote Database

Even with VMware Consolidated Backup and the release of VMware Site Recovery Manager around the corner, there is still a (gaping) hole left by these 2 products - the ability to failover a single guest to a regularly sync'ed and otherwise offline DR partner.

In the physical world there were 2 main reasons to invoke DR for a single node - physical hardware failure, or a configuration error that couldnt be recovered from fast enough. Virtual hardware obviously doesn't fail however (I know, I know, the underlying host could fail but you've all got HA enabled clusters to allow quick recovery from that don't you!), which leaves us pretty much with the configuration error problem. Whether it be due to OS or application patching, a change request gone pear shaped, rogue developers or administrators, whatever... the probability of a configuration error causing an outage is much greater than that of a datacenter outage (one would hope!). There is definitely a need to offer such a capability, as companies like Platespin (or Novell now I guess), Veeam and Vizioncore know all too well.

Veeam Backup is something I've been messing around with lately, and it certainly impresses in a number or areas, like being able to manage the inventories of multiple VirtualCenters and backup / replicate VM's or ESX host files between them from a single interface. I'm still waiting for someone in this space to eliminate the need for direct ESX host access though, but I digress...

NOTE that the following is pure hackery, and should not be used in production environments!

Anyway, onto the topic of the post. By default, Veeam Backup installs a SQL 2005 Express instance named 'VEEAM' on the box. Thankfully, they have built their product with enterprise scalability in mind (rare for a version 1 product from a small relatively new software company), you just need to know how to configure it. Which would be as follows:

1. Extract the files from the Veeam Backup single install binary.

2. Run 'setuplight.exe'. Make sure you have a generic domain account ready to use for the Veeam Backup service.

3. Go to the install directory, and modify the 'dbconfig.xml' file to point to your remote database host and instance.

4. While you're in the install directory, copy the 'DBcreate.sql' file to the remote database host.

5. On the remote database host, create a login for the generic domain account used in step 2.

6. Create a database named 'VeeamBackup' and set the owner to be the generic domain account login created in step 5. Dont just grant the login the db_owner role - the login should be directly mapped to the dbo account for the VeeamBackup database.

7. Open up the 'DBcreate.sql' file you copied over in step 4, add the following at the start of the file and execute it in Query Analyzer:

USE VeeamBackup
GO

8. Now jump back on the box where Veeam Backup was installed, and fire up the GUI - you're done!

Video coming soon... in the meantime, go grab the bits and check it out for yourselves!

UPDATE The good folk at Veeam got in touch regarding this post... turns out I inadvertently stumbled upon something they're planning for a future Enterprise focused release. So don't deploy this configuration in your production environments just yet, and keep an eye out for the upcoming Enterprise product release. I'll be sure to run that through its paces too :-)

Clustering Guests - no longer possible with Windows Server 2008?

It seems Microsoft have ditched the ability to use a parallel SCSI device as a shared disk resource in Windows Server 2008 failover clusters, which now require a SAS, FC or iSCSI LUN for the job.

I have yet to work somewhere that clustered a lot of VMs, but potentially this and the 'one physical node / one virtual node' are not possible anymore with 2008 unless you have iSCSI available. Coincidentally perhaps, I have also yet to work somewhere with a lot of iSCSI deployed :-)

I haven't tried it in ESX (yet) but the Windows Server 2008 cluster validation wizard fails with a Workstation 6 based cluster. If anyone else out there knows / tests it out before me, let me know!

UPDATE: As expected, it doesn't work on ESX either. Wonder if it will work on Hyper-V...

VirtualCenter 2.5 Update 1 - Upgrade Process Still Sucks!

Well what can I say... the old days, the bad days, the all-or-nothing days - they're back!

Once again, the VirtualCenter upgrade process is the equivalent of a full uninstall and reinstall. Someone better let the developers of the VMware management tools suite (or at least whoever writes the installers) that VMware are supposed to be an enterprise software company.

I have a video of the process, but until I find that elusive video host that allows resolutions of 800 x 600, I can't post it up. But the usual caveats apply... here are the main ones:

1. The VirtualCenter service is removed and reinstalled to run under the context of LocalSystem. So for all you ppl who have enterprise class deployments with the database on another host and vpxd running as a domain account in order to use Windows Authentication to the database, you need to reset the service account credentials. I would strongly advise doing this after the VC installer has run but BEFORE proceeding with the database upgrade wizard.

2. The VirtualCenter database user must have sysadmin rights on the entire database server in order for the upgrade the work. Why this is the case is anyone's guess - a clean install only requires 'dbo' rights on the VC database itself. But that isn't enough for the upgrade.

3. The bug with ADDLOCAL=VPX is still present if you're doing a clean install. I can't remember if I logged a bug report for that already, but looks like I need to raise another one. We don't use WebAccess in my organisation, and given the choice I'd rather not have to install Tomcat on the VC box - it's just one more bit of software that will require security patches. Might need to log a feature request for an IIS based web console.

The support matrix document still isn't updated to reflect whether managing ESX 3.5 U1 with VirtualCenter 2.5 GA is a supported configuration. It will be a small consolation if it is, and an absolute disappointment if it is not.

For anyone running VC 2.5 already, it's not worth upgrading IMHO. The release notes really don't provide any compelling reason to upgrade unless you're running a lot of ESXi and want non-experimental HA (which suggests that the HA code has been moved into the VC agent). For anyone doing fresh installs or upgrading from 2.0.x it would be worth going straight to Update 1 though.

VI 3.5 Update 1 released - ESXi finally has full HA support!

I thought VMware were going to make a liar of me for a minute, but as reported here VirtualCenter 2.5 Update 1, ESX 3.5 Update 1 and ESXi 3.5 Update 1 have been released on April 10th. Several other plugins (Update Manager, Converter) have also received Update 1's.

Doesn't look to be anything mind blowing in the release notes, but one thing that stands out is that 3i finally has full HA support (albeit a homogenous cluster is required). Well done VMware, I'm glad we didn't have to wait for the next major release for that!

I'll go through an upgrade over the weekend and report back any findings, unless Mike beats me to it of course :-)

Download the bits here

VMware Workstation 6.5 - ESX / ESXi will not run as a Guest :-(

Thank god the beta has gone public, because for the past 6 months that I have been on the private release program, I have been dying to get this into the community at large.

Workstation has undergone significant changes in 6.5, one of which is that it is no longer possible to run ESX as a VM. See this thread in the VMware forums for details, and I urge anyone else who cares to request the feature be brought back like I did in the 7th post of that thread :-)

VirtualCenter 2.5 & ESX / ESXi 3.5 Update 1 imminent - what caveats will there be?

I must admit, I'm a bit sceptical that VMware will really deliver on it's new naming convention. My understanding is that they want to move away from dot-dot releases and go to 'updates' in order to imply a common code base or something, kind of like Microsoft are doing with their 'no new functionality in service packs' mantra. But like Windows Service Packs, I have no doubt 'updates' will still have all the hardware vendor re-certification requirements of dot releases. Here's hoping they can at least do away with forking patches based on 'update' level... otherwise, what is the bl__dy point?

Dot-dot releases have long been called 'maintenance' releases by VMware, but somehow they are considered significant enough to have serious interop requirements. For example, managing ESX 3.0.2 hosts with VC 2.0.1 is an unsupported configuration. If updating VirtualCenter to a new 'maintenance release' level wasn't tantamount to a full uninstall and reinstall of VC (kissing goodbye to your database in the process if you're not careful), it wouldn't be such a big deal... but we all know that is not the case! I really hope the same upgrade pain and support requirements don't carry through with the new 'update' nomenclature, guess we'll know in a couple of days ;-)

Use Passthrough Authentication in VMware SDK Apps

It turns out that it was staring me in the face all along... there's a new method mentioned right inderneath the standard Login() method in the online VMware API reference, called LoginBySSPI().

But using it isn't quite as straight forward as replacing Login() with LoginBySSPI() on it's own... you first need to grab a security context via C++ and pass that as an argument to LoginBySSPI(). I can barely read C++, let alone write it, so I need to get my head around the easiest way to use it. When I work that out, I'll post back a few samples from the VI SDK using LoginBySSPI() instead of Login(). Unless someone like Andrew Kutz happens to read this and expend 0.00001% brain power to provide the answer :-)

I'll also be asking VMware when SSPI will be moved out of 'experimental support' status... things like VCB could really benefit from it, as could a raft of other 3rd party apps.