The Myth of Infrastructure Contention

Back... caught u lookin for the same thing. It's a new thing, check out this... oh no wait. It ain't a new thing, it's just another Sunday Arvo Architecture And Philoshopy post. This time I'm going to focus on a long time thorn in many of our sides - the myth of infrastructure resource contention.

This ugly beast rears it's head in many ways, but in particular when consolidating workloads or changing hardware standards (such as standardising on blade). Of course, the people raising these arguments are often server admins with little or no knowledge of the storage and network architecture in their environments, or consultants who have either never worked in large environments or also do not know the storage and network architectures of the environment they have come into. Which is not their fault - due to the necessary delineation of responsibility in any enterprise, they just don't get exposure to the big picture. And again, I should say from the outset that I'm talking ENTERPRISE people! Seriously, if I cop shit from one more person who claims to know better based on their 20 "strong" ESX infrastructure or home fucking lab, I am going to break out the shuriken. YES THE ENTERPRISE IS DIFFERENT. If you have never worked in a large environment, you can probably stop reading right now (unless you want to work in a large environment, in which case you should pay close attention). Can you tell how much these baseless concerns get to me? Now where was I...

OK a few more disclaimers. In this post I will try to stay as generic as possible to allow for broader applicability, and focus on single paths and simplistic components for the sake of clarity. Yes I know about Layer 3 switches and other convergent network devices and topologies, but they don't help to clarify things for those who may not know of such things. Additionally, the diagrams below are a wierd kind of mish-mash of various things I've seen in my time in several large enterprises, and I suck at Visio. Again, I have labelled things for clarity more than accuracy, and chopped stuff down in the name of broad applicability. Keep that in mind before you write to me saying I've got it all wrong.

IP Networks
So lets tackle the big one first, IP networks. Before virtualisation, your network may have looked something like this:

Does that surprise you? If it does, go ask one of your network team to draw out what a typical server class network looks like, from border to server. I bet I'm not far off. Go and do it now, I'll wait for you to get back.

OK, enlightened now? And in fact if you are, the penny has probably already dropped. But in case it hasn't, lets see what happens when the virtualisatoin train comes rolling in, and your friendly architecture and engineering team propose putting those 100 phsyicals into a blade chassis. It is precisely at this point that most operations staff, without a view of the big picture, start screaming bloody murder. "You idiot designers, how the hell do you think you can connect 100 boxes with only 4GB (active) links!!! @$%#@%# no way I'm letting that into production you @#%$%!!!". However, when we virtualise those 100 physical boxes and throw them all into a blade chassis, our diagram becomes:

OK, _now_ the penny has definitely dropped (or you shouldn't have administrative access to production systems). IT DOESN"T MATTER WHAT IS BELOW THE ACCESS LAYER. Because a single hop away (or 2 if you're lucky), all that bandwidth is concentrated by an order of magnitude. The networks guys have known this all along. They probably laughed at the server guys demands for GbE to the endpoints, knowing that in the grand scheme of things it would make fuck all difference in 90% of cases. But they humoured us anyway. And lucky for them they did, because the average network guy's mantra of "the core needs to be a multiple of the edge" needs to be tossed out on it's arse, for different reasons. But that's another post :-).

Fibre Channel Networks
I know I know, I really don't need to be as blatant about it this time, because you know I'm going to follow the exact same logic with storage. But just to drive the point home, here again we have our before virtualisation infrastructure:

And again, after sticking everything onto a blade chassis:

I don't think the above needs any futher explanation.

I'm sure there are a million variations out there that may give rise to what some may think as legitimate arguments. You may have a dedicated backup network, it may even be non-routed. To which I would ask, what is the backup server connected at? What are you backing up to? Whats the overall throughput of that backup system? Point is, there will _always_ be concentration of bandwidth on the backend, be it networking or storage, and your physical boxes don't use anywhere near the amount of bandwdith that you think they do. You may get the odd outlier, sure. Just stick it on it's own box, but still put ESX underneath it - even without the added benefits of SAN and cluster membership, from an administrative perspective you still get many advantages of virtualising the OS (remember, enterprise. We don't pay on a per-host basis, so the additional cost of ESX doesn't factor in for enterprises like it would for smaller shops).

OK, time to wrap this one up. Your environment may vary from the diagrams above, but you _will_ have concentration points like those above, somewhere. That being the case, if you don't have network or storage bandwidth problems before virtualisation, don't think that you will have them afterwards just because you massively cut the aggregate endpoint connectivity.

The Rise of the Stupid Endpoint

Welcome to the first SAAAP of 2009. And boy am I feeling philosophical today :-). The similarities between this post and David Isenberg's article Rise of the Stupid Network end with the title, but if you haven't read Isenberg's article you should do so - the principles he outlined over 10 years ago are just as applicable today to network infrastructure.

The Failure of Distributed Computing
The proliferation of distributed computing in the datacenter was essentially driven by the low cost of x86 hardware. Rather than spending huge sums on mainframe or proprietary Unix infrastructure and have it sit there underutilised for a long period (hmmm, sound familiar?) or use it for non-critical applications, why not buy smaller, cheaper x86 servers instead? The managment of these many small systems was never really factored into the equation however, and before too long there were more applications for management of this infrastructure than you could poke a stick at. But for a long time, those management systems were generally lacking in intelligence. And because of that, more staff were needed to manage the management systems, respond to alerts etc etc. x86 didn't turn out to be as cheap as promised, not because of the hardware but because of the lack of intelligent management systems.

What We Learned from Management Systems
The distributed computing management systems taught a new generation of system administrators (as was I) what had existed in the mainframe infrastructure for decades - the advantages of centralisation. Monitoring baselines, patch and software distribution, backup, job scheduling... these things have always been centralised in enterprise distributed computing environments. Such environments would be extremely expensive to operate otherwise. Yet for some reason this train of thought never really made it all the way to the personality of the endpoint. Because of this, system level backup and restore is of paramount importance in todays datacenter. DR and lifecycle management is a painfully manual and labour intensive process. Administrative overhead is exacerbated.

Endpoint Stupidity, Centralised Intelligence
You cannot make something stupid without simultaneously making something else intelligent. And that intelligence needs to be centralised. A single endpoint has many touch points in the enterprise. Are all these touch points still required? Is the cost of removing them greater than the cost of living with them? Is the pain simply due to a lack of intelligence in the tools, or the processes that you follow? Orchestration tools may be the solution for this, but do not fall into the trap of merely alleviating the pain and not addressing the underlying cause.

Infrastructure Like Water
"The height of cultivation is really nothing special. It is merely simplicity; the ability to express the utmost with the minimum." - Bruce Lee
That's what it all boils down to. In order for distributed computing infrastructure to work, we need to simplify the endpoint. When the endpoint is simple, it can be formless like a cloud (sorry, I couldn't resist), empty like a cup. You cannot have elasticity in the datacenter without this simplification. A piece of hardware may be running ESX today, Windows tomorrow, and shutdown the next day. In order to do that, the personality of the hardware cannot exist on the hardware itself. Likewise with a workload. It may ask for one level of resources today and another tomorrow, it doesn't matter. It may be running in one location today and another tomorrow, it shouldn't matter. Virtualisation has obviously made this much easier than it would have been otherwise, and I'm taking virtualisation as whole here - not just hypervisors, but VLANs, VSANs, VIPs, DDNS etc etc. But the same principles need to apply to both physical and virtual infrastructure.

End Game
So that's the what, it's up to people like you and me to figure out the how. With problems like these to solve, how can 2009 not be a good year :-)

It's Beginning to Look a Lot Like Citrix...

VDI that is. Technically that should say 'terminal services', but I had to get in a Christmas themed post title before the 25th :)

Remember back when VDI reached that tipping point in the enterprise? It was hailed as offering unprecedented levels of flexibility. Hailed as the slayer of terminal server based environments with their draconian mandatory profiles and restricted application sets. And biggest of all, hailed as finally providing us with a unified environment for application packaging, support and delivery. And so we all embarked on this VDI journey. But of course, there is a difference between knowing the path and walking the path. And having travelled this road for a while now, a sense of déjà vu is creeping in.

Yes, it is decidedly feeling a lot like Citrix. In an effort to drive down costs, the VDI restrictions and caveats are coming out of the woodwork. Scheduled desktop "refreshes", locking down desktop functionality in the name of stopping writes to the system drive, redirecting profiles to other volumes, squeezing a common set of required applications into a base 'master' template and disallowing application installs, etc etc. Software solutions are being ushered in to address these issues (brokers, app virtualisation, user environment virtualisation) - all for a premium of course. The same way it happened with Terminal Services / Citrix. We need to buy Terminal Server CALs from Microsoft, Citrix CAL's from Citrix, pay additional for enterprise features like useful load balancing, even more for decent reporting capabilities, then we need to do something about profile management, we need separate packaging standards, there are separate support teams, separate application compatibility and supportability issues, etc etc.

If we continue to slap all these restrictions and add all these infrastructure / software (ie cost) requirements on top of VDI, we run the risk of turning it into the very thing we were trying to get away from in the first place. The problem is without them, VDI doesn't provide a cost effective solution for _general desktop replacement in well connected environments_ yet. I'll emphasize the important bit in that sentence, in bold caps so I'm not misunderstood. _GENERAL DESKTOP REPLACEMENT IN WELL CONNECTED ENVIRONMENTS_. There are of course loads of other use cases where VDI makes great sense. But a desktop replacement for the average office worker in a well connected environment ain't one of them.

And the desktop requirements for the average office worker is on the rise. In this new world of communication, IM based voice and video is coming to the fore. And yet none of the big players in the VDI space have a solution for universal USB redirection or high quality bi-directional audio/video to remote machines. But does it even make sense to develop such things? Either way, you still have a requirement for local compute resources and a local operating environment in which these devices can be used. Surely it makes more sense to use that local machine for the minimal desktop and applications that use this hardware (VOIP clients, etc), and remote all the other apps into the physical machine from somewhere else?

Maybe we'd be better off parking those plans for VDI world domination for the moment, and focusing on next generation application delivery and user environment virtualisation for our current desktop infrastructure (both physical and virtual). Once those things are in place, we will be in a much better position to assess just how much sense VDI really makes as a general desktop replacement.

Kicking ESX Host Hardware Standards Down a Notch

The hardware vs software race is becoming more and more a hare vs tortoise these days, with the advent of multicore. And you can understand why - concurrency is hard. _Very_ hard. But oddly enough, I don't see much on the intertubes about people changing their hardware standards, except for the odd bit of marketing.

Although the HP BL 495c is in some respects an interim step to where we really want to go (think PCIe-V), the CPU / memory design of the blade is pretty much spot on. That is, as more cores become available, we should be moving to less sockets and more slots.

I'm not going to entertain the whole blade vs rackmount debate. I totally understand that blades may not make much sense in small shops (places with only a few hundred servers). I probably should change the description of my blog... I only talk enterprise here people. Not small scale, not internet scale, but enterprise scale (although to be honest a lot of the same principles apply to enterprise and internet scale architectures, on the infrastructure side at least).

The next release of VMware Virtual Infrastructure will make this even more painfully obvious. VMware Fault Tolerance was never designed for big iron - it's designed to make many small nodes function like big iron. OK, so maybe the hare vs tortoise comparison isn't really fair with regards to VMware ;-).

But this doesn't only apply to ESX host hardware standards - it should apply to pretty much any x86 hardware standards, _especially_ those targetted at Windows workloads. We've seen it time and time again - even if the Windows operating system did scale really well across lots of cores (and it won't until 2008 R2), the applications just don't. We only need to look at the Exchange 2007 benchmarks that were getting attention around the time of VMworld Europe 2008 for evidence of this. If Exchange works better with 4 cores than it does 16, you can bet your life that 99.999% of Windows apps will be in the same boat. Giving your business the opportunity to purchase the latest and greatest 4 way box will do nothing but throw up artifical barriers to virtualisation. The only Windows apps that require so much grunt are poorly written ones.

So if you haven't revisited your ESX host hardware standards in the past year or so, it's probably time to do so now, so you can be ready when VI.next finally drops. Concurrency may be hard, but I wouldnt call distrubted processing easy either - the more the underlying platform can abstract these difficult programming constructs, the more easy it will be to virtualise.

Why Times Like These are _Great_ for Enterprise IT

First, my heart goes out to out to all those who have found themselves out of a job during these dark financial times. I hope you won't find this post antagonistic in any way, it is obviously written from the perspective of someone who is still gainfully employed in a large enterprise (but what the future holds is anyone's guess). If you are in a tight situation right now, I can only hope that this post will give you some ammunition for your next interview, or help give some focus to your time between jobs.

OK, I'll lighten up a little now :-). So the title of this post may seem a little odd. All we seem to be hearing about these days is lay offs and cutbacks. I know of several institutions who have mandated all contractors to not be renewed, and others who are chopping full time staff alongside contractors. Either way, doesn't sound too great at all, and in fact I'm beginning to think I may change that title.

But then I remember what's great about restriction - it's a lightning rod for innovation. Look at all the success stories on the web - the vast majority of them were born from tight circumstances, from the poor student to the unemployed developer. And if you're on top of your game in the enterprise, it's no different - you can write your own internal success story.

Some of the things I hear around my office are no doubt typical of any architecture & engineering group in a large investment bank. The focus from on high is on keeping the lights on, and cutting costs at any available opportunity. But what's maybe different is the approach to cost cutting, which is absolutely grounded in fundamental mathematical principles. What do I mean by that? If you need to spend $1 to save $2, then you're still saving $1 so it's worth doing.

But why would you need financial dire straits for such behaviour? Surely you would follow the same principles in the good times as well as bad? Damn straight we do - the difference is in these times, _everything_ can be challenged. And when you can challenge everything, you can innovate like an engineer possessed.

Case in point, clouds. And let's be honest, there's only one viable player in the market currently, Amazon. And boy are the eyes of the business on them. But what's been lacking until now is the ability for an internal cloud to compete with them on purely a cost basis. Amazon's EC2 pricing includes a bucket of compute resources, SLA's for those compute resources, and a choice of preconfigured AMI's (I'm conveniently ignoring the ability to roll your own AMI and upload it, for the time being).

Here's what Amazon's pricing doesn't include. Guest support, backup, monitoring, antivirus, patching, and auditing. Where is the greatest cost associated with enterprise compute resources? Those very same things.

Now, yes I am conveniently ignoring the ability to roll your own AMI and upload it to EC2, but even if you did include all the aforementioned agents in your image, how practical would that actually be? I have yet to see an enterprise with a monitoring system that was intelligent enough to know the difference between "down because someone shut me down because they don't need me right now" and "down because of a hardware / software error". So if, in order to get Amazon's cheap pricing, we're willing to forgo monitoring within the guest, then surely I can do the same for my internal cloud machines? And the same goes for backup. Machines on a backup schedule don't only attract backup software licensing costs, there is monitoring overheads, and storage costs associated with the backed up data. And again, backup systems are generally inflexible. A backup missed is a backup missed, and generally the operational decree will be that it needs to be run at the first opportunity. But in order to get Amazon's cheap pricing (there's additional charges for network IO), we're willing to forgo backups. Hey, there's another cost I can strike off the list for my internal cloud offering! How about authentication? Domain membership has all sorts of implications for creation / deletion / archiving or machines, snapshots that get rolled back beyond 30 days etc. Windows based EC2 machines with "authentication services" attract something like a 50% premium. Perhaps my internal cloud machines should too. After all, does every development box _really_ need to be on the domain? How about grid nodes?

I won't bore with going through the rest of the list, but you get the picture. In order to get this rock bottom pricing, there is a _lot_ of functionality that is generally taken for granted in the enterprise that needs to be stripped out.

One of the biggest challenges in the enterprise is reworking the charging model to strip out all these things that now look like "extras" in comparison to Amazon's EC2. And this is where the "great times for enterprise computing" comes into the picture - it's about time these things were fucking well treated as extras, and that an _accurate_ pricing model was available for the business that broke down all these costs, and made them optional. That's the only way we'll get an apples to apples comparison to the likes of EC2.

And you know what? It's actually happening where I work. All the restrictions that would've hamstrung us from ever offering something on par with EC2 are being lifted. Finally, cheap utility _compute_ (I can't stress that enough - COMPUTE) is something we're actually going to be able to offer for the first time ever. And it's all thanks to the financial crisis, because the laser like focus on cost would never have happened otherwise.

This of course also has implications for VMware. When I say _everything_ can be challenged, I mean _everything_. Would it be cheaper for us to pay for VMware, or for someone like me to be given a few internal developers and infrastructure resources and take a shot at building something like EC2, right down to the Xen part of things?

Right now in many companies, it's survival of the fittest. If all we're going to be left with internally is a skeletal staff of absolute guns in their respective fields, along with a mandate to drive down costs through innovation and to hell with how things used to be done, then you can bet your bottom dollar something like that is entirely possible.

So if you do find yourself out of work, maybe it's time to further develop those automation skills. Get familiar with web services, pick any language you like - C#, Java, Python etc. PowerShell is a great option too, check out the PowerShell 2.0 CTP. Sign up for an Amazon Web Services account and figure out how to do stuff (you only pay for what you use, it can be cheap). Think about what they offer, find the strengths and the weaknesses, and then think about you might implement something similar in an enterprise, and what would be required to do it better. Think about how you might burst into something like EC2 from an internal cloud. What layers need to be loosely coupled in order to do such a thing in the most efficient way? What implications do external clouds have for internal cloud architecture and operations? Now take all that, and expand your scope to Google's cloud offering, what's coming with Azure, what other players there are in the field.

Yes sir, exciting times ahead in the next few years, even moreso than they were before the financial meltdown occurred.

Microsoft Azure Infrastructure Deployment Video

I've been a bit quiet of late, mainly due to the digestion of Microsoft's PDC 2008 content (what a great idea to provide the content for free after the conference - VMware could take a leaf out of that book). I meant to post this up yesterday as another SAAAP installment but missed the deadline... but what the hell, I'll tag this post anyway!

One of the more infrastructure oriented PDC sessions was "Under the Hood - Inside the Windows Azure Hosting Environment". Skip forward to around 43 minutes into the presentation, where Chuck Lenzmeier goes into the deployment model used within the Azure cloud (you can stop watching at around 48 minutes).

Conceptually, this is _exactly_ the deployment model I and ppl like Lance Berc envisage for ESXi. Rather than put that base VHD onto local USB devices ala ESXi, Microsoft PXE boot a Windows PE "maintenance os", drop a common base image onto the endpoint, dynamically build a personality (offline) as a differencing disk (ie linked clone), drop that down to the endpoint, and then boot straight off the differencing disk VHD (booting directly off VHD's is a _very_ cool feature of Win7 / Server 2008 R2). I'm glad even Microsoft recognise the massive benefits of this approach - no installation, rapid rollback, etc.

Now ESXi of course has one *massive* advantage over Windows in this scenario - it is weighs in at around the same size as a typical Windows PE build, much smaller than a Hyper-V enabled Server Core build.

And if only VMware would support PXE booting ESXi, you could couple it with Lance's birth client and midwife or our statelesx, and you don't even need the 'maintenance OS'. You get an environment that is almost identical conceptually, but can be deployed much much faster due to the ~150MB or so total size of ESXi (including Windows, Linux and Solaris vmtools iso's and customised configuration to enable kernel mode options like swap or LVM.EnableResignature within ESXi that would otherwise require a reboot (chicken, meet egg :-)) versus near a GB of Windows PE and Hyper-V with Server Core. Of course I don't even need to go into the superiority of ESXi over Hyper-V ;-)

With Windows Server 2008 R2 earmarked for a 2010 release, it will be sometime before the sweet deployment scenario outlined in that video is available to Microsoft customers (gotta love how Microsoft get the good stuff to themselves first - jeez, what a convenient way to ensure your cloud service is much more efficient than anything your customer could build with your own technology). But by that time they will have a helluva set of best practice deployment knowledge for liquid infrastructure like this, and you can bet aspects of their "fabric controller" will find their way into the System Center management suite.

The liquid compute deployment gauntlet has been thrown VMware, time to step up to the plate.

VMware: Unshackle ESXi, or Follow Microsoft Into the Cloud...

One of the things that will be interesting to observe as the cloud matures will the influence the internal and external clouds will have on each on each other.

To this end, I think Microsoft have benefited from years of experience with the web hosting industry, and as a result have completely nailed the correct approach for them to take into the cloud. VMware doesn't have the benefit of this experience, and should watch Microsoft closely as the key to their strategy has more in common with what VMware needs to be successful than one might think. That key being cost.

For years, Microsoft never made a dent in the web hosting space. It would be naive to attribute this solely to the stability and security of the platform - IIS 5 had a few howlers sure, but so has Apache over the same time. IIS 6 however was entirely different - in it's default configuration, not a single remotely exploitable vulnerability has been publicly identified to date. But there was the cost angle - Windows based hosting charges were far greater than comparable Linux based offerings, and they pretty much stayed that way until SWsoft came along with Virtuozzo for Windows in 2005. The massive advantage Virtuozzo had (and still has) over other Virtualisation offerings from a cost perspective is that only a single Windows license is required for the base OS - all the containers running on top of that base don't cost anything more. Now all of a sudden web hosting companies could apply the same techniques to Windows as they had used for Linux based platforms for years, and achieve massive cost reductions.

I can see a similar problem for VMware in the external cloud space. Let's compare 2 companies currently offering infrastructure based cloud services - Terremark and Amazon.

Terremark's 'Infinistructure' is based (among other things) on hardware from HP, a hypervisor from VMware (although it's not clear if this is the only hypervisor available), all managed by an in house developed application called digitalOps which apparently had a developmemt cost of over $10mil USD to date.

Amazon's EC2 on the other hand uses custom built hardware from commodity components, uses the free open source Xen (and likely their own version of Linux for Dom0), and of course a fully developed management API based on open web standards.

Comparing those 2, which do you think will _always_ cost more? If VMware want to seriously compete in the external cloud space, they need to address this. Right now I see 2 options for them. One is to unshackle the free version of ESXi, so that the API is fully functional and hence 3rd parties could write their own management tools to sit on top and not have to pay for the VMware tools (and then have to write their own stuff anyway, as Terremark have done). The other is for VMware to enter the hosting space themselves, as they would be the only company in a position to avoid paying software licenses for ESX and their VI management layer.

I'm sure VMware realise they only have a limited time in which to establish themselves in this space, because 2 trends are in motion that have the potential to render the operating system insignificant, let alone the underlying hypervisor: application virtualisation, and cloud aware web based applications. But if will likely be some time before either of those 2 will be adopted broadly in the enterprise, and hence VMware are pushing the fact that their take on cloud is heavily infrastructure focused, and thus doesn't require any changes in the application layer.

Sure, unshackling free ESXi would mean missing out on some revenue from external cloud providers, but it will go a long way towards insulating them from any external cloud influence creeping into the internal cloud. What do I mean by that? Say I was a CIO, and I had 2 external cloud providers vying for my business. While they both offer identical SLA's surrounding performance and availability, one of them offers a near seamless internal/external cloud transition based on the software stack in my internal cloud. It is however 3 times the cost of the other provider. After signing a short-term agreement with the more expensive cloud provider, I would be straight on the phone to my directs asking for an investigation into the internal cloud, with a view to making it more compatible with the cheaper external offering.

Which is why Microsoft have got the play right. They know that no 3rd party host can pay for Windows and compete with Linux on price. So instead of go after it with Hyper-V on an infrastructure level, they're taking a longer term approach and going after the cloud aware web based application market. Personally, I can't ever see VMware becoming a cloud host, which means they need to do something about unshackling ESXi. And fast - the clock is ticking.

Clouds. Plenty of Them Around This Sunday Afternoon...

... and I'm not just talking about London. The topic of cloud or liquid or utility compute is perfectly suited for an edition of Sunday Afternoon Architecture & Philosophy.

Note that I said compute there. Indeed the cloud has many other components, but infrastructure guys like me will mainly be concerned with compute. And maybe storage. And of course what connects things to either of those 2. Hmmm. OK, maybe it's more than just compute, but for all intents and purposes I'm going to try and stay focused on compute.

So lets talk about apps. See what I did there? At the end of the day, compute is only good for one thing - running apps. But I can't see us getting to this fluffy land of external federated clouds without a fundamental change to how most applications in an enterprise environment run. Now don't get me wrong - the whole point of cloud compute from infrastructure up is that we don't have to change the apps. I get that. But in reality, they're going to have to change - they have to become portable, and any state needs to move out of the endpoint and into somewhere central. Why? Because it's gonna be a looooong time before _data_ is held in an external cloud, if ever IMHO. As Tim O'Reilly points out in his recent post Web 2.0 and Cloud Computing:

The prospect of "my" data disappearing or being unavailable is far more alarming than, for example, the disappearance of a service that merely hosts an aggregated view of data that is available elsewhere.

And I think that's the key point that a lot of skeptics are missing. As an individual I hold exactly the stance that Tim describes, and all I really care about are some photos, videos, my resume and my password database. None of that stuff is required in order for me to live. What stance do you imagine companies are going to have, who's very existence depends upon data and the manipulation thereof?

Cloud compute means exactly as the name implies. COMPUTE. As in run up a compute engine (ie. an OS + App + State stack), throw some data at it, get some data back, job done. At no point did that data originate from, persist in (for any meaningful amount of time), or return to the external cloud. The security, availability and integrity of that data during transit and processing is by no means trivial, but compared to storing that data in the external cloud it is.

Which leads me in a roundabout way back to why apps need to change in order for internal compute clouds to reach their full potential and for external compute clouds to become really viable. Apps need to be delivered predictably and efficiently, that we may throw data at them. Whether that is achieved by virtualising them, streaming them, packaging them the traditional way, the choice is yours. But start thinking about it now, lest your clouds do nothing but rain on you.

Future ESXi Version Call to Arms!

While there wasn't a lot of concrete details that came out of VMworld, one thing is for certain - VMware are working on a _lot_ of new functionality for their next major release. Of particular relevance to us at vinternals is the host profiles feature, as it provides the same functionality as Statelesx, but is done in a much more user friendly way. Which is fine - it's not like we threw in out jobs and tried to position statelesx as a commercial product, and in some ways it's a validation that our idea was a good one.

But VMware need to ensure they implement the other half of the solution, ala the client initiator. Without this, there is still an unnecessary manual task of joining a host to a cluster in the first place. You may think this isn't such a big deal, but if your hosts are PXE booted then a client initiator is an absolute requirement.

Fortunately for us, there is at least one senior guy at VMware who gets all this - Lance Berc (I get a laugh eveytime I see his 'novice' status in the VMware forums - for fucks sake, this is the guy who wrote the original esxtop). So far most attention has been on his post-build configuration scripts, but Lance put his C skills to work and released the crucial client initiator piece as well as a 'midwife' which essentially performs the same duties as statelesx. Put all this together and you have a truly automated, scalable, liquid environment - the likes of which is usually only seen in compute clusters. Except this time it's a _virtual_ environment.

VMware seem to be taking notice of the need for cluster wide configuration automation with host profiles and the distributed virtual switch, we as customers need to make sure they understand the need for the client initiator functionality as well. The 'midwife' functionality would also be nice if it was wrapped into VirtualCenter, but if it isn't then maybe Statelesx will have some longevity after all, even if it is somewhat more limited to automating the application of a host-profile that is defined elsewhere.

So here's the call to arms. We as customers have the strongest sway with regards to product features. And with a new version of the platform in development, now is the time to start hitting our account managers inboxes with requests for this functionality. Hell, just send 'em a link to this post if it makes it easier - the important thing is to push this through the account channels, not Lance (you'd just be preaching to the converted), and the time to act is now!

How a stateless ESXi infrastructure might work

Yep, it's Sunday afternoon, and thus time for another installment of Sunday Afternoon Architecture and Philosophy! Advance Warning: Get your reading specs, this post is a big 'un.

I've mused several times on the whole stateless thing, especially with regards to ESXi, today I'm going to take it a bit further in the hope that someone out there from VMware may actually be reading (besides JT from Communities :-).

Previously I've showed how you can PXE boot ESXi. While completely unsupported, it at least lends itself to some interesting possibilities, as with ESXi, VMware are uniquely positioned to offer such a capability. The Xen hypervisor may be 50,000 lines of code but it's useless without that bloated Dom0 sitting on top of it. Check out the video (if you can be bothered to register, are they so desparate for sales leads that you need to register to watch a video???) of the XenServer "embedded" product - it still requires going though what is essentially a full Linux install, except instead of reading from a CD and installing to a hard drive it's all on a flash device attached the mainboard. But i digress...

So lets start at the top, and take a stroll through how you might string this ESXi stateless infrastructure together in your everyday enterprise. And I'll say upfront, I'm a Microsoft guy so a lot of the options in here are Microsoft centric. In my defense however, every enterprise runs Active Directory and it's easy to leverage some peripheral Windows technologies for what we want to achieve.

First up, the TFTP server. RIS (or WDS) is not entirely necessary for what we want to do - a simple ol TFTP server will do, even the one you can freely install from a Windows CD. In this example we'll use good ol' pxelinux, so our bootfilename will be 'pxelinux.0' and that file will be in the root of the TFTP server. The directory structure TFTP root could be something as follows:

In the TFTP root pictured above I have 3 directories named after the ESXi build. The 'default' file in the pxelinx.cfg directory presents a menu so I can select which kernel to boot. I could also have a file in the pxelinux.cfg directory named after the GUID of the client, which would allow me to specify which kernel to boot for a particular client.

If you already have RIS / WDS in your environment, things are a little less clunky... can simply create a machine account in AD, enter the GUID of the box when prompted and then set the 'netbootMachineFilePath' attribute on computer object to the file on the RIS box that you want to boot.

Onto DHCP. Options 66 (TFTP server hostname) and 67 (bootfile name) need to be configured for the relevant scope. DHCP reservations for the ESXi boxen could also be considered a pre-requisite. The ESXi startup scripts do a nice job of picking that up and handling it accordingly.

So all this stuff is possible today (albeit unsupported). If ESXi doesn't have a filesystem for scratch space, it simply uses an additional 512MB of RAM for it's scratch - hardly a big overhead in comparison to the flexibility PXE gives you. Booting of an embedded USB device is cool, but having a single centralised image is way cooler. As you can see, there's nothing stopping you from keeping multiple build versions on the TFTP server, making rollbacks a snap. With this in place, you are halfway to a stateless infrastructure. New ESXi boxes can be provisioned almost as fast as they can be booted.

After booting, they need to be configured though... and that's where we move onto theory...

The biggest roadblock by far in making this truly stateless, is the lack of state management. There's no reason why VirtualCenter couldn't perform this function. But there's other stuff that would need to change too in order to support it. For example, something like the following might enable a fully functioning stateless infrastructure:

1) Move the VirtualCenter configuration store to Lightweight Directory Services (what used to be called ADAM), allowing VirtualCenter to become a federated, mutli-master application like Active Directory. The VMware Desktop Manager team are already aware that lightweght directory services make a _much_ better configuration store than SQL Server does. SQL Server would still be needed for performance data, but the recommendation for enterprises these days is to have SQL Server on a separate host anyway.

2) Enhance VirtualCenter so that you can define configurations on a cluster-wide basis. VirtualCenter would then just have to track which hosts belonged to what cluster. XenServer kind of works this way currently - as soon as you join a XenServer host to a cluster, the configurations from the other hosts are replicated to it so you don't have to do anything further on the host in order to start moving workloads onto it. This is probably the only thing XenServer does _way_ better than VI3 currently. Let's be honest - in the enterprise, the atomic unit of computing resource is the cluster these days, not the individual host. Additionally, configuration information could be further defined at a resource pool or vmfolder level.

3) Use SRV records to allow clients to locate VirtualCenter hosts (ie the Virtual Infrastructure Management service). Modify the startup process of ESXi so that it sends out a query for this SRV record everytime it boots.

4) Regardless of which VirtualCenter the ESXi box hit, since it would be federated it can tell the ESXi box which VirtualCenter host is closest to it. The ESXi box would then connect to this closest VC, and ask for configuration information.

By now all the Windows people reading this are thinking "Hmmm, something about that sounds all too familiar". And they'd be right - Windows domains work almost exactly in this way.

SRV records are used to allow clients to locate kerberos and LDAP services, ie Domain Controllers. The closest Domain Controller to the client is identified during the logon process (or from cache), and the client authenticates to this Domain Controller and pulls down configuration information (ie user profile and homedrive paths, group membership information for the user and machine accounts, Group Policy, logon scripts etc). This information is then applied during the logon process, resulting in the user receiving a fully configured environment by the time they logon.

I haven't had enough of a chance to run SCVMM 2008 and Hyper-V through their paces to see if they operate in this manner. If they don't, VMware can consider themselves lucky and would do well to get this functionality into the managment layer ASAP (even if it means releasing yet another product with "Manager" in the title :-).

If Microsoft have implmented this kind of functionality however, VMware needs to take notice and respond quickly. Given that the management layer will become more and more important as virtualisation moves into hardware, VMware can't afford to slip on this front.

Congratulations if you made it this far. Hopefully you've enjoyed reading and as always for this kind of post, comments are open!

SAAAP!? VDI - Why is Storage Such an Issue?

A rather contentious title for a post, but there are a few things on the storage front that have been bugging me lately. The body of this post will probably have very little to do with storage, just bear with me on this little derailed train of thought.

The most common thing we see around the interwebs lately on the storage front all seem to be targeted at this issue of the storage cost with VDI. Single instancing, de-duplication, linked cloning, disk streaming... you all know what I'm talking about. But is this storage issue best solved by the storage vendors?

Maintaining state on the endpoints is the reason why we need so much storage for VDI. Even with some jedi mind trickery on the storage side to reduce duplicate OS files, there's still the applications. Which could be addressed to a similar degree with the same tricks and some kind of de-duplication, but I honestly don't think the technology is there yet. Not when Microsoft is releasing security patches on a monthly basis. The de-dup functionality from the likes of NetApp is way cool no doubt, but it's a post-processing operation. And while it may be pure philosophy on my part, I just can't see that approach scaling too well - prevention is better than cure, after all. And while de-dup via post processing is certainly a more viable option than what is currently available in a pre-processing engine, all I can say is keep an eye on that space.

Another kind of de-dup I'm sure we'll see bandied about more is Citrix Provisioning Server, the rebranded Ardence product. But again, while very, very cool, it's not very practical currently due to the non-persistence of the cache. One reboot and you're back to square 1 - which is great for a grid compute farm, but not so great for VDI.

Application virtualisation obviously goes a long way towards solving this problem. VMware knows this, otherwise the Thinstall acquisition wouldn't make much sense. With app virtualisation and presentation virtualisation and we're finally starting to address the storage problem. But a major piece is still missing - environment virtualisation.

Environment virtualisation is something that people in the presentation virtualisation (read:Citrix folk) space are very familiar with. I'm not talking about simple profile redirection or mandatory profiles - I'm talking about real environment virtualisation. The kind of thing that allows you to logon to a desktop, fire up Excel, then logon to a VDI machine on the other side of the world and open Excel on that, then fire up another Excel session from a presentation server in New York, and then another one from a presentation server in Sydney, and have all your application preferences individually delivered and saved accordingly. The kind of technology that streams your profile to wherever it needs to go. Mark it for replication around the globe or don't - the choice is yours.

Which leads me inexorably back to the title of this post. Why is storage such an issue for VDI? If the endpoint was stateless, would this issue still exist? In such a world, could we use, dare I say it, local storage on a large scale? Does this have greater implications - with a fully virtualised (machine, app, and environment) stack, could Microsoft finally have a leg to stand on with the "Quick Migration vs VMotion" debate? Comments are open - what do you think?

NOTE: I changed the title of this post after finishing it and realising the original title was just plain wrong. So apologies if you came here via an RSS feed!

SAAAP!? //Sunday Afternoon Architecture And Philosophy

I really wanted to keep this blog strictly technical, but alas I'm going to have to indulge in some ranting philosophy on at most a weekly basis. Hence a new post category, which is a rather poor wordplay on the abbreviated form of "what's up". I couldn't think of anything classier unfortunately.

But with a GTA IV PS3 pack on the way to my place next week and GT5 and MGS4 only a few months away, I'm sure this little excursion won't be too painful for you all - I'll probably be lucky to get something out once a month!