I've got a cake. It's my favourite variety, caramel mud. As much as I'd like to eat the whole cake myself, in a shameless attempt to increase my blog audience I'm going to eat half and give half to the first person to email me after reading this post*.
Question: How much cake is left for the second person to email me after reading this post?
None. That's right, I ate half and gave the other half to the first mailer.
But if I was HA configured to allow a single host failure, some recently observed behaviour indicates that the first mailer would actually only get 1/4 of the cake, and there would be some cake left for every person who mailed me until infinity. This is because HA doesn't appear to have any concept of a 'known good state' - instead, it seems that if a single host fails (half cake eaten by me), it then re-evaluates the cluster size (the half cake left is now considered the whole cake) and adjusts it's resource cap to allow for another single host to fail (first mailer takes half of the new cake size, the cake size is recalculated, next mailer gets half the new cake size, the cake size is recalculated, ad infinitum).
In my mind, this behaviour is not correct. If I tell HA I want to allow for a single host failure, then it should have some idea of what my known good state is. When one host actually does fail, then my risk criterion has been met and all resource guarantees in the event of a subsequent host failure are off. But it seems that all HA knows is how many hosts are in a cluster at any point in time, and how many failures to allow for at any point in time. This obviously has some implications for cluster design, in particular the "do not allow violation of availability constraints" option. Previously I was a strong advocate of taking this option - in the enterprise, soft limits are hardly ever honoured. But now I'm looking at using a "master" resource pool to achieve my resource constraint and switching to the other option.
I'm hoping the first person to mail me after reading this will be Duncan or Mike telling me I don't know wtf I'm talking about, but I've come to this belief after some conversations with VMware during a post mortem of a production incident so I'm not entirely to blame if I'm wrong :-P. If anyone else out there has seen similar behaviour, email me now - that half-a-cake could be yours!
*The cake is a lie.