Sorry for the delay in posting this last part, but vacation took priority.  To finish the series out, I want to provide you with a short list of things not to do.  These are the things I’ve seen customers not do properly that eventually come back to haunt us and make us all look bad (IT staff and consultant).  Obviously, anything that would be opposite of any items above would fit here as well.

1. Don’t pay for a consultant to come in, then stick him in a room and say “come on out when you’re done.”  There’s no way he can understand your objectives or the special quirks your organization has.  This approach reduces the valuable knowledge transfer and experience you could gain by doing the work yourself while the consultant guides you through the process.
2. Make sure your house is in order first.  If your infrastructure is not prepared, don’t rush the project in.  Manny of these items have gone into my pre-engagement checklist.
3. Nothing in IT is perfect, so don’t expect VDI to be.  As much as we depend on computers these days, we should all realize by now that they don’t always work as we plan them to.  Be prepared for this and don’t blame anyone for it.  Work through it together.  Collaboration is key, because one person does not always have every answer.
4. Cutting corners saves money, but cutting the wrong corners too deep will destabilize the building.  Work with your partner to determine when, where and how much the quote can be cut back.

Many of the items in all three parts rely on you trusting your partner, which is why I put that at the very beginning.  Go back and read each of my recommendations and you’ll realize that most of them involve trusting your partner to some extent.  You may have a lot of expertise with virtualization and desktop management, but in this case 1 + 1 doesn’t equal 2.

Hopefully my role as a consultant doesn’t make you instantly dismiss my recommendations.  I have only spent the last two years as a consultant, so I can still put myself into the mindset of a customer.  While I was a customer, we did a mix of learning on our own and relying on consultants for many different non-VDI projects.  While most projects were successful, I always felt the ones that were the most successful were the ones where we had a trustworthy advisor.

VDI really is a paradigm shift and will require you to think in different ways.  That’s not to say you or any of the customers I’ve worked with couldn’t do it on their own, after all, I and every other VDI pioneer had to figure it out somehow.  The advantage a seasoned consultant can bring is experience, which manifests as quicker, more decisive success for you.

In part one of this three part series, I covered the first five items I have found to help make VDI deployments successful.  Now let’s cover the final four items:

1. Plan for future growth.  Don’t spend all the money up front.  Design the infrastructure for growth and budget for cyclical upgrades.  Perhaps you skimp on the storage during the pilot, but set aside money in subsequent quarters for additional drives/shelves.  Leave room for additional RAM in the hosts and set aside money to purchase that RAM later.  You will never know for sure where all the bottlenecks are during the design phase, so having a quarterly upgrade budget will allow for easier resolution of unexpected bottlenecks.
2. Don’t expect the solution to be perfect on day one.  Of course we will all strive for it, but prepare for a rocky road to start out with.  Hope for the best, but expect the worst.  How will you handle grumpy accountants?  What is the best approach for the doctors who don’t feel like they need to comply?  Do you plan to teach the customer service representatives all about their new environment before you even turn it over to them?  Secondarily, budget or prepay for your consultant to come back on site during or after the launch.  This way you can have your (hopefully trusted) consultant available to help with any odd issues that may pop up.  My sales team will often pitch a block of hours to our customers during the presales phase for this specific purpose.  If all goes well, our customers can use this prepaid time to do a six month review of the implementation, help design the next phase, clean up group policy, or help with something else totally unrelated to the VDI environment.
3. Manage your users as closely as you manage your infrastructure.  Train them ahead of time on any changes they’ll need to endure.  Be ready to have technical feet on the floors when they go live on the new system and make sure those feet carry a smile along with them.  Try and utilize superusers in each department to augment the technical staff.  These people will best understand the intersection of business process and technical infrastructure and will be more trusted by the end users.
4. Ask your consultant ahead of time what you can do to prepare your current environment to make the implementation go smoothly.  If you can predefine all the IPs you need before hand, you’ll save valuable time that will greatly benefit everyone later.  I have a document I have developed over the last couple of years that details out exactly what will need to be done to their existing environment so I don’t have to deal with it when I get on site.  I simply make sure the document gets into their hands a week before the project starts and we can start right away with installing the VDI components.

In the final post of this series, I’ll cover a few things I’ve found that can severely harm a VDI project.

I have recently wrapped up my most successful View deployment yet.  As I look back on the project and reflect on its success, I found this blog post writing itself.  In fact, it wrote itself so well, I plan on breaking it up into three different parts.  

Here are the first five items I have found to be critical success factors I in this and several other projects I consider successful:

1. Trust your consultants.  Keep them on board and informed through the entire process.  Make sure they understand both IT’s goals and the business’s goals.
2. Deliver a complete infrastructure.  It doesn’t have to be completely greenfield, but it should be well planned and completely integrated.  Duct tape should not be taken out of the toolbox for this project.
3. Run a complete and thorough proof of concept and pilot.  Nothing beats running the proposed infrastructure for real.  You might find it won’t work and have to throw away all the time and money, but that’s better than building the entire environment and have to make it work due to the size of the investment.  This also helps you to see around all the vendor half-truths and smokescreens and get a true appreciation of the capabilities of each of the products in the solution.
4. Don’t mold VDI into your current processes, take a fresh approach to both and design and deliver them as a single package. This includes both business and IT processes.  If you can, introduce it along with another major process disrupter.  If you’re introducing a new CRM package that completely changes the way the organization will manage its data, introduce your new VDI processes at the same time.  One is bound to fail if you try to retrain your users twice, so why not completely turn them upside down and only train them once?
5. Completely understand your infrastructure.  A consultant may have designed and built most of your solution, but you need to support it.  Learn all you can from the consultant while you can.  Attach to their hip and don’t allow yourself to be distracted.  Ask where the weak spots are and where the bottlenecks will be.  If you develop a good rapport with the engineer, you’ll learn stuff that the pre-sales team won’t give up easily.  This will lead us into the next tip, but you’ll have to wait for the next post.

Part 2 >>

Fear, Uncertainty and Doubt, or F.U.D., is a term I seem to have really come to use a lot lately.  It’s a marketing approach that seems to transcend industries.  You see it in politics as one candidate tries to undermine his/her opponents (in politics it’s usually called mud-slinging), in the battles between the cable/Internet/phone providers (here in Omaha it’s between Cox Communications and Qwest Communications), in advertisements for cleaning products (what percentage of germs does Clorox kill compared to Lysol) and of course within our own beloved IT industry.

Recent examples of IT FUD (at least in the infrastructure realm) have included, but definitely not limited to:

• Microsoft/Citrix v. VMware –hypervisor and virtual desktop/application infrastructure
• Vizioncore v. Veaam – The especially heated virtualization backup and management realm
• HP v. Dell v. IBM v. Cisco – “<insert company name> is the best, purpose-built platform for virtualization”
• HP v. EMC v. NetApp – the ongoing storage wars

All of the above mentioned companies deserve their place towards the top of the heap in their respective piles, which of course leads to very heated debate.  This debate unfortunately tends to boil over into FUD territory.

Having spent time as both a customer and a partner (though not a vendor), I definitely appreciate one company telling me the pros and cons of both their product and their competitor’s product.  In fact, I would doubt the aptitude of a vendor who didn’t have a competitive fact sheet for each of their products.  Where FUD comes into play is when the comparisons are overly tilted, based on half-truths or flat out lies and presented to a customer as an unerring truth.

 

Let’s take the recent trend of the Tolly reports and similar vendor-sponsored “independent” studies.  I use quotes around the word independent due to the fact that the third party is receiving money from only one of the competitors.  That breeds an inherent perception of a conflict of interest, which in my opinion instantly taints the report whether or not one truly does exist.

It is also well known that when a vendor performs or commissions someone else to perform a head-to-head comparison, that the tests that will be run will favor the features of that vendor’s product.  As a nonspecific example, let’s look at a storage bake-off.  Vendor A sets up a performance test between its array and Vendor B’s array.  The load that is put against both arrays could favor the caching algorithm that Vendor A’s array uses, thereby ensuring which array will perform the best.

Another aspect of FUD is the constant hammering of a competitor’s flaws, while totally ignoring any advantages their competitor may have.  You see this in the ever popular side-by-side feature comparison tables.  Two of my favorites are the VMware View v. Citrix XenDesktop feature set battles.

 

Wow, View sure is more feature rich, isn’t it?

Hold on there, apparently Citrix offers me more unique features.

 

See what I mean?  Clearly each vendor is focusing solely on the negatives of their competition, and in some cases not highlighting their best features. Also notice that VMware favors the least impactive of the competitors.

Next is the outright lie or deception.  This clearly falls into the Dirty category.  This will always occur, but in any healthy community it should quickly be knocked out of the sky.  So much of the IT industry is based on facts and numbers, so this doesn’t happen much, but it is definitely resident within politics (ever heard of a dirty politician?) where the facts and issues fall more into shades of gray.  These shades of gray make it harder to use facts to counter claims, especially when people’s emotions are thoroughly invested.

Finally, I’d like to highlight some fun that can be had with FUD.  As an example, I’d like to point out Doug Hazelman’s post here: http://veeammeup.com/2010/05/fud-for-thought.html.  Sure he’s being blatantly competitive and even admits to FUD flinging, but he also isn’t pretending that he’s giving an unbiased opinion.  FUD can also lead to great debates like we saw between VMware’s Scott Drummonds and Citrix’s Simon Crosby.  There is also a trend on Twitter and on some blogs of individuals who can transcend the Kool-Aid and have fun with their respective employer’s marketing companies along with their competitors’.  The Twiiter jabbing between Chad Sakacc and Vaugn Stewart is a perfect and perpetual example of this.  Both give credit where it is due and use sound technical arguments when disagreements appear.  Outcomes like these can turn FUD into something that actually benefits the community as a whole, but take a special set of individuals and circumstances.

So what can we do to wade through the mire that FUD creates in our decision making process?  To me it’s always been a matter of using the purely marketing information as a guidepost; a way to decide where my time would be best spent researching a set of competing products.  If Vendor A says their product is better because they don’t rely solely on SATA disks like Vendor B does, then I know I need to spend some time with the facts, trusted blog sites, a spreadsheet and possibly a lab to determine if there is some merit to using a large array of SATA disks verses a smaller array of SAS disks.  Companies do things because they think it’s a better way to do them, it’s up to us to determine if it really is a better mousetrap.

I like the way Wikipedia states it: "To dispel FUD, the easiest way is to ask for details and then provide well researched, hard facts which disproves the details.”  That clearly was written to help dispel a competitor’s FUD, but how does a consumer dispel the FUD coming from both directions?  I suggest not depending on marketing materials and asking for details from both (or all) parties, and then provide your own research and hard facts to make a fair comparison.

Anyone trying to sell you something is guilty of some of these things, though some may be more trustworthy than others.  Just remember, you must use your best judgment to make an informed decision that you can live with and defend, just like you should be doing in politics. 

I first want to apologize to all of you who were looking forward to additional posts about our VMUG setup back in March.  I ended up only having my complete environment for about a week after the VMUG before I had to yank it out of my data center and set it up at a customer site for a POC.  I still hope to get my hands on it again and do some posts on it.

In addition, I've been busy studying for certifications lately, which has left little time for blogging.  The good news here is that I now have a Microsoft MCITP Enterprise Administrator certification and will soon have an HP Master ASE certification for Blades and Clustering.

The main reason I'm finally getting around to posting an update (and apology) is that today was the announcement of the first set of vEXPERTs for 2010 and I was lucky enough to be selected for a second year in a row!  The award says nothing about my technical skills directly, but is definitely still an honor in that it acknowledges my ability to communicate the awesomeness of VMware's products and vision. 

Thank you very much to John Troyer, the man-not-so-behind-the-curtain in all things VMware social media, for heading up and pushing this award program and all its benefits.  Another special thank you to VMware and the team of vExpert judges.

I already have another post in the works, so keep an eye out for that…

All our hard work seems to have paid off.  Based on the conversations and survey results we’ve received I think everything went very well. 

Labs

The labs were a huge success and for the most part went off without a hitch.  There were some issues with the SRM labs that lead me to believe that SRM doesn’t much like having many different people connected to it at one time.  Generally this won’t be a problem as SRM will be managed by a few people who will be doing very little outside of setup and actual tests/failovers.

 

 

 

 

 

 

 

 

 

 

Architecting Sessions

The architecting sessions were probably the most challenging sessions.  The Architecting vSphere session was very heavily attended, which led to some noise issues from side conversations, and spread people out far enough that it was difficult for them to hear.  Capping the attendance for these sessions would definitely pay off in the future.

Due to a major snowstorm in Denver, the VMware security specialist was unable to attend.  This left a major gap in the session I was promoting the heaviest.  I deeply apologize to those who came out specifically for that session.  We made do with who we had on hand, and I hope they were able to provide something valuable.  I’m happy to report that we are already actively working on scheduling him to come out for a future VMUG meeting.

The Architecting View session was surprisingly the least attended of the three architecting sessions.  Based on what I’ve heard, it sounds like there were a lot of people who were simply interested in learning about View.

A general piece of feedback we received was that these sessions should have been more structured and maybe a bit more specific with the topics.  I have discussed a few ideas with the VMUG leaders on how to continue this format in different ways, so keep an eye open for more information.

 

 

Presentations

The presentations seemed to be well attended, though I admit to not having spent any time listening to them due to supporting the labs and architecting sessions.  Survey results varied, but I attribute these variations to potentially different expectations and skill levels.  Hopefully everyone found the content somewhat useful and relevant.

 

 

 

 

 

 

 

 

 

 

Data Center Tours

I’m happy to report that the data center tours went over well and were way more popular than I expected.  I think CoSentry gained a lot from this offering as it seems that many people in attendance did not know about CoSentry before the meeting.

 

 

 

 

 

 

 

 

 

 

 

 

Food, facilities, etc.

I felt the food was excellent and well presented.  Better availability of drinks would be my only request.  Other than that, I didn’t hear any complaints.

The facility, I thought, was perfect.  The open space didn’t provide the ideal acoustics, but I think that was more than balanced out by the feeling of being in a nice open area (no claustrophobia).  Registration was a bit long, but was a necessary evil.  I’m glad people showed up plenty early to sign in.

 

 

 

 

 

Overall, I hope everyone enjoyed themselves and found something worthwhile.  Based on the survey results we received (about 1/3 of attendees) everyone did find the event worthwhile and particularly enjoyed the labs and the networking aspect.  This is encouraging, since the labs took the most effort to develop and networking is one of the primary goals of the VMUG.

If you have any specific feedback, please feel free to post it on the Omaha-Area VMUG Forum or send it to me directly here.

Thank you all for coming out!

This week is the culmination of months of work for me and the many others that have been working to get the next Omaha-Area VMUG event ready for an AMAZING time!  Everything is on track and the finishing touches are going on right now.  Last I heard we'll have close to 200 people registered!  If you haven't registered yet and would like to attend, please sign up here.  If you need more information, check out my previous post with all the details (almost all of the details are still accurate).

For those of you who will be in attendance, I'd like to thank a few people before the event so you have the opportunity to find them at the event and thank them yourselves for making this event possible for you.  Write these names down and hunt for them at the event.  Most of them should be there.  If you can't find them, then hunt me down and I'll point you in their direction.

Cisco and EMC get my first thanks for helping us secure the hardware necessary to host these labs.  Curtis Hayworth (EMC) and Dale Dewitt (Cisco) have been the frontmen to our requests, and our advocates, but I know they have several people behind them who have worked diligently to get us what we need in time to pull it all off.

A special thanks to Chris Simpson from VMware.  He has been instrumental in helping me align VMware resources.  He has also been willing to respond quickly to a lot of random questions I've had over the last few weeks.  Mike Bullerdick and Sheng Sheen have been my local support and I know they've promoted the living daylights out of this thing, which is no small part in its eventual success. 

Thank you to CoSentry for providing the facility, rackspace for the lab infrastructure, cool giveaways, and a large portion of the food that will be there.  Specifically, I'd like to thank Jason Phipps who has been my right-hand man through most of this.

Last, and most importantly, I'd like to thank Jodi Shely, Warren Dugas and David Olig who run the User Group together.  Thank you for taking up the huge mantle of running this group, thank you for asking Vital to help set this up, and thank you for all the support you've given me while building these labs.

I'd also like to thank Vital for handing me such an important project to our company and trusting me to pull it off.  I sure hope I've lived up to their expectations, because I've enjoyed it and would love to do it again in the future.

With the Thank You's out of the way, I'd like to apologize to my readers for my lack of posts on the VMUG Lab setup lately.  Building these labs has been a priority that has severely dug into my writing time.  I'm hoping to be able to keep a hold of the hardware for a few extra days so I can play a bit more (and of course blog about it).  Hang on, there'll definitely be more to come.

One of the labs we’re creating for the upcoming Omaha-Area VMUG is SRM.  It is posing some interesting challenges.  The biggest challenge is figuring out how to best provide a lab to 15 pairs of people without creating 30 sites.  We will cut some corners on that issue by having the instructor perform some of the core tasks like pairing sites and configuring the arrays.  This leaves users to work directly with the Protection Groups and Recovery Plans, which is where people need to pay the most attention anyways.

The other interesting challenge for us is one you find each time you setup SRM for a different array than you’ve done in the past.  In this case it was setting it up to use a CX4 using MirrorView. 

For those who aren’t familiar with why this is a big deal, let me explain.  SRM ties directly into the storage array in order to affect the status of the replication and to create LUN level snapshots during failover tests.  In order to do this, SRM uses a Site Recovery Adapter (SRA) to talk to the array.  The twist is that the storage vendor is responsible for creating the SRA for each array that SRM supports.  The difficulty lies in the different implementations and prerequisites each vendor has for their SRAs.

Here are a few things I discovered about using the EMC MirrorView SRA:

• Not only do you need to install the SRA on the SRM server, but you also need to install EMC’s Solutions Enabler.  As I understand it, Solutions Enabler is the foundation for software that needs to talk to the CX4 storage processors (somewhat analogous to the .NET framework).  EMC folks correct me where I might be wrong.
• After creating the remote mirror of the LUN, you need to create a snapshot of the destination LUN (at the recovery site).  The name you give this snapshot must include the substring “VMWARE_SRM_SNAP”.  Make sure reserve LUNs are created for this snapshot, which should be done when using the “Configure SnapView Snapshots” wizard.  The SRA will NOT create the snapshot on the fly for you like some SRAs will.
• A Consistency Group is not necessary unless your protected VMs cross multiple LUNs or you want multiple LUNs in a single Protection Group

 

Useful links:

EMC_MirrorView_Adapter_for_VMware_SRM_Release_Notes_v1.4

After about one week of playing with the lab gear EMC has loaned us to use for the upcoming VMUG, I’m ready to share my first impressions.

First off, let me say that these drives deliver on the promise the SSD drives offer.  They are FAST.

Generally, I’m seeing an XP template deploy in ~2 min (template is a thin-provisioned 10GB hard drive that is ~2GB on disk).  Windows 2003 templates (10GB thin provisioned HDD, ~4GB on disk) are deploying anywhere between 2 and 20 min depending on how many I’m deploying at once.  The average is around 8 min.

Boot times are even more impressive.  Windows 2003 VMs are powering on in 20-50 seconds, while Windows XP VMs are coming up in about 15 seconds (or less).  This is the time between the completion of the vCenter “Power on Virtual Machine” task and the Ctl-Alt-Del prompt.  There have been several times when I don’t even see the BIOS screen or the Loading Windows screen because they boot faster than the VM console can refresh.  This obviously has huge implications for VDI or DR environments that require tens, hundreds or even thousands of machines to power up in a short timeframe.

We’ve done some rough IOMeter testing as well.  I’m not ready to talk specific numbers, but generally we’re seeing almost 4x the IOPS on SSD disks than on 15k disks in the same array.  Latency increases from sub-millisecond times on the EFDs to tens of milliseconds on the 15k drives.

Overall, very impressive.  Of course, the EMC engineer that was helping us setup the arrays thought we should’ve seen better results.  I’d say good enough for now.

I wanted to start my series of blog posts about the Omaha-Area VMUG lab infrastructure with the base of the infrastructure (or any virtual infrastructure): the storage.

We were lucky enough to work out a deal with EMC to let us borrow a couple of Clariion arrays.  Specifically they are CX4-120 arrays.  Not that impressive you say?  What if I told you they were configured with six 200GB Enterprise Flash Drives (EFD) (see the first photo)?  Now do I have your attention?

 

 

Anecdote: I had an EMC sales rep ask me why I don’t have the face plates on our devices.  I told him it was because I care more about geeking out about the devices than about marketing the devices.

To summarize, we have two CX4-120s with about 1TB of usable EFD storage.  Both arrays have EMC’s mirroring software, MirrorView enabled so we can replicate between the two arrays (for the SRM lab).

The array consists of three distinct parts: the standby power supplies (SPS), which act as power distribution, but also contain batteries to allow for a graceful shutdown of the array should a loss of power occur; the storage processor enclosure (SPE), which contains the storage processors (SPs) (some may call these the controllers) and I/O modules (including fiber and iSCSI connectivity); and the disk array enclosure (DAE), which contains up to 15 drives (the first five of which contain the operating system for the array).  More technical details can be found here.

I would love to post a comparison of the HP EVAs that my company is used to dealing with (see the next picture with an EVA below and above the CX4 in our rack), but I have to admit to not knowing enough about either array to do such a comparison justice.  Not to mention the fact that I’d be comparing an array with EFDs to one with mostly 10k disks.  So, I’ll just discuss my experiences with the CX4 for what they are, not as a comparison to the EVA (sorry EMC).

The racking of the equipment wasn’t terribly complicated and was easily figured out using the enclosed quick start guide, though there is a mess of cables required to get all the pieces properly connected (see the next photo).

 

 

At this stage, we have only racked, cabled and powered on these two arrays, so I can’t judge performance yet.  That’s it for now, but keep a look out for a future post where I will hopefully be gushing on how awesome performance is on this array.