I received the following message from a coworker of mine today (Todd Dresser), and thought it was well written and could be of great use to others:

I had an issue where my web services would
keep crashing thus creating a ton of dump files (30,000 x 8 KB files)
and also a high CPU load.  I learned some good troubleshooting steps
and also things to avoid.  So the best log file to read from was the
catalina.yyyy-mm-dd.log in the directory C:\Program Files
(x86)\VMware\Infrastructure\tomcat\log.  I found I was receiving errors
like this

SEVERE: StandardServer.await: create[8005]:

java.net.BindException: Address already in use: JVM_Bind

So the vCenter web services uses tomcat apache
as the web server.  If there is another tomcat server running on the
same server you must make manual changes to a configuration file to
allow both of them to work.  During the vCenter install there are
options to change ports but that is not all of them, in the file

C:\Program Files (x86)\VMware\Infrastructure\tomcat\config\server.xml

There is this line

<Server port="8005" shutdown="SHUTDOWN">

I guess 8005 is default for apache, so if you
have two instances on a server one of those instances will fail to
start every time.  In the end I just had to adjust these ports during
the vCenter installation

8080 = 8081

8443 = 8444

And then post installation I edited the
server.xml and adjusted port 8005 to 8006.  After that my web server
was functioning correctly.  Without the web services you will lose your
storage view and hardware tab reports and cannot run VMware vCenter
SDK.  By the way the software that was conflicting with vCenter was
Symantec Endpoint Protection Manager Console.  Hopefully this can help
someone figure out a web services problem faster than it took me :D

Looks like this is a documented on VMware's site in KB 1009168.  

Thanks for the great information Todd!

The last day of boot camp went at a nice (relatively) slow pace, but not any less mind-bending.  We started with an open-ended lab where our goal was to get ESX up and running on our blades.  The process was pretty easy (as it should have been at this point).  The interface doesn’t make it entirely intuitive, but isn’t difficult once you learn it.

Today, Cisco officially announced the B250 full-width blade, Palo adapter and rack-mounted servers.  Rodos has summarized the news release well on his site: http://rodos.haywood.org/2009/10/ucs-palo-and-c-series.html.  Several points that stood out to me:

• The production name of the Palo adapter has been named the Virtual Interface Card, or VIC, which is a rather poor choice given the emphasis Cisco has given to the virtualization capabilities of the system.  I am of course referring to the acronym they share with the Virtual Infrastructure Client.  I suspect that it may continue to be referred to as Palo.
• The Palo adapter will be made for both blade (mezzanine) and rack mount server (PCI-Express x16).  I wonder if it would work or be supported in non-Cisco servers.  Could open up a whole new arena for Cisco to dive into.
• The C Series of rack-mount servers will not initially be able to tie into the UCS 6100 Fabric Interconnects.  This one surprised me, but confirmed a suspicion I had that they would not require the Fabric Interconnects (like UCS blade chassis do).

We then had a web meeting with Netformx to cover the DesignXpert product that Cisco has decided to use as their exclusive UCS configuration tool.  It is a nice tool with no major surprises, other than the fact it indicates to me that I’ll have to do more sales-y functions.  That I don’t look forward to, but I guess that’s life.

After that, we did a quick whiteboarding session to explain Fibre Channel and the differences and similarities to FCoE.  We didn’t cover anything that hadn’t already been covered at some point in the week, but helped to clarify and cement the concepts in all our minds.  The main diagram we centered the discussion around can be found in my first day’s report.

A few miscellaneous notes:

• Only QoS can be defined on the Palo vNICs, but you can define minimum guaranteed bandwith.  This is a difference with HP Virtual Connect Flex-10 that has you configure the specific bandwidth for each vNIC .  This is definitely to Cisco’s advantage (imagine that coming from a networking company) as it does a better job of not wasting bandwidth and guaranteeing services.
• Multi-hop FCoE in the Nexus line is currently scheduled to arrive about the same time as northbound FCOE in the UCS 6100 Fabric Interconnects.  This will solve the two roadblocks to end-to-end FCoE for the UCS system.
• NPV is F-port virtualization on the switches, which currently can participate in only a single VSAN.  NPIV is N-port virtualization on the HBA/CNA.  The two together allow multiple WWNs talk through a single FC connection, which was traditionally limited to a single WWN.

That finished up our week at UCS Boot Camp, a very aptly named training course.  It was very intense, but extremely educational, entertaining and insightful.  A big thanks go to our instructors and the developers, product managers, engineers and technical marketers who stopped by.  They were all very gracious to take the time to explain to us the components, philosophies and deep technical details, and put up with our many questions, some of which may have seemed heated at times.  They were always willing to keep at a topic until we understood it, and would pull in another resource to help deliver the message.  I’d also like to thank my fellow classmates for their open exchanges and willingness to share their experiences and thoughts.  It was one of those precious moments that bring several very intelligent people from different backgrounds and expertise together into a single room to hash out a new and technically innovative platform.  I believe that good things will come from this week to all of us who were involved.

Please provide comments below to let me know your thoughts, questions or concerns.

Our first lesson for today was on multitenancy, or setting up the UCS for multiple administrative users.  The first step to delegating authority and setting up Role Based Access Controls (RBAC) is to create an organizational chart.  This organization doesn’t need to match any company hierarchies, but should make sense on how you will delegate authority (not unlike approaching an Active Directory design).  Locales are created on a global level to group multiple organizational nodes together (no default locales exist).  Users are also created at a global level, and can be either local or from an external directory (i.e. LDAP, RADIUS).  The users are then given multiple roles (several built-in roles are available) and locales.  The roles assigned to a user are universal across all the organizational nodes in the locales they are assigned.  This means a user can’t have the server-equipment role in one locale and the server-security in another.

When creating and tying together all these pieces, keep in mind these design factors:

1. Policies can be used in a Service Profile from a higher node (toward the root), but if the same names are used at multiple levels, only the closest of these policies will be available
2. When pools (server or identity) are exhausted at a leaf, then a service profile will attempt to use resources from a pool with the same name in the next node up (all the way to the root)
3. While Server Pools can be created at any level of the organization, servers themselves are not affected by or exist within the organization
4. When adding a node into a locale, all dependent nodes will be included in the locale
5. There is no blocking of inheritance for any elements (including organizational nodes within a locale)

These elements, particularly the last two, sparked of a series of debates that lasted the entire morning.  It got so deep that we had one of the developers of the RBAC and one of the product managers in the front of the room using the whiteboard to explain how it works and answering some pretty difficult questions.  We also discussed a few elements in the UCSM GUI that did not seem to fully match the functionality that the logic behind it was doing.  I really enjoyed the deep two-way discussion with Cisco engineers & programmers it allowed us.  This kind of interaction makes having to travel all the way to San Jose to do the training in Cisco’s office (where most of the UCS team sits) completely worthwhile.

After lunch we dug into maintaining the UCS, starting with performing backups and restores.  Backups can be one of the following:

• Config-system – All information pertaining to authentication, authorization and accounting (AAA)
• Config-logical – All configuration details not associated with AAA
• Config-all – everything in the config-system and config-logical
• Full-state - The complete DR backup

Currently backups cannot be scheduled through the GUI, but could be setup as a cron job through the CLI.

The final section we covered today was upgrading UCS firmware.  Firmware updates are available as individual downloads, but the impression I had was that Cisco would prefer that the entire bundle, which contains firmware for all components, be downloaded for consistency of versions across the domain.  The recommended update order is the following (this may be an updated list for previous UCS Boot Camp attendees):  Mezzanine cards, BMC, BIOS, CMC, Fabric Interconnects., LSI/SCSI controllers.  Just like in the HP c-Class (maybe even more so), all the components work together to implement a solution, so the firmware versions need to stay in sync to some extent.

Some of the updates could be disruptive to the system, so release notes should be used to determine which ones could cause outages.  If a Fabric Interconnect or IOM outage would be necessary, we were told that the hardware failover within the Palo & Menlo cards will take about 5 milliseconds. 

All firmware versions of all components loaded into UCSM will be maintained in a library, but the individual components will store up to two versions each: a backup version (previously running version) and a startup version (the version to be loaded on next boot).  There will also be a version listed for the current running state, which could be different then these two.

The firmware piece felt pretty well screwed together, without any major gotchas or concerns.  However, it can be a bit intimidating given the number of components that have updatable firmware, especially when you can consider how many enclosures the domain can scale to.

Overall, I find the UCSM GUI to be not as well refined as the hardware itself.  There seemed to be several vestigial GUI elements and other elements that didn’t fully reflect the functions they are providing behind the GUI.  You are also unable to modify many elements once they have been created.  Some refinement is needed, no doubt, but there is a lot of power in there too.

Miscellaneous things learned today:

• Do not use the “Recover Corrupt BIOS Firmware” functionality.  Apparently it has caused major problems in the past.  Instructors were not quite sure what it was intended to do in the first place.
• When using the CLI, you must always commit a change after executing the command.  Another apparent hold over from switch CLI.
• The erase command on the CLI only wipes out the local Fabric Interconnect, so you’d have to run it on both devices to completely wipe out the entire UCS domain.
• UCS blades do not have a current option for embedded ESXi (as far as I have been told).
• I got confirmation that there is a promotion that when you buy VMware licenses and an UCS you will receive the Nexus 1000v for free.  No one knew any details on the promotion, though.

A few useful links encountered today:

http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/GUI_Config_Guide_chapter9.html

https://supportforums.cisco.com/docs/DOC-6143

One clarification from yesterday’s post, based on a question by Rodos: UCSM does contain some logic and code from BMC, but the GUI and the rest of the code (probably a majority) was created by Cisco.  There is a separate SKU that is a full OEM version of BladeLogic to provide the “Manager of managers” functionality.

Sorry about the fake post last night.  Several classmates were introduced to my blog today and aApparently they were so impressed with my posts from the first two days that they are now relying on my posts to be their notes.  So I had to pull a practicle joke on them.   Today was also big day for Twitter. I helped to hook three of my classmates into the community and discovered that Brad Hedlund covers my territory for Cisco.  This is why I put up a fake post last night, as a joke to them.  Hopefully I didn’t put too many of you off with it.

Today we got to dig into the UCSM interface some more.  One thing we discussed pretty heavily is that the Service Profile assignment is confusing when slot and server are used interchangeably.  Questions like “Do you apply a Service Profile to a server or a slot” came up (answer: slot).  But then, when stepping through the creation process in UCSM, we came across this:

Figure 1: Associating a Service Profile

Note that your two options are to associate it to a slot or a server.  Confused?  We were too.  The end result: you apply a service profile to a slot, not a server.  If a server is removed from one slot and inserted into another slot, the profile does NOT move with it.  Our advice to Cisco was to clean up the interface by eliminating the use of the word “server” and ensure consistency with the different parts of the interface. 

Why eliminate the word “server?”  Well, what is a server?  That can be a very subjective term.  In UCS terms, I view the server as the convergence of the blade hardware, the Service Profile and the Operating System.  Before virtualization and UCS (or HP Virtual Connect) this debate was a moot point because the three parts were inseparable.  Nowadays we may need to reconsider how we use some of these terms.

The process of associating a Service Profile to a blade is a two step process.

1. Assign Service Profile to a slot.
2. When blade is inserted, or if already present immediately after step one, it is powered on and booted by a UCS internal PXE server to a PnuOS.  This utility OS provides UCS access to the system for writing BIOS values (UUID, MAC, WWPN, etc.).  The blade is then rebooted to the Service Profile defined boot device.

When disassociating a Service Profile from a slot that has a running blade in it, you are given an option to gracefully shutdown the OS.  This can be useful if you use the “Reassign Service Profile” option to automate the shutdown>disassociate>associate (to different slot)>power on process.

We got our first exposure to the KVM functionality today.  Overall it is similar to the traditional (Java-based) Remote Console on HP’s iLO.  You have the same cursor speed mismatch and a lack of power options and remote media built into the KVM window.  However, you do get a menu option to open the remote media window in the KVM window.  One thing I noticed is that it will allow at least two connections to the same KVM, which is very useful for collaborating.  Overall, the UCS KVM is a bit lacking compared to iLO’s new Integrated Remote Console, but most of the shortcomings could be considered bells and whistles.

Another quirk of the UCS GUI we noticed is that in the Equipment tab of the Navigation pane, the icon of each blade does not change based on the status of the hardware.  A simple change of color to indicate power state would save a ton of clicks.  The number of clicks can quickly become a problem given the huge amount of data provided by UCSM.

One side question that I received an answer to today was that the UCS chassis has no cooling zones like the HP c-Class.  Apparently Cisco felt that no zones is more efficient, but is still studying the merits of both approaches to determine which is truly more efficient.

When creating a Service Profile to enable stateless computing you must specify MAC addresses and WWN in order to achieve the aforementioned separation of hardware, physical identity and OS.  When defining the MAC address, you can choose a pool of MAC addresses or define a specific one.  In either case the default Vendor ID is owned by Cisco so it should only show up within UCS systems.  Of course if you have multiple UCS domains, you will have to manage the uniqueness between the multiple UCS Managers.  When creating a pool of MACs or WWNs, there are no pre-defined ranges like Virtual Connect provides.  However it is very easy to define your own pool.  Simply enter a starting MAC, define the number of address you want in the pool and it creates a pool of sequential MACs.

There are actually two different types of pools that can be created: Server pools, which contain blades, and identity pools, which are actually several different pools for MAC, WWN, UUID, etc.  In a stateless computing paradigm, the Service Profile becomes the glue to hook all these different pools together.  It is possible to attach more Service Profiles to a server pool then there are physical blades in the pool.  You’ll simply end up with Service Profiles that are unassigned.

The server blade pool can be setup manually or in an automated fashion by defining the characteristics (CPU type, # of CPU, amount of RAM, slot #) of the blades that should belong to the pool.  As blades are discovered by UCSM, they are added to all automatic pools for which they meet the criteria.  This means that a blade can be part of multiple pools at once, but it still can only have one Service Profile associated with it at a time.

This Server Farm approach will have a rather dramatic effect on the approach to data center management.  In many organizations the hardware is owned by the business unit whose budget was used to purchase the hardware.  This has led to an “I own that server, you can’t use it for anything else” mentality to datacenter assets.  Virtualization has started to break down this approach due to the shared nature of the resources.  UCS will further erode this philosophy due to the dynamic nature of the use-what-is-need-when-needed approach that a Server Farm enables.  We had a good discussion around this topic and specifically what it will mean to our customers and how they can best adjust to it (hint: VMware is already providing some of the necessary tools).

Over lunch, Chris Haynes (Cisco) stopped in to help us understand how Cisco architected the memory in the B250 to allow up to 384GB.  There was a lot of information to absorb, so I think I may leave this detail to a later post.  Suffice it to say, I was very impressed and now feel that Cisco can make it as a viable server company.

Another question that was clarified today was the effect of UCSM and CMC failovers.  Each UCSM (remember one UCSM per Fabric Interconnect, configured in an Active/Passive cluster) has a SAM controller.  Each SAM controller has a unique ID and communicates with it’s peer to determine who is the Active node.  The UCSM Active/Passive roles are completely independent of the Active/Passive roles of the CMCs on the IOM.  In other words, you can have the Active UCSM on Fabric Interconnect A, but all the Active CMCs on the IOMs in Fabric B.  The cross channel communication between the Fabric Interconnects allows the communication to flow properly.  I’m not sure what would happen should this cross communication channel between Fabric Interconnects fail, but it’s worth investigating.

The UCSM is an OEM version of BMC BladeLogic, but this version of the BMC product is only available within UCS.  James Hollinger, a BladeLogic for UCS specialist at BMC, showed us some of the capabilities of a full implementation of BladeLogic: 

• Manager of mangers functionality across multiple UCS domains
• Create profile templates within BladeLogic in order to create Service Profiles across multiple UCS domains
• Very granular Role Based Security (i.e. user has permissions to stop one service, but no other permissions to the server itself)
• Additional workflow, including OS and application tasks

We finished the day performing the stateless computing lab where we created Service Profiles with defined WWNs for connectivity to boot volumes on the SAN.  The lab didn’t initially go smoothly since our boot from SAN Operating Systems consistently crashed during boot up for all lab teams.  After pulling in additional Cisco resources, it was discovered that you cannot migrate a Service Profile that boots from SAN and was built on a Menlo card to a system with a Palo card (the same applies in reverse).  This is due to incompatible drivers between the two devices, so the OS starts to load, but cannot access it’s own boot disk since it doesn’t have the proper drivers.

After class, I made an excursion up to VMware’s headquarters where I received a quick tour of the campus by John Troyer and had a nice dinner with John and Mike Coleman.  Thanks guys for a very enjoyable evening.

Today we started with a brief discussion around Cisco’s C-Series rack mount servers.  They are based on the technology used in the B-Series blades, including the Fabric Interconnects and presumably (this is an educated guess on my part) the entire Service Profile model.  They will have an optional CAN called Monterey Park which will provide the same functionality as the Palo adapter in the blades (multiple virtual adapters from a single physical port).  There will be three initial rack mount offerings: Los Angles (3U server equivalent to the B250 full-width blade, including the Catalina chipset) and the San Diego 1U and 2U servers (based on the B200 half-width blades).  Squeezing as much life out of Catalina seems to be the big driver for introducing rack mount servers today, but they do fill the small gap that blade servers don’t cover (small business with <4 servers).

We also discussed the five key value propositions for the UCS:

1. Virtualization – Server hardware enhancements to extend the virtualization capability of a server using technologies like Palo and Catalina
2. Scalability – Within a single server (memory and NICs) or within the management software (up to 40 chassis & 320 blades through one pane of glass)
3. Management – Single point of management for all aspects (server, network, storage, chassis) and multiple modes of access to the management  (GUI, console, API)
4. Statelessness – Complete decoupling of the identity of the server from the hardware (Service Profiles)
5. Converged Fabric – I think we all know what this one means by now: FCoE running at 10+Gbps

Another limitation that was brought up with the 6100 Fabric Interconnects (FI) is the fact that today there are no 1GbE uplinks to existing Ethernet networks, which requires a company to upgrade or replace switches as part of a UCS implementation.  This is not necessarily a cheap thing for a customer to do.  It seems like this may be a software limitation and may be relatively easy to correct.

After all the product talk, we finally got to our first lab where we got to play in the GUI and console interfaces for the UCS Manager (UCSM).  Cisco took an interesting approach by having the UCSM GUI be deployed as a downloadable Java application.  Though it is a very powerful tool, I have to admit I like the HP C-class’s web-only interface better.  

I didn’t take any screenshots (I’ll remember to do so tomorrow), so there’s not a lot to share regarding the interface.  One common theme that came up several times throughout the day was the fact that the tab used to contain all the Service Profiles is called “Server” which is very confusing.  The word server indicates a physical asset (body), not the Service Profile (soul).  Given the focus on the Service Profile within UCS, I’d have thought this would’ve been an easy design element. 

Another thing I noticed is that many settings within UCSM are saved, but then must be committed as a separate step.  This process eliminates the potential for error, and I believe this has worked well for many years in the networking realm.  However, it may come as a bit tedious to many server admins.  It could also be a point of error since each chassis needs to be committed separately and this could be easily overlooked.

I recommend checking out Rodos’s and Rich Brambley’s posts for some good posts that include screenshots of the GUI interface.   

One of the more powerful features of the UCSM is the complexity masking that occurs with the GUI.  For example, when you add a VLAN within UCSM GUI, it is a simple one screen operation.  Behind the GUI, however, a couple of tasks are completed.  The VLAN is added to both FI and is applied to all uplinks when running in automatic End Host Mode.  Similar functionality exists when creating Service Profiles.

After the lab, we hit the whiteboard again to discuss FCoE in more detail.  If you don’t currently have a good grasp of how Ethernet and Fiber Channel work to begin with, I’d suggest doing some studying.  If you want to know how FCoE really works, you’re going to need to know how those two protocols work.  The good news is you may not need to know it all that deeply.  The UCSM GUI seems to do a good job of hiding a lot of the intricacies of the protocol and it’s implementation between the different components of UCS.  This knowledge will still be crucial for anyone architecting a complete FCoE solution.

I’ll try and tackle some of the basics.  FCoE is made up of two parts: Data, which is SCSI commands inside of Fibre Channel packets and Control information using the FCoE Intialization Protocol (FIP).  FIP is used to assign and maintain FCoE MAC addresses.  There is a process for setting this MAC address that is very similar to the process that DHCP uses.  In fact, they’re similar enough that I won’t take it any further than that.

One thing I came away with is the accuracy of my drawing from yesterday’s post (presented again here for discussion):

Figure 1: FCoE Switching (again)

I have not modified this drawing, since it is still theoretically possible (and in my opinion very probable in the near term).  What isn’t 100% correct for today is the hop between the vE-Ports.  This is considered a multi-hop FCoE communication, and is currently not a capability.  This is where things start getting into a nuts and bolts discussion.  Take what I’ve posted yesterday and today and then go read Scott Lowe’s post on the same topic.  That should get you a pretty good feel for the situation.  If you survive all that, then you may be brave enough to read through http://fcoe.com/. 

You could make this picture work by simply sending the communication through native Fibre Channel between the two Nexus 5000 switches instead of via FCoE.  This is something that will no doubt be fixed at some point in the future.

VNTag is another important piece of the UCS puzzle.  I don’t fully understand how it works, so I won’t dig too deep into it here for fear of providing incorrect information.  What I do understand it to be is a wrapper that exists only in the UCS that is used to route packets through the different components (FI, IOM, Adapters and optionally the hypervisor).

Another related technology is VNLink, which I again don’t fully understand.  One function VMLink does provide is the ability to shutdown downstream ports if there is a connection loss.  HP provides similar functionality with SmartLink.

VNTag and VNLink are two technologies that do not need to be fully understood in order to manage a UCS.  Once again, the UCSM GUI does a great job of hiding this complexity behind a relatively simplistic interface.

The 6100 FI can operate in two different modes, which can be chosen during configuration of the devices:

• Ethernet Switching + NPV – This mode offers full Ethernet switching, but all fibre storage connections are pinned directly to NPV ports.  No fibre switching.  This is part of the reason why FCoE outside of the UCS is not possible.
• End Host Mode – This mode pins vEth/vHBA ports to specific uplinks.  The pinning can be done in static or dynamic mode.  Static mode is mostly self explanatory.  Dynamic mode requires that all uplinks be trunks that allow the same set of VLANs/VSANs.  Traffic is then routed out in a round robin style.

End Host Mode is the default and least complicated approach, but Ethernet Switching + NPV allows for more control over your uplink usage.  In the future a third mode will be possible: Ethernet Switching + FC Switching, which should enable FCoE to be routed outside of the UCS.

One odd setting we found today was a global policy that defined the number of server connections out of the FI: 1, 2 or 4.  It can only be set at the top of the Equipment tree and cannot be defined at a lower level.  We debated the merits of such a policy to no definitive resolution.  Later in the day one of our instructors came back with the answer.  This policy defines when an automatic discovery should occur: when 1^st, 2^nd or 4^th cable is connected between the FI and IOM.  I guess a quick response from knowledgeable people is one of the many advantages to having the boot camp in one of Cisco’s main buildings.

The final section for the day introduced us to the Service Profile.  The configuration settings in the Service Profile are what allow the mobility of a server from one blade to another.  For example, if you’re booting from a SAN LUN, then that LUN is probably attached to a WWN on an HBA.  The Service Profile will mask the factory WWN and use the WWN it has defined.  Therefore, whatever blade the Service Profile is attached to will assume this WWN and will boot off the related SAN LUN.  Given the nature of these types of settings, a Service Profile can only be attached to a single blade at a time. 

We had an interesting debate as to whether the Service Profile represents the Operating System or the hardware.  Here are the two strongest counter arguments:

• You have 320 blades and 500 OS instances defined on the SAN.  In order to define these OS instances, you’d need 500 Service profiles.  This would indicate that the Service Profile is a representative of the OS.
• The Service Profile defines hardware configuration details, including the BIOS settings and firmware versions; therefore the Service Profile is representative of a hardware configuration.

I found myself somewhere in the middle.  To me, the Service Profile exists to not represent either the OS or the HW, but to join the two together.   In a UCS system the blades and the OS (and the network connectivity) are each useless without a Service Profile to tie it all together.  Ultimately, I think it’s greater than any of these views and contains elements of each.

We didn’t have time to finish the Service Profile section, so you’ll have to wait until tomorrow for the rest (just like me!).

Toward the end of the day, I received a Direct Message on Twitter from Rodos asking me to help him validate the existence of an error he discovered in the UCS Boot Camp courseware.  If you’ve been to UCS Boot Camp, you should check out the details here: http://rodos.haywood.org/2009/10/error-on-cisco-ucs-pinning-training.html  Great catch Rodos, glad I could help out.

I have been given the distinct privilege of being able to attend an exclusive partners-only boot camp training put on by Cisco for their UCS system (Cisco’s blade implementation).  Like Scott Lowe, Rich Brambley and Rodney Haywood, I will be blogging my thoughts and technical details (along with the occasional tweet).  I may skip past many technical details simply due to the fact they’ve been covered by one of the above blogs, or because I simply have to leave something for my customers to pay for.

The class started off pretty slow with general housekeeping and an overview of the pains of current blade infrastructures and a high level explanation of FCoE.  Then the fun began.

The UCS chassis includes eight half-width blade slots, four power supplies, eight fan modules and two Fabric Extenders (also referred to as I/O Modules).  Two power supplies can power the entire chassis with the third and fourth providing redundancy.  (Similarly, the HP c-Class enclosures can be fully powered by three power supplies with the fourth, fifth and sixth providing redundancy.)  There is no power domain concept in the UCS chassis as there is in the IBM and the HP p-Class enclosures.

External to the chassis, but still required, are a pair of Fabric Interconnects (FI), each of which can support up to 20 or 40 UCS chassis, depending on the FI model.  The UCS Manager (UCSM) runs in an Active/Passive mode on these FI.  The UCSM is built on a XML database that stores all configuration details and settings for the entire solution, including the Service Profiles.  This database can be accessed through many different ways, including the client Java GUI, CLI, SNMP, and XML APIs.

The UCSM works through the Chassis Management Channel (CMC) on the I/O Modules (IOM) within each chassis.  There are two IOMs in each chassis and they run Active/Active for the data communication and Active/Passive for the CMC.  The CMC is really nothing more than a proxy for talking to the individual chassis, blade and switch components.  Also on the IOMs is the Chassis Management Switch (CMS), which provides communication to the Baseboard Management Controller (BMC) on the individual blades (think iLO).  These are 100MB connections that flow through a dedicated port on the chassis.

Currently there is only one blade available, the half-width B200, which contains two Xeon 5500 (Nehalem) processors, 12 DIMM slots (up to 96GB), 2 SAS/SATA HDD and a single mezzanine socket.  The DIMMs are limited to only Registered 4GB and 8GB 1333Mhz DIMMs (yes, only two types of DIMMs are supported based on what we were told).  The BIOS is currently limiting the bus speed to 1066Mhz, no matter how many DIMM slots are populated.  It is possible for certain processors to push the bus speed down to 800Mhz.

The next blade to be released will be the full-height B250, which will contain two Xeon 5500 (Nehalem) processors, 48 DIMM slots (up to 384GB), 2 SAS/SATA HDD and 2 mezzanine sockets.  The extra DIMM slots are made possible by the Cisco-exclusive Catalina chip.  How this works was beyond the scope of our class, but I am supposed to be getting a whitepaper that describes the nuts and bolts, but essentially it’s able to put four times the DIMMs in each channel (8 DIMMs x 3 Channels x 2 Processors) without affecting bus speed and only incurring minimal additional latency (6 ns).  The same limits to DIMM types and speeds apply to this blade as in the B200.

All the pieces are well laid out in Rodos’ post here: http://rodos.haywood.org/2009/08/ucs-schematic-sketch.html

One thing I learned today is that FCoE is not simply a Fibre Channel packet wearing an Ethernet Halloween costume.  FCoE requires a special handling and flow control.  FC works by not allowing packets to be dropped, and FCoE must still abide by this rule.  The Nexus switches do this by actually embedding MDS functionality directly into switch.

Figure 1: FCoE Switching

In Figure 1 I have depicted a server with a CNA adapter connecting through two Nexus 5000 switches to a native FCoE storage array.  The blue lines depict the FCoE traffic, which by the nature of the NX-OS will be lossless, and the orange lines depict native FC traffic.  Note the use of a MAC address as the Source and Destination for the FCoE packets and the fact that they are unwrapped upon arrival at their destination.

Given the losses requirement (among others, as our instructor was quick to point out), FCoE packets cannot be routed through a Catalyst switch.  In other words FCoE should only flow through FCoE-aware switches (read Nexus) because they are not normal Ethernet packets.

One point that has been pretty well documented, and I have confirmed this to still be true (for a little while longer at least) is the fact that FCoE cannot be sent upstream (toward the SAN appliance) from the Fabric Interconnects (FI).  Note that Figure 1 describes a non-UCS server.  There currently is no way to do native FCoE from a UCS blade completely to the Storage Array.  See Scott Lowe’s description for a good summary.  Essentially, it comes down to this: the FI does not contain the embedded MDS functionality to forward on the FC packets.  All the FI can do today is strip off the FCoE wrapper and send the native FC out an NPIV enabled port.  This will change in the future, but today is a limitation.

Service Profiles are a big differentiator for the UCS system.  Those familiar with VitualConnect may realize that Service Profiles are very similar to VirtualConnect Server Profiles, but UCS does add some unique capabilities, such as the ability to define the Firmware and Quality of Service properties.  Another unique feature is the ability to wipe the local drives when applying a Service Profile to a blade.

When it comes to configuring interconnects and adapters, Cisco seems to have moved most functionality and choice into the adapters instead of the interconnects.  Ultimately, this seems like a simpler solution, since now you just pick which adapter you want and the I/O Module (IOM) is the same regardless of your choice.  There are (or will soon be) three choices for mezzanine adapters: Palo (CNA built for virtualization), Menlo (CNA with two 10GbE and two FC) and Oplin (standard dual port 10GbE with support for OS-implemented FCoE).

Cisco’s upcoming Palo adapter mezzanine card provides similar functionality to HP’s VirtualConnect Flex-10.  I was in awe when I first realized what Flex-10 could do.  Using the Palo adapter within UCS really blew me away.  This is where Cisco’s networking expertise really shows through.  Here’s a quick comparison:

Similarities

• Split a single 10GbE connection into multiple instances that the OS sees as individual devices
• Ability to define bandwidth of each device

• Splitting of connection occurs completely within the Palo adapter, rather than a combination of the adapter and interconnect
• Palo can create 128 connections, as opposed to Flex-10’s four
• Palo can define many more characteristics for each logical connection
• Palo has built-in hardware failover between the two uplinks, eliminating the need to implement failover within the OS/software layer (mezzanine card is still single point of failure)
• Palo is a CNA, meaning those 128 connections can be any combination of vNICs and vHBAs
• Palo can enable direct 1:1 mapping of VM vNICs to Palo vNICs using VN-Link
• The Palo adapter actually runs a Linux OS and an unmanaged switch in order to manage all this magic

As we dug in deeper into the actual data paths when using Palo, FCoE, 6100 Fabric Interconnects (FI) , 2104 Fabric Extenders (FEX) and Nexus switches (primarily 1000v and 5000), I began to wonder: Did Cisco create a complicated UCS (w/ FCoE and Palo adapter) to sell more Nexus 1000v?  It essentially comes down to this: Ethernet best practice is to not route a packet back down the same port it came in on.  In the case of an ESX host, this could be a possible scenario.  In order to avoid this, VN-Link creates virtual Ethernet ports on the FI in order to treat them as two separate ports, thereby allowing routing between them.  At the end of the long, hard to grasp discussion it was stated that the Nexus 1000v would avoid all of this by simply routing the traffic within the host and avoiding the FI completely.  Good selling point for the Nexus 1000v.

Two great pictures we actually used in our class for understanding how traffic flows out of the UCS using Palo can be found here: http://www.internetworkexpert.org/2009/08/11/cisco-ucs-nexus-1000v-design-palo-virtual-adapter/ and here: http://www.internetworkexpert.org/2009/07/05/cisco-ucs-vmware-vswitch-design-cisco-10ge-virtual-adapter/.  Both are by Brad Hedlund, who appears (based on my limited exposure to the Cisco Data Center world) to be an IT rockstar (perhaps the Duncan Epping of UCS?).

This leads me to a final general point about the UCS system.  Ultimately, there is a lot to love about the UCS system.  It was clearly designed with Network and Storage I/O in mind (as you would expect from Cisco), and with little innovation needed on Nehalem systems, this helps Cisco stand apart.  They have also made an effort to truly unify all the management interfaces, though based on the screenshots I’ve seen so far they’re not as nice as HP’s.  At the same time I worry that the UCS system is simply just too complicated to sell to the general customer.  As a HP reseller and implementer, I find the whole VirtualConnect and Flex-10 conversation can go over many technical people’s heads.  UCS is even harder to understand (note to self: practice UCS whiteboarding skills thoroughly). 

Some additional comparisons to HP blades:

• Cisco’s blade slot architecture seems similar to HP p-class (4 slots that can be divided w/half height blades) as opposed to c-class (16 half height slots that can be converted to full height).
• Cisco’s Baseboard Manager is equivalent to HP’s iLO

Miscellaneous final notes:

• UCS certifications will be available early next year for design and implementation
• Storage redundancy is not handled in the UCS hardware and should be implemented within the OS/Application layer
• If multiple uplinks are used to connect IOM and FI, they are completely separate connections and cannot be combined with a port group
• An IOM can only be connected to a single FI

I guess that’s it for today.  Not enough to digest?  Check back tomorrow and the rest of the week for more.  Don’t worry; I’m pretty sure this will be the longest post since the rest of the week will involve more labs and less architecture.

Please feel  free to leave comments to ask questions, make corrections or provide additional information.