Recently, our work has been able to get some Equallogic PS6100 series SANs from Dell. With the front cover off, it honestly reminds me of Optimus Prime for some reason as each drive caddy was made with chrome covered metal alloy. I was excited to get it set up with XenServer to see what all the magic was all about. To my disapointment, I had a bad experience with it.
To skip all my rants and get to the setup details, simply go to the end of this post.
My initial struggle...
After my colleague had gotten it set up initially, I started looking for documentation regarding how the networking should be set up. To my surprise, the included documentation failed to discuss this in detail. Looking for documentation on their website resulted in only the same documentation in PDF format. Based on the documentation saying to connect with your iSCSI initiator without any other details, I then assumed that we set up each of the four interfaces into different subnets, how tradditional iSCSI multipathing is done. XenServer did connect to the volume, but with only a single path.
Next step was to search online for all documentation regarding Equallogic and XenServer. This resulted in a documentation from Dell, for XenServer 5.0 Dell Edition. It referred to a graphical setup in XenCenter for Equallogic that is no longer available. What's also no longer available is XenServer Dell Edition, they stopped producing this after version 5.6, which wasn't even available as an installer. Everything had referred to a software level Equallogic Adapter of some sort. Searching for this brought me back to the Dell XenServer 5.0 documentation.
XenServer 6.0 documentation did not say how to set up Equallogic at all. Forums did not yeild anything useful, nor did any other Internet search.
I found a RedHat 5.0 guide for Equallogic. With it, I could set up multipathing in XenServer. However, XenServer overwrites configurations on startup so this went nowhere.
Since we paid for the best support there was, I decided to give them a call. In my mind, I thought "This is enterprise support, it's Equallogic, they know what they're doing."
To start, I thought it was weird that when I called the number for Equallogic Array support, it asked me what I'm calling about (servers, desktops, arrays). Once I finally spoke to someone, they did not know how the network should be set up. They also did not know anything about Linux or XenServer iSCSI connections either. What they also did not know was the term "bonding". So now I assumed the person I spoke to knew only Windows so I started using the terms "teaming" and "trunking" (trunking's actually misused in this situation) and immediately he knew what I was talking about...well not really. He still did not know how the networking should be set up on neither the array nor the client.
I asked for documentation, pointing out that the documentation form Equallogic was useless. He went silent for a while and then sent me an email with "documentation". Turns out, when he went silent for half an hour, he was searching online. All the "documentation" he found, I had already seen!
At this point I asked him why Equallogic does not have documentation on this particular implementation in which XenServer is advertised as supported and even is listed as one of the solutions on their website. His response:
"Sir, just because we support it, doesn't mean we have documentation."
I was shocked!!! But then again, it's Dell, I'm not that shocked. I was a little more shocked when he told me he's talking with two other support personnel and an engineer to find a solution. This is the best they can come up with???
After over two hours, he sent me another email, one of the links was to XenServer 5.6 documentation. He pointed out on page 41 there was some information. To my surprise, it was commandline instructions to connect to Equallogic array! I was hopeful. But I still required a lot more information regarding the networking portion. So a request for a level 2 support was put in and I had to wait for a call back.
Back to setting up the array without support...
Using the instructions in old XenServer documentation, I was close, but it was difficult still. Eventually some information showed that all Equallogic storage interfaces should be in the same subnet. Another important piece of information was to disable dedicated managment interface.
After this, what do you know...it worked! But...how do I make sure there's redundancy if everything's in the same subnet? There can only be one interfaced configured on the single subnet due to routing. Additional interfaces set up does not help as no traffic is directed through them. Once the interface that is currently communicating with the array has a problem, there is no routing rule to say "use another interface".
The best way I could think of was bonding...
Problem: XenServer doesn't support bonding of more than two ports.
Possible solution: Create two bonds in separate subnets, then set up two of the interfaces on the array into separate subnet. This could theoretically work but I had no time to test. So I only set up a single bond for the time being and everything worked as they should.
Level 2 support calls back...
My remaining question at this point was, how should the network be set up? And to my surprise, he didn't know either. He also didn't know the term "bonding" so I had to resort to Windows term "teaming" and the misused "trunking". His idea was that any iSCSI initiator can connect to the array and that was it. He did confirm that all the NICs on the array should be in one subnet however. After some exchange of conversation, he couldn't provide me with anymore help.
Experience with the array once it was working...
This is a very different concept compared to regular iSCSI. You don't set up a volume in the Equallogic management UI. XenServer will manage volumes and snapshots for you! Each VM has their own volume and snapshots that can be viewed inside Equallogic's management UI or console. Pretty cool stuff.
Detailed Setup Info
- Equallogic array should have no volumes created (unless you share it with another iSCSI initiator)
- Equallogic network interfaces should be in one subnet
- Dedicated management interface should be disabled
- XenServer client should use bonding for storage backend (this was later confirmed in the Dell XenServer 5.0 Equallogic doc, page 15)
- Use sr-create command in xe to create the SR (refer to XenServer 5.6 documenation, page 41 and 42)
My Additional Comments Regarding Setup
Array Mangement Interface: The Equallogic array adapter in XenServer can use a different IP address for managment interface. This means that the dedicated managment interface can be enabled. However, consider this: What if that interface is no long accessible? Since XenServer uses that interface to manage everything, XenServer loses the ability to control the array and thus you will lose control of your VMs.
Client Side Bonding: Supported bonding methods in XenServer doesn't seem to allow client side storage to scale in performance. It's possible that LACP bonding could allow more than two NICs as well as distribute traffic between the different IP addresses of the Equallogic array.
What I did like...
Once things are set up with XenServer, management is flawless. There should be no reason to go to the Equallogic management afterwards. It's one less thing to think about.
Array expansions should be without hassle in the future. Since XenServer creates individual volumes on the array, additional storage will show up automatically without jumping through hoops to get XenServer to recognize it. This cannot be done with just iSCSI array because you connect to the volume and create VMs in there. Unless you want to create one small volume for each VM manually that is, which I've found to slow down VM start/shutdown linearly as additional volumes are created (applies to versions 5.6FP1 and up).
My Comments to Dell
Get your act together! This array is not cheap by any means. I expect that proper testing and documentation is done for all supported solutions! I don't blame the support personnels in this case. They had as much information as I did so how can they help me properly?