Krystalmods.com

After hearing about the possibility of removing Apache completely, I took a look and found the following link:

http://www.howtoforge.com/installing-nginx-with-php5-and-php-fpm-and-mysql-support-on-opensuse-12.1

Following these instructions was fairly simple. I skipped mysql server setup in the instructions as I prefer to separate my database servers from the application layer. There was one issue though. The following warning started appearing on php pages. For b2evolution version 3.x, it showed up so many times that the page was impossible to read.

PHP Warning: phpinfo(): It is not safe to rely on the system's timezone settings. You are *required* to use the date.timezone setting or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'America/Newyork' for 'CEST/2.0/DST' instead in /usr/share/nginx/html/info.php on line 2

Now then, some of the guides for Centos mentioned the same issue. It suggested modifying php.ini. However, I couldn't really find one for php-fpm. The suggested change to php.ini does not work in php-fpm.conf file. To resolve this, add the following to /etc/php5/fpm/php-fpm.conf:

php_admin_value[date.timezone] = 'America/New_York'

This resolves the problem.

As for b2evolution's system check, a couple settings need to change to ensure security. Add the following to /etc/php5/fpm/php-fpm.conf:

php_admin_flag[magic_quotes_gpc] = off

We also need "allow_url_fopen", however, the override in php-fpm.conf doesn't work. To get around this, we have the option of adding the following to /etc/nginx/nginx.conf:

fastcgi_param       PHP_VALUE       "allow_url_fopen=off";

One final issue exist. B2evolution has been configured to use "extra-path" instead of "param" for URLs like this: http://www.krystalmods.com/index.php/2006/blah

This is due to nginx looking at it as a directory rather than php script now. It goes to find the file but cannot find it. To resolve this issue, we switch to "param" so it says: http://www.krystalmods.com/index.php?title=blah&more=1

Although this solves the issue, consider if your website has been live for a while with "extra-path" set up. By changing to "param", the URL now changes which means it will take some time for search engines to crawl through. This also renders referring links from other websites useless. I don't have time to look into a solution now. Hopefully I'll find a solution for this in the future.

Results

If you are looking at this page right now, then the results are right here. Memory usage is low to start, however, it increases as php-fpm get some requests. Its memory footprint increases for each child process, possibly for caching. Memory usage while serving requests is equivalent to apache sitting idle unless if you decide to have a lot of php-fpm children running. Nginx it self uses very little memory. The best part is the reduced CPU usage. If I find some time, and remember, I'll test on an isolated system for both scenarios to get some performance numbers for comparison.

Recently, our work has been able to get some Equallogic PS6100 series SANs from Dell. With the front cover off, it honestly reminds me of Optimus Prime for some reason as each drive caddy was made with chrome covered metal alloy. I was excited to get it set up with XenServer to see what all the magic was all about. To my disapointment, I had a bad experience with it.

To skip all my rants and get to the setup details, simply go to the end of this post.

My initial struggle...

After my colleague had gotten it set up initially, I started looking for documentation regarding how the networking should be set up. To my surprise, the included documentation failed to discuss this in detail. Looking for documentation on their website resulted in only the same documentation in PDF format. Based on the documentation saying to connect with your iSCSI initiator without any other details, I then assumed that we set up each of the four interfaces into different subnets, how tradditional iSCSI multipathing is done. XenServer did connect to the volume, but with only a single path.

Next step was to search online for all documentation regarding Equallogic and XenServer. This resulted in a documentation from Dell, for XenServer 5.0 Dell Edition. It referred to a graphical setup in XenCenter for Equallogic that is no longer available. What's also no longer available is XenServer Dell Edition, they stopped producing this after version 5.6, which wasn't even available as an installer. Everything had referred to a software level Equallogic Adapter of some sort. Searching for this brought me back to the Dell XenServer 5.0 documentation.

XenServer 6.0 documentation did not say how to set up Equallogic at all. Forums did not yeild anything useful, nor did any other Internet search.

I found a RedHat 5.0 guide for Equallogic. With it, I could set up multipathing in XenServer. However, XenServer overwrites configurations on startup so this went nowhere.

Calling support...

Since we paid for the best support there was, I decided to give them a call. In my mind, I thought "This is enterprise support, it's Equallogic, they know what they're doing."

To start, I thought it was weird that when I called the number for Equallogic Array support, it asked me what I'm calling about (servers, desktops, arrays). Once I finally spoke to someone, they did not know how the network should be set up. They also did not know anything about Linux or XenServer iSCSI connections either. What they also did not know was the term "bonding". So now I assumed the person I spoke to knew only Windows so I started using the terms "teaming" and "trunking" (trunking's actually misused in this situation) and immediately he knew what I was talking about...well not really. He still did not know how the networking should be set up on neither the array nor the client.

I asked for documentation, pointing out that the documentation form Equallogic was useless. He went silent for a while and then sent me an email with "documentation". Turns out, when he went silent for half an hour, he was searching online. All the "documentation" he found, I had already seen!

At this point I asked him why Equallogic does not have documentation on this particular implementation in which XenServer is advertised as supported and even is listed as one of the solutions on their website. His response:

"Sir, just because we support it, doesn't mean we have documentation."

I was shocked!!! But then again, it's Dell, I'm not that shocked. I was a little more shocked when he told me he's talking with two other support personnel and an engineer to find a solution. This is the best they can come up with???

After over two hours, he sent me another email, one of the links was to XenServer 5.6 documentation. He pointed out on page 41 there was some information. To my surprise, it was commandline instructions to connect to Equallogic array! I was hopeful. But I still required a lot more information regarding the networking portion. So a request for a level 2 support was put in and I had to wait for a call back.

Back to setting up the array without support...

Using the instructions in old XenServer documentation, I was close, but it was difficult still. Eventually some information showed that all Equallogic storage interfaces should be in the same subnet. Another important piece of information was to disable dedicated managment interface.

After this, what do you know...it worked! But...how do I make sure there's redundancy if everything's in the same subnet? There can only be one interfaced configured on the single subnet due to routing. Additional interfaces set up does not help as no traffic is directed through them. Once the interface that is currently communicating with the array has a problem, there is no routing rule to say "use another interface".

The best way I could think of was bonding...

Problem: XenServer doesn't support bonding of more than two ports.

Possible solution: Create two bonds in separate subnets, then set up two of the interfaces on the array into separate subnet. This could theoretically work but I had no time to test. So I only set up a single bond for the time being and everything worked as they should.

Level 2 support calls back...

My remaining question at this point was, how should the network be set up? And to my surprise, he didn't know either. He also didn't know the term "bonding" so I had to resort to Windows term "teaming" and the misused "trunking". His idea was that any iSCSI initiator can connect to the array and that was it. He did confirm that all the NICs on the array should be in one subnet however. After some exchange of conversation, he couldn't provide me with anymore help.

Experience with the array once it was working...

This is a very different concept compared to regular iSCSI. You don't set up a volume in the Equallogic management UI. XenServer will manage volumes and snapshots for you! Each VM has their own volume and snapshots that can be viewed inside Equallogic's management UI or console. Pretty cool stuff.

Detailed Setup Info

• Equallogic array should have no volumes created (unless you share it with another iSCSI initiator)
• Equallogic network interfaces should be in one subnet
• Dedicated management interface should be disabled
• XenServer client should use bonding for storage backend (this was later confirmed in the Dell XenServer 5.0 Equallogic doc, page 15)
• Use sr-create command in xe to create the SR (refer to XenServer 5.6 documenation, page 41 and 42)

My Additional Comments Regarding Setup

Array Mangement Interface: The Equallogic array adapter in XenServer can use a different IP address for managment interface. This means that the dedicated managment interface can be enabled. However, consider this: What if that interface is no long accessible? Since XenServer uses that interface to manage everything, XenServer loses the ability to control the array and thus you will lose control of your VMs.

Client Side Bonding: Supported bonding methods in XenServer doesn't seem to allow client side storage to scale in performance. It's possible that LACP bonding could allow more than two NICs as well as distribute traffic between the different IP addresses of the Equallogic array.

What I did like...

Once things are set up with XenServer, management is flawless. There should be no reason to go to the Equallogic management afterwards. It's one less thing to think about.

Array expansions should be without hassle in the future. Since XenServer creates individual volumes on the array, additional storage will show up automatically without jumping through hoops to get XenServer to recognize it. This cannot be done with just iSCSI array because you connect to the volume and create VMs in there. Unless you want to create one small volume for each VM manually that is, which I've found to slow down VM start/shutdown linearly as additional volumes are created (applies to versions 5.6FP1 and up).

My Comments to Dell

Get your act together! This array is not cheap by any means. I expect that proper testing and documentation is done for all supported solutions! I don't blame the support personnels in this case. They had as much information as I did so how can they help me properly?

I've been having issues with a motherboard. In my case, the Areca ARC-1230 and the low cost Highpoint 2720 controllers both have problems on the Gigabyte MA770T-UD3P motherboard. The Areca controller prevented booting from any disk while the Highpoint controller causes boot hang after Linux kernel load whenever an Array is configured. This is meant to be a testing Xenserver as well as for backup purposes.

I've resorted to using Linux RAID now. I was never happy with Linux RAID performance, especially since I'm running RAID5. The parity building is looking to be going at best 56MB/s. When creating an ext3 FS, this slowed down to below 1MB/s!!!

After searching for some tweaks, I found two sources worth mentioning:

First tweak: http://kerneltrap.org/mailarchive/linux-kernel/2007/1/11/44612

In the above, Justin had changed the stripe cache and readahead cache. This for me resulted in the following during rebuild with a modest 8MB stripe cache and 16MB readahead cache:

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde1[4] sdd1[2] sdc1[1] sdb1[0]
1465151808 blocks level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
[=======>.............]  recovery = 36.6% (178959240/488383936) finish=55.7min speed=92564K/sec

Disks uses were four Seagate 7200.4 SATA 500GB drives connected to the Highpoint controller as passthrough disks.

Second tweak:

This one comes from a number of sources and basically says to set "noatime" and "nodiratime". I have yet to test this one out, however, it makes sense. This should significantly improve small file access. It will not show up in large file access based benchmarks.

Windows always get all the tools to run benchmarks because majority of the population uses it. In linux, there are few testing methods when it comes to disk performance. I've used the following in the past:

• hdparm command
• dd command in a shell script to simulate different types of create, read, write, and delete operations
• bonnie++ benchmark

Problem is, none of them allow testing for IOPs directly which makes it difficult to judge if the results would be accurate by deriving IOPs from transfer rate at specific read sizes (for dd and bonnie++). IOPs is much more important in a workstation or server environment compared to sustained transfer rate of larger read sizes.

FIO was built to solve this issue. I've been moving around some of my servers so I don't have my configurations for FIO so I'll leave that till the next post. Here's a good post on FIO testing: http://www.evanhoffman.com/evan/2011/07/22/benchmarking-disk-io-on-ext3-vs-ext4-with-fio/

What is not addressed in that post is how to test higher queue depth. Queue depth is extremely important in workstation and server environment. While a personal computer may reach as high at QD10, a server may easily reach well over QD64. It is important to test controller's ability to handle this. Typically all controllers will show increase in IOPs with increased QD until the controller's own limits. A good controller will show linear increase in IOPs until QD64 or sometimes QD128. An inferior controller will show its limits at QD32 or sometimes even lower. Keep in mind that on a RAID controller, the RAID level will affect IOPs behaviours greatly.

I'll do a followup to this at a later date once I have my config file again to show benchmarking IOPs in Linux with higher than QD1.