Krystalmods.com

Those familiar with Compliance for their organizations know that it's very hard to prevent users from stealing data or making malicious attempts to gain access to resources. One can only hope to limit the channels to which these things can happen, especially for data leaks. If they can view it, they can take the data away somehow. What then? Tracking!

It's important to realize that it is more important to track user action on your Linux/Unix servers so in the event that an incident happens, you can figure out who did it and what was done. Most of the tools for this have either been outdated or doesn't record enough information.

For example, this guide talks about linux auditing which includes the sa command to show what commands users have issued. Problem here is that it doesn't list any arguments and therefore, the data collected cannot be used to successfully identify malicious activity.

Why not just use .bash_history?

Well, that file has a limit to the number of commands being one reason but most importantly, it is writeable by the user so it's susceptible to tampering not to mention it is written to on the exit command so if the user just disconnects, nothing is written to this file for that session.

I stumbled onto Snoopy Logger, first released in 2000, another 2004 release and has now been updated a few more times in 2010. This builds a library file that is loaded on system startup that outputs to syslog. Installation is very simple:

# ./configure
# make && make install && make enable

Since I'm implementing this in SLES11SP1, I must now configure syslog-ng to separate this into another log file. To do so, we add the following lines to the syslog-ng.conf file:

destination snoopy_all { file("/var/log/snoopy.log"); };
filter f_snoopy_all {
match("snoopy");
};
log {
source(src);
filter(f_snoopy_all);
destination(snoopy_all);
};

After this, we need to filter out Snoopy output from the main log by changing the f_messages filter to this:

filter f_messages   { not facility(news, mail) and not filter(f_iptables) and not filter(f_snoopy_all); };

Now restart the server to activate snoopy and you should start seeing activity in /var/log/snoopy.log like this:

Aug 26 14:26:54 intactprd1 snoopy[8086]: [uid:1000 sid:5857 cwd:/home/user path:/usr/bin/who]: who
Aug 26 14:26:56 intactprd1 snoopy[8087]: [uid:1000 sid:5857 cwd:/home/user path:/bin/ls]: ls -N --color=tty -T 0
Aug 26 14:26:58 intactprd1 snoopy[8088]: [uid:1000 sid:5857 cwd:/home/user path:/bin/cat]: cat /etc/passwd

The reason the server needs a reboot is that this library is loaded for every program run in the system, which means currently running processes won't use the library.

What about in a chroot???

Well, I thought about that too. In a chroot, you need to deploy it again or if you want, you could deploy this only in the chroot.

# cp /etc/ld.so.preload /chroot/etc/
# cp /usr/local/lib/snoopy.so /chroot/usr/local/lib/snoopy.so

Okay, now the module is loaded for chroot users, how do we get the logs? The programs inside the chroot needs a socket at /dev/log. Again, we modify the syslog-ng config but before that, if you are running Apparmor, make the follwoing changes first.

Change the following line in /etc/apparmor.d/sbin.syslog-ng:


@{CHROOT_BASE} = ,

To this:

@{CHROOT_BASE} = "/chroot",

That will give syslog-ng write access to /chroot/dev/log. However, I've found that to be somewhat unreliable at times so if you have problems with it, add this line instead:

/chroot/dev/log w,

Then issue the command...

# /etc/init.d/boot.apparmor restart

Add in /etc/syslog-ng/syslog-ng.conf:

source chroot { unix-dgram("/chroot/dev/log"); };
destination messages_chroot { file("/var/log/messages_chroot"); };
log { source(chroot); filter(f_messages); destination(messages_chroot); };

Now add Snoopy Logger configuration for chroot:

destination snoopy_chroot_all { file("/var/log/snoopy_chroot.log"); };
log {
source(chroot);
filter(f_snoopy_all);
destination(snoopy_chroot_all);
};

Then issue the command...

# /etc/init.d/syslog restart

Now, there is a bit of an issue. The system doesn't clean up /chroot/dev/log like it does with /dev/log. So when the system is restarted, syslog will fail to start. For now the quick and dirty workaround would be to modify /etc/init.d/syslog to remove that file on startup if it exists. Add the following to /etc/init.d/syslog just under "start" case:

if test -e /chroot/dev/log ; then
rm -f  /chroot/dev/log
fi

Unfortunately for now we have to turn off earlysyslog service as it interferes with this process.

# chkconfig earlysyslog off

A fellow reader asked the following question:

"Hey Eric, took a minute to read your blog this morning... Good Stuff ! Keep it up.

One question that I have for you though. How do you manage to keep temperature down in your server room ? Do you have special cooling equipment? You should write a blog post about this :)

Thanks,

Justin"

The key for keeping temperature down in a home setup is not about how to remove heat or introduce cold air.

It's very important to remember that the more power you use, the more heat you generate. Do you need a quad core? Probably not. Do you need a dual core? Maybe...but maybe not. Do you have power management and processor throttling turned on? Do you need the fastest drives you can find if you are limited by your network to serve file? Probably not. Essentially, choose components you can find that will get the job done with a bit of head room. However, that's not to say one should spend a huge amount on energy efficient processors either because often times, it can cost a lot more than it's worth so research on these processors is a must.

One should remember also that in a home and even in many business environment, these servers are at idle most of the time so power usage while idle is very important. For example, the Xeon X3440 processors used in this setup with Supermicro motherboard, two onboard Intel gigabit NICs, a dual port Intel PT giabit NIC, and an SSD to install OS, only runs 40 Watts when idle! The two storage servers use Atom processors and 2.5" SAS drives to reduce power use while maintaining performance.

Those who know about the "beast" in this picture I call the StorageTank would know that it is not exactly a power saver. I believe it idles at 160 Watts. One of the things on the to-do list is to change that Core 2 Quad into an Atom setup and when needed to go to 2TB drives, the current 10 x 1TB drives will be replaced with energy efficient ones instead. The two OS drives will be replaced by one SSD. This should get its power usage below 100 Watts, optimistically around 80 Watts. It actually is the primary heat producer in this setup.

Switching equipment. A lot of switching equipment, while meets your requirements, may not be energy efficient. Previously, this setup ran on a Linksys SRW224G4, a 24x100Base-T and 4xGigabit managed switch. What was surprising was that when measured, it used 25 Watts of power. The two new HP Procurve 1810G uses 11 Watts combind.

Can't enable CPU throttling due to the Linux distro you are using? (in my case, XenServer) If you need to reduce heat output and power consumption, try under-clocking your processor.

In the end, it always comes down to..."Keep it simple". That's the rule of thumb.

When I say keep it simple, don't dismiss the use of server/workstation products. A Xeon motherboard with quad Intel gigabit NICs can cost you $240 - $260 where as buying a cheap consumer motherboard and then 4 Gigabit can cost you about the same. The difference is in quality and space savings. Server motherboards will also have the minimal reqired components on it, no sound card, a minimal video card, gigabit NICs that are not made by Realtek or Marvell, etc. They are generally more stable and if you know how to choose, they can utilize normal desktop memory (non-ECC) to save you some money. A supermicro motherboard with Atom D510 and dual Intel Gigabit NICs will cost you $200 and comes with a better chipset, bringing better storage controller. Adding IPMI is a $25 premium as an onboard option, allowing remote console, power control, mounting of ISOs as virtual CD/DVD, monitoring, etc.

So to summarize, choose the right components for the job and keep the cost down but don't be cheap, buy smart.

Migration of this blog from Opensuse 11.0 to version 11.3 is about to complete. One last thing needed is to point the load balancer to the new servers. With this change, the main website will be replaced by b2evolution as it is much easier to maintain. One of the things I had back in opensuse 11.0 was eAccelerator compiled from scratch. In this post, I will show you how much simpler this has gotten.

What is eAccelerator?

"eAccelerator stores compiled PHP scripts in shared memory and executes code directly from it." (eaccelerator.net)

This was first used in the old days on a Pentium III based web server farm over 4 years ago to overcome performance problems when supporting more than one user. The difference is drastic to say the least.

Why not APC? It's got support in b2evolution doesn't it?

eAccelerator is simply superior. Take a look at the benchmark done by 2bit which also includes xcache.

So with that said, let's begin.

Requirements:

• Opensuse 11.1 - 11.3 (tested with 11.3)
• Apache (distro bundled)
• PHP (distro bundled)
• You've done the configuration and can run PHP code in apache now
• You have the package "patterns-openSUSE-yast2_basis" installed

Instructions:

1. We start with going to http://software.opensuse.org and searching for "eaccelerator". This will show you php5-eaccelerator package.
2. Right click 1-Click Install and copy the link. (make sure it's from a stable repository)
3. Go to your server and type OCICLI <link>. Follow the instructions.
4. Optional: Edit /etc/php5/conf.d/eaccelerator.ini and set eaccelerator.compress="0". We don't care about compression on today's servers in most cases and this will improve performance.
5. Restart apache: service apache2 restart

Verify that it is indeed running by creating a php file in your htdocs directory with the following code and then open it from a browser:

<?php
phpinfo();
?>

Scroll down and you should see that eAccelerator has been activated! Don't forget to remove that php file.

The H has an article regarding the vulnerability of Linux kernel since at least 2004 (possibly since first release of 2.6 kernel).

http://www.h-online.com/open/news/item/Root-privileges-through-Linux-kernel-bug-Update-1061563.html

They mentioned that Suse provided a fix back in 2004 and all Suse based distros since version 9 are patched for this vulnerability (SLES/SLED version 9+ and all Opensuse versions). However, the troubling matter is that it did not make it into the kernel sources and other distros likely are affected.

That makes me feel even better about my choice of Linux distro. hehe.

While the net is flooded with news of the broken iPhone 4, many still have no idea of what is going on. Others, deny that the list of problems are true and think that it's all hyped up by those who like to bash Apple. Those in denial likely have purchased or ordered the new iPhone 4 by new after Steve Jobs gave his speech about how the phone is actually better quality than all other phones on the market and that the complaints are isolated cases. They did however, offer free cases for those who wanted it (even though the device is flawless).

While chatting with a friend recently whom had no idea of this whole thing as he's not in the smartphone market yet, I was told that his friend working for Apple Canada has been trying to get a working iPhone 4 for a while now. It turns out that a couple couldn't make calls and another two had screen failures.

I personally am not against the concept of the restrictions imposed on iPhone users. I can see that it has a positive effect of stability, quality and hopefully security. However, that whole concept went out the window with the iPhone 4 because many parts of the hardware have problems, so the software side doesn't even make a difference anymore.

With that said, it seems that Apple has decided to file another patent. I wonder if they have a facility dedicated to patenting because they sure make of lot more of those than each of the components they get Foxconn to make. The new patent filed is named "SYSTEMS AND METHODS FOR IDENTIFYING UNAUTHORIZED USERS OF AN ELECTRONIC DEVICE". Outlined here It specifies methods of remotely disabling the device, takes a picture of the user, and provides GPS tracking. This is quite puzzling as I'm not sure how they will actually use it since jailbreaking is now legal. Maybe jailbreakers will get a visit by Steve Jobs him self + his mob.

So like many out there, I've been eagerly waiting on Cable service in my area. I'm too lazy to go checking every day if it's available so I've put together this shell script for all you linux/mac users:

#!/bin/bash

if [[ $1 == *teksavvy.com* ]] && [[ $2 != "" ]] && [[ $3 == *@* ]]
then
URL=$1
POSTALCODE=$2
EMAIL=$3
else
echo "Syntax: ./teksavvy_cable_avail.sh <URL> <POSTAL_CODE> <EMAIL>"
exit 1
fi

OUTPUT=`wget -qO- $URL --post-data="postal=$POSTALCODE&submit=Submit" |grep Congratulations`
if [[ $OUTPUT == *"is available"* ]]
then
echo $OUTPUT |mailx -s "Teksavvy Cable Internet Is Here!" $EMAIL
fi

So then I'd add the following in my crontab to check at 8 am everyday:

0 8 * * * /home/ewu/teksavvy_cable_avail.sh http://teksavvy.com/en/checkontariocable.asp <postal_code> <me@myemail.com>

Once it is available, I'll get an email. Remember your system has to have working outgoing mail transport.

Note: Please don't check every minute guys...we don't want to give Teksavvy workers a hard time...lol.

Some discussions here: http://www.dslreports.com/forum/r24685085-Cable-Automatically-check-cable-availability-on-LinuxMac

Still waiting for some more parts to come in. Don't know why it's taking so long.

I just mounted the monitor to make room for the servers. Also done was a second APC power bar for the second switch and the future backup Internet connection. Power bar 1 would connect to a new UPS with higher capacity meant for extended run and power bar 2 would connect to the current UPS meant for safe shutdown of servers.

The larger 20" monitor is much more comfortable to work with than the 15". The difference between a year 2000 display and one from 2005 is contrast and brightness, the key to not hurting my eyes.

Here's a shot...

If there was a way to burn everything solaris and related to solaris, I'd do it right now. I can't even remove unnecessary packages without the whole OS install getting bricked! So now I'm reinstalling with Core Networking group and I'm customizing packages, it tells me things like fonts require audio. Then it tells you where and what packages are for dependencies, but it makes you do them one by one manually searching for them!!! Resolving 60 dependencies manually is  a real PITA!!! ARG!!!!!!!!!!!

I don't understand it, all these years, latest Solaris 10 still can't do things properly. Last time I remember dealing with dependencies manually in Linux was sometime before the year 2000!

I'm waiting on one more motherboard and my ECC memory. However, I figured I'd try the Xeon X3440 with 4GB non-ECC memory that I had in another system...well what do you know? It works despite Supermicro's lack of official support!

I've never messed around with X3400 series since we only have X5000 series at work. So here's the scoop on my first impressions. Xeon X3440, based on the Core i5, runs at 2.53Ghz, has HT and Turbo Boost but lacks the triple channel memory of the Core i7 or Xeon X5000 series. However, one of its advantages seems to be power consumption in idle, which is most of the time for a home server. This may also be due to the fact that on a server motherboard, there's no nonsense of desktop boards which helps a lot and that this chip just has good power management.

At first I wasn't exactly sure of what I was getting my self into but it didn't look too bad from the test run outside of a chassis. It was around 65 watts in the BIOS which is actually quite impressive already. However, I realized that after POST, where it was waiting for a boot loader, it was consuming well over 80 watts! Finally, once XenServer was installed and booted up, the system used a mere 40 watts of power on idle!

40 Watts is actually quite fantastic. However, we do have to remember that this system will likely spike beyond 100 watts but this shouldn't happen often. One other thing to keep in mind - XenServer does not have CPU throttling at all. This means the idle power could possibly be reduced by another 10 watts maximum.

System config:

Intel Xeon X3440
Supermicro X8SIE-F w/IPMI
OCZ 4GB DDR3-1600 (Waiting for 8GB FB-DIMM)
Kingston SSDNow 64GB (gen 2)
Intel Server PT Dual Port Gigabit NIC
3Com 100Base-TX NIC (3C905B)
OCZ ModXstream-Pro 500W
Lian-Li PC-A05N

Had a recent failure that forced me to fsck an 8TB partition formated with EXT4 because the superblock was corrupted. The same partition used to be a 4TB EXT3 partition and was migrated over and doubled in size.

Well I was about to go nuts because last time on the 4TB EXT3 partition, it took 9 hours to do this on one of the fastest RAID controllers on the market. I was surprised that it took only 1 hour 40 minutes!

There you have it, another reason to move to EXT4. Though if you run web servers, EXT3 still outperforms EXT4 by maybe 30% if I remember correctly.

Some more hardware came in...including eight Seagate Constellation 2.5" SAS 500GB drives. Unfortunately one of them is DoA so I'll need a replacement.

So here's some simple comparison between SAS and SATA using Linux hdparm:

	2 hdparm instances	4 hdparm instances
SATA	160 MBps	300 MBps
SAS	295 MBps	375 MBps

Mirror setup on both sides so the only variable is this:

- 4x Seagate Momentus 2.5" 500GB 7200.4 SATA
- 4x Seagate Constellation 2.5" 500GB 7200rpm SAS

The reasons I used mutiple hdparm sessions was because the Highpoint RocketRAID 2720 doesn't really output to its full potential with single threaded read unlike Areca controllers. I wonder if this will make much difference in iSCSI performance.

I finally get to mount up the switches the way I wanted to. I don't have a cable management unit since I forgot to order one so I did what I could to make it less of a mess. Here's before:

And after:

And when it gets dark: