Archive for the ‘Networking’ Category

Hello, this is more a post to remind myself on how to install MPIO on Hyper-V R2. I’ve been experimenting with MPIO and my WSS server’s to make a more robust cluster, and need to use MPIO to have proper failover when maintaining the iSCSI /SAN network.

By default Hyper-V R2 does not habe the MPIO feature installed, so this needs to be enabled by issuing the command

Dism /online /enable-feature:MultipathIo

Copy and paste the above, as it is case sensitive. It doesn’t end here however, we still need to register the iSCSI initiator to use MPIO. Start up the MPIO control panel

mpiocpl

Check the box “Add support for iSCSI devices”

Then click “Add”

Reboot when ready. Once rebooted, simply add a new session to your iSCSI target, and you can then use the MCS/MPIO to verify and set the MPIO configuration (round robin or failover only). Using this, I have verified I can get double the throughput when using two interfaces on an WSS target.

Here are my notes on creating a LAMP server with FastCGI/SuExec on the latest CentOS 6.1. I prefer to use CentOS since it is fully Hyper-V happy, and quiet well supported.

This assumes the “Basic Server” installation on CentOS 6.1. MySQL is installed, but not configured – but that’s an easy one to do. This also includes a bit at the end to run SSH for SFTP on port 24 so that you can allow external users in without them messing around with other users data.

# Centos 6.1 LAMP Server with Fast CGI and SuExec

# Install prereqs
yum install httpd httpd-devel php-mysql mysql-server php gcc

# Fast CGI
cd /opt
wget http://www.fastcgi.com/dist/mod_fastcgi-current.tar.gz
tar -zxvf mod_fastcgi-current.tar.gz
cd mod_fastcgi*

cp Makefile.AP2 Makefile

make top_dir=/usr/lib64/httpd
make install top_dir=/usr/lib64/httpd

echo "LoadModule fastcgi_module modules/mod_fastcgi.so" > /etc/httpd/conf.d/mod_fastcgi.conf
echo "DirectoryIndex index.php default.php" >> /etc/httpd/conf.d/mod_fastcgi.conf
echo "" >> /etc/httpd/conf.d/mod_fastcgi.conf
echo "<IfModule mod_fastcgi.c>" >> /etc/httpd/conf.d/mod_fastcgi.conf
echo "  AddHandler fastcgi-script .fcgi" >> /etc/httpd/conf.d/mod_fastcgi.conf
echo "  FastCgiWrapper /usr/sbin/suexec" >> /etc/httpd/conf.d/mod_fastcgi.conf
echo "  FastCgiIpcDir fcgi/" >> /etc/httpd/conf.d/mod_fastcgi.conf
echo "  FastCgiConfig -singleThreshold 1 -autoUpdate -idle-timeout 240 -pass-header HTTP_AUTHORIZATION" >> /etc/httpd/conf.d/mod_fastcgi.conf
echo "</IfModule>" >> /etc/httpd/conf.d/mod_fastcgi.conf

chmod 755 /usr/sbin/suexec
chmod +s /usr/sbin/suexec

mkdir /etc/httpd/fcgi
mkdir /etc/httpd/fcgi/dynamic
chmod 777 /etc/httpd/fcgi -R

# Switch to worker mode
echo "HTTPD=/usr/sbin/httpd.worker" >> /etc/sysconfig/httpd

# Create the skeleton
cd /opt
mkdir skel
mkdir skel/logs
mkdir skel/homepage
mkdir skel/cgi-bin
echo $’#\x21/bin/sh’ > skel/cgi-bin/php.fcgi
echo "PHP_CGI=/usr/bin/php-cgi" >> skel/cgi-bin/php.fcgi
echo "PHP_FCGI_CHILDREN=2" >> skel/cgi-bin/php.fcgi
echo "PHP_FCGI_MAX_REQUESTS=1000" >> skel/cgi-bin/php.fcgi
echo "### no editing below ###" >> skel/cgi-bin/php.fcgi
echo "export PHP_FCGI_CHILDREN" >> skel/cgi-bin/php.fcgi
echo "export PHP_FCGI_MAX_REQUESTS" >> skel/cgi-bin/php.fcgi
echo "exec \$PHP_CGI" >> skel/cgi-bin/php.fcgi

chmod 755 skel/cgi-bin/php.fcgi

# Vhost template
mkdir /etc/httpd/conf.d/hosts
echo "<VirtualHost *:80>" > /etc/httpd/conf.d/template
echo "        ServerAdmin webmaster@DOMAIN" >> /etc/httpd/conf.d/template
echo "        ServerName DOMAIN" >> /etc/httpd/conf.d/template
echo "        ServerAlias *.DOMAIN" >> /etc/httpd/conf.d/template
echo "        DocumentRoot /var/www/USER/homepage" >> /etc/httpd/conf.d/template
echo "" >> /etc/httpd/conf.d/template
echo "        SuexecUserGroup USER USER" >> /etc/httpd/conf.d/template
echo "" >> /etc/httpd/conf.d/template
echo "        <Directory />" >> /etc/httpd/conf.d/template
echo "                Options FollowSymLinks" >> /etc/httpd/conf.d/template
echo "                AllowOverride None" >> /etc/httpd/conf.d/template
echo "        </Directory>" >> /etc/httpd/conf.d/template
echo "" >> /etc/httpd/conf.d/template
echo "        <Directory /var/www/USER/homepage/>" >> /etc/httpd/conf.d/template
echo "                Options -Indexes FollowSymLinks -MultiViews" >> /etc/httpd/conf.d/template
echo "                AllowOverride all" >> /etc/httpd/conf.d/template
echo "                Order allow,deny" >> /etc/httpd/conf.d/template
echo "                Allow from all" >> /etc/httpd/conf.d/template
echo "        </Directory>" >> /etc/httpd/conf.d/template
echo "" >> /etc/httpd/conf.d/template
echo "        ScriptAlias /cgi-bin/ /var/www/USER/cgi-bin/" >> /etc/httpd/conf.d/template
echo "        <Directory "/var/www/USER/cgi-bin/">" >> /etc/httpd/conf.d/template
echo "                AllowOverride None" >> /etc/httpd/conf.d/template
echo "                Options ExecCGI -MultiViews +SymLinksIfOwnerMatch" >> /etc/httpd/conf.d/template
echo "                Order allow,deny" >> /etc/httpd/conf.d/template
echo "                Allow from all" >> /etc/httpd/conf.d/template
echo "        </Directory>" >> /etc/httpd/conf.d/template
echo "" >> /etc/httpd/conf.d/template
echo "        AddHandler php-fastcgi .php" >> /etc/httpd/conf.d/template
echo "        AddType application/x-httpd-php .php" >> /etc/httpd/conf.d/template
echo "        DirectoryIndex index.html index.php" >> /etc/httpd/conf.d/template
echo "        Action php-fastcgi /cgi-bin/php.fcgi" >> /etc/httpd/conf.d/template
echo "" >> /etc/httpd/conf.d/template
echo "        ServerSignature On" >> /etc/httpd/conf.d/template
echo "        ErrorLog logs/USER/error.log" >> /etc/httpd/conf.d/template
echo "        CustomLog logs/USER/access.log combined" >> /etc/httpd/conf.d/template
echo "</VirtualHost>" >> /etc/httpd/conf.d/template

# turn on vhosting
echo "NameVirtualHost *:80" > /etc/httpd/conf.d/00-EnableVirtualHost.conf
echo "Include conf.d/hosts/*" >> /etc/httpd/conf.d/00-EnableVirtualHost.conf

# create a shortcut script to create webs
# usage: createweb username domainname.tld
echo $’#\x21/bin/sh’ > /usr/sbin/createweb
echo "useradd -b /var/www -d /var/www/\$1 -m -k /opt/skel -s /bin/false \$1" >> /usr/sbin/createweb
echo "chmod 755 /var/www/\$1" >> /usr/sbin/createweb
echo "chmod 755 /var/www/\$1/cgi-bin/php.fcgi" >> /usr/sbin/createweb
echo "ln -s /var/www/\$1/logs /var/log/httpd/\$1" >> /usr/sbin/createweb
echo "cp /etc/httpd/conf.d/template /etc/httpd/conf.d/hosts/\$2" >> /usr/sbin/createweb
echo "replace DOMAIN \$2 — /etc/httpd/conf.d/hosts/\$2" >> /usr/sbin/createweb
echo "replace USER \$1 — /etc/httpd/conf.d/hosts/\$2" >> /usr/sbin/createweb
echo "passwd \$1" >> /usr/sbin/createweb
echo "chown root.root /var/www/$1" >> /usr/sbin/createweb
chmod 700 /usr/sbin/createweb

# chroot jail ssh.
cd /etc/ssh

cp sshd_config sshd_config24
nano sshd_config24

# Change the following
Port 24
PermitRootLogin no
ChrootDirectory /var/www/%u
Subsystem       sftp    internal-sftp

# Startup on reboot
echo "/usr/sbin/sshd -f /etc/ssh/sshd_config24" >> /etc/rc.d/rc.local

#Start Now
/usr/sbin/sshd -f /etc/ssh/sshd_config24

Quick notes on installing asterisk 1.8 on Centos 6.0 x64. I used this to connect Microsoft Lync to my SIP providers.

yum install wget
yum install kernel-devel gcc make gcc-c++ libxml2-devel
yum install perl ncurses-devel

wget http://downloads.asterisk.org/pub/telephony/asterisk/releases/asterisk-1.8.5.0.tar.gz
wget http://downloads.asterisk.org/pub/telephony/dahdi-linux-complete/releases/dahdi-linux-complete-2.5.0+2.5.0.tar.gz
tar -xvzf asterisk-1.8*
tar -xvzf dahdi-linux-complete-*
cd  dahdi-linux-complete*
make all

make install
make config
chkconfig dahdi on
service dahdi start
cd ..
cd asterisk-*
make clean
./configure
make menuselect
make
make install
make samples
make config
chkconfig asterisk on

Quick post on a discovery I made. I setup a complete infrastructure on an Hyper-V host for a customer (co-located at a datacenter), and one of the components of this infrastructure was Lync 2010. Lync is a phenomenal piece of software that allows for full collaboration for an enterprise without using third party services.

After the complete setup, we noticed that external users were not getting a great experience using desktop sharing / application sharing features from within Lync. Upon further investigation, it turned out the firewall was getting slammed (a virtual instance of pfSense on the hyper-v host). It appears that Lync floods the servers with as much udp traffic as configured to allow, and this causes a problem for the pfSense install which uses emulated network cards. My theory is that the software interrupts that are getting triggered for the emulated network card eats up all the cpu time on the guest OS giving it little time to do the routing/natting it needs to accomplish.

We solved this by using a very simple Linux firewall, although I could have easily just moved the pfSense install to a real machine and would have accomplished the same result. I also wonder if using a CentOS based firewall on the Hyper-V host would have solved the problem since it is supported using Hyper-V’s synthetic nics.

Just a quick post wit ha download link to a Zabbix template which has the profile for a SNMP enabled Netgear switch, 24port (GS724T)

Here it is

For years, I’ve ran many small servers running the popular ICH/ISW Intel Storage Matrix RAID in Raid-1 configuration. For many years this has worked absolutely perfectly with no issues on both Windows and Linux. But something has always really bugged me. What do i do when (and they will) a drive fails? How does ISW handle it?

On Windows, this is simple, you launch the Storage Matrix software and click rebuild (if it isn’t rebuilding automagically). But how do you do this on a Linux server which has no Storage Matrix software? After hours of Googling, i came across the command “dmraid -R”. But that didn’t work in my test environments.

So i spent a whole afternoon figuring this out. This is what i found.

DMRaid Works. Sort of

DMRaid is the linux implementation of popular onboard RAID setups. Your raid can be from Intel, Nvidia, Promise and a few others who do implement it. Intel is the most common one, and that’s the one i generally have on all my Intel servers. What *you* may find is that your implementation is different, but this posting should help you.

My test setup was a simple ICH6R machine with two 160gb Seagate hard drives. I booted up the machine, went into the Intel raid setup, and created a 20gb mirror partition called “System”. I then installed CentOS 5.5 32bit on this machine, and went to work.

Initial results

The first thing i did, was find out what i’ve got. Running “dmraid -s” gave me

[root@nasri ~]# dmraid -s
*** Group superset isw_djhffiddde
–> Active Subset
name   : isw_djhffiddde_System
size   : 41942528
stride : 256
type   : mirror
status : ok
subsets: 0
devs   : 2
spares : 0

Then running “dmraid -r” gave me

[root@nasri ~]# dmraid -r
/dev/sda: isw, "isw_djhffiddde", GROUP, ok, 312581806 sectors, data@ 0
/dev/sdb: isw, "isw_djhffiddde", GROUP, ok, 312581806 sectors, data@ 0

This tells me, my mirror set is running, and has two drives attached and all is happy.

Broken results

I then, turned the machine off, and yanked a drive, inserted a different drive, and turned it back on. After fiddling with the bios for a few minutes (my machine wanted to boot form the newly installed drive, not the raid) i got back in, and this is what i saw

[root@nasri ~]# dmraid -s
ERROR: isw: wrong number of devices in RAID set "isw_djhffiddde_System" [1/2] on /dev/sda
*** Group superset isw_djhffiddde
–> *Inconsistent* Active Subset
name   : isw_djhffiddde_System
size   : 41942528
stride : 256
type   : mirror
status : inconsistent
subsets: 0
devs   : 1
spares : 0

and

[root@nasri ~]# dmraid -r
/dev/sda: isw, "isw_djhffiddde", GROUP, ok, 312581806 sectors, data@ 0

So, dmraid tells me that the raid is broken and inconsistent. Great. That’s what i want to see when a disk fails in my raid sets. According to the man pages, and the Google, to repair it you use “dmraid -R <raid id> /dev/<device>”

So, here goes.

[root@nasri ~]# dmraid -R isw_djhffiddde_System /dev/sdb
ERROR: isw: wrong number of devices in RAID set "isw_djhffiddde_System" [1/2] on /dev/sda
isw: drive to rebuild: /dev/sdb

RAID set "isw_djhffiddde_System" already active
device "isw_djhffiddde_System" is now registered with dmeventd for monitoring
Error: Unable to write to descriptor!
Error: Unable to execute set command!
Error: Unable to write to descriptor!
Error: Unable to execute set command!

Hrm. Error’s. I don’t like errors. What’s happened? To be honest, I’ll never know – but it seems like it was not working. dmraid thinks its working, but i cant see it. I cant really hear any grumblings from the drive, nor can i see the LED’s flash. dmraid tells me the following:

[root@nasri ~]# dmraid -s
*** Group superset isw_djhffiddde
–> Active Subset
name   : isw_djhffiddde_System
size   : 41942528
stride : 256
type   : mirror
status : nosync
subsets: 0
devs   : 2
spares : 0

Ok, so its not inconsistent now, but it is “nosync”, which i cannot figure out what it means. I should look at the source code, but i cant be bothered.

Alright, so it appears that its not working.

Plan B

To figure out if its doing something, i turned the machine off and removed the new drive, and put in a Western Digital Raptor. Something that makes sounds. Booted up, and dmraid still showed the same stuff, inconsistent raid set. Now, i added the new WDRaptor to this set.

[root@nasri ~]# dmraid -R isw_djhffiddde_System /dev/sdb
ERROR: isw: wrong number of devices in RAID set "isw_djhffiddde_System" [1/2] on /dev/sda
isw: drive to rebuild: /dev/sdb

RAID set "isw_djhffiddde_System" already active
device "isw_djhffiddde_System" is now registered with dmeventd for monitoring

Oh wow, much better. On top of that, i could hear the grumblings of the WD, and i could see LED activity. So, it works!

I also found a command to monitor this progress. Its called “dmsetup status”

[root@nasri ~]# dmsetup status
isw_djhffiddde_Systemp2: 0 41720805 linear
isw_djhffiddde_Systemp1: 0 208782 linear
isw_djhffiddde_System: 0 41942776 mirror 2 8:16 8:0 928/1280 1 AA 1 core
VolGroup00-LogVol01: 0 4128768 linear
VolGroup00-LogVol00: 0 37552128 linear

[root@nasri ~]# dmsetup status
isw_djhffiddde_Systemp2: 0 41720805 linear
isw_djhffiddde_Systemp1: 0 208782 linear
isw_djhffiddde_System: 0 41942776 mirror 2 8:16 8:0 936/1280 1 AA 1 core
VolGroup00-LogVol01: 0 4128768 linear
VolGroup00-LogVol00: 0 37552128 linear

[root@nasri ~]# dmsetup status
isw_djhffiddde_Systemp2: 0 41720805 linear
isw_djhffiddde_Systemp1: 0 208782 linear
isw_djhffiddde_System: 0 41942776 mirror 2 8:16 8:0 1280/1280 1 AA 1 core
VolGroup00-LogVol01: 0 4128768 linear
VolGroup00-LogVol00: 0 37552128 linear

And finally

[root@nasri ~]# dmraid -r
/dev/sdb: isw, "isw_djhffiddde", GROUP, ok, 312581806 sectors, data@ 0
/dev/sda: isw, "isw_djhffiddde", GROUP, ok, 72303838 sectors, data@ 0
[root@nasri ~]# dmraid -s
*** Group superset isw_djhffiddde
–> Active Subset
name   : isw_djhffiddde_System
size   : 41942528
stride : 256
type   : mirror
status : ok
subsets: 0
devs   : 2
spares : 0

So. This is why it “sort of” works. It didn’t work with another Seagate drive, but it worked with a different drive. Consequently, i yanked the good 80gb drive from this set, and plugged in a 750gb Seagate, and was able to mirror back to that without a problem. Maybe initially it was my drives.

Conclusion

To fix your broken Raid1′s on your Intel raid’s, use “dmraid -R <raidid> <dev>” and watch “dmsetup status” and wait for the ratio to be 1.

Another quick post/reminder to myself. I’ve been experimenting with the idea of using Cacti to monitor the performance of my Hyper-V servers, so i needed SNMP on my HyperV machines. However there is no UI to add that feature into the core installs. So, to install SNMP on HyperV R2, use the following command line

start /w ocsetup SNMP-SC

That’s it!

An update on my previous posting about the Adaptec 3805 and my troubles with getting compatible drives.

I’ve been running a RAID-5 with the 3085 using 4 Samsung SpinPoint F3′s for about 2 weeks, and two days ago it started to give problems. The fourth disk in the array just dropped out, with no visible SMART issues or physical defects.

So, after waking up to this news I added it back into the array (probably not a great idea, but usually its fine). The moment the full initialization completed, something occurred that caused one of my Virtual Machines running from that RAID array to stop functioning (it was a mail server). There was another machine on there with much lesser activity, that kept running without a problem – but it did all but destroy the other virtual machine. Luckily, i have backups.

Not only did it knock off one of my virtual machines, it kicked out a different disk from the array, and started complaining again. So then I broke down, and ordered 2 ES.2 disks from Newegg which are on the HCL for this controller. However, 8hrs later, the controller barfed up completely and started giving timeouts to the host to the logical drive – even with 3 perfectly useable drives. The host became very unstable, i had to shut down the virtual machines, and reset the machine (thank god for Lights out control).

Upon reboot, the logical raid-5 array was unusable, and had to be forced online. I copied the two files that i absolutely had to have, and finally destroyed the array. When i go up to the data center again, I’m removing this card and burying it. Going to go back to my trusty Highpoint RR2224 which I’ve had for over 5 years now, without a single glitch.

Done.

Hello, a quick post / announcement that the Adaptec 3805 SAS Raid controller has compatibility issues with Western Digital Caviar Black drives. I don’t know if its an issue with all capacities of the Black series, but the ones i had were the 1TB ones, model number WDC-WD1001FALS-0.

After doing some research, it appears that the WD Blacks are NOT on the HCL for the Adaptec 3805 controller, but it took me some time to find this. I hope this post helps anyone who was going to go down this route.

Since i’d already gotten the WD Blacks, I ended up using Samsung Spinpoint F3′s from some of my servers. The WD Blacks work fine on Intel ICH Raid. I think next time, I’ll stick with these Samsungs. Cheap, good warranty, and FAST.

Now in general, why can’t Western Digital make standards compliant drives? I could easily blame Adaptec as well, but in this case, i think it has to be WD’s fault. They are known to make ATA drives that don’t work the same way as everyone else. How hard can it be? Even Samsung got it right.

Update, see my second post on this

Quick Reminder to myself on how to fix this time speedup issue with Centos 5.4 and HyperV

nano /boot/grub/grub.conf

append

divider=10 clocksource=acpi_pm

to the end of the current kernel (or all of them actually)