Archive

Archive for the ‘CentOS’ Category

How To remove all messages from a sendmail mail queue

 
Problem description:

Somehow your sendmail outgoing message queue got flooded by e.g. a job and you want to remove all of the messages to be delivered.

 
Problem resolution:

Outgoing messages in the sendmail queue are stored in /var/spool/mqueue. In order to get an overview of the messages you could execute the following sendmail command:

sendmail -bp

 
As your sendmail queue got flooded with a lot of messages, this command might take forever. So instead of querying all of the stored messages you can also check the files directly:

[root@server01 mqueue]# pwd
/var/spool/mqueue
[root@server01 mqueue]# 

[root@server01 mqueue]# ls -al
...
-rw-------  1 root smmsp    1285 18. Mai 10:35 qfr4I8ZSBn009790
-rw-------  1 root smmsp    1242 18. Mai 10:35 qfr4I8ZSu1009786
-rw-------  1 root smmsp    1233 18. Mai 10:35 qfr4I8ZumY009830
-rw-------  1 root smmsp    1253 18. Mai 10:35 qfr4I8ZUOF009794
-rw-------  1 root smmsp    1257 18. Mai 10:35 qfr4I8ZXPu009798
-rw-------  1 root smmsp    1253 18. Mai 10:35 qfr4I8ZZhj009802
...
[root@server01 mqueue]# 

[root@server01 mqueue]# ls -al | wc -l
34439
[root@server01 mqueue]#

 
There is no explicit command for removing all messages. You can directly delete them with “rm”. In order to remove all messages, we just delete the queued files:

[root@server01 mqueue]# pwd
/var/spool/mqueue
[root@server01 mqueue]# 

[root@server01 mqueue]# rm -f *
[root@server01 mqueue]# 

[root@server01 mqueue]# ls -al
insgesamt 1128
drwx------  2 root mail 1138688 20. Mai 11:06 .
drwxr-xr-x 16 root root    4096 11. Mai 2011  ..
[root@server01 mqueue]#

 
“sendmail -bq” should now report an empty mail queue:

[root@server01 mqueue]# sendmail -bp
/var/spool/mqueue is empty
          Total requests: 0
[root@server01 mqueue]#

 

Categories: CentOS, Linux, Oracle Linux, Red Hat Tags:

CentOS: yum operations fail with errors like “HTTP Error 404: Not Found” or “550 os: No such file or directory”

Problem description:

You want to update / upgrade your CentOS / RHEL-based Linux installation, but you are receiving errors like “HTTP Error 404: Not Found” or “550 os: No such file or directory”:

[root@centos01 ~]# yum update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.fraunhofer.de
 * extras: mirror.fraunhofer.de
 * updates: centos.bio.lmu.de
ftp://mirror.fraunhofer.de/centos.org/5.7/os/i386/repodata/repomd.xml: [Errno 4] IOError: [Errno ftp error] 550 os: No such file or directory
Trying other mirror.
http://centos.intergenia.de/5.7/os/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
http://centos.kiewel-online.ch/centos/5.7/os/i386/repodata/repomd.xml: [Errno 4] IOError: <urlopen error (-2, 'Der Name oder der Dienst ist nicht bekannt')>
Trying other mirror.
http://centos.mirroraustria.at/5.7/os/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
http://centos.psw.net/centos/5.7/os/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
http://centos.vieth-server.de/5.7/os/i386/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
...
...

 
Problem resolution:
The repomd.xml file listed as missing comes from the Continuous Refresh Repo (CR) for your CentOS version, but your release has been superseded by a new one. In order to get yum to refresh its metadata, just clean up the existing one:

[root@centos01 ~]# yum clean all
Loaded plugins: fastestmirror
Cleaning up Everything
Cleaning up list of fastest mirrors
[root@centos01 ~]#

 
Now yum should determine the new mirror information and allow you to continue your update, upgrade or install operation by using the correct mirror-URLs:

[root@centos01 ~]# yum update
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: tweedo.com
 * extras: centos.mirroraustria.at
 * updates: centos.mirroraustria.at
base                                                                                                                                                                      | 1.1 kB     00:00     
base/primary                                                                                                                                                              | 967 kB     00:00     
base                                                                                                                                                                                   2725/2725
extras                                                                                                                                                                    | 2.1 kB     00:00     
extras/primary_db                                                                                                                                                         | 171 kB     00:00     
updates                                                                                                                                                                   | 1.9 kB     00:00     
updates/primary_db                                                                                                                                                        | 564 kB     00:00     
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package acl.i386 0:2.2.39-8.el5 set to be updated
---> Package acpid.i386 0:1.0.4-12.el5 set to be updated
...
...

 

Categories: CentOS, Oracle Linux, Red Hat Tags:

Red Hat: How to check and repair your root-filesystem in rescue mode

November 24th, 2011 Matthias Pölzinger No comments

Problem description:

Your Linux server had some troubles with it’s underlying storage devices and put all your filesystems into read only mode. You corrected the issue and are booting up your Linux server. The boot process determines some problems with your root filesystems and requires a manual intervention by logging in with the root password. Unfortunately your are unable to login because the root password is not accepted (although you are using the correct one for the 25th time ;-) ).

You are unable to boot because of filesystem issues. At the same time your are unable to fix the filesystem, because your root password is not accepted.

 
Problem resolution:

Take a Linux installation media of your distribution and boot into linux rescue mode by typing the following command when asked for boot options:

linux rescue



 
Select your language and keyboard layout:

 



 
Network configuration is not necessary and can be skipped:

 


 
A very important step is to SKIP the mount of your Linux installation. Do not use MOUNT or READ-ONLY. For a filesystem check it is required that the filesystem is unmounted. Once mounted it is nearly unpossible to unmount your root-filesystem in rescue mode.

 


 
If you are not using a Logical Volume Manager for your filesystems you can directly jump to execute the “fsck” command. Otherwise you will have to first scan for your Physical Volumes and review them:

lvm pvscan
lvm pvdisplay

 


 
If your Physical Volumes were listed correctly continue to review the detected Volume Groups:

lvm vgdisplay

 
Now continue to activate your Volume Groups in order to create your Logical Volume devices in the “/dev” filesystem. In this example a Volume Group with the name “VolGroup00″ has to be activated:

lvm vgchange -a y VolGroup00

 


 
After activating your Volume Group(s), you will be able to perform a filesystem check and correct problems:

fsck /dev/VolGroup00/LogVol00

 


 
Once the filesystem check finishes successfully, you should be able to reboot without any further complication.

 

Categories: CentOS, Linux, Oracle Linux, Red Hat Tags:

Linux: How to query the WWPN of Fibre Channel HBA ports

October 8th, 2011 Matthias Pölzinger No comments

Problem description:

The WWPNs (World Wide Port Numbers) of your HBAs are required e.g. in order to configure Fibre Channel LUN Access. You want to use Linux instruments to access this information in order to avoid rebooting the servers and access the information from BIOS or any other utility.

 
Problem resolution:

Current Linux kernel versions provide relevant information about FC HBAs through the sysfs Filesystem mounted at /sys. The location for detailed information depends on your Linux OS version. For CentOS / Red Hat Enterprise Linux / Oracle Linux 5 the path is /sys/class/scsi_host/host*/device/fc_host*/, but for Version 6 it changed to /sys/class/fc_host/host*/ which makes it easier to separate between SCSI controllers and FC HBAs.

Example for Centos / RHEL / Oracle Linux 5:

[root@initso01 ~]# ls -al /sys/class/scsi_host/host5/device/fc_host:host5/
total 0
drwxr-xr-x 4 root root    0 Oct  5 17:03 .
drwxr-xr-x 6 root root    0 Oct  5 17:03 ..
lrwxrwxrwx 1 root root    0 Oct  5 17:03 device -> ../../../devices/pci0000:00/0000:00:03.0/0000:15:00.0/host5
-r--r--r-- 1 root root 4096 Oct  7 10:07 fabric_name
--w------- 1 root root 4096 Oct  7 10:07 issue_lip
-r--r--r-- 1 root root 4096 Oct  7 10:07 max_npiv_vports
-r--r--r-- 1 root root 4096 Oct  7 10:07 node_name
-r--r--r-- 1 root root 4096 Oct  7 10:07 npiv_vports_inuse
-r--r--r-- 1 root root 4096 Oct  7 10:07 port_id
-r--r--r-- 1 root root 4096 Oct  7 10:07 port_name
-r--r--r-- 1 root root 4096 Oct  7 10:07 port_state
-r--r--r-- 1 root root 4096 Oct  7 10:07 port_type
drwxr-xr-x 2 root root    0 Oct  7 10:07 power
-r--r--r-- 1 root root 4096 Oct  7 10:07 speed
drwxr-xr-x 2 root root    0 Oct  7 10:07 statistics
lrwxrwxrwx 1 root root    0 Oct  5 17:04 subsystem -> ../../fc_host
-r--r--r-- 1 root root 4096 Oct  7 10:07 supported_classes
-r--r--r-- 1 root root 4096 Oct  7 10:07 supported_speeds
-r--r--r-- 1 root root 4096 Oct  7 10:07 symbolic_name
-rw-r--r-- 1 root root 4096 Oct  7 10:07 system_hostname
-rw-r--r-- 1 root root 4096 Oct  7 10:07 tgtid_bind_type
-rw-r--r-- 1 root root 4096 Oct  5 17:03 uevent
--w------- 1 root root 4096 Oct  7 10:07 vport_create
--w------- 1 root root 4096 Oct  7 10:07 vport_delete
[root@initso01 ~]#

 
These fc_host directories can be used to to determine the port speed:

[root@initso01 ~]# for i in `ls /sys/class/scsi_host/host*/device/fc_host*/speed`; do echo $i; echo "==============="; cat $i; done
/sys/class/scsi_host/host5/device/fc_host:host5/speed
===============
unknown
/sys/class/scsi_host/host6/device/fc_host:host6/speed
===============
4 Gbit
/sys/class/scsi_host/host7/device/fc_host:host7/speed
===============
unknown
/sys/class/scsi_host/host8/device/fc_host:host8/speed
===============
4 Gbit
[root@initso01 ~]#

 
or to query the WWPN of each port:

[root@initso01 ~]# for i in `ls /sys/class/scsi_host/host*/device/fc_host*/port_name`; do echo $i; echo "==============="; cat $i; done
/sys/class/scsi_host/host5/device/fc_host:host5/port_name
===============
0x21000024ff2e30ce
/sys/class/scsi_host/host6/device/fc_host:host6/port_name
===============
0x21000024ff2e30cf
/sys/class/scsi_host/host7/device/fc_host:host7/port_name
===============
0x21000024ff2e30cc
/sys/class/scsi_host/host8/device/fc_host:host8/port_name
===============
0x21000024ff2e30cd
[root@initso01 ~]#

 
Just omit the hexidecimal prefix “0x” and you have the WWPN for each Fibre Channel HBA port.

 
Example for Centos / RHEL / Oracle Linux 6:

[root@initso02 ~]# ls -al /sys/class/fc_host/host1/
total 0
drwxr-xr-x. 4 root root    0 Aug 11 07:11 .
drwxr-xr-x. 3 root root    0 Aug 11 07:11 ..
-rw-r--r--. 1 root root 4096 Aug 13 10:29 dev_loss_tmo
lrwxrwxrwx. 1 root root    0 Aug 13 10:29 device -> ../../../host1
-r--r--r--. 1 root root 4096 Aug 13 10:29 fabric_name
--w-------. 1 root root 4096 Aug 13 10:29 issue_lip
-r--r--r--. 1 root root 4096 Aug 13 10:29 max_npiv_vports
-r--r--r--. 1 root root 4096 Aug 13 10:29 node_name
-r--r--r--. 1 root root 4096 Aug 13 10:29 npiv_vports_inuse
-r--r--r--. 1 root root 4096 Aug 13 10:29 port_id
-r--r--r--. 1 root root 4096 Aug 13 10:14 port_name
-r--r--r--. 1 root root 4096 Aug 13 10:29 port_state
-r--r--r--. 1 root root 4096 Aug 13 10:29 port_type
drwxr-xr-x. 2 root root    0 Aug 13 10:29 power
-r--r--r--. 1 root root 4096 Aug 13 10:29 speed
drwxr-xr-x. 2 root root    0 Aug 13 10:29 statistics
lrwxrwxrwx. 1 root root    0 Aug 11 07:11 subsystem -> ../../../../../../../class/fc_host
-r--r--r--. 1 root root 4096 Aug 13 10:29 supported_classes
-r--r--r--. 1 root root 4096 Aug 13 10:29 supported_speeds
-r--r--r--. 1 root root 4096 Aug 13 10:29 symbolic_name
-rw-r--r--. 1 root root 4096 Aug 13 10:29 system_hostname
-rw-r--r--. 1 root root 4096 Aug 13 10:29 tgtid_bind_type
-rw-r--r--. 1 root root 4096 Aug 11 07:11 uevent
--w-------. 1 root root 4096 Aug 13 10:29 vport_create
--w-------. 1 root root 4096 Aug 13 10:29 vport_delete
[root@initso02 ~]#

 
As with Version 5 these fc_host directories can be used to to determine the port speed:

[root@initso02 ~]# for i in `ls /sys/class/fc_host/host*/speed`; do echo $i; echo "==============="; cat $i; done
/sys/class/fc_host/host1/speed
===============
4 Gbit
/sys/class/fc_host/host2/speed
===============
4 Gbit
[root@initso02 ~]#

 
or to query the WWPN of each port:

[root@initso02 ~]# for i in `ls /sys/class/fc_host/host*/port_name`; do echo $i; echo "==============="; cat $i; done
/sys/class/fc_host/host1/port_name
===============
0x24000024ee09a545
/sys/class/fc_host/host2/port_name
===============
0x24000024ee09a578
[root@initso02 ~]#

 
Just omit the hexidecimal prefix "0x" as with Version 5 and you have the WWPN for each Fibre Channel HBA port.

Linux/Nagios: sudo: sorry, you must have a tty to run sudo

Problem description:

If you are using Nagios or other monitoring products to check your servers, you might come into the situation running commands which require special permissions (e.g. like running them as the root-user). You have configured sudo permissions and your check runs fine on the terminal, but if the monitoring process/service tries to execute it, you are receiving the following message:

sudo: sorry, you must have a tty to run sudo

Cause:

By default sudo only allows to be executed with a tty:

[root@linux01 ~]# cat /etc/sudoers | grep requiretty
Defaults    requiretty
[root@linux01 ~]#

Problem resolution:

You can either disable requiretty or just disable it for the user you require to use sudo without tty (PREFERRED):

[root@linux01 ~]# cat /etc/sudoers | grep requiretty
Defaults    requiretty
Defaults:myosuser !requiretty
[root@linux01 ~]#
Categories: CentOS, Monitoring, Nagios, Oracle Linux, Red Hat Tags:

Linux: HOWTO add an additional IP address to an interface

Works for:
Red Hat Entperise Linux, Oracle Linux, CentOS or any other Red Hat based distribution

HOWTO:
You can bind multiple IP addresses to one interface. Just go to your /etc/sysconfig/network-scripts directory and copy your existing interface configuration file to an additional one:

[root@linux01 ~]# cd /etc/sysconfig/network-scripts/
[root@linux01 network-scripts]# ls -al ifcfg-eth0 
-rw-r--r-- 1 root root 201 Dec 30  2009 ifcfg-eth0
[root@linux01 network-scripts]# cp ifcfg-eth0 ifcfg-eth0:1
[root@linux01 network-scripts]#

Now adapt your newly create configuration file (DEVICE, remove HWADDR, adapt network information, etc.). Example:

[root@linux01 network-scripts]# cat ifcfg-eth0:1
DEVICE=eth0:1
BOOTPROTO=static
BROADCAST=192.168.2.255
IPADDR=192.168.2.233
NETMASK=255.255.255.0
NETWORK=192.168.2.0
ONBOOT=yes
[root@linux01 network-scripts]#

Either use ifup or restart the network service to update the interface configuration:

[root@linux01 network-scripts]# service network restart
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:                                [  OK  ]
[root@linux01 network-scripts]#

Check the status of your newly created “virtual” interface:

[root@linux01 network-scripts]# ifconfig eth0:1
eth0:1    Link encap:Ethernet  HWaddr 00:0A:2A:4A:EA:4A 
          inet addr:192.168.2.233  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:59 Base address:0x2024 
 
[root@linux01 network-scripts]#
Categories: CentOS, Oracle Linux, Red Hat Tags:

Linux: HOWTO flush filesystem cache

Some time ago, I had to flush all cached filesystem pages in order to diagnose memory usage related problems. On Linux this is quite easy to achieve by writing to the /proc/sys/vm/drop_caches file:

  • To free the pagecache:
    echo 1 > /proc/sys/vm/drop_caches
  • To free dentries and inodes:
    echo 2 > /proc/sys/vm/drop_caches
  • To free pagecache, dentries and inodes:
    echo 3 > /proc/sys/vm/drop_caches

This is a non-destructive operation and will not remove any dirty pages. So, if you want to clear the complete cache, you should sync all dirty objects first:

[root@linux01 ~]# free
             total       used       free     shared    buffers     cached
Mem:       1035064     942004      93060          0      21000     808428
-/+ buffers/cache:     112576     922488
Swap:      2097144        116    2097028
[root@linux01 ~]# sync; echo 3 > /proc/sys/vm/drop_caches
[root@linux01 ~]# free
             total       used       free     shared    buffers     cached
Mem:       1035064     102320     932744          0        192      22036
-/+ buffers/cache:      80092     954972
Swap:      2097144        116    2097028
[root@linux01 ~]#
Categories: CentOS, Oracle, Oracle Linux, Red Hat Tags: