Why is file system usage still high after housekeeping?

In this example, my user couldn’t understand why filesystem usage was still so high considering he had removed and zipped up some files. It was still showing 99% used.

Checking errpt showed:

LABEL:          J2_FS_FULL
IDENTIFIER:     F7FA22C9
 
Date/Time:       Sat 14 Oct 13:50:00 2017
Sequence Number: 47470
Machine Id:      00C9A2F04C00
Node Id:         xxxxxxxxxxx
Class:           O
Type:            INFO
WPAR:            Global
Resource Name:   SYSJ2
 
Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
 
Probable Causes
FILE SYSTEM FULL
 
        Recommended Actions
        INCREASE THE SIZE OF THE ASSOCIATED FILE SYSTEM
        REMOVE UNNECESSARY DATA FROM FILE SYSTEM
        USE FUSER UTILITY TO LOCATE UNLINKED FILES STILL REFERENCED
 
Detail Data
JFS2 MAJOR/MINOR DEVICE NUMBER
000A 0009
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/hd1, /home

So using that last recommended action when we use fuser as follows we see :

fuser -dV /home
/home:
inode=12409  size=633888747    fd=1      7274534
inode=12409  size=633888747    fd=1      12648680
inode=12409  size=633888747    fd=1      13762622

This is telling us there are 3 files that have been removed but are being used by an existing process, the PID of which is that last number.

Indeed, checking my process table I could see those three PID’s still exist. In order for the space to be released these processes need to be stopped.

Building Cobbler server using centos63

A little while ago I started a little project to build a cobbler server using Centos 6.3 on a VirtualBox VM to test builds. Here are my notes for that, though they may, of course, be out of date now. Hopefully still useful for reference though.

 

Ingredients for VirtualBox:

  • Minimal Centos 6.4 install.
  • 512M RAM
  • 2 x network cards
  • One is host-based and will have the static IP 192.168.56.102
  • One is bridged to iMac eth0 and will have DHCP enabled
  • 1 x 8G filesystem for the OS
  • 1 x 10G filesystem for /opt/cobber_data
  • Defaults for all else.

Method:

After cloning, edit /etc/udev/rules.d/70-persistent-net.rules deleting the entries for eth0 and eth1 (this parent VM’s NICs) and changing the remaing eth2 and eth3 to eth0 and eth1. Edit /etc/sysconfig/network-scripts/ifcfg-eth0 and ifcfg-eth1 to use the same MAC addresses. Then reboot.

Once restarted and networking is running correctly we need to add the EPEL repo in order to get the cobbler packages.

rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

Make sure the Centos DVD iso is mounted at /cdrom:

mount -o ro /dev/cdrom /cdrom

Create the /opt/cobbler_data filesystem using the second PV:

pvcreate /dev/sdb
vgcreate cobblervg /dev/sdb
lvcreate -L 9G -n cobblerlv cobblervg
mkfs -t ext4 /dev/cobblervg/cobblerlv
mkdir /opt/cobbler_data
add mount to /etc/fstab
mount /opt/cobbler_data
rm -rf /var/www/cobbler/ks_mirror
ln -s /opt/cobbler_data /var/www/cobbler/ks_mirror/

Update the O/S and install cobbler:

yum update
yum -y install cobbler cobbler-web dhcp

Configure cobbler:

sed -i 's/authn_denyall/authn_configfile/g' /etc/cobbler/modules.conf
htdigest /etc/cobbler/users.digest "Cobbler" cobbler

Change the password as required.

More configs:

sed -i 's/server: 127.0.0.1/server: 192.168.56.103/g' /etc/cobbler/settings
sed -i 's/pxe_just_once: 0/pxe_just_once: 1/g' /etc/cobbler/settings
sed -i 's/anamon_enabled: 0/anamon_enabled: 1/g' /etc/cobbler/settings
sed -i 's/manage_dhcp: 0/manage_dhcp: 1/g' /etc/cobbler/settings

Edit /etc/cobbler/dhcp.template for network subnet to administer (if there isn’t already a DHCP server doing this)

sed -i 's/next_server: 127.0.0.1/next_server: 192.168.1.1/g' /etc/cobbler/settings

Stop selinux by editing /etc/selinux/config and change from enforcing to disabled.
Reboot.

Set startups for services:

service start httpd
chkconfig httpd on
service cobblerd start
chkconfig cobblerd on

Start cobbler:

cobbler get-loaders

Install debmirror. pykickstart and cman (ensure CentOS cdrom is mounted):

yum -y install debmirror pykickstart cman

Comment ‘dists’ and ‘arches’ on /etc/debmirror.conf

Save and stop iptables (for testing):

/etc/init.d/iptables save
/etc/init.d/iptables stop
chkconfig iptables off

Create a new encrypted password for the /etc/cobbler/settings_default_password_crypted entry:

openssl passwd -1

Sync and restart Cobbler:

/etc/init.d/iptables save
/etc/init.d/iptables stop
chkconfig iptables off

In /etc/xinetd.d/tftpd change ‘disable’ to ‘no’, then restart with:

service xinetd restart
chkconfig xinetd on

Import the CentOS DVD with (user minus minus for the options here):

cobbler import –name=centos63 –path=/cdrom

Create a new cobbler system called centos63 with the centos63-i386 profile and the target IP address.

Create a new virtualbox guest, boot and F12, select l for LAN and it should get an IP from the DHCP server, then present a cobbler menu for you to choose the profile, and off it goes.

Now you could also add puppet:

yum -y install puppet puppet-server

And if you wanted to add the desktop environment to Centos:

Ensure the CentOS DVD is mounted and the cdrom repo enabled (/etc/yum.repos/cdrom) and do

yum groupinstall basic-desktop desktop-platform x11 fonts

 

Migrating to AWS

So I’ve just migrated my websites to the AWS platform, using an EC2 instance with Amazon Linux and it works nicely.

Monitoring is in place, DNS via Route53 done, and backups running.

 

Hoping to save something like £40 a year on my previous hosting provider, which will be something like a 50% saving.

Time will tell.

I also have my first client running on a WordPress AMI on AWS too.

Nearly time to update my CV!

XIV HAK Commands

Just a couple of quick AIX XIV commands that can be used if you have the HAK (Host Attachment Kit) installed for reference:

Show XIV attached disks:

xiv_devlist

Show which XIV arrays are attached:

xiv_syslist