This was an interesting exercise in a sense that I have never had a similar filesystem issue in the past. For some unknown reason so far one of the RHEL 5.9 ESXi guests decided to remount /var as read-only. Taking a look at output from dmesg confirmed ext3 had some ordeal going on:

EXT3-fs error (device dm-3): ext3_free_blocks_sb: bit already cleared for block 364592
Aborting journal on device dm-3.
ext3_abort called.
EXT3-fs error (device dm-3): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
EXT3-fs error (device dm-3) in ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device dm-3) in ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device dm-3) in ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device dm-3) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-3) in ext3_truncate: Journal has aborted
EXT3-fs error (device dm-3) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-3) in ext3_orphan_del: Journal has aborted
EXT3-fs error (device dm-3) in ext3_reserve_inode_write: Journal has aborted
__journal_remove_journal_head: freeing b_committed_data

I figured mount -o remount /var would do the job, as usual. But no:

[root@vm-prd-039 ~]# mount -o remount /var
mount: block device /dev/system/var is write-protected, mounting read-only

Interesting, so I decided to take at volume groups and logical volumes:

[root@vm-prd-039 ~]# vgs
  File-based locking initialisation failed.
[root@vm-prd-039 ~]# lvs
  File-based locking initialisation failed.

Fine, /var was read-only and LVM kept lock files there, so LVM could not work them. One more shot:

[root@vm-prd-039 ~]# vgs --ignorelockingfailures
  VG     #PV #LV #SN Attr   VSize   VFree
  system   1   5   0 wz--n-  24.47G     0

That looked OK. I did a quick Google search and it looked like I was going to have to drop the filesystem journal, check the filesystem and then create a new filesystem journal as follows:

umount /dev/mapper/system-var
fsck -y /dev/mapper/system-var
tune2fs -O ^has_journal /dev/mapper/system-var
fsck -f -y /dev/mapper/system-var
tune2fs -j /dev/mapper/system-var

So, I dropped the server into single user mode, killed processes that had open files on /var, unmounted the filesystem and just ran fsck to fix up the filesystem. That’s it.