Performance Issues / fork() errors

[abrist]

You are safe to go ahead and apply those updates. If you let yum update your kernel, it might install one without grsec. FYI, CentOs does not ship with grsec on, but there is a chance that the yum kernel update might pull in the grsec module to replicate the old config. If you do update the kernel (and reboot) and the problem still exists, report back and I will walk you through building a new kernel.

[chrisp]

OK cool,

I've just done: -

Code: Select all

# yum -y update && yum install kernel

Code: Select all

Dependencies Resolved

================================================================================================================================================
 Package                              Arch                        Version                                    Repository                    Size
================================================================================================================================================
Installing:
 kernel                               x86_64                      2.6.32-279.22.1.el6                        updates                       25 M
Installing for dependencies:
 dracut                               noarch                      004-284.el6_3.1                            updates                      112 k
 dracut-kernel                        noarch                      004-284.el6_3.1                            updates                       21 k
 kernel-firmware                      noarch                      2.6.32-279.22.1.el6                        updates                      8.7 M
 libdrm                               x86_64                      2.4.25-2.el6                               base                          65 k
 plymouth                             x86_64                      0.8.3-24.el6.centos                        base                          89 k

Installed:
  kernel.x86_64 0:2.6.32-279.22.1.el6                                                                                                           

All I need to do is fugure out the way to make that kernel the live one...

[abrist]

You should be all set. Just reboot
And then run:

Code: Select all

uname -a 

To verify that you are on the updated one.

[chrisp]

Yeah, that exactly what I'd expected, but the service provider provided kernel persists and the standard one is not in the grub menu.

I had a go at adding it, but now I'm sitting at home trying to rescue it, because gurb is broken

[abrist]

This can be a dicey process if the provided kernel and system rely heavily on initrd. Do they have any documentation for changing the kernel? What is wrong with grub, what error are you receiving?

[chrisp]

Well, the provider pointed me to a forum post regarding Ubuntu, not CentOS, so I was a bit hampered & the fact that I'd never touched grub has not helped either

The system boots to grub & leaves me here: -

vKVM_GRUB_useless.png

The system is using software RAID1, so I've struggled to get anything more than a half booted system, which kernel panics or other crap outcome...

I had a go at using Super Grub Disk to help me repair the problem but it seems like it was unable.

I also tried the CentOS NetInstall/Recovery CD/ISO which showed some promise (where I thought I'd be able to do "chroot /mnt/sysimage ; grub-install /dev/md2" or something), but the system crashes just after it says you're about to be able to do that stuff.

I tried manually going to the CLI (skipping the auto mounting etc), but I couldn't find appropriate instructions to follow in order to dig myself out of the hole.

OK, I've (with a LOT of trial & error) got the system to boot, but now I need to re-install grub, as it's definitely not happy with the status quo...

The original /boot/grub/grub.conf was like this: -

Code: Select all

default=0
timeout=5

    title linux centos6_64
    kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md2  ro
    root (hd0,0)

However, when attempting to run that from the "grub>" prompt like this: -

Code: Select all

grub> kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md2  ro
grub> root (hd0,0)
grub> boot

It doesn't go to plan, as there only seems to be /dev/md127 (/) & /dev/md126 (/boot). When I try to use one of the new/available "md" devices, I get an incomplete boot (probably obviously).

Right, the latest is that I have used the netinstall.iso shell, to individually mount the separate partitions and manually edit the /boot/grub/grub.conf and /etc/fstab for the changed /dev/md* devices.

I booted up again & manually entered the worky details like so: -

Code: Select all

grub> kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md127  ro 1
grub> root (hd0,0)
grub> boot

This allowed me to boot up (into single user mode for now) and from there I tried to reinstll grub with: -

Code: Select all

# grub-install --recheck /dev/hda

vKVM_grub-install.png

But on reboot, it still fails to boot automatically. Now I'm stumped

And this is just in order to get the provider's crappy grsec kernel back into action, not even started to figure out how to boot the one I installed with yum...

[abrist]

Well, the provider pointed me to a forum post regarding Ubuntu, not CentOS, so I was a bit hampered & the fact that I'd never touched grub has not helped either

What was the forum post link? There may be some hints in there.

The original /boot/grub/grub.conf was like this: -

Code: Select all

default=0
timeout=5

    title linux centos6_64
    kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md2  ro
    root (hd0,0)

However, when attempting to run that from the "grub>" prompt like this: -

Code: Select all

grub> kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md2  ro
grub> root (hd0,0)
grub> boot

It doesn't go to plan, as there only seems to be /dev/md127 (/) & /dev/md126 (/boot). When I try to use one of the new/available "md" devices, I get an incomplete boot (probably obviously).

udev/mdadm most likely have assigned a different name for the arrays, this is not uncommon, what is uncommon is the kpanic after changing grub to the new devices.

Right, the latest is that I have used the netinstall.iso shell, to individually mount the separate partitions and manually edit the /boot/grub/grub.conf and /etc/fstab for the changed /dev/md* devices.

I booted up again & manually entered the worky details like so: -

Code: Select all

grub> kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md127  ro 1
grub> root (hd0,0)
grub> boot

This allowed me to boot up (into single user mode for now) and from there I tried to reinstll grub with: -

This is good, that means we should be able to get it working again. Have you tried to boot without the "ro 1" flag?

Code: Select all

# grub-install --recheck /dev/hda

vKVM_grub-install.png

But on reboot, it still fails to boot automatically. Now I'm stumped

If you booted in to read only mode (ro 1), you probably cannot install anything . . .What is the exact error received on reboot? Was there an initrd image associated with this kernel?

And this is just in order to get the provider's crappy grsec kernel back into action, not even started to figure out how to boot the one I installed with yum...

Lets get the box booting first, then we will work on moving away from the provider's kernel.

[chrisp]

I gave up & started afresh (losing all my previous changes, but I'd documented them in our wiki, so I'm already back where I started (mostly), but with a less rubbish kernel)...

This is what I did (mainly using http://evcz.tk/blog/2008/10/21/remote-centos-install/ as inspiration): -

Bypass the wonky default kernel you get when you install an OVH CentOS build as follows...

• Install a blank CentOS 64bit OS (unfortunately this isn't as vanilla as you're lead to believe, hence the next steps)
• Log in to the new system as instructed by the email OVH sends out after install
• Issue the following commands to get the system ready for, then initiate the "real" install: -

Code: Select all

# yum update -y && \
mkdir /centos_install && \
cd /centos_install && \
wget http://mirror.ox.ac.uk/sites/mirror.centos.org/6/os/x86_64/images/pxeboot/initrd.img && \
wget http://mirror.ox.ac.uk/sites/mirror.centos.org/6/os/x86_64/images/pxeboot/vmlinuz && \
cp vmlinuz /boot/vmlinuz.cent.pxe && \
cp initrd.img /boot/initrd.img.cent.pxe && \
IPADDR=`grep IPADDR /etc/sysconfig/network-scripts/ifcfg-eth0 | cut -f2 -d'=' | sed -e 's/"//g'` && \
NETMASK=`grep NETMASK /etc/sysconfig/network-scripts/ifcfg-eth0 | cut -f2 -d'=' | sed -e 's/"//g'` && \
GATEWAY=`grep GATEWAY /etc/sysconfig/network-scripts/ifcfg-eth0 | cut -f2 -d'=' | sed -e 's/"//g'` && \
(
  echo 
  echo 'title CentOS Install (PXE)'
  echo '    root (hd0,0)'
  echo "    kernel /boot/vmlinuz.cent.pxe vnc vncpassword=qkMHItqL headless ip=${IPADDR} netmask=${NETMASK} gateway=${GATEWAY} dns=8.8.8.8 ksdevice=eth0 method=http://mirror.ox.ac.uk/sites/mirror.centos.org/6/os/x86_64/ lang=en_GB keymap=uk"
  echo '    initrd /boot/initrd.img.cent.pxe'
  echo 
) >>/boot/grub/grub.conf && \
sed -i -e s'default=0/default=1/' /boot/grub/grub.conf && \
reboot

Once ${IPADDR} becomes pingable, connect to "${IPADDR}:1" via VNC Viewer & install as desired.

The code above could probably be cleaned up and more elegant, but it serves its purpose and is wiped from the system as part of the install

[scottwilkerson]

With this does everything seem to be running normally?

[chrisp]

I'll let you know tomorrow, when we put it "live"...