Page 3 of 5

Re: Performance Issues / fork() errors

Posted: Thu Feb 14, 2013 11:05 am
by abrist
You are safe to go ahead and apply those updates. If you let yum update your kernel, it might install one without grsec. FYI, CentOs does not ship with grsec on, but there is a chance that the yum kernel update might pull in the grsec module to replicate the old config. If you do update the kernel (and reboot) and the problem still exists, report back and I will walk you through building a new kernel.

Re: Performance Issues / fork() errors

Posted: Thu Feb 14, 2013 3:30 pm
by chrisp
OK cool,

I've just done: -

Code: Select all

# yum -y update && yum install kernel

Code: Select all

Dependencies Resolved

================================================================================================================================================
 Package                              Arch                        Version                                    Repository                    Size
================================================================================================================================================
Installing:
 kernel                               x86_64                      2.6.32-279.22.1.el6                        updates                       25 M
Installing for dependencies:
 dracut                               noarch                      004-284.el6_3.1                            updates                      112 k
 dracut-kernel                        noarch                      004-284.el6_3.1                            updates                       21 k
 kernel-firmware                      noarch                      2.6.32-279.22.1.el6                        updates                      8.7 M
 libdrm                               x86_64                      2.4.25-2.el6                               base                          65 k
 plymouth                             x86_64                      0.8.3-24.el6.centos                        base                          89 k

Installed:
  kernel.x86_64 0:2.6.32-279.22.1.el6                                                                                                           
All I need to do is fugure out the way to make that kernel the live one...

Re: Performance Issues / fork() errors

Posted: Thu Feb 14, 2013 4:00 pm
by abrist
You should be all set. Just reboot :)
And then run:

Code: Select all

uname -a 
To verify that you are on the updated one.

Re: Performance Issues / fork() errors

Posted: Thu Feb 14, 2013 4:08 pm
by chrisp
Yeah, that exactly what I'd expected, but the service provider provided kernel persists and the standard one is not in the grub menu.

I had a go at adding it, but now I'm sitting at home trying to rescue it, because gurb is broken :(

Re: Performance Issues / fork() errors

Posted: Fri Feb 15, 2013 10:29 am
by abrist
This can be a dicey process if the provided kernel and system rely heavily on initrd. Do they have any documentation for changing the kernel? What is wrong with grub, what error are you receiving?

Re: Performance Issues / fork() errors

Posted: Fri Feb 15, 2013 4:25 pm
by chrisp
Well, the provider pointed me to a forum post regarding Ubuntu, not CentOS, so I was a bit hampered & the fact that I'd never touched grub has not helped either :(

The system boots to grub & leaves me here: -
vKVM_GRUB_useless.png
The system is using software RAID1, so I've struggled to get anything more than a half booted system, which kernel panics or other crap outcome...

I had a go at using Super Grub Disk to help me repair the problem but it seems like it was unable.

I also tried the CentOS NetInstall/Recovery CD/ISO which showed some promise (where I thought I'd be able to do "chroot /mnt/sysimage ; grub-install /dev/md2" or something), but the system crashes just after it says you're about to be able to do that stuff.

I tried manually going to the CLI (skipping the auto mounting etc), but I couldn't find appropriate instructions to follow in order to dig myself out of the hole.

OK, I've (with a LOT of trial & error) got the system to boot, but now I need to re-install grub, as it's definitely not happy with the status quo...

The original /boot/grub/grub.conf was like this: -

Code: Select all

default=0
timeout=5

    title linux centos6_64
    kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md2  ro
    root (hd0,0)
However, when attempting to run that from the "grub>" prompt like this: -

Code: Select all

grub> kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md2  ro
grub> root (hd0,0)
grub> boot
It doesn't go to plan, as there only seems to be /dev/md127 (/) & /dev/md126 (/boot). When I try to use one of the new/available "md" devices, I get an incomplete boot (probably obviously).

Right, the latest is that I have used the netinstall.iso shell, to individually mount the separate partitions and manually edit the /boot/grub/grub.conf and /etc/fstab for the changed /dev/md* devices.

I booted up again & manually entered the worky details like so: -

Code: Select all

grub> kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md127  ro 1
grub> root (hd0,0)
grub> boot
This allowed me to boot up (into single user mode for now) and from there I tried to reinstll grub with: -

Code: Select all

# grub-install --recheck /dev/hda
vKVM_grub-install.png
But on reboot, it still fails to boot automatically. Now I'm stumped :(

And this is just in order to get the provider's crappy grsec kernel back into action, not even started to figure out how to boot the one I installed with yum...

Re: Performance Issues / fork() errors

Posted: Mon Feb 18, 2013 11:01 am
by abrist
chrisp wrote:Well, the provider pointed me to a forum post regarding Ubuntu, not CentOS, so I was a bit hampered & the fact that I'd never touched grub has not helped either :(
What was the forum post link? There may be some hints in there.
The original /boot/grub/grub.conf was like this: -

Code: Select all

default=0
timeout=5

    title linux centos6_64
    kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md2  ro
    root (hd0,0)
However, when attempting to run that from the "grub>" prompt like this: -

Code: Select all

grub> kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md2  ro
grub> root (hd0,0)
grub> boot
It doesn't go to plan, as there only seems to be /dev/md127 (/) & /dev/md126 (/boot). When I try to use one of the new/available "md" devices, I get an incomplete boot (probably obviously).
udev/mdadm most likely have assigned a different name for the arrays, this is not uncommon, what is uncommon is the kpanic after changing grub to the new devices.
Right, the latest is that I have used the netinstall.iso shell, to individually mount the separate partitions and manually edit the /boot/grub/grub.conf and /etc/fstab for the changed /dev/md* devices.

I booted up again & manually entered the worky details like so: -

Code: Select all

grub> kernel /bzImage-3.2.13-xxxx-grs-ipv6-64 root=/dev/md127  ro 1
grub> root (hd0,0)
grub> boot
This allowed me to boot up (into single user mode for now) and from there I tried to reinstll grub with: -
This is good, that means we should be able to get it working again. Have you tried to boot without the "ro 1" flag?

Code: Select all

# grub-install --recheck /dev/hda
vKVM_grub-install.png
But on reboot, it still fails to boot automatically. Now I'm stumped :(
If you booted in to read only mode (ro 1), you probably cannot install anything . . .What is the exact error received on reboot? Was there an initrd image associated with this kernel?
And this is just in order to get the provider's crappy grsec kernel back into action, not even started to figure out how to boot the one I installed with yum...
Lets get the box booting first, then we will work on moving away from the provider's kernel.

Re: Performance Issues / fork() errors

Posted: Mon Feb 18, 2013 11:13 am
by chrisp
I gave up & started afresh (losing all my previous changes, but I'd documented them in our wiki, so I'm already back where I started (mostly), but with a less rubbish kernel)...

This is what I did (mainly using http://evcz.tk/blog/2008/10/21/remote-centos-install/ as inspiration): -

Bypass the wonky default kernel you get when you install an OVH CentOS build as follows...
  • Install a blank CentOS 64bit OS (unfortunately this isn't as vanilla as you're lead to believe, hence the next steps)
  • Log in to the new system as instructed by the email OVH sends out after install
  • Issue the following commands to get the system ready for, then initiate the "real" install: -

Code: Select all

# yum update -y && \
mkdir /centos_install && \
cd /centos_install && \
wget http://mirror.ox.ac.uk/sites/mirror.centos.org/6/os/x86_64/images/pxeboot/initrd.img && \
wget http://mirror.ox.ac.uk/sites/mirror.centos.org/6/os/x86_64/images/pxeboot/vmlinuz && \
cp vmlinuz /boot/vmlinuz.cent.pxe && \
cp initrd.img /boot/initrd.img.cent.pxe && \
IPADDR=`grep IPADDR /etc/sysconfig/network-scripts/ifcfg-eth0 | cut -f2 -d'=' | sed -e 's/"//g'` && \
NETMASK=`grep NETMASK /etc/sysconfig/network-scripts/ifcfg-eth0 | cut -f2 -d'=' | sed -e 's/"//g'` && \
GATEWAY=`grep GATEWAY /etc/sysconfig/network-scripts/ifcfg-eth0 | cut -f2 -d'=' | sed -e 's/"//g'` && \
(
  echo 
  echo 'title CentOS Install (PXE)'
  echo '    root (hd0,0)'
  echo "    kernel /boot/vmlinuz.cent.pxe vnc vncpassword=qkMHItqL headless ip=${IPADDR} netmask=${NETMASK} gateway=${GATEWAY} dns=8.8.8.8 ksdevice=eth0 method=http://mirror.ox.ac.uk/sites/mirror.centos.org/6/os/x86_64/ lang=en_GB keymap=uk"
  echo '    initrd /boot/initrd.img.cent.pxe'
  echo 
) >>/boot/grub/grub.conf && \
sed -i -e s'default=0/default=1/' /boot/grub/grub.conf && \
reboot
Once ${IPADDR} becomes pingable, connect to "${IPADDR}:1" via VNC Viewer & install as desired.

The code above could probably be cleaned up and more elegant, but it serves its purpose and is wiped from the system as part of the install ;)

Re: Performance Issues / fork() errors

Posted: Mon Feb 18, 2013 11:20 am
by scottwilkerson
With this does everything seem to be running normally?

Re: Performance Issues / fork() errors

Posted: Mon Feb 18, 2013 12:38 pm
by chrisp
I'll let you know tomorrow, when we put it "live"...