2010-11-01 20:37:14

by John 'Warthog9' Hawley

[permalink] [raw]
Subject: Re: [kernel.org users] [KORG] Master / hera upgrade & downtime announcement

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey All,

Just a heads up I'm running a little behind but should be starting shortly.

- - John 'Warthog9' Hawley
Chief Kernel.org Administrator

On 10/29/2010 01:40 PM, J.H. wrote:
> Afternoon Everyone,
>
> Thought I'd give everyone a quick heads up, that since Kernel Summit
> starts on the 1st of November, which means I've got my one chance to do
> system maintenance on master / hera for the year.
>
> I'm expecting to do the upgrade on the 1st itself, likely start time
> will be around Mon Nov 1 19:00:00 UTC 2010, and I'm hoping for the
> entire downtime to take no longer than 2 hours.
>
> Things that are going to happen during the upgrade:
>
> - OS getting upgraded to Fedora 13
> w/ custom kernel to solve XFS and disable IMA
> - Hardware change
> Google and HP have been very generous enough to
> get us a new machine for the master / hera machine.
> The new machine is a dual, quad core Xeon X5550 box
> with 32G of ram. Beyond that it nearly identical to the
> current machine.
>
> The current machine will become a 'live' backup machine
> that will sync semi-continuously, and be available
> should catastrophic failure of master / hera occur.
>
> Think that's the main big changes. I'm already working on the upgrade
> since I have multiple boxes to do this with, and things are looking good
> so far.
>
> As a note the start time is estimated, if there is a major change to the
> start time I will let everyone know.
>
> As always, if you have questions, comments or concerns please e-mail me
> or [email protected]
>
> - John 'Warthog9' Hawley
> Chief Kernel.org Administrator

_______________________________________________
Users mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/users
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkzPJPAACgkQ/E3kyWU9dic+zgCdHf0zF7EXIn6BwmqboBE25Nqr
AJQAoINWVSNuF9lyUFPhkSsdN89DtYpF
=AkT+
-----END PGP SIGNATURE-----


2010-11-01 22:10:57

by John 'Warthog9' Hawley

[permalink] [raw]
Subject: Re: [kernel.org users] [KORG] Master / hera upgrade & downtime announcement

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey All,

Just a heads up the box is back up. There's a few sundry items that
still need to be dealt with and there might still be another reboot in
the next couple of days but I'll send a separate e-mail if/when that has
to occur.

All in all things look good, and for the most part it went smoothly
(well other than the typical disaster that grub throws at me).

If you see anything amiss, please give me a holler or e-mail
[email protected] with details.

Again thanks to Google and HP, specifically Chris DiBona and Bdale
Garbee, for the new box. It is very nice. If you get a chance cat
/proc/meminfo or cat /proc/cpuinfo - you won't be disappointed.

- - John 'Warthog9' Hawley

On 11/01/2010 01:37 PM, J.H. wrote:
> Hey All,
>
> Just a heads up I'm running a little behind but should be starting shortly.
>
> - John 'Warthog9' Hawley
> Chief Kernel.org Administrator
>
> On 10/29/2010 01:40 PM, J.H. wrote:
>> Afternoon Everyone,
>
>> Thought I'd give everyone a quick heads up, that since Kernel Summit
>> starts on the 1st of November, which means I've got my one chance to do
>> system maintenance on master / hera for the year.
>
>> I'm expecting to do the upgrade on the 1st itself, likely start time
>> will be around Mon Nov 1 19:00:00 UTC 2010, and I'm hoping for the
>> entire downtime to take no longer than 2 hours.
>
>> Things that are going to happen during the upgrade:
>
>> - OS getting upgraded to Fedora 13
>> w/ custom kernel to solve XFS and disable IMA
>> - Hardware change
>> Google and HP have been very generous enough to
>> get us a new machine for the master / hera machine.
>> The new machine is a dual, quad core Xeon X5550 box
>> with 32G of ram. Beyond that it nearly identical to the
>> current machine.
>
>> The current machine will become a 'live' backup machine
>> that will sync semi-continuously, and be available
>> should catastrophic failure of master / hera occur.
>
>> Think that's the main big changes. I'm already working on the upgrade
>> since I have multiple boxes to do this with, and things are looking good
>> so far.
>
>> As a note the start time is estimated, if there is a major change to the
>> start time I will let everyone know.
>
>> As always, if you have questions, comments or concerns please e-mail me
>> or [email protected]
>
>> - John 'Warthog9' Hawley
>> Chief Kernel.org Administrator
>
> _______________________________________________
> Users mailing list
> [email protected]
> http://linux.kernel.org/mailman/listinfo/users

_______________________________________________
Users mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/users
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkzPOqEACgkQ/E3kyWU9dicAzQCfdEeFOo5hUUQ5ZSIXePAGR47U
YEgAn3sdNcSfeDo2gc6aOta8zcBlOzDA
=o0kl
-----END PGP SIGNATURE-----

2010-11-01 23:18:44

by John 'Warthog9' Hawley

[permalink] [raw]
Subject: Re: [kernel.org users] [KORG] Master / hera upgrade & downtime announcement

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ergh,

Heads up apparently the server lied to me and claimed the drives were
fully in sync, when they were not.

A couple of people have caught that the last sync that occurred
apparently was the 29th.

I'm going to do a full resync of /pub - this will squash anything that
was there with what was present around 1900 UTC on Nov 1st. Home dirs,
mail, etc are going to have to get dealt with on a per-case basis.

I might just setup an NFS mount across the two machines and let people
snag what they need on their own with respect to that.

Right now I'm working on bringing the old box back up so we can get at
the data.

Just a heads up, stop by #korg on LinuxNet or #kernel.org on Freenode if
you need to corner me on something *NOWish*

Sorry about that, I did double check that stuff was in sync, so I'm not
sure what happened there.

- - John 'Warthog9' Hawley

On 11/01/2010 03:09 PM, J.H. wrote:
> Hey All,
>
> Just a heads up the box is back up. There's a few sundry items that
> still need to be dealt with and there might still be another reboot in
> the next couple of days but I'll send a separate e-mail if/when that has
> to occur.
>
> All in all things look good, and for the most part it went smoothly
> (well other than the typical disaster that grub throws at me).
>
> If you see anything amiss, please give me a holler or e-mail
> [email protected] with details.
>
> Again thanks to Google and HP, specifically Chris DiBona and Bdale
> Garbee, for the new box. It is very nice. If you get a chance cat
> /proc/meminfo or cat /proc/cpuinfo - you won't be disappointed.
>
> - John 'Warthog9' Hawley
>
> On 11/01/2010 01:37 PM, J.H. wrote:
>> Hey All,
>
>> Just a heads up I'm running a little behind but should be starting shortly.
>
>> - John 'Warthog9' Hawley
>> Chief Kernel.org Administrator
>
>> On 10/29/2010 01:40 PM, J.H. wrote:
>>> Afternoon Everyone,
>
>>> Thought I'd give everyone a quick heads up, that since Kernel Summit
>>> starts on the 1st of November, which means I've got my one chance to do
>>> system maintenance on master / hera for the year.
>
>>> I'm expecting to do the upgrade on the 1st itself, likely start time
>>> will be around Mon Nov 1 19:00:00 UTC 2010, and I'm hoping for the
>>> entire downtime to take no longer than 2 hours.
>
>>> Things that are going to happen during the upgrade:
>
>>> - OS getting upgraded to Fedora 13
>>> w/ custom kernel to solve XFS and disable IMA
>>> - Hardware change
>>> Google and HP have been very generous enough to
>>> get us a new machine for the master / hera machine.
>>> The new machine is a dual, quad core Xeon X5550 box
>>> with 32G of ram. Beyond that it nearly identical to the
>>> current machine.
>
>>> The current machine will become a 'live' backup machine
>>> that will sync semi-continuously, and be available
>>> should catastrophic failure of master / hera occur.
>
>>> Think that's the main big changes. I'm already working on the upgrade
>>> since I have multiple boxes to do this with, and things are looking good
>>> so far.
>
>>> As a note the start time is estimated, if there is a major change to the
>>> start time I will let everyone know.
>
>>> As always, if you have questions, comments or concerns please e-mail me
>>> or [email protected]
>
>>> - John 'Warthog9' Hawley
>>> Chief Kernel.org Administrator
>
>> _______________________________________________
>> Users mailing list
>> [email protected]
>> http://linux.kernel.org/mailman/listinfo/users
>
> _______________________________________________
> Users mailing list
> [email protected]
> http://linux.kernel.org/mailman/listinfo/users

_______________________________________________
Users mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/users
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkzPSrMACgkQ/E3kyWU9dicreQCghOiKTEnmeAE0iVQKrYX1FRHJ
+3AAn0G1XAIl+eefZ2il7wPvyeNxzmz0
=Z//P
-----END PGP SIGNATURE-----

2010-11-02 18:19:12

by John 'Warthog9' Hawley

[permalink] [raw]
Subject: Re: [kernel.org users] [KORG] Master / hera upgrade & downtime announcement

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Latest update

> I'm going to do a full resync of /pub - this will squash anything that
> was there with what was present around 1900 UTC on Nov 1st. Home dirs,
> mail, etc are going to have to get dealt with on a per-case basis.

/pub was reset last night, which basically means anything that may have
been pushed from about 19:00 Nov 1 UTC to 02:56 Nov 2 UTC could be
squashed and will need to be re-pushed, but this should clean up the
vast majority of the issues there were present there.

This will likely mean a few more extra tip e-mails for those of you on
that mailing list (again sorry about that), and possibly some other
e-mails from various cron scripts that individual users are running and
maintaining.

> I might just setup an NFS mount across the two machines and let people
> snag what they need on their own with respect to that.

I've gone ahead and created /home/restore - it has have a copy of the
home directory as it existed around 19:00 Nov 1 UTC. I will garauntee
it being around until Nov 30th. I will need to delete it sometime after
that, mainly because of the backup bloat it will cause. If you made
changes to your home directory between Sept. 29th and Nov 1, you will
need to deal with the merging of the data manually.

With respect to those users who use IMAP against the machine I've copied
your INBOX mail spools to /home/restore/MAILSPOOL. The easiest way to
deal with merging this in with your current inbox is toe copy your spool
to your ~/mail directory, load up thunderbird and copy all the mail from
the old inbox to your current inbox and then use something like "Remove
Duplicate Messages (alternate)" to eliminate the duplicates.

That should be everything. I'm going to keep poking around trying to
find anything else that's off kilter, but I haven't seen any new reports
of anything else amiss yet and things seem to be working otherwise. So
if you see something amiss, please let me know ASAP.

- - John 'Warthog9' Hawley
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkzQVhAACgkQ/E3kyWU9dic1ZwCeL0IdwITS2lYcmQhmOK/kevk5
l2wAnAzfynrADpgv+I7KL0rvQQU8Fmxe
=6g1w
-----END PGP SIGNATURE-----