RainDrop Laboratories Status

The copy off local storage was taking too long, which among other things implies that there really is something wrong with the new ssd or file server, so I aborted the attempt to upgrade vmware and flopped it over to the replicas on the other fileserver.  Things seem to be running normally, though this is a disk based system, so it has its limits.  Things seem to be back to normal though…

It’s never easy: it looks like the esxi update is going to trash the local storage, so I’m having to backup up a couple things I want to save first, which will take about an hour…

Since the old readynas server is performing normally, I’m copying a snapshot of agora’s vm files over to it and when that’s done, will restart agora there.  Not speedy, but should be useable.

I have narrowed the problem down to nfs performance between both vms and vmware itself, and the freebsd file server (as in 40KB/sec!)…

Filesystem check ran much faster than expected, agora coming back up now.  Bad news is that probably won’t have made any difference…

I’m running a filesystem check that will fix some problems; not sure if they’re affecting performance, but I’m grasping at straws…the system probably won’t be back up until late tonight

There is something doing a lot of writes that appears to be causing vm performance issues, and I’m having trouble finding out what it is.  At this point, I’m going to investigate moving rdrop mail to Peak and start phasing out web hosting (Peak offers as good or better rates on a better platform).

The vm host fought me a bit, but everything should be operating normally again.

The VM server has gotten terribly confused in the ordeal, and it looks like I’m going to have to go in and hard reset it. Hopefully things will be back up 9ish…