RainDrop Laboratories Status

The other day, I backed up the other virtual machines, last night I started a backup of agora itself.  This is loading the fileserver which agora uses for /home, and agora does not handle that well (other vms work with the fileserver fine for some reason).  This will be the last of these, in the future, I’ll use a different fileserver for backups, and will shortly be setting up a new mailserver that works better than the primitive one running on agora as well…

I started a backup process early this evening and it seems to be hammering the fileserver.  Looks like I’ll have to use a separate fileserver for that in the future…  Things should recover by tomorrow morning…

The RAID resilver worked this time, so Agora is now running on redundant local disks again.  If you are missing critical mail from the last couple of days *that was delivered*, let me know and I’ll grab it from the other instance.  There are a large number of messages queued up that are going to be catching up…

Running on the cheap little ReadyNAS fileserver just isn’t working.  I’ve shut off incoming mail and will give another go at rebuilding the local mirror on the vm server, then revert to the original agora there.  I will be able to restore any interim mail that people want from the copy after the fact…

I’ve opened up mail processing some to try to get it caught up overnight, so system will be pretty slow for a while…

The file copy finished ahead of schedule: system is coming up now!  Note that agora will get hammered with email, so will be very unresponsive for a while until it gets caught up…  I’ve throttled the mail server as much as I can, hopefully it will still be *able* to get caught up, but the system should at least be useable…

The copies kill performance so much, I’m moving agora now, as I think it will run better off the fileserver while the other copies are happening (though it’ll still be slow).  It will probably take a full 24hours to copy it as it’s large and vmware is not very fast about copying – sorry about that…

Update 9:43am: The largest virtual disk is 52% copied…

Update 1:30pm: 72% copied – on schedule to finish around 8ish

Things are running slow while I’m migrating machines to the file server.  It looks like the resilver failed, so we’re still running on one drive.  More as things develop…

UPDATE: expect agora to be down all night the way things are moving (crawling)…

I figured out which disk was acting up and removed it, so things are back online, however with only half the mirror (which means if that disk goes, all is lost, except agora’s files are backed up independently).  I will try to replace the disk today, though I’m not sure if I’ll have to take the system down to get the mirror to resilver or not.

Update: The disk replacement has been done and is resilvering with no further action needed.  System should be back to normal by tomorrow…

The virtual machine server that agora (and the other virtual machines) run on is getting disk errors that are severely impacting performance.  I’m suspending the virtual machines to move them to the file server, but this will likely be an all-day project…

Update 1pm: the disks are both readable in another machine, I’m not sure what’s going on, still investigating…