It seems the problem was the zfs fileserver got a little too full and performance went to hell. Freed up some space and things are running normally again…
Virtual infrastructure out to lunch
vmware is having serious problems, still working on it…
Agora is growing increasingly unstable
The filesystem on the agora server is degrading and the system is becoming unstable. It is recommended that you make sure anything you want is copied elsewhere. I’m not sure when I’m going to get the chance to get things moved to a new server.
Services restored
All services should be operational now.
UPS Failure
Data center UPS failure – services will resume later this morning
Reboot taking a long time
A reboot of agora is taking a long time while it cleans the filesystem…
Services restored
It turned out the fileserver wasn’t down, but apparently had a glitch last night the upset most of the vms…
Fileserver down
The primary fileserver is down, heading in to take a look at it…
System recovered
It looks like the corruption was mostly /tmp and the system is back in operation.
File system corruption
agora’s filesystem corrupted last night; it’s trying to repair, but I may have to revert to a snapshot…