RainDrop Laboratories Status

Network outage

root — Thu, 25 Jun 2009 18:13:48 +0000

Peak replaced its core routers last night, and in the process, the route to my network got lost due to an obscure configuration issue. It should be fixed shortly…

New shell.rdrop.com nearly ready

root — Mon, 15 Jun 2009 01:27:42 +0000

I’m finishing up the next step in the process of retiring the antique system that is the current “agora”: a new shell server. It can be accessed at shell2.rdrop.com, or at shell.ipv6.rdrop.com (or, shell6.rdrop.com for easier typing). Yes, Peak has IPv6 connectivity, and I’ve started adding connections to the various systems. When you login to the new system, you can “ping6 ipv6.google.com” and be doing the latest and greatest internetworking!

When might you be able to do that? Well, technically, now. Logins are enabled, however, I’ve just started copying /home from agora to the file server that the shell server now uses, so you won’t have a home directory, or all of your files, until that completes. And many of the tools you use are probably not installed yet, and it’s a Centos system instead of FreeBSD, so it’ll be a little different. Getting logins to work with the database backend has been the hard part, however, and now that that’s operational, the rest is nearly trivial…

So the bravest of you can give it a whirl now, and once the files are all copied in a day or so, I’ll get some of the basic addons installed and then it’ll be in a proper Beta state for real testing before making it the production shell server…

vm server upgrade complete

root — Sun, 08 Feb 2009 04:04:54 +0000

There were some issues getting the equipment into the rack, but managed a work around, copied vm files and up it all came. Only problem is an aggravated time issue with the mail server; I’m building a new version with a more forgiving setting so it’ll be happy again… All services should be operational now…

vm services stopping for a couple hours

root — Sun, 08 Feb 2009 01:12:45 +0000

Shutting down services for hardware swap, should be back up by 7, hopefully…

vmserver upgrade postponed - Feb 7

root — Fri, 30 Jan 2009 07:49:37 +0000

I’ve postponed the vmserver upgrade a week in order to get a couple more pieces of hardware for it. I’ll be working on the new services on it instead, so no disruptions this weekend…

Next phase: vmserver upgrade

root — Thu, 29 Jan 2009 03:03:36 +0000

Saturday afternoon, I’ll be migrating the virtual machines to a new host machine. This requires format conversion, which takes about 1/2hr-1hr for each one (but each one will be up until its conversion process begins, and will then come back up immediately). The main impact will be a nameserver and outgoing mail. Once this is done, I will be able to move additional services off the old agora and onto new vms.

For the technically inclined, I’m upgrading from a 2.8Ghz P4/Xeon w/3G ram running vmware server to a 2.83Ghz Q9550 quad-core w/8G ram running ESXi. The new vms will reside on an opensolaris zfs/nfs server as well. The only storage on the vmhost is a 32G ssd.

Performance and emu update

root — Mon, 26 Jan 2009 20:54:01 +0000

There was another problem with the nameserver, relating to earlier work on supporting secure dns. This caused the problem with emu, and probably the performance problems some people have been seeing. Note that while emu is now working, it will be retired when I get the new mail server built…

Emumail retirement

root — Mon, 26 Jan 2009 18:36:43 +0000

Emumail seems to have lost its mind in the move; I’m not sure how, as nothing inside the network should be able to tell that it didn’t just reboot. In any case, it’s been on its last legs for some time, and this is a good opportunity to retire it for good, and good incentive for me to get the new mail server going…

Data center move complete

root — Sun, 25 Jan 2009 03:58:46 +0000

All went about as smoothly as could be expected. Agora is a little finicky about its power up sequence these days, and the virtual machines noticed things had changed (not sure what though, they shouldn’t have seen any change at all!), and so needed some comforting to start up, but all are happy now. I will be finalizing a new virtual machine host over the next week, and plan to move the vms to it next weekend. Once that’s done, I’ll probably move the web server from agora to its own vm next, followed by a new shell server and finally mail, which should allow the retirement of the antique hardware finally!

Data center move for Saturday, January 24, confirmed

root — Thu, 22 Jan 2009 23:09:35 +0000

I’m getting a deal I can’t refuse at Peak, so at about 2pm Saturday, Jan 24, most Raindrop services will go down for a few hours while the servers are moved 90 miles south. Everything *should* be back up by 5pm, but these things never go according to plan… The secondary nameserver is separate, as is status.rdrop.com, so DNS serving for hosted domains will continue, and you’ll be able to get updates if there are unexpected events.

As always, any questions to support@rdrop.com, or, while mail is down, post as comments here…