RainDrop Laboratories Status

Apparently, my current colocation provider missed sending me mail in December, resulting in about a week’s notice that he’s moving and dropping colocation services because of a substantial rate hike.  He was reselling service from the data center we’re in, and their direct prices do turn out to be prohibitive, so I will have to move as well.  The current plan is to move the equipment to Peak in Corvallis, which will make it much more accessible and easier to manage, but means that all services will be down for most of Saturday afternoon.  These are tentative plans, so check back here through Friday for the latest info…

There will be a brief network outage this afternoon while I upgrade
the router firmware in preparation for supporting IPv6 (a new Internet
protocol).  There *may* be a more extended outage for agora itself,
depending on if a tool to copy it into a virtual machine works or not.
This may be a way of getting it off the questionable hardware it’s
running on currently…if it works…

Another bullet dodged — it just rebooted…

I’ve got the colo manager checking it out, but it is likely the complete hardware failure it’s been threatening to do, which means it will be late tonight at the earliest before it’s back up…

Agora seems to have mostly stabilized, and with the weather looking bad for travelling, I’m working getting things moved to the virtual machines instead of moving the actual hardware, as that’s the real end goal. I’m still limited by the memory in the vmhost, but if I can get things built, it will be a big step forward…

I’m starting by getting /home moved to the vmhost machine, where it will become an nfs mount (i.e. it can be shared between machines), and getting incoming mail setup on vmail.rdrop.com.

I suspect a power outage to the rack, as it was maxed out when I installed it. That issue is being worked on, but people are in the process of bringing things back up…

Update: Yup, it was power in the rack and will be addressed tomorrow.  Agora is having hardware issues aggravating things (and probably causing the crashes this weekend), and will be moved into new hardware this weekend.  Hopefully it won’t force me to do so sooner!

Saturday afternoon, Jan 12, 2008, the Raindrop equipment will be moved into a more accessible rack and the memory in the virtual machine server will be upgraded to accomodate more vm’s.  Everything should be back up by late afternoon or early evening.

Since September, the spam load on agora has jumped dramatically, with the result that the mail system has been faltering. To deal with the onslaught, a couple of changes have been made:

1. I’ve made arrangements with Peak Internet, the ISP I work full time for, to use their spam filter, a Barracuda Spam Firewall 600 (actually, a pair clustered to look like one). This does a very effective job, with only a few annoyances. For more info, there is some documentation, and you can see the current statistics.

   One of the things the barracuda is bad about is that users can’t manage their passwords.  It wouldn’t be good to flood Peak with calls to set them to something known (they shouldn’t be called about anything rdrop related), so I’ll handle it as needed.  It actually shouldn’t be necessary: When you get your daily quarantine notice (assuming you have something quarantined, but that’s almost a given), there will be a “click here” link at the bottom of the message.  That will, for about a day, work to get you into it and you can twiddle things from there.

   Note that because the barracuda is a separate, independent, system, it can’t tell what addresses are aliased to the same mailbox, so each email address you use will be managed separately.  That’s one of the other things I plan to fix when I get a new one built… (I plan to build my own eventually to fix some of the shortcomings of the Barracuda).

2. To keep spammers from going directly to the mail server and bypassing the spam filter, as well as to start the move to a new, updated, mail server, I’ve put a block so that only the spam filter can send mail into the rdrop mail servers. For similar reasons, the new dialup service blocks standard mail access, so users need to update their outgoing mail server configuration. I have documentation for configuring Outlook to help, though I still need to update it: for now, change references to smtp.rdrop.com to vmail.rdrop.com, and port 25 to port 465. You will also want to visit a page to install a new certificate so your mail program will trust the new mail server. NOTE: If you have trouble logging into vmail.rdrop.com, go to the User Account Management page and reset your password. Some accounts have an old format password that the new system doesn’t understand, and this will update it to the new format. Be sure to use a good password — spammers are getting really sophisticated and cracking accounts so they can send authenticated spam using hijacked accounts.

I’ll be moving incoming mail to vmail early next year, but I’ll have updated docs and separate info for that change…

As previously mentioned, dialup usage has dropped off to the point where it’s no longer practical to continue offering dialup service (all other services will continue unchanged).  As a result, I’ve made arrangements with Peak Internet, the ISP I work full time at, to pickup my dialup users.  They will continue accounts as-is: accounts will expire at the same time, and the rates will be the same.  Only the dialup service is moving; in order to minimize disruption, I’ll continue email service at rdrop.com as long as you have your Peak account.  You should already have the new connect info, if not, use the support form to let me know and I’ll get you all setup.

There’s a number of updates happening, long overdue; here’s an overview:

1. There are only 50 people actively using dialups, and dropping, and that isn’t covering the $600 phone bill, so the dialups are being handed off to Peak Internet, the ISP I work for for a day job in Corvallis, OR (they have Portland numbers).
2. The level of spam took a dramatic increase in September, causing things to break on agora all too often.  As a result, I made arrangements with Peak to route Raindrop’s mail through their spam filter.  I’m also making other related changes to keep spammers from bypassing the spam filter (this is why pop/imap clients need to use vmail.rdrop.com:587 to send mail).
3. Agora is several years out of date, and I’m making progress on getting new versions of the various services up and running.  webmail has already moved to new software, vmail for outgoing is the next step

After that, I’ll be moving mailboxes to a new imap server on it (that, unfortunately, will be another painful transition, but should allow for transparent upgrades afterwards).  After that, I’ll be moving web sites to a new server (that *should* be mostly transparent), and finally a new shell server for shell users.  Once I get everything moved off the old server, I can rebuild it, and make some more improvements behind the scenes to improve things, but after the initial transitions, which should be done by this spring, everything should be pretty transparent.