If you didn’t know, this has been a tumultuous week for clients of Westhost, my internet service provider. Their Primary data center is located in Utah and they share that space with a sister brand VPS.net. The datacenter is a Tier IV center managed by Consonus. Saturday afternoon there was a yearly fire equipment/alarm/suppression system test. The third party technician failed to follow procedures and one actuator remained on the output system for the gas that is designed to suppress fires in the building. When the system was re-armed there was a sudden release of the gaseous fire suppressant. At that same moment hundreds of hard drives died. Now, Inergen is what was used and the gases themselves shouldn’t be a problem. In this case, and judging from what I’ve read, the problem was with the sudden and intense change in air pressure caused by the release. That point is somewhat moot though, the end result is hundreds of dead and damaged hard drives.
Tag: outage
-
Strange Outage
We’re back up after a very strange outage. The 500 internal server error was because our .htaccess file was sideways.
What I mean by sideways is that part of a line had been truncated which led to an incorrect htaccess rule and all sorts of error pages thrown up.
Interestingly the only thing happening prior to the first errors was a page edit here in WordPress. I updated my dd-wrt page last night right before the errors started getting tossed up. Is it possible that wordpress 2.6.3 stomped on my htaccess??? Don’t know, but we’re back up now.
-
Your own wikipedia….
I’ve made quite a bit of use out of the wikipedia in recent years. I know it has it’s flaws (I’ve run across some first hand), but I’ve found typos in textbooks as well. However that doesn’t mean that it can’t be a very useful reference. In fact, in some of my browsing I’ve gone through the spanish language version of the wikipedia putting some of my spanish reading skills to the test. Anyway, in the last couple days I became curious for various reasons about actually downloading a copy and installing the wikipedia locally. Now, I know one of the benefits of the wikipedia is that it’s collaborative and this way I’ll miss out on current and changing/improving/updating articles. But I can see some reasons to want to have a “snapshot”.
-
Apache2 not starting because of ssl_scache file
I mentioned this a while back, but I didn’t go into much detail on a long term solution. Let me re-set the situation. Linux server running apache2. It’s Mandrake (now Mandriva) (an older version.) When the system has suffered abrupt outage (power loss). Everything starts up normally with the exception of httpd2. It claims that it’s running but gives an error message. (For reference here’s the old article. Basically when you try to manually restart you see..
Cannot allocate shared memory: (17)File exists apache
-
Dejavu for worldnic dns servers….
It’s been a long day out and I didn’t have but a few moments to check mail this afternoon. I did happen to check stats and saw a lot of people visiting an old article (November 28th) on a Worldnic DNS server outage. I thought it was a bit odd and this evening, now I know why. Incidents.org has some details on the outage.
-
Bellsouth mail.lig.bellsouth.net server phasing out?
I haven’t had much time to look into this, but one of the mailservers I administer is typically configured to relay through mail.lig.bellsouth.net, with mail.averyjparker.com as a fallback. Sometime overnight, mail.averyjparker.com started getting heavy use and on checking this morning was getting all of the outbound traffic. So, I did a bit of investigation mail.lig.bellsouth.net is no longer found and I’ve switched the configuration to mail.bellsouth.net and all is churning along well.
-
Connectivity issues
Our ISP here was out this afternoon. (Cable and internet) for a bit before I had to run to an appointment, so I got a bit behind in entries. It’s interesting though, Charter has been really pushing their new telephone service lately. Which is all well and good, but I’ve thought many times, if I were to get phone service from Charter how many times a year I’d be without phone service? How would I have called to report the outage today? (Carrier pigeon) – cell phone is the expected answer I’m sure, but….
-
Worldnic DNS server outage teaches lesson…
Incidents.org has a post on a DNS server outage for Worldnic. Which effects a number of Network Solutions customers. Apparently they’re aware of the problem and are working on a fix. It doesn’t affect EVERY Network Solutions customer, there are some specifics…
To clarify the impact to the casual reader:
Not all customers of Network Solutions are affected.
No root or TLD servers are known to reside on these machines.
It’s “just” individual domains that are affected, but it might be a lot of them.
Only domains that have all their namervers on these machines will have significant impact.
-
Cogent cut takes down major internet backbone
Cogent has suffered a major outage of one of their main internet backbone connections. It appears that this link is having a big affect on the “internet health”. Comcast seems to be relatively hard hit with connectivity issues from this. It appears that the Northeast US and Southeast may have sporadic outages depending on the ISP. I’ve had a few peculiar net experiences this morning, but I’m not sure if this connectivity problem is what I’ve seen.
-
Server Outage
There is one domain that I host which are on a server that has had some major “issues”, backups are in the process of being reloaded and the current estimate is that they will be back up in about two and a half hours. It appears as though there were some data corruption problems, possibly with the first attempt to restore from backup. It appears that the backup restored will be from midnight the night before the outage.