Damn it has been a stressfull couple of weeks at work, so I have not been around here much. We did an upgrade of our AS/400's OS and then hardware last week, it did not go smoothly and I spent a lot of time over the weekend at work. This system is our primary business system - MRP, finance, etc.
Then Monday and Tuesday we migrated/ upgraded our phone system - we basically went away from a PBX and migrated to a server based system. This new system gives us the ability to use VoIP phones, it uses VoIP connections to our external offices too. So we had to change a lot of our network infrastructure to use VLans and trunking. Big learning curve for me. We went live Tuesday evening - long day that day. Wednesday we found some issues with a few things, basically Avaya did not properly convert our old PBX settings to the new system, or the backup they got had been corrupted somehow. So we had lots of things to backtrack and check. Another long day.
Then Thursday I'm thinking things will go smooth and so I would leave early, since I was taking Friday off as vacation. That would make up for the long days and weekends. But about 8:30 am Thursday a thunderstorm moved through and we had two lightning hits near our building. We lost power. No problem, we had just spent $100k on a generator and department battery backup system a few months ago. Opps! They had not connected our computer room air conditioner to the generator! We got to 90 degrees before an hour was up and had to shut every server and system down. I hung around for a couple hours and then went home. Got a call an hour later that power was back, so I went back to start all systems up.
No phone calls were working in or out of the building. Was it the new system? Was it AT&T's T1 circuit? Our phone vendor tested and said it was AT&T's circuit. I called in the repair, AT&T tested the circuit and said it was OK and closed the ticket, WTF! I called in again. I got a phone call from AT&T at 9 pm last night saying it was OK, I said it was not and to escalate it and get a tech onsite. I let my boss know first thing this morning to expect a tech. When none showed by 10 am I called our phone vendor and got an engineer to show up, since I'm suppostedly on vacation today. My boss and this engineer opened a new ticket (somehow my ticket from the evening before was closed - heh) and finally got AT&T to do an invasive test on the T1. Wow, AT&T said they found a problem on that circuit!
They fixed the problem and my company is now live with phone service. 30 hours without phones (well limited service using our few backup trunk lines) and AT&T was at fault the entire time. Lack of concern, closed repair tickets without notice, BAH! Time for a rebate for lost service and thus lost business. Time for a new phone circuit provider too.
So now it is afternoon on Friday, I hope I can forget about work and post fun stuff again, and maybe play something online
