Downtime Notice (Wednesday Night)

[Andy R]

I just wanted to let you all know that we are going to have to turn off the site for up to a couple of hours tonight while the server admins turn on replication on our database servers.

I know the servers have been spotty a few times in the last couple of weeks with the 500 Errors. I am working to make sure that things stabilize and are as fast as possible. We should be in very good shape soon.

Thanks for your patience and understanding during this process!

Regards,

Andy

 

[Nords]

for up to a couple of hours tonight

Andy, if they give you any indication of their schedule, could you tell us what time zone this is in?

 

[Andy R]

Nords,

When I hear back I will post all the info. The issue is I am communicating with them via the help desk and the admin that is going to help tonight is not online right now. I was going to wait to do this a few days from now so I could make more precise plans but I also have to change a bunch of IP stuff in the load balancer (different admin for that) and he is available tonight. So we can do both things in the same window.

I learned a lot in the last two weeks about more complex server setups like this and decided I did a few things in a way that could be cleaner for easier management moving forward. So that is why I have to make the IP changes. The DB is to have Replication on our database server which will spread the reads to one server and the writes to the other. It will also give us a hot backup in the even the master database server fails.

I was hoping to have all this done by now but it's taking longer for me to understand everything and get the pieces setup just right. If any of you ever have to setup a system like this, shoot me an email and I will share with you what I have learned.

Andy

 

[Andy R]

Well it looks like is going to be earlier then expected because the DBA is only available at 7:30 EST. So that means the site will be down in about 1.5 hours from now.

Sorry about the late notice, it's something that really needs to get done.

Andy

 

[Andy R]

It was not the prettiest down time, my apologies. The database replication went much faster then expect and is now in running. The load balancing setup took a bit longer then anticipated (which caused some server cannot be found messages) but it is now all setup in a clean manner. As I mentioned above, I have learned a lot in the last couple of weeks so adding servers moving forward (as needed) is going to be much easier.

At the moment we are limping by on one server as the second huge web server did not come back online after changing the IPs to reflect the new load balanced IPs. The admins are working on this and should have the server back up shortly.