Uncategorized

Microsoft Azure Outage

By September 4, 2018November 10th, 2020No Comments
2:35pm MDT — RESOLVED:

Our Engineering team has been hard at work testing the system, and we’re happy to report that CampMinder and CampInTouch are available again.

We do expect initial slowness as we’re spinning up servers to normal levels. Normal system speeds should return within the next few hours.

 

1:25pm MDT — UPDATE:

We have no additional information to share, but we continue to see improvement to our Azure services. Our Engineering team continues to monitor the situation and work to prepare the system to go live once Microsoft gives the “all clear”.

We will provide another update a 3:00 pm MDT, or sooner, if warranted.

 

11:03am MDT — UPDATE:

While we have no additional information from Azure, we are continuing to see our Azure services come back to life which is encouraging. We still do not have a reliable ETA to offer you.

Once all our servers and services come back to life, we will test our system to ensure it’s stable, and we will wait for an OK from Microsoft to go live. As eager as we are to restore service, we want to be appropriately cautious. Please continue to check here for regular updates.

Thank you for your patience and understanding through this ordeal. It has been incredibly heartwarming for us to have received a number of messages of care and support from so many of you. We love serving camps because we get to work with people like you.

We will provide another update at 1pm MDT, or sooner, if warranted.

9:03am MDT — UPDATE:

Unfortunately, Microsoft Azure does not have any additional information to provide in this most recent update.

We will provide another update here at 11am MDT, or sooner, if warranted.

7:03am MDT — UPDATE:

There is no new information at the time, although Microsoft’s recent update indicates that they are making progress. The next scheduled update from Azure will be at 9am MDT at which time we will provide another update here. 

 

3:09am MDT — UPDATE:

There is no new information at this time. The next scheduled update from Azure will be at 5am MDT at which time we will provide another update here.

1:22am MDT — UPDATE:

Microsoft Azure’s latest update is that they are making progress on recovering services but no current timeline available.

The next expected update from their team is scheduled for 3am MDT.

11:25pm MDT — UPDATE:

Microsoft Azures latest update contains no new relevant information. We will continue to monitor and update as information becomes available.

The next expected update from their team is scheduled for 1am MDT.

9:00pm MDT — UPDATE:

CampMinder Engineers are continuing to monitor our servers and will do so throughout the night until all necessary Azure services come back online. Once this occurs we will take immediate action to get CampMinder up and running as soon as possible. Rackspace, who is our intermediary with Azure, will also continue to monitor and give updates to us.

The next expected update from their team is scheduled for 11pm MDT.

4:30pm MDT — UPDATE:

Engineers continue to implement the necessary mitigation steps and have started to see some services to return. We will continue to update you as we receive information from Azure.

2:30pm MDT — UPDATE:

Microsoft Azure Engineers have restored power to the data center and recovered a majority of network devices. Some services are starting to see signs of recovery, mitigation efforts to restore service are continuing as their highest priority. 

The next expected update from their team is scheduled for 4pm MDT.

12:00pm MDT — UPDATE:

Microsoft Azure engineers continue to work to resolve the outage. The next expected update from their team is scheduled for 2pm MDT.

9:45am MDT — UPDATE:

Microsoft Azure, the services that CampMinder uses to host its platform, continues to be unavailable. The Microsoft engineers have isolated an issue with cooling one part of their data center, and as their procedure dictates, an automated power down process went into effect in order to ensure data and hardware integrity. The Microsoft engineers continue to work to resolve the issue.

We will provide an update when the issue is resolved.

6:20am MDT:

Microsoft Azure, the services that CampMinder uses to host its platform, began experiencing networking issues in its South Central US region early this morning. This affects a countless number of Azure customers, who all rely on the service to host their applications, so Microsoft is treating it with the appropriate level of urgency.

Due to this issue, CampMinder and CampInTouch are currently unavailable. We are monitoring the situation with Microsoft Azure closely and will provide an update when the issue is resolved.