Database capacity issue causing unresponsive apps

Incident Report for SweetHawk

Postmortem

Yesterday all our apps were impacted by increased usage of a variety of under-optimised app features resulting in the database server to be overloaded for a prolonged period. We have since made efficiency improvements to the database queries in question to alleviate this. We are also working on a process to better identify similar potential issues in advance.

During the course of the incident, we had to put apps into maintenance mode, making them unavailable a couple of times to be able to carry out needed maintenance to the database server. At other times, performance was impacted and apps could have been intermittently unavailable. Due to the length of the outage, this incident was of major impact and we apologise for the inconvenience these service interruptions have caused.

Posted Oct 22, 2021 - 09:08 UTC

Resolved

Maintenance has concluded and services are running smoothly for over 2 hours. We will continue to work on the root cause analysis and resolution. Once again we apologise for the interruptions to the apps today.

Posted Oct 21, 2021 - 22:57 UTC

Monitoring

We're still monitoring our server and re-enabling the Notify app. Apologies for the disruptions today.

Posted Oct 21, 2021 - 17:36 UTC

Update

We are continuing to work on a fix for this issue.

Posted Oct 21, 2021 - 17:34 UTC

Update

We are still managing the issue, apps are generally working well. Unfortunately the Notify app is partly disabled and may be for some time (the feed is unavailable currently).

Posted Oct 21, 2021 - 16:56 UTC

Identified

We are continuing to investigate slow response times and intermittent issues.

Posted Oct 21, 2021 - 16:02 UTC

Monitoring

Maintenance has been completed and apps are running again. We're continuing to monitor the servers at this time.

Posted Oct 21, 2021 - 15:30 UTC

Identified

We are sorry this is taking longer than expected, we're carrying out some emergency maintenance on the database server at this time.

Posted Oct 21, 2021 - 15:20 UTC

Investigating

We're looking into an issue with long running queries placing a high load on the database server, resulting in web request queueing, causing apps to appear unresponsive and not functioning. Apologies for the inconvenience, this is being investigated.

Posted Oct 21, 2021 - 14:15 UTC