Further processing delays
Incident Report for SweetHawk
Postmortem

Usage of some apps have recently increased in ways that was under anticipated. We found that the server was overloaded due to a combination of factors placing a high load on our database server. While we were able to reduce the congestion and get things back up and running in a first incident, however the incident recurred hours later. We were monitoring the service and have been able to keep downtime to a minimum however we are aware customers have been seeing capacity issues and processing delays. A side effect of issues is that app packaged targets are automatically disabled.

All services have now been restored with the exception of our secondary database server which is being reset at this time. We have re-enabled any app packaged targets automatically once the incident was resolved.

What we are changing:

  • Increased network throughput for the database server
  • Improved processing queue monitoring to identify issues faster
  • A range of internal optimizations based on observed issues

We are dedicated to providing a reliable service. If you have any further concerns, please contact support@sweethawk.com.

Posted Jan 25, 2022 - 00:25 UTC

Resolved
All functionality has returned to normal for the last couple of hours.
Posted Jan 24, 2022 - 23:22 UTC
Update
All apps are functioning again as normal and we are working to re-enable any targets that may have been automatically disabled as a result of outages.
Posted Jan 24, 2022 - 21:45 UTC
Monitoring
Most of our services are back up and running, we're still looking into some remaining issues with SLA Timers and Notify.
Posted Jan 24, 2022 - 17:44 UTC
Identified
The incident that happened earlier seems to have been unresolved, we are continuing to investigate issues.
Posted Jan 24, 2022 - 16:17 UTC