TrekkSoft global issue affecting several production functionalities

Incident Report for TrekkSoft

Postmortem

What happened?
One of external services, our system is using, was experiencing issues. This affected our web servers which were overloaded and finally crashed as the result of it.

What we did?
When our developers identified the issue, they fixed it by restarting failed services and servers for which it took some time to get back to the normal operating state.

Impact
During the estimated 2h time frame (12.30 to 14.30 CEST) of the incident no bookings have been able to be processed as well as our customers had issues with accessing our web and mobile application.

Learnings
In order to prevent similar incidents in the future we added additional monitoring system to the before mentioned external service as well as servers that the service is used on. With this in place, we will have a better overview in case similar issue happens again and this will allow us to act faster and minimize the impact this has on our customers.

We apologize for any inconvenience this might have caused you.

Posted Aug 30, 2021 - 16:53 CEST

Resolved

The incident has been resolved and all the functionalities are performing as expected.

We are continuing to investigate what encountered this unusual spike in network traffic and will provide postmortem once the investigation is finished.

Posted Aug 27, 2021 - 16:36 CEST

Update

Issues with bookings and payments have been addressed as well.

Trekksoft functionalities are again performing as expected.

Our developers are continuing to monitor the issue closely to ensure there are no further issues.

Posted Aug 27, 2021 - 14:56 CEST

Update

We are still facing some issues with taking bookings and payments.

Our developers are continuously working towards finding a fix for this issue as well.

Posted Aug 27, 2021 - 14:01 CEST

Monitoring

Our developers identified the root cause of the issue and already found a fix for it. Issue occurred due to unusual spike in network traffic which affected performance of our servers.

Trekksoft functionalities are again performing normally.

Our developers are continuing to monitor the issue closely to ensure there are no further performance issues.

Posted Aug 27, 2021 - 13:50 CEST

Investigating

We are currently experiencing global issues in several Trekksoft production functionalities.

Our developers are already investigating on finding the root cause of the issue.

We will keep you updated and apologize for the inconvenience caused.

Posted Aug 27, 2021 - 13:16 CEST

This incident affected: TrekkSoft Application, TrekkSoft API, Backend Mobile Applications, POS Desk, and TrekkSoft Website.