Global issues in several Trekksoft production functionalities

Incident Report for TrekkSoft

Postmortem

Incident Date: February 15, 2024
Incident Duration: Approximately 1 hour
Affected Services: All services

Incident Description:
At approximately 11:30 AM CET on February 15, 2024, a bug fix caused an unexpected slowdown of our database. As a result, all services dependent on this database were impacted.

Impact:
The incident rendered it impossible to perform operations on the database, leading to a halt in functionality for all of our services. This disruption lasted for around one hour until our developers were able to mitigate the issue.

Resolution:
The incident was resolved by rolling back the release.

Preventive Measures and Recommendations:

Testing in our test environment: Improve options to test database impacts on our test environment, incl. smoke/regression/load tests.
Speed up our roll-back process

Despite our strict release policy which includes thorough tests on our test environment, for this specific bug fix, our test environment exceptionally didn't cover all implications. By implementing these measures, we can proactively address and mitigate similar incidents in the future, ensuring the continued reliability of our services.

Posted Feb 19, 2024 - 10:33 CET

Resolved

The incident has been resolved and all the Trekksoft functionalities are again operational as normal.

We will provide a postmortem of the incident in the following days.

Once again we want to apologize for any inconvenience this might have caused you.

Posted Feb 15, 2024 - 15:16 CET

Monitoring

The issue has been identified and a fix has been implemented. All system components are operational. We'll keep monitoring it.

Posted Feb 15, 2024 - 12:42 CET

Investigating

We are currently experiencing global issues in several Trekksoft production functionalities (a.o. Backoffice, POS Desk).

Our developers are already investigating on finding the root cause of the issue.

We will keep you updated and apologize for the inconvenience caused.

Posted Feb 15, 2024 - 12:00 CET

This incident affected: TrekkSoft Application, TrekkSoft API, Backend Mobile Applications, POS Desk, TrekkSoft Website, and Channel Manager.