Registered system slowdown December the 2nd 2024

Incident Report for TrekkSoft

Postmortem

Incident Date: December 2nd 2024
Incident Duration: Approximately 5 hours (intermittently)
Affected Services: TrekkSoft Backoffice, POS Desk, TrekkSoft Mobile App (mPOS)

Incident Description:
On Monday, Dec 2nd, at 12:13 PM CET, system slowness was reported, leading to an inability to process bookings. Earlier that morning, an alert indicated web response times exceeding 1.0 seconds for over 10 minutes.

Impact:
Investigation revealed it was caused by resource-heavy queries, making the system unresponsive.

Resolution:
This issue coincided with increased demand for year-end reports, likely contributing to the slowdown. Queues for reporting and scheduling were overloaded, with pending messages being gradually consumed after being moved to temporary queues.

Preventive Measures:
Further monitoring and optimization are ongoing to prevent similar issues.

Posted Dec 09, 2024 - 15:36 CET

Resolved

The incident has been resolved, and all TrekkSoft functionalities are operating as expected.

Our investigation revealed that the system slowdown, which caused error messages during booking attempts and prevented some bookings from being processed, was triggered by an unexpected increase in system requests (e.g. a high volume of reporting requests).

We have implemented a fix that appears to have resolved the issue and are actively exploring measures to prevent similar issues in the future. A detailed postmortem of the incident will be shared in the coming days.

We sincerely apologize once again for any inconvenience caused.

Posted Dec 03, 2024 - 13:52 CET

Update

We are continuing to monitor for any further issues.

Posted Dec 02, 2024 - 18:27 CET

Monitoring

We have identified the root cause, applied a fix, and are currently monitoring the results.

Posted Dec 02, 2024 - 13:39 CET

Update

We are continuing to investigate this issue.

Posted Dec 02, 2024 - 13:37 CET

Investigating

We are currently experiencing a system delay that is causing a general slowdown of the system, this might result in error messages being shown and also might prevent some bookings from being placed. The affected areas include the booking widget, PoS desk and the mPoS mobile app.

Our developers are actively investigating the root cause of this issue.

We sincerely apologize for any inconvenience this may have caused and appreciate your patience as we work toward a resolution. If you have any questions or concerns, please feel free to reach out to us.

We will keep you informed of any updates through our https://status.trekksoft.com page. If you haven't subscribed yet, we encourage you to do so to stay updated.

Posted Dec 02, 2024 - 12:40 CET

This incident affected: TrekkSoft Backoffice, TrekkSoft Mobile App (mPOS), and POS Desk.