Trekksoft Channel Manager (ExperienceBank) Partial Outage

Incident Report for TrekkSoft

Postmortem

The root issue: On the evening of January 2nd 2023 scheduled maintenance work was done on some of our domain names. As a result, suppliers and partners connected to the ExperienceBank channel manager, lost their connection to OTAs that were using the old trekkconnect.com domain name. These include Viator, GYG and Expedia among some other smaller OTAs. Sales on those OTAs were no longer reflecting on the booking software (including TrekkSoft), and availabilities were not being updated to the OTAs correctly. In addition our URL Shortener solution for SMS messages stopped working, getstarted.trecksoft.com redirect to signup page stopped working. Some long standing merchants started receiving booking notifications with links that are not working anymore.

Identification: Our automated observability system detected and alerted on errors coming in. Our engineers quickly escalated the situation to critical.

Steps taken to solve the issue: DNS addresses for trekkconnect.com were reconfigured. These changes can often take up to 48 hours to take effect. In addition, the affected OTAs were contacted as some reconfiguration work was required on their side so that their connection to the ExperienceBank channel manager was reestablished.

Resolution: While in most cases bookings began to reflect on the booking systems within a few hours, some OTAs took an additional 12 hours to get all suppliers reconfigured and running.

By early morning on January 4th all connections to OTAs were back and bookings were being reflected normally. We worked with individual partners to reactivate those connections where necessary. The critical situation was deemed resolved and communicated to all stakeholders. Some follow up housekeeping work came in the form of fixes for minor issues on the WEBAPP related to legacy domain deprecation were released on 5th of January in the evening.

Posted Jan 09, 2023 - 14:43 CET

Resolved

The incident has been resolved as all OTAs are again connected to our channel manager.

Bookings from OTAs are coming trough and are reflected properly to the baskets in TrekkSoft system. If you observe any more issues with the activities on the OTAs please check their mapping configuration and if further help is needed, feel free to reach out to support@experiencebank.travel.

We will provide a postmortem of the incident in the following days.

Once again we want to apologise for any inconvenience this might have caused you.

Posted Jan 04, 2023 - 11:09 CET

Update

The major OTAs (e.g. Expedia, GYG, Viator) have been notified about the changes made with domain names and are now making the necessary changes on their end to establish again the connection with our channel manager (ExperienceBank).

Changes are rolling out gradually and will be applied to all our customers one by one. Some suppliers are already back on line with their respective OTAs while others may still need to wait while the OTAs take action.
Please note that in order for the fix to be applied to all the customers it can take up to 72h.

We apologize for any inconvenience this may have caused and appreciate your patience until this is resolved. In the meantime, we recommend that you keep checking if the connection has been reestablished already.

We will make sure to keep updating you on further progress.

Posted Jan 03, 2023 - 16:15 CET

Identified

We are currently facing a partial outage in our channel manager ExperienceBank which is impacting bookings coming from the following OTAs: Expedia, GYG, Viator, Musement, Tiqets, Project Expedition and Grayline. In some cases bookings from these sources are not reflected properly in baskets in the TrekkSoft system.

The issue arose after scheduled maintenance to our domain names, namely “trekkconnect.com” redirection which is used by ExperienceBank to handle bookings coming from the OTAs.

We want to assure you that we are working diligently to resolve this situation. We have already identified the issue and restored the domain redirect and settings. However, it may take some time for these changes to take effect.

We apologize for any inconvenience this may have caused and appreciate your patience as we work to resolve the issue. If you have any questions or concerns, please don't hesitate to contact us. We will make sure to keep you updated on the progress of this issue.

Posted Jan 03, 2023 - 11:51 CET

This incident affected: Channel Manager.