INCIDENT DESCRIPTION:

At 12:00 pm NZST on Saturday, 6 September 2025, an AWS security certificate did not automatically renew as expected, which caused it to expire. As a result, system access was disrupted. At 12:07 pm, our Support team received the first client notification of the outage. By 12:10 pm, the weekend Support team had escalated the matter to the IT Department for immediate investigation. 

At 12:38 pm, the IT Department had identified the root cause as an expired AWS certificate and commenced investigating possible solutions. The certificate expiry caused a complete system outage affecting domain which hosts the BookIt staff and operator consoles, and the webAPI and booking gadgets. 

AS A RESULT:

• Staff, affiliates and operators were unable to access the BookIt console. 
• Customers were unable to search for availability or complete bookings online.

BookIt has multiple layers of detection and monitoring designed to ensure redundancy if one layer fails. In this case, automated alarms were triggered, however only after the certificate had expired.

By 1:15 pm, IT had attempted the renewal process but encountered failures, prompting them to pivot to issuing new certificates.   

At 1:30 pm, IT began preparing notifications on the gadgets to inform customers of the outage. The server load balancer was also restarted to support the resolution process.  

At 1:38 pm, IT confirmed the installation of newly issued certificates. By 1:39 pm system tests were underway, and by 1:40 pm NZST, all systems were fully restored. 

Following restoration, the Support team initiated customer communications to notify affected users. System monitoring continued throughout the day to confirm stability and prevent recurrence. 

MITIGATION:

To prevent this type of incident in the future, we are upgrading all remaining core domains currently using email-based authorisation for certificate renewals to DNS-based certificate resolution. This will eliminate the risk of renewal delays caused by failed email notifications. Additionally, we are reviewing and enhancing our monitoring systems to make them more proactive and preventative, capturing potential expirations well in advance. 

We sincerely apologise for the inconvenience this outage may have caused you and your customers. We understand how important reliable systems are to your operations and are committed to providing consistent support. Our friendly BookIt Support team is available to assist with any questions or concerns.