Sites Down

Incident Report for Quintype

Postmortem

This downtime was caused by an internal load balancer that was deleted by an engineer on the Quintype team.

Our internal system detected the deletion and created a new load balancer for traffic, which came up within a minute. However, a proxy server was still pointing to the old load balancer, and needed to be reloaded manually.

To prevent these issues in future, we are making the following changes

We are ensuring that all load balancers have delete protection enabled by default.
We are automating the reload of HAProxy to resolve such issues quicker, by updating a DNS record

Posted Dec 31, 2019 - 14:29 IST

Resolved

The issue has been resolved, and was due to a deleted load balancer.

We apologise for the technical glitch. Please see the attached post mortem for more details.

During this downtime, most publishers were unaffected due to the presence of `stale-if-error` is present in the Cache-Control headers.

However, we did see that some publishers saw an error on the homepage if the failure scenario wasn't handled correctly by the front end app. Please contact support@quintype.com if this is the case.

Posted Dec 31, 2019 - 11:18 IST

Investigating

We are currently investigating the issue.

Posted Dec 31, 2019 - 11:14 IST

This incident affected: API.