2nd Printavo Slowness 05/15/23
Incident Report for Printavo
Postmortem

Hello Printavo Users,

Earlier today the Printavo platform experienced a significant degradation of service which resulted in the inability of many of our customers to access the platform in a timely manner. The incident occurred from 10:51am to 12:15pm Central, followed by a much smaller but still significant increase in request queue wait time from 1:07pm to 1:37pm central time. The root cause of these incidents stemmed from an issue with our firewall which incorrectly identified our marketing page as down rerouting marketing page traffic (malicious and otherwise) through the main platform's front door. The end result was that this deluge of requests ended up taking the slots of legitimate requests to the Printavo platform. To make matters worse the marking site experienced slowness of its own which exacerbated the issue.

As a result of the above, we have opened a ticket with our firewall provider and have set a more aggressive timeout for marketing content for the time being. This means that in the future should something like this occur again we will favor serving requests to the platform over those of the marketing pages. This strategy was the reason why the second degradation was both shorter and less intense than the first. That said as a platform which helps our customers run their business neither is ideal nor acceptable. We have historically taken great pride in the performance of Printavo as well as its uptime. Starting with the last week of April through today we have experienced numerous attacks as well as scaling challenges as the number of customers has grown. We apologize for the frustration these outages have caused and are committed to keeping extra resources on the situation until it fully stabilizes.

Our engineering team is continuing to monitor the situation to ensure that should anything new come up we can promptly respond to keep our customers up and running. Again, we apologize for the disruption to your business. When your business is disrupted so is ours and as we stated above we don't take either lightly. Thank you for your continued support.

Sincerely,

Piotr Biegaj
Inktavo CTO

Posted May 15, 2023 - 17:12 CDT

Resolved
We have not received any reports or seen any high queue times since 1:50pm CST so we'll be marking this incident as resolved.

I sincerely apologize again for these issues and rest assured, the engineering team is looking into it.

A post-mortem will be provided as soon as we're able.

Best,
Alex Worrall & The Printavo Team
Posted May 15, 2023 - 14:16 CDT
Monitoring
We are seeing that no more high queue times have occurred for the last 15 minutes and are on the lookout for any more occurrences.

Our engineering team is continuing to investigate the root cause of this incident.
Posted May 15, 2023 - 14:05 CDT
Identified
Printavo appears to be suffering a second wave of slowness - our Engineering team has confirmed that the issue is not causing as much of an issue as the previous one but we are investigating it and will be addressing the incident ASAP.
Posted May 15, 2023 - 13:32 CDT
This incident affected: Website.