So far we've been seeing a big improvement after the deploy today. We will keep monitoring things to make sure they are stable, but the "brownouts" seem to have ceased. Please let us know if you do continue to see any issues with sites not loading.
Posted Jan 21, 2025 - 14:53 PST
Update
We have been continuing to work on the 503 errors affecting the frontend websites. We will be deploying an update in the next hour or so that we are expecting to help with the site performance.
We also have some other avenues we are looking into as well with our data center contacts today to improve things. Will post more as we have more updates.
Posted Jan 21, 2025 - 10:45 PST
Update
We are continuing to work on a fix for this issue.
Posted Jan 18, 2025 - 15:21 PST
Identified
The issue has been identified and a fix is being implemented.
Posted Jan 18, 2025 - 10:00 PST
Investigating
We’ve identified that system brownouts were being caused by intensive database queries. To address this, we’ve implemented a performance solution that has already resulted in a 90% improvement, and will provide long-term stability improvements for the system. We’re actively investigating the root cause and exploring ways to prevent this from happening again in the future. Thank you for your patience and continued support!
Posted Jan 18, 2025 - 10:00 PST
This incident affected: Admin, POS, Frontend, and Batch Update.