• Status
  • Events
  • Monitors

America/New_York

powered by openstatus.dev

OpenPanel

Oct 01, 2025

Issues with automatic kubernetes update

  • API

Resolved

Our api and website was down after a automatic node upgrade on one of our DB nodes. We noticed that we only had one read replica for our DB which caused pgbouncer to throw a connection error while the upgrade was running.

This caused a bunch of issues.

Steps to improve:

  • We'll introduce a much better readiness and liveness for our API, since our API tried to scale (and restart all existing pods) to solve the problem, this just made the issue worse.
  • We have increase our instances of the DB from 2 -> 3 to always have 1 read-replica available.
  • Have a better plan when automatic upgrades should be triggered

These two changes will improve similar issues in the future.

Dec 17, 2024

Cluster Maintenance

  • API

-

Maintenance

Around midnight we’ll perform some changes to our nginx ingress. Expecting some down time.

Couple of seconds to max 10 minutes (depending on outcome)

Sep 01, 2024

Moving API

    -

    Maintenance

    Last piece will be to move the API as well. Might be a couple of minutes down time but should go relative smooth.

    Aug 31, 2024

    Migration of redis and db

      -

      Maintenance

      Need to move our database and redis during Saturday morning. Expecting to have couple of minuts down. Services might be unreliable during this period.