{"ok":true,"db":"up","sha":"a4d24a68bfaf","incidents":[{"date":"2026-09-06","title":"Planned database resize window","status":"resolved","duration_minutes":2,"cause":"The production database was resized from 256 MB to 1 GB of memory. The platform restarts a database to apply a plan change, and the requests that reached it in that window failed with \"the database system is shutting down\".","fix":"Planned. This window was part of the same night's memory remediation, done together with the API instance's own resize from 512 MB to 2 GB.","impact":"No customer data was lost. About 75 seconds of requests failed: session refresh and the demo entry point received errors.","timeline_utc":["03:19 - the database was restarted to apply the plan change.","03:19-03:20 - requests to the database failed.","03:21 - the database was serving on the new plan and the API reconnected."]},{"date":"2026-09-06","title":"Database restarted after exceeding its memory limit","status":"resolved","duration_minutes":1,"cause":"During a manually triggered run of the daily background job, the database exceeded its 256 MB memory limit and was stopped by the platform. PostgreSQL recovered from its write-ahead log automatically.","fix":"Database memory was raised from 256 MB to 1 GB the same night, and the daily job's demo-reseed step gained a memory ceiling so it yields instead of exhausting the instance.","impact":"No customer data was lost. The database was unavailable for about two seconds; API requests in that moment received errors and the connection pool reconnected.","timeline_utc":["02:59:28 - the database exceeded its memory limit and was stopped.","02:59:30 - the database was back up after automatic crash recovery, and API connections were re-established."]},{"date":"2026-09-06","title":"API instance restarted a third time, during a manually triggered daily job run","status":"resolved","duration_minutes":6,"cause":"The daily job was run by hand, with its once-per-day guard bypassed, to trace memory. The instance's memory rose from about 470 MB after start, to about 505 MB after the tenant walk, to about 536 MB during the public-demo reseed, past the 512 MB limit, and the platform restarted it.","fix":"A redeploy was forced at 02:45 to restore service. The instance was then resized from 512 MB to 2 GB and the database from 256 MB to 1 GB, and the reseed step gained a memory ceiling.","impact":"No customer data was lost. Requests that arrived during the restart window received errors.","timeline_utc":["02:40 - the API instance exceeded its memory limit during the demo-reseed step.","02:45 - the forced redeploy was live and serving at 02:45:46."]},{"date":"2026-09-05","title":"API instance restarted again at the second daily job run","status":"resolved","duration_minutes":5,"cause":"The same root cause as the 10:04 restart, at the duplicate scheduler's second fire of the day. The daily background job was still being triggered twice per day by a legacy duplicate scheduler entry, and the fix had been written but had not finished deploying when the second run started.","fix":"The same fix as the 10:04 incident, deployed later the same day (deploy b314c5546846): a single-daily-run guard so the job can only fire once per day, and a per-tenant memory release in the worker so the job's footprint does not grow as it walks tenants.","impact":"No customer data was lost. Requests that arrived during the restart window received errors.","timeline_utc":["13:02 - the API instance exceeded its memory limit during the second daily background job run.","13:03 - the platform restarted the instance automatically.","13:07 - the instance was serving normally again and the external health check confirmed recovery at 13:07:49."]},{"date":"2026-09-05","title":"API instance restarted after exceeding its memory limit","status":"resolved","duration_minutes":5,"cause":"The API instance exceeded its memory limit during the daily background job and was automatically restarted. The job was running twice per day rather than once, because a legacy duplicate scheduler entry was still firing the same work a second time, and the instance's memory headroom had shrunk over the same period.","fix":"Shipped the same day (deploy b314c5546846): a single-daily-run guard so the job can only fire once per day, and a per-tenant memory release in the worker so the job's footprint does not grow as it walks tenants.","impact":"No customer data was lost. Requests that arrived during the restart window received errors.","timeline_utc":["10:04 - the API instance exceeded its memory limit during the daily background job run.","10:05 - the platform restarted the instance automatically.","10:09 - the instance was serving normally again and the external health check confirmed recovery at 10:09:14."]}]}