WaterMark

Service status

This page reports current service reachability, and every incident WaterMark has recorded since it started keeping this log.

WaterMark is reachable
The application answered its own health check at 09:30 UTC, including a live database query.
Current deploy: a4d24a68bfaf

WaterMark does not publish historical uptime percentages here. It has not been measuring one long enough for that number to mean anything, and a number on this page is a promise. The incidents below are the record instead: what broke, why, and what was changed.

Get told, instead of checking

Give us an address and WaterMark emails you when it records an incident, carrying the same cause, fix and timeline the log below carries. We send a confirmation link first, so nobody can put your address on this list but you. One click stops it, from any of those emails.

No marketing, no newsletter, no uptime percentages. This address is used for incident notices and nothing else.

Incident history

Newest first. Each entry is written after the incident is resolved and ships with a deploy, so the log is versioned in the same place as the code.

Planned database resize window

2026-09-06 · Resolved · 2 minutes

Impact. No customer data was lost. About 75 seconds of requests failed: session refresh and the demo entry point received errors.

Cause. The production database was resized from 256 MB to 1 GB of memory. The platform restarts a database to apply a plan change, and the requests that reached it in that window failed with "the database system is shutting down".

Fix. Planned. This window was part of the same night's memory remediation, done together with the API instance's own resize from 512 MB to 2 GB.

  1. 03:19 - the database was restarted to apply the plan change.
  2. 03:19-03:20 - requests to the database failed.
  3. 03:21 - the database was serving on the new plan and the API reconnected.

Database restarted after exceeding its memory limit

2026-09-06 · Resolved · 1 minute

Impact. No customer data was lost. The database was unavailable for about two seconds; API requests in that moment received errors and the connection pool reconnected.

Cause. During a manually triggered run of the daily background job, the database exceeded its 256 MB memory limit and was stopped by the platform. PostgreSQL recovered from its write-ahead log automatically.

Fix. Database memory was raised from 256 MB to 1 GB the same night, and the daily job's demo-reseed step gained a memory ceiling so it yields instead of exhausting the instance.

  1. 02:59:28 - the database exceeded its memory limit and was stopped.
  2. 02:59:30 - the database was back up after automatic crash recovery, and API connections were re-established.

API instance restarted a third time, during a manually triggered daily job run

2026-09-06 · Resolved · 6 minutes

Impact. No customer data was lost. Requests that arrived during the restart window received errors.

Cause. The daily job was run by hand, with its once-per-day guard bypassed, to trace memory. The instance's memory rose from about 470 MB after start, to about 505 MB after the tenant walk, to about 536 MB during the public-demo reseed, past the 512 MB limit, and the platform restarted it.

Fix. A redeploy was forced at 02:45 to restore service. The instance was then resized from 512 MB to 2 GB and the database from 256 MB to 1 GB, and the reseed step gained a memory ceiling.

  1. 02:40 - the API instance exceeded its memory limit during the demo-reseed step.
  2. 02:45 - the forced redeploy was live and serving at 02:45:46.

API instance restarted again at the second daily job run

2026-09-05 · Resolved · 5 minutes

Impact. No customer data was lost. Requests that arrived during the restart window received errors.

Cause. The same root cause as the 10:04 restart, at the duplicate scheduler's second fire of the day. The daily background job was still being triggered twice per day by a legacy duplicate scheduler entry, and the fix had been written but had not finished deploying when the second run started.

Fix. The same fix as the 10:04 incident, deployed later the same day (deploy b314c5546846): a single-daily-run guard so the job can only fire once per day, and a per-tenant memory release in the worker so the job's footprint does not grow as it walks tenants.

  1. 13:02 - the API instance exceeded its memory limit during the second daily background job run.
  2. 13:03 - the platform restarted the instance automatically.
  3. 13:07 - the instance was serving normally again and the external health check confirmed recovery at 13:07:49.

API instance restarted after exceeding its memory limit

2026-09-05 · Resolved · 5 minutes

Impact. No customer data was lost. Requests that arrived during the restart window received errors.

Cause. The API instance exceeded its memory limit during the daily background job and was automatically restarted. The job was running twice per day rather than once, because a legacy duplicate scheduler entry was still firing the same work a second time, and the instance's memory headroom had shrunk over the same period.

Fix. Shipped the same day (deploy b314c5546846): a single-daily-run guard so the job can only fire once per day, and a per-tenant memory release in the worker so the job's footprint does not grow as it walks tenants.

  1. 10:04 - the API instance exceeded its memory limit during the daily background job run.
  2. 10:05 - the platform restarted the instance automatically.
  3. 10:09 - the instance was serving normally again and the external health check confirmed recovery at 10:09:14.

Prefer to pull instead of being pushed? Subscribe in any feed reader: RSS feed. Machine readable: /status.json. For an active problem, email support@watermarkapp.online.