Our nightly backup job started failing structurally at a certain point when writing to our external storage provider. The error occurred after the backup itself (database and files) was created successfully, specifically during the upload to the external, redundant storage location. At no point was any data lost: local backups remained available as normal; only the copy to the external location could temporarily not be written.

Further investigation revealed that the issues with the storage provider were not limited to backups alone, but also affected the uploading and downloading of files by users of our platform.

We investigated this extensively, ruled out all possible causes on our own end, and ultimately decided to structurally move to a different, more reliable storage provider to permanently resolve the issue.

Because files could not be uploaded or downloaded, this caused disruptions to the service. For example, our users were unable to add quotes and invoices, while their customers were unable to view or use PDFs that had already been added.

We checked and ruled out all possible causes on our own end, step by step:

• Validity of our access keys and permissions

• Network configuration and our server's IP address (including testing with a completely new IP address)

• Configuration and security settings at our cloud provider

• Payment status of our accounts with the storage provider

None of these turned out to be the cause. All diagnostic signals ultimately pointed to an issue on the storage provider's side, which we were unable to resolve ourselves.

• We contacted the support team of the provider in question to have the cause investigated further.

• We had a new IP address assigned to our server. Unfortunately, this did not resolve the issue.

• Ahead of receiving support assistance, we decided to migrate our backup and storage infrastructure to a different, reputable provider, so that we would no longer remain dependent on an issue outside our control.

• Following the migration, our nightly backups were validated and are now running normally again.

• We are improving our internal monitoring so we can more quickly distinguish whether an outage is a network, configuration, or provider issue, speeding up future diagnoses.

• We periodically evaluate whether our infrastructure choices remain the most reliable option for critical processes such as backups.