Hamina Wireless - Connection issues with all services and instances – Incident details

Connection issues with all services and instances

Resolved
Major outage
Started about 1 month agoLasted about 5 hours

Affected

Hamina app (EU region)

Major outage from 4:00 PM to 9:14 PM

Hamina app (US region)

Major outage from 4:00 PM to 9:14 PM

Integrations

Major outage from 4:00 PM to 9:14 PM

Updates
  • Postmortem
    Postmortem

    On 29 October 2025, a widespread outage in Microsoft Azure Front Door caused Hamina Planner to become temporarily unreachable for approximately four hours. Our monitoring alerted us immediately, and we confirmed the issue both through internal checks and customer reports. Because the outage occurred within Azure’s global routing service, it was outside our direct control, and we provided status updates while monitoring Azure’s progress toward mitigation.

    Azure resolved the issue later that evening, restoring full access to Hamina Planner. We are now evaluating additional failover options to improve service resilience in the future, including alternative CDN/DNS routing paths and improved status visibility.

  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    Our services have recovered and applications are operating normally. We’re continuing to monitor closely.

  • Update
    Update

    Update from Azure:

    At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025. We will provide another update on our progress within two hours, or sooner if warranted.

    This message was last updated at 19:57 UTC on 29 October 2025

  • Update
    Update

    Update from Azure:

    We have pushed our ‘last known good’ configuration, and customers may begin to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement.

    We are continuing to monitor progress closely and will provide an ETA for full mitigation within the next 20 minutes as we assess recovery across the AFD service.

    This message was last updated at 19:01 UTC on 29 October 2025

  • Identified
    Identified

    Update from Azure:

    We have initiated the deployment of our 'last known good' configuration. This is expected to be fully deployed in about 30 minutes from which point customers will start to see initial signs of recovery. Once this is completed, the next stage is to start to recover nodes while we route traffic through these healthy nodes.

    We do not have an ETA for full mitigation, we will update this communication within 30 minutes, once the deployment is completed.

    This message was last updated at 18:11 UTC on 29 October 2025

     

     

  • Update
    Update

    Update from Azure:

    Starting at approximately 16:00 UTC, we began experiencing Azure Front Door issues resulting in a loss of availability of some services. We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and disabling a problematic route that we found to be related to this, and at the same time rolling back to our last known good state.

    We are taking several concurrent actions: Firstly where we are blocking all changes to the AFD services, this includes customer configuration changes as well. At the same time, we are rolling back our AFD configuration to our last known good state. As we rollback we want to ensure that the problematic configuration doesn't re-initiate upon recovery.

     

    We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update.

  • Investigating
    Investigating

    We’re investigating a widespread outage of our cloud provider that is affecting our infrastructure.