This incident is resolved, all systems are back online and operational.
Engineers are running updates to pull in any orders or product updates that happened during the time of this outage. If we are not able to recover some data, we will reach out with whatever information we have, so manual updates can be made.
If your team was working between the hours of 16:32 UTC and 18:09 UTC, we recommend checking shipment and inventory logs to see if updates were saved. Do not make manual updates unless we reach out to you - doing so may cause changes to be duplicated.
Posted Jul 12, 2020 - 20:46 UTC
Monitoring
ShipHero applications are currently back up and running. Engineers are working on identifying orders that were impacted by the 12-minutes of data loss. For customers who are using the API, please ensure that API requests during the past few hours are resent.
Posted Jul 12, 2020 - 20:13 UTC
Update
A ShipHero database has been corrupted, resulting in several hours of downtime on the application.
Due to the database failure, we lost some data during a 12-minute window from 12:20-12:32 Eastern. Data on the platform -such as new orders - has not been lost. Data lost is what happened in ShipHero (for example, if you received a PO or shipped an order - that information will be lost). We are working on providing information as to what orders might have been shipped and recommend not shipping at this time so that orders are not double shipped.
Please monitor status.shiphero.com for updates, we expect the system to be back online and functional in the next hour or two.
Posted Jul 12, 2020 - 19:47 UTC
Identified
ShipHero has identified the issue as a corrupted AWS database. We are in the process of rebuilding the database and have been able to recover all but 15-minutes of data. Engineers are still trying to recover the remaining data before restoring services.
Posted Jul 12, 2020 - 19:16 UTC
Update
We are continuing to investigate this issue.
Posted Jul 12, 2020 - 19:14 UTC
Update
We are continuing to investigate this issue.
Posted Jul 12, 2020 - 18:48 UTC
Investigating
ShipHero is experiencing an AWS Database failure and is attempting to recover data before putting systems back online. We will continue to keep you abreast of updates via the status page.
Posted Jul 12, 2020 - 18:45 UTC
This incident affected: ShipHero Mobile App, ShipHero Dashboard, Store Integrations, Label Printing, ShipHero Public API, Miscellaneous Services, and ShipHero Ticket Support.