On July 22nd, 2015, the Heap collector and dashboard were unavailable for approximately 30 minutes, and we lost all event data submitted during that period (between 2:56 and 3:32pm PDT). Website and apps using Heap were not affected by this downtime, so your users should not have been impacted.

This data loss is absolutely unacceptable for us, and we’re changing our policies to stop this from happening again. As some background on our server infrastructure, our event collector cluster is isolated from the rest of our stack, and can continue to collect data even if other servers have failed. However, the code change that caused this outage overloaded our servers to the point that they were completely unresponsive to administrative tools and required a manual restart from the on-call engineer.

We're changing our deploy policies to require staged deploys for all changes affecting our collectors. We've also set up logging at our load balancer which would allow us to recover collector data even if our entire cluster is down.

If you have any questions about this, please reach out to support@heapanalytics.com.