Elevated error rates and dashboard errors

Incident Report for Heap

Resolved

The dashboard and collector are now fully available.

During the period between 4:40pm and 4:54pm PDT, certain dashboard pages were unavailable, and the collector endpoint had significantly elevated error rates and latency. The root cause was a maintenance task that obtained an exclusive lock on a metadata table, blocking other queries from completing. We have suspended the task pending implementation and verification of a fix.

Posted Sep 28, 2015 - 17:29 PDT

Monitoring

The collector errors are limited to identify endpoints, and collector error rates appear to be returning to normal. We are continuing to monitor the situation, and are investigating the root cause.

Posted Sep 28, 2015 - 16:58 PDT

Investigating

Our systems team is investigating elevated error rates on all collector endpoints, and queries or dashboards failing to load for some users.

Posted Sep 28, 2015 - 16:48 PDT