We have finished backfilling data across all products: all data received during the incident that had been successfully buffered but unprocessed, is now fully accessible on the platform. Due to the nature of this outage, you may see some residual gaps in the data we received within the first few hours after the start of the incident.
We truly appreciate your patience and understanding during this incident.
Posted Mar 10, 2023 - 00:02 EST
Update
We have completed backfill of data for the following products
We are now in the process of validating and verifying data across all customers in those products.
For other products, we are actively working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 21:09 EST
Update
We have also completed backfilling data for the following products:
Log Management Metrics RUM CI Visibility
We are now in the process of validating and verifying data across all customers in those products.
For other products, we are actively working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 18:02 EST
Update
We have completed backfill of data for Network Performance Monitoring and Profiling, and are now in the process of validating and verifying data across all customers in those products. For other products, we are actively working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 15:20 EST
Update
All Datadog services are now available and able to receive, query, and report on live data. Monitors continue to be evaluated correctly since live data has been restored. Some customers may still observe gaps in historical data for parts of the last 24 hours.
We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 10:09 EST
Monitoring
Log management is operational. NPM is operational. Serverless is operational. Monitors are evaluated correctly since live data has been restored.
Unless noted otherwise, all Datadog services are now available and able to receive and query live data. Some customers may still observe gaps in historical data for certain products for parts of the last 24 hours. We are now working on backfilling data and will provide updates every 2 - 3 hours until the backfill effort is complete and the incident is fully resolved.
Posted Mar 09, 2023 - 10:06 EST
Update
APM Traces is operational. CI Visibility is operational. Error Tracking is operational. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 07:59 EST
Update
Log Management and NPM data are starting to recover and appear in recent queries, we will give another update once the services are fully operational. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 05:55 EST
Update
APM Services are operational. Profiling is operational. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 04:53 EST
Update
Metrics are operational. SLOs are operational. Cloud Integrations are operational. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 02:08 EST
Update
RUM is fully operational. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 01:14 EST
Update
The Test and Test Runs pages of CI Visibility are available. We're investigating issues with Log Management and Profiling. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 09, 2023 - 00:21 EST
Update
We're seeing partial recovery across several products including Real User Monitoring, Profiling, and Error Tracking. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 08, 2023 - 23:30 EST
Update
The Synthetics product is fully operational. We're seeing partial recovery for Security Monitoring, as well as metrics from our cloud provider integrations. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 08, 2023 - 22:12 EST
Update
Monitors for Logs and Service Checks are operational. We're seeing partial recovery for Watchdog. We will continue to monitor progress towards recovering the remaining services.
Posted Mar 08, 2023 - 21:13 EST
Update
Live data is now available for Logs. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Posted Mar 08, 2023 - 19:56 EST
Update
We are continuing to work on a fix for this issue.
Posted Mar 08, 2023 - 18:50 EST
Update
Live Search on last 15 mins for APM Traces is recovered. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Posted Mar 08, 2023 - 18:50 EST
Update
Processes are fully operational. We're seeing partial recovery for CI Visibility and Network Performance Monitoring. These products may have gaps in data and partial limitations based on data available to monitors. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Posted Mar 08, 2023 - 18:23 EST
Update
We're seeing partial recovery across several products including SLOs and Logs. These products may have gaps in data and partial limitations based on data available to monitors. We will continue to monitor progress towards recovering the remaining services. Data ingestion and monitor notifications remain delayed across non-metric data types.
Posted Mar 08, 2023 - 17:50 EST
Update
We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 17:07 EST
Update
We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 16:22 EST
Update
At 06:00 UTC on March 8th, 2023 the Datadog platform started experiencing widespread issues across multiple products and regions . The web application was unavailable or intermittently loading, and data ingestion & monitor evaluation were delayed.
We will share a more detailed analysis post-recovery, but at a very high level: A system update on a number of hosts controlling our compute clusters caused a subset of these hosts to lose network connectivity As a result a number of the corresponding clusters entered unhealthy states and caused failures in a number of the internal services, datastores and applications hosted on these clusters.
Our current status is: We identified and mitigated the initial issue, and rebuilt our clusters We also have recovered a number of our applications and services, including our web portals We are now working on recovering and catching-up the rest of our data systems for metrics, traces and logs across the regions that are still affected (see region-specific status pages). The recovery work is currently constrained by the number and large scale of the systems involved.
What to expect next: We are focusing on bringing back live data for all customers and all products before catching-up on any historical data we may have stored during the outage We expect live data recovery in a matter of hours (not minutes, and not days) We will continue to issue regular updates as the situation unfolds
We understand how critical Datadog is to your business, we sincerely apologize for the inconvenience and we are working hard to resolve this issue.
Posted Mar 08, 2023 - 15:39 EST
Update
We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 15:13 EST
Update
We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 14:28 EST
Update
We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 13:44 EST
Update
We are continuing to make progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 13:16 EST
Update
We continue progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 12:31 EST
Update
We continue progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 11:49 EST
Update
We continue progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 11:09 EST
Update
We are still working on the identified issue and are making continued progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 10:34 EST
Update
We are still working on the identified issue and are making continued progress towards recovering all services. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 09:48 EST
Update
We are still working on the identified issue and are making continued progress towards recovery. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 09:06 EST
Update
We are still working on the identified issue and are making continued progress towards recovery. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 08:29 EST
Update
We are still working on the identified issue and are making continued progress towards recovery. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 07:43 EST
Update
We have identified the issue, and are making continued progress towards recovery. Data ingestion and monitor notifications remain delayed across all data types.
Posted Mar 08, 2023 - 07:08 EST
Identified
We are seeing reduced error rates for the web application. We are continuing to work on mitigating and investigating the issue causing delayed data ingestion across all data types. Monitor notifications are delayed, and you may observe delayed data throughout the app.
Posted Mar 08, 2023 - 06:22 EST
Update
We are continuing to work on mitigating and investigating the issue causing delayed data ingestion across all data types. Monitor notifications are delayed, and the web application continues to be unavailable.
Posted Mar 08, 2023 - 05:32 EST
Update
We are continuing to work on mitigating and investigating the issue causing delayed data ingestion across all data types. Monitor notifications are delayed, and the web application continues to be unavailable.
Posted Mar 08, 2023 - 04:35 EST
Update
We are continuing to investigate this issue.
Posted Mar 08, 2023 - 03:51 EST
Update
We are still investigating issues causing delayed data ingestion across all data types. Monitor notifications may be delayed, and you may observe delayed data throughout the web app.
Posted Mar 08, 2023 - 03:37 EST
Update
We are still investigating issues causing delayed data ingestion across all data types. Monitor notifications may be delayed, and you may observe delayed data throughout the web app
Posted Mar 08, 2023 - 03:04 EST
Update
We are investigating issues causing delayed data ingestion across all data types. As a result monitor notifications may be delayed, and you may observe delayed data throughout the web app.
Posted Mar 08, 2023 - 02:19 EST
Investigating
We are investigating loading issues on our web application. As a result, some users might be getting errors when loading the web application.
Posted Mar 08, 2023 - 01:34 EST
This incident affected: APM, CI Visibility, Cloud Security Management, Incident Management, Log Management, Metrics and Infra Monitoring, Mobile Application, Monitors, NPM, Profiling, RUM, Serverless, Synthetics, and Web Application.