Internet Outages impacting customers in Northeast and Midwest

Incident Report for DPS AdTracker

Postmortem

One of our ISPs, CrownCastle, had discovered a routing instability within their network impacting multiple customers. A peering point failure was identified with one of our partner carriers. Network traffic has been routed to an alternate provider. CrownCastle Network Analysts have confirmed routing issues have cleared. We will continue to monitor network health to confirm stability.

‌

This only impacted a small subset of customers in the Northeast and Midwest that were also co-located in the NY and VA Datacenters. This did not impact Azure hostings in the USA, Hong Kong, or West India.

Posted Jun 27, 2022 - 17:04 EDT

Resolved

It appears the handoff issue may be resolved. We will continue monitoring and re-open if the issue recurs.

If you have any "stuck" ads that are on a DELIVERY RETRY/DELIVERY FAIL, you should attempt to re-approve them.

Posted Jun 27, 2022 - 16:25 EDT

Investigating

We are currently investigating a network outage which appears to be occurring with a handoff between Spectrum, Level3/CenturyLink, and CrownCastle. We will have more information as this becomes available.

This is only impacting a subset of customers -- within that subset, only customers hosted in the NY and VA Datacenter.

This does not appear to be impacting Azure Hosted Datacenter.

Posted Jun 27, 2022 - 16:13 EDT

This incident affected: DPS AdTracker®:Cloud Platform Services (Order Entry / Business System Feeds, AdTracker:Online (ATOL) Websites, Third-Party Vendor Integrations, Customer Provided Systems / Other, FTP / SFTP Intake Services), DPS AdTracker®:Customer Support + Development (DPS Connect Remote Support (connect.dpsadtracker.com)), and Microsoft Azure + Office Cloud Services (Datacenter / Internet Connectivity, DPSCLOUD - USA-based Hosting).