ChannelLife Australia logo
Industry insider news for Australia's technology resellers
Story image

Cloudflare outage in 19 data centers worldwide due to own error

By Jessie Chiang
Wed 22 Jun 2022

Content delivery network Cloudflare says its outage for 19 of its data centers yesterday was because of a change in a long-running project to increase resilience in its busiest locations.

Many sites went down as a result, including Discord and Shopify. The company says even though these locations, which include Mumbai, Osaka, Singapore, Sydney, and Tokyo, are only 4% of its total network, the outage impacted 50% of total requests. 

Cloudflare says the first outage started at 06:27 UTC on June 21. At 06:58 UTC, the company brought the first data center back online, and by 07:42 UTC, all data centers were online and working correctly.

In a statement on its blog, Cloudflare says, "We are very sorry for this outage. This was our error and not the result of an attack or malicious activity."

The company says over the last 18 months, it has been working on converting all of its busiest locations to a more flexible and resilient architecture. During this time, Cloudflare converted 19 of its data centers to this architecture, internally called Multi-Colo PoP (MCP).

Cloudflare says this new architecture provides significant reliability improvements, allowing it to run maintenance in those locations without disrupting customer traffic. But those locations also carry a substantial proportion of the Cloudflare traffic, so any problem there can have a wide impact.

In order to be reachable on the internet, networks like Cloudflare make use of a protocol called BGP. As part of this protocol, operators define policies that decide which prefixes (a collection of adjacent IP addresses) are advertised to peers (the other networks they connect to) or accepted from peers.

Cloudflare says these policies have individual components, which are evaluated sequentially. The result is that any given prefixes will either be advertised or not advertised. A policy change can mean a previously advertised prefix is no longer advertised, known as being "withdrawn", and those IP addresses will no longer be reachable on the internet.

The company says while deploying a change to its prefix advertisement policies, a re-ordering of terms caused it to withdraw a critical subset of prefixes.

Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. However, the company says it has backup procedures for handling such an event and used them to take control of the affected locations.

But F5 Labs threat research lead David Warburton says this should be a reminder of the dangers of the centralisation of large cloud solution providers.

"In a traditional internet app deployment model, an outage of a server or misconfigured application might take out a single website," he says.

"But similar problems with a cloud solution provider can end up taking out all of their customers, resulting in not one website being taken offline, but hundreds or thousands. The impact can potentially affect organisations' digital experiences, revenues and reputations." 

Warburton says cloud solution providers provide immeasurable benefits to their users, but the re-centralisation of the internet through these cloud solutions is now causing the very problems the original design of the internet was intended to avoid through redundancy. 

ServerChoice commercial director Adam Bradshaw says outages can cause businesses serious reputational and financial damage. 

"For a smaller or medium-sized business, the risk associated with such failures in service can be critical. Even short outages in the digital economy could cause damage," he says.

"Diversifying an IT environment reduces the likelihood of third-party outages negatively impacting a business. Owning the hardware gives an organisation more control, with IT components such as colocation services providing an organisation with a backup when cloud services fail."

Related stories
Top stories
Story image
Microsoft
TO THE NEW unveils A/NZ Managed Services for Microsoft Azure
TO THE NEW has released Managed Services for Microsoft Azure to meet the growing demand in the A/NZ market and globally.
Story image
Sustainability
Honeywell launches new carbon energy management software for buildings
The new Carbon & Energy Management service allows building owners to track and optimise energy performance against carbon reduction goals, down to a device or asset level.
Story image
Infrastructure
Global investment in data centers more than doubled in 2021
DLA Piper's latest global survey finds the total investment in data center infrastructure worldwide rose from USD $24.4 billion in 2020 to USD $53.8 billion in 2021.
Story image
Tech job moves
Tech job moves - ActiveCampaign, Arcserve, LogRhythm & Qlik
We round up all job appointments from June 17-22, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Cybersecurity
Aqua Security, CIS create software supply chain security guide
Aqua Securityand the Center for Internet Security have together released the industry’s first formal guidelines for software supply chain security.
Story image
Review
Hands-on review: Philips Hue White and Color Ambiance Starter Kit A60
Philips’ range of Hue smart lighting promises to make any home a smart home. Techday’s Darren Price checks out the Philips Hue White and Color Ambiance Starter Kit A60.
Story image
Gaming
Intel Arc A380 graphics units launched in China, global release imminent
New Intel Arc A380 graphics units are set to bring next-generation technologies to gamers and content creators in the coming months. 
Story image
INTERPOL
Hundreds arrested, millions seized in global INTERPOL investigation
A two-month-long investigation by INTERPOL this year involved 76 countries and clamped down on organised crime groups behind telecommunications and social engineering scams.
Story image
Talend
Forrester names Talend Leader in enterprise data fabric
Forrester has named Talend a leader among enterprise data fabric providers in the Forrester Wave: Enterprise Data Fabric, Q2 2022 report.
Story image
Cybersecurity
Ingram Micro launches vendor-backed security program
Ingram Micro has unveiled a new program intended to give resellers the effective offerings their customers need to stay safe in the evolving threat landscape.
Story image
inDriver
New ride service inDriver arrives in Melbourne - pick your own fares
inDriver has launched in Melbourne, Australia and says it's set to disrupt the ride-hailing industry by allowing passengers and drivers to set and pick their own fares.
Story image
B2B
Corpay partners with supply chain platform PracBiz Exchange
Corpay's new partnership with PracBiz’s allows more than 4000 B2B suppliers on the latter's platform to use Corpay's global payments services.
Story image
Partnerships
Microsoft expands APAC Enabler Mentorship Program
"Mentors are the key to success for every professional. A good mentor is a coach, a guide, as well as a vocal advocate."
Story image
Cloud
Cloudera harnesses the power of cloud and data management for digital transformation
The data management landscape has changed significantly in recent years, with customer demands shifting and evolving in light of new technology and work processes.
Wiise
What are the 5 critical impacts of outdated systems? As the landscape continues to shift, having the right ERP will make or break Australian businesses. Discover 5 ways your current ERP might be letting you down.
Link image
Story image
Artificial Intelligence
Abnormal Security finds financial supply chain under threat
New research by Abnormal Security has found a rising trend in financial supply chain compromise as threat actors increasingly impersonate vendors.
Wiise
3 success factors for your distribution business in FY23. With growing challenges facing Australia’s supply chain, understanding your key priorities as a wholesale distribution business is crucial. Discover the 3 success factors for your distribution business in FY23.
Link image
Wiise
Is your on-premise ERP holding you back? Many Australian businesses are moving from on-premise to the cloud so they can streamline processes and access up-to-date business data on the go. Here’s 5 signs that you’re equipped to migrate from on-premise to the cloud.
Link image
Story image
Cloud
Cloudflare outage in 19 data centers worldwide due to own error
Cloudflare says its outage for 19 of its data centers yesterday was because of a change in a long-running project to increase resilience in its busiest locations.
Story image
Identity and Access Management
Ping Identity launches corporate venture capital fund
Ping Identity has launched a corporate venture capital fund to foster innovative offerings for the identity security market.
Story image
Cybersecurity
Australia worst in the world for mobile app threats
Australia is the country with the highest percentage of mobile app threats detected, according to new research.
Story image
Hybrid workforce
Why hybrid working is here to stay and how to ace it
Citrix's new report reveals hybrid workers are more productive and engaged at work than their office and completely remote counterparts.
Story image
Internet of Things
Domino's Pizza: A blueprint for secure enterprise IoT deployment
Increasingly, organisations are embracing smart technologies to underpin innovations that can enhance safety and productivity in every part of our lives, from industrial systems, utilities, and building management to various forms of business enablement.
Story image
Cloud
SAS and Greater Bank launches new AML and fraud service
Greater Bank and SAS have joined forces to deliver a new fraud detection and anti-money laundering solution that bolsters the protection of its 270,000 plus customers.
Story image
trust
Consumers want personalisation, but don't trust brands with their data
Customers expect personalisation during every brand interaction but they don't trust brands to keep their personal data secure and to use it responsibly. 
Story image
SaaS
Varonis strengthens security capabilities for AWS and S3
Varonis has strengthened and expanded its cloud and security capabilities, with a critical aim of improving safety and boosting data visibility in Amazon Simple Storage Service (S3).
Story image
Citrix
The best ways to attract young talent during labour shortages
New research from Citrix reveals hybrid working and ventures into the metaverse are top of mind for Gen Z workers.
Story image
Attack
Phishing attacks are making a comeback
No matter what approach or tool cybercriminals use to breach a network, they all have one thing in common: access.
Story image
Cybersecurity
Threat actors ramp up their social engineering attacks
As people get better at identifying potential threats in their inbox, threat actors must evolve their methods. Their new M.O? Social engineering.
Story image
Infrastructure
Lenovo announces launch of Retail Solutions portfolio in A/NZ
Lenovo has announced the launch of its Lenovo Retail Solutions portfolio across the A/NZ region, which they say will help retailers transform their business and face future retail demands.
Story image
FAQ
Techday's new Quickfire interview process explained
Techday is introducing a fantastic new streamlined editorial interview process. We explain the process and thoughts behind it.
Story image
DDoS
Flashpoint unveils security offering for school boards
Flashpoint has released its K-12 risk management and security offering to provide school boards and education security practitioners with tools to recognise, prevent and manage cyber and physical threats.
Story image
Digital Transformation
Apptio adds portfolio enhancements to promote digital strategy
"While digitalisation creates opportunities, it also makes budgeting far more complex, leading many companies to waste substantial funds."
Story image
CRM
Microsoft launches app for modern selling experience
The new release is designed to enhance CRM systems with customer engagement data from Microsoft 365 and Microsoft Teams.
Story image
NaaS
Survey finds 94% of Australian IT leaders looking at NaaS
Aruba’s latest survey reveals a rising interest in NaaS among Australian technology leaders as they re-evaluate their current infrastructure and network setup.
Story image
10 Minute IT Jams
Video: 10 Minute IT Jams - An update from Rimini Street
Today we welcome back Daniel Benad, who is the GVP & regional GM for Oceania at Rimini Street.
Story image
Consumer
Adobe survey reveals link between brand trust and consumer buy-in
Adobe has announced results from a study, which finds a strong correlation between brand trust and consumer behaviour.
Wiise
Project accounting not adding up? Successful job-based businesses all have one thing in common, it's the way they manage their job and project accounting. Discover the 4 success factors now.
Link image
Story image
3D Printing
Fleet Space, Konica Minolta partner for 3D printer offering
Fleet Space has partnered with Konica Minolta to implement 3D printers from 3D Systems to support the commercial small satellite manufacturer’s offerings.
Story image
Manhattan Associates
New late-stage order cancellation to improve customer service
Manhattan Associates launches new service allowing orders to be cancelled up to the point of manifested/loaded status, preventing unwanted shipments and costly returns.
Story image
Tech job moves
Tech job moves - Boomi, Limepay, Thales, VMware & Zoom
We round up all job appointments from June 6-16, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Healthcare
Helping Western Sydney Local Health District provide a new model of care in the fight against COVID
Western Sydney Local Health District (WSLHD) and PwC’s Consulting Business came together to solve through the challenges of COVID-19.
Story image
Apple
Jamf updates healthcare IT to protect data on Apple devices
Jamf has rolled out new functionality to help healthcare and IT teams protect patient data and streamline clinical access for their Apple fleet.
Story image
Cloud
NetApp announces winners of APAC Partner Excellence Awards
These awards recognise partners that have demonstrated success in delivering new cloud experiences, unlocking greater business value for their customers.