AWS Route 53 DDoS Attack Shows You’re Responsible for Availability
The recent AWS Route 53 DNS attack should make you consider who is responsible for availability when we outsource key services. The attack highlights how dependent businesses have become on our wide swath of communication service providers (CSPs) and how this reliance impacts our business availability. As with Route 53, hosting domain name system (DNS) services have been one of the fastest-growing areas of outsourcing. In a 2018 Harvard Library study, the authors pointed to the industry-wide risk associated with having 66 percent of domains entirely managed by a small number of external service companies.
We’re investigating reports of intermittent DNS resolution errors with Route 53 & our external DNS providers. We’re working towards resolution & will post updates here: https://health.aws.amazon.com/phd/status. 🚦
— AWS Support (@AWSSupport) October 22, 2019
We saw the report’s risk assessment play out in the real-world with the AWS Route 53 DDoS attack, just as it did with the colossal 2016 Mirai IoT DYN DDoS attack. Failed or partially mitigated DNS attacks leave a wake of collateral damage on the rest of us innocent bystanders who are dependent on these services. Interestingly, both of these DNS attacks leveraged a common, yet challenging to defend, DNS attack strategy cleverly named “water torture.” We previously explained this attack strategy in our Investigating Mirai Botnet paper.
DNS Water Torture
Attack Floods ISP recursive DNS servers with randomized queries to a victim base domain name, causing the ISP’s DNS servers to perform the attack on the victim’s authoritative DNS servers.
How Water Torture Works
Water torture cyber attacks can be difficult to defend because attackers use volumes of fake queries that look real to DNS services. Attackers exploit the label structure of DNS’s fully qualified domain name (FQDN) to create legitimate-looking queries that exercise the whole domain name tree with resolution misses. In this attack, the query will have a legitimate first label, like a10networks.com managed by the victim’s designated authoritative server. A small pool of DDoS drones spray spoofed recursive requests to any of the millions of unwitting DNS resolvers exposed on the internet. These fake queries are composed of pseudo-randomized second, third, fourth, or up to 50th labels adding up to a total of 253 ASCII characters. That’s 253256 possible permutations; yikes, a big number.
These fake queries are randomized so that each request results in full DNS tree climb since the FQDN will not be in any of the DNS servers’ hit or miss cache. The whole process keeps going as long as the drones spew out more queries, or when one of the servers in the path falls over from resource exhaustion.
We will discuss how to defend against “water torture” later in this blog.
Attribution and Ambiguous Targets
Hopefully, we will find out the whodunit and their motivation in the AWS incident, but rarely are the providers the intended victims of the DDoS attack. Instead, the mark is usually a tenant downstream inside the CSP’s infrastructure that has caught the ire of an attacker. Apparently, the attacker saw DNS as a lower-effort alternative to a volumetric assault against AWS’s Shield volumetric defenses. It is natural for attackers to gravitate to the weakest link, and they will exploit weaknesses regardless of a direct or indirect path to the target. Unfortunately for the rest of us, indirect attacks have a large blast radius that absorbs everyone in the connected vicinity.
Isn’t it Better Security?
It is ironic. We outsource for many reasons. In some instances, we believe going to vendors, especially bigger ones, will give us better security, hoping that being a part of the masses – hiding in anonymity is better than going it alone. Unfortunately, being a part of the swarm has its consequences even when we are not the target.
AWS told customers, “…but these mitigations are also flagging some legitimate customer queries at this time.” Source: Securityweek.com
We Do Have Someone to Blame, Right?
If you were one of those companies impacted by AWS Shield’s massive over response that blocked legitimate requests, you can get reimbursed for the lost availability but does that really overcome the downtime?
Figure 1: AWS Route 53 SLA
That level of compensation for your service downtime may seem a little underwhelming, but that is what we must accept when we relinquish availability to service companies.
Your Role in Outsourced Services Defense
It is a given that enterprises will continue to outsource IT functions to CSPs. AWS wasn’t perfect in its attack response, but it gets the importance of DDoS resilience. However, AWS isn’t your only vendor. Are your other CSPs taking enough responsibility for their subscriber’s defenses?
Unfortunately, many enterprises don’t have the resources, time, or people to operate their own DDoS defenses. The CSP is in the best position to defend its tenants. After all, it is the conduit to legitimate and attack-traffic delivery to your applications, services, and infrastructure. The CSP’s network edge is the right place to scrub the traffic, not downstream in our infrastructure, or intercepted by yet another service provider. If they do the right thing by taking the step to protect us, we might even be willing to help them build a business service that takes a bite out of the multi-billion-dollar cloud scrubbing industry. Interestingly, these types of services instill our loyalty,
“Since deploying intelligent, automated DDoS protection, Leaseweb also has seen an 11 percent reduction in support tickets. Leaseweb has improved the experience of both its customers and help desk agents, and lowered its OpEx.”, Bart van der Sloot, at Leaseweb Network.
However, that doesn’t mean we aren’t responsible for our businesses’ availability. When we outsource, we have to heighten our awareness of what constitutes an effective defense against our number-one asset, AVAILABILITY. With an understanding of threats and appropriate mitigation strategies, we are in a better position to ask the right questions of providers to assess how effective their defenses work in protecting the critical services you count on. As President Reagan said, “trust but verify.” Wise advice when your outsourced services matter.
Learn About Water Torture Defense
DNS water torture attacks are not going away any time soon. For attackers, water torture is easy to initiate, and nothing inspires repetition like success.
For DNS authoritative servers to be resilient to water torture, the server has to be able to answer an infinite number of bogus queries without impeding legitimate requests. Infinite resources aren’t possible, and even AWS has limits, so let’s look at a different strategy. A10 Networks’ Thunder Threat Protection System (TPS) DDoS defense is one such alternative approach. Here is a partial list of some mitigation capabilities that can be used independently or in combinations to achieve DNS DDoS resilience
- Reduce the attack surface by defining the DNS query policy that includes the max length of an FQDN and the total number of acceptable labels. Then, drop requests that don’t conform to legitimate FQDN structure for this specific service. Now, we have reduced the attack surface.
- Limit the queries rate of each requester, which will lower the impact of the noisy requesters.
- Dynamically learn the FQDN requested up to the specified label count and then applying a query limit to requesters asking for an unusually high number of similar FQDN. This works because TPS knows DNS FQDN structure and can learn what is being requested, then enforce limits over an interval. This is a more advanced technique available in the Thunder TPS product.
- Only answer to legitimate queries during an attack. This is one failsafe way to ensure DNS resilience Here, Thunder TPS escalates the defense posture by initiating a DNS zone transfer from the authoritative server. The transfer teaches Thunder TPS all the legitimate hosted FQDN and in effect creates an FQDN whitelist. During an attack, Thunder TPS only passes legitimate queries to the authoritative server and rejects the fakes. Once the attack has subsided, Thunder TPS automatically deescalates back to normal mode.
Additional Resources
- Watch A10’s on-demand webinar, “How to Defend DNS Services from DDoS Attacks of all Types“
- Read The State of DDoS Weapons Special Report, for more insights into DDoS attacks
- Find out more aboutt A10’s DDoS Protection solutions
- Learn more about our DDoS Mitigation & Detection product, Thunder TPS