[AWS](EN) Case of throttling due to spike(sudden traffic increase) when using auto scaling in DynamoDB
Post about case of DyanmoDb throttling
Environment and Prerequisite
- AWS
- DynamoDB
Background
- Investigated and summarized the reasons for throttling when using auto scaling in DynamoDB provisioned mode.
Issue
- Throttling occurs in DynamoDB even though auto scaling is set in provisioned mode.
Reason
Summary of DynamoDB Related Concept and Word
- Before summarize content, let me clarify some concepts and words.
On-demand capacity mode and Provisioned capacity mode
- On-demand capacity mode: A serverless billing option that can serve millions of requests per second without capacity planning. Easily say pay what I used.
- Provisioned capacity mode: Specify the number of reads and writes per second that you require for your application. We can use auto scaling to adjust your table’s provisioned capacity automatically in response to traffic changes.
- Related Link: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/capacity-mode.html
WCU and RCU
- Specify the number of data reads and writes per second that you require for your application. Unit of read and write in provisioned mode.
- WCU: Each API call to write data to your table is a write request.
- RCU: Each API call to read data from your table is a read request.
- Each size of WCU and RCU is on official document.
- Related Link: https://aws.amazon.com/dynamodb/pricing/provisioned/
Auto Scaling
- We can set minimum, maximum provisioned capacity unit as mentioned in above and target utilization.
- Capacity is controlled between minimum and maximun following to target utilization. For example if target utilization is 70% and use 700 RCU, then provisioned would be 1000 RCU. I will explain later but rest 300 RCU are called padding.
- Related Link: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html
Padding, Burst Capacity and Spike
- Padding: Rest read and write capacity which excludes target utilization capacity.
- Burst Capacity: In official document it says “DynamoDB provides some flexibility for your throughput provisioning with burst capacity.”.
- Spike: Sudden request traffic increase.
- Related Link: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/burst-adaptive-capacity.html
- Related Link: https://aws.amazon.com/blogs/database/handle-traffic-spikes-with-amazon-dynamodb-provisioned-capacity/
Reason of Throttling When Spike Occurs
- When a spike(sudden request traffic) exceeds the provisioned capacity, requests may be processed in the following order.
Use RCU under Target Utilization
-> Use All RCU(RCU under Target Utilization + Padding)
-> Use Burst Capacity
- In spike, as shown in the above graph, firstly burst capacity handles requests and then throttling occures after use all prepared burst capacity. After auto scaling is done, then throttling disappears.
- In document it says “Auto scaling triggers when your consumed capacity breaches the configured target utilization for two consecutive minutes. CloudWatch alarms might have a short delay of up to a few minutes before triggering auto scaling.”.
- In DynamoDB auto scaling occurs when consumed capacity breaches the configured target utilization for two consecutive minutes. Therefore, delays occur for the following reasons, leading to throttling in reponse to spike in requests.
- There is a short delay of up to CloudWatch.
- Auto scaling will be started when consumed capacity breaches the configured target utilization for two consecutive minutes.
- Throttling occurs in process of auto scaling over above “Use RCU under Target Utilization -> Use All RCU(RCU under Target Utilization + Padding) -> Use Burst Capacity” requests.
- Related Link: https://aws.amazon.com/blogs/database/handle-traffic-spikes-with-amazon-dynamodb-provisioned-capacity/
- Related Link: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html
Author’s Thoughts
- Due to scaling timing detection and setting time in auto scaling, it is good to consider about it when designing.
- I learned many things about DynamoDB auto scaling.
- Make alarm for unexpected sudden increase traffic. Make habit of review capacity when there is plan to have more traffic.
Reference
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/capacity-mode.html
- https://aws.amazon.com/dynamodb/pricing/provisioned/
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/AutoScaling.html
- https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/burst-adaptive-capacity.html
- https://aws.amazon.com/blogs/database/handle-traffic-spikes-with-amazon-dynamodb-provisioned-capacity/