Conversation
When a CreateFleet response that launched nothing carries RequestLimitExceeded next to other real errors, retrying the throttling only helps if the pools EC2 did not get to try can still serve the batch. That holds when every other error is confined to one instance type and subnet pool: a capacity shortage, an unsupported type in one AZ, a full or missing subnet, an AZ that lacks the requested volume type or is temporarily unavailable. Any other error, such as an account quota or a permission problem, would fail the retry on every pool alike, so it is now reported immediately instead of waiting for the rate limit to refill while the instances already launched for the job sit idle. The pool-scoped set reuses EC2_ICE_ERROR_CODES and adds six subnet and AZ level codes. Codes outside the set, including unknown ones, are not retried: the batch fails with that code, as it did before the throttling fix minus the invented insufficient capacity. Responses without RequestLimitExceeded are unchanged.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #740 +/- ##
===========================================
- Coverage 91.06% 91.05% -0.01%
===========================================
Files 20 20
Lines 3213 3210 -3
===========================================
- Hits 2926 2923 -3
Misses 287 287
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
At first I thought this fix deserves an entry in the changelog. However it seems to me it is included in the line below: Just double check this is intentional |
| | { | ||
| "InsufficientFreeAddressesInSubnet", | ||
| "InvalidSubnetID.NotFound", | ||
| "InvalidSubnet", |
There was a problem hiding this comment.
I've never seen this error code. Always seen something like InvalidSubnetSOMETHING.
Where did we get this error code?
There was a problem hiding this comment.
I got it from EC2 public doc:
https://docs.aws.amazon.com/ec2/latest/devguide/errors-overview.html
Search InvalidSubnet in it. And you get:
InvalidSubnet:
The specified subnet ID is not valid or does not exist.
| # fail the compute resource over. | ||
| # A single cause is normally left. Should there be several, throttling is retried only when every | ||
| # other cause is confined to a pool, since the throttled pools can then still serve the batch once the | ||
| # rate limit refills. Any other cause would fail the retry on every pool alike and is reported instead. |
There was a problem hiding this comment.
Can you specify an example in the comment of an error that when mixed with the throttling will not trigger a retry?
There was a problem hiding this comment.
Added:
# A single cause is normally left. Should there be several, throttling is retried only when every
# other cause is confined to a pool, since the throttled pools can then still serve the batch once the
# rate limit refills. Any other cause would fail the retry on every pool alike and is reported instead.
# For example:
# - [RequestLimitExceeded, InsufficientInstanceCapacity] -> retry, other pools can still serve the batch
# - [RequestLimitExceeded, VcpuLimitExceeded] -> report VcpuLimitExceeded, a retry hits the same limit
|
The current change is basically fixing a missed part in #734. |
Yes, it's intentional, the new changes don't change the changelog wording. |
|
Linked: the Reference section of the description now points to #734. |
…ome: throttling next to InsufficientInstanceCapacity is retried, throttling next to VcpuLimitExceeded is not.
2 participants
Description of changes
When a CreateFleet response that launched nothing carries RequestLimitExceeded next to other real errors, retrying the throttling only helps if the pools EC2 did not get to try can still serve the batch. That holds when every other error is confined to one instance type and subnet pool: a capacity shortage, an unsupported type in one AZ, a full or missing subnet, an AZ that lacks the requested volume type or is temporarily unavailable. Any other error, such as an account quota or a permission problem, would fail the retry on every pool alike, so it is now reported immediately instead of waiting for the rate limit to refill while the instances already launched for the job sit idle.
The pool-scoped set reuses EC2_ICE_ERROR_CODES and adds six subnet and AZ level codes. Codes outside the set, including unknown ones, are not retried: the batch fails with that code, as it did before the throttling fix minus the invented insufficient capacity. Responses without RequestLimitExceeded are unchanged.
Reference
This is a follow-up PR of #734: Fix EC2 throttling during computenode launch is not retried and is reported as ICE when using multi InstanceTypes or SubnetIds
Tests
slurm_plugin.overrideshook: throttling alone was retried and launched as before, while throttling next to an injectedVcpuLimitExceededwas not retried — the nodes went DOWN with(Code:VcpuLimitExceeded), the job was requeued without an insufficient-capacity fail-over and completed on its next resume.VcpuLimitExceededentry was injected rather than produced by EC2; that a single response can carry several real errors, with the rate limit being the last pool tried, was confirmed with the EC2 Fleet team.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.