Mixing on demand and Spot instances in Auto scaling groups

For some time now it’s been possible to mix OnDemand and Spot instances in Auto Scaling Groups so here is a little example of such a setup with Terraform.

Some reasons

OnDemand instances are marked with the base instance pricing we can find in AWS pricing : EC2 Instance Pricing – Amazon Web Services (AWS) . But AWS give us a few possibilities to lower the cost of our instances by allowing us to either reserve instances for one year or to grab instances that can go away at any time but that are discounted rate since otherwise … well they would earn no money at all to AWS.

So one general strategy to lower costs while insuring you do have a minimum capacity present is to mix both Reserved On Demand instances (full price minus a discount because of a up front payment for a year worth of use) and Spot instances (market price at the time of purchase in the pool of ready but not used instances of our type).

And now, instead of having a complex setup with different resources aside our ASG we can just specify the ratio of OnDemand vs Spot in your ASG, including price limits.

Required reading

There are some small lines to read and things to know before going down that road.

The first is that spot instances are meant to go away at any time so you should be prepared to see instances go down without notice because suddenly there is a surge in demand and your max price has been reached.

The second is that not all instance types are available in the spot market. We can to check the Spot Instance Advisor to find instances types available as spot instances in each region. That page also tell you good information about the savings and frequency of interruption. If you keep an eye on EC2 Instance Pricing – Amazon Web Services (AWS) you can check the max price for different instance types.

Terraform time

One important thing to note : you will need to use Launch Templates and not Launch Configurations with your ASG. We can check our post AWS Launch Templates to read more on that topic.

We will consider that you do have a Launch Template already in the following example of AutoScalingGroup configuration in Terraform. Two important things to note :

• we cannot have the setting disable_api_termination set to true in the Launch Template.
• we don’t need to have a launch_template section at the root of the AutoScalingGroup resource definition. Instead it will be present in the mixed_instances_policy section.

Here is an example.

resource "aws_autoscaling_group" "hello-world-blue" {
  vpc_zone_identifier = [
    "${aws_subnet.priv-subnets.0.id}",
    "${aws_subnet.priv-subnets.1.id}",
    "${aws_subnet.priv-subnets.2.id}"
  ]

  name = "hello-world-blue"
  max_size = 2
  min_size = 0
  desired_capacity = 1
  health_check_grace_period = 300
  health_check_type = "ELB"
  force_delete = true
  load_balancers = ["${aws_elb.hello-world.id}"]

  mixed_instances_policy = {
    instances_distribution = {
      on_demand_base_capacity = 1
      on_demand_percentage_above_base_capacity = 0
    }

    launch_template {
      launch_template_specification {
        launch_template_id = "${aws_launch_template.hello-world-blue.id}"
        version = "$$Latest"
      }
      override {
        instance_type = "t3.nano"
      }

      override {
        instance_type = "t3.micro"
      }
    }
  }
}

The mixed_instances_policy section is the new part. As you can read in Terraform AWS: aws_autoscaling_group documentation we need to specify two sections within it :

• instances_distribution
• launch_template

The instances_distrubtion part needs to have at least two parts :

• on_demand_base_capacity : defining the number of OnDemand instances that you want as a basis (our minimum of instances to be safe with your minimum traffic for example).
• on_demand_percentage_above_base_capacity : defining the percentage of OnDemand instances compared to Spot instances beyond the base capacity (0 : 0% OnDemand, 100% Spot).

That section can also contain a spot_max_price attribute to mark the max price we are willing to pay for the Spot Instances. Again for that it’s best to refer to the OnDemand pricing as reference.

The launch_template part is a bit odd to me but … it needs to be there. First that’s where we will put a reference to the Launch Template to use and the version, just like we would in a regular ASG. Then we need to specify at least two override sections.

These are intended to give us the possibility to suggest at least two instance types that can be used from the Spot market in replacement of the one defined in the Launch Template. So in our case we specified t3.nano (the same one as in the Launch Template) and t3.micro (a type that is just a bit bigger).