Subscribe
By Thomas Riboulet in aws —

Building releases with AWS CodeBuild and Lambda

Testing another way to run tests and build releases by using AWS Codebuild.

Many of us have been using commercial CI tools such as Codeship, Semaphore and CircleCI to run the test suite of our code bases and subsequently build releases for them. Some rely on “on-premise” versions of those tools or OpenSource ones instead (Jenkins, Travis, Concourse).

They do work, but they might cost you an arm. At Imfiny we are testing another way lately : using AWS Codebuild to run tests and build releases.

This is about the general reasoning behind it and the bricks used to do that.

An integration game

If we take the case of commercial CI tools running in the provider’s infrastructure it’s possible to plug it with an AWS ECR (AWS Docker images repository) but it’s not handily doable at the same time as we do our complete AWS setup.
As our infrastructure plan rely on AWS heavily we don’t want to have more to do than creating AWS resources and allowing access between them. When using terraform the integration can actually be made right away as we build the infrastructure, without leaving the terminal.

The main aspects

First the code base needs to be reachable by the CI, Codebuild, and that is covered by IAM roles and policies since our code base is hosted in AWS CodeCommit, at least the production and pre production branches.

If we were to use an on premise CI such as Concourse we would have a similar need : we would assign a role to the instances running Concourse and there wouldn’t be a problem.

Next the code base must include instructions for Codebuild to do its job. There is plenty to read on that topic but it boils down to a buildspec file with a sequence of instructions to follow. This is pretty much similar to all the other CI task files.
If the build works then one can pass to the release part. In our case we rely on ECR to host images for our services and giving CodeBuild access to ECR repository is a matter of IAM roles and policies.
The only trick is to rig the CodeCommit repository with a trigger upon creation or update of content. This trigger is actually telling an AWS Lambda task to run to signal CodeBuild to start a build.

Diving in the details

Terraform : setup

Nah, just kidding, we won’t go there. We are just saying what you need setup with terraform :

  • a VPC with public and private subnets, their internet gateway and a NAT for the private subnets. (Make sure instances seating on the private subnets are able to connect to internet)
  • an IAM user for you and one called lambda_dev (or something else but we will refer to it as lambda_dev in the following)
  • an SSH key for use with CodeCommit linked to your IAM user
    We might publish some examples later on about this if you ask nicely but it’s also fairly easy to find the Terraform documentation and the AWS one on those topics.

Some basic code base

We need to get a basic code base to be used. For tests purposes we will rely on the following Dockerfile and buildspec files.

# Dockerfile
from mcansky/sinatra_hello

This is very minimal, it relies on a public image we keep out there ourselves for exactly this purpose. It’s simple HelloWorld Sinatra app.

version: 0.2

phases:
  install:
    commands:
      - ps aux | grep docker
      - env | grep CODEBUILD
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Hello World Docker image...
      - cd $SERVICE_SRC_DIR
      - docker build -t $SERVICE_IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $SERVICE_IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$SERVICE_IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION
      - docker tag $SERVICE_IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$SERVICE_IMAGE_REPO_NAME:$IMAGE_TAG
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$SERVICE_IMAGE_REPO_NAME:$IMAGE_TAG
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$SERVICE_IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION

This is a bit bigger. You should refer to Build Specification Reference for AWS CodeBuild - AWS CodeBuild for the explanations about the phases in more details. Still we think it’s pretty self explanatory.
All $VARIABLES in there are provided either by CodeBuild itself or through the setup of the CodeBuild task with terraform (more on that later). So just trust that the name reflects the content of it. Environment Variables in Build Environments - AWS CodeBuild.
The install phase here is merely to check we do have what we want available in the VM that is running the build.
The Pre Build phase is where we login to ECR. On its own that command doesn’t do much if the IAM policy isn’t set before. More on that later.
The Build phase is where the matter is. We start by going into the source code directory and then run docker build, and docker tag . If we had tests to run, we would do so here too.
The Post Build phase is dedicated to pushing the prepared images to ECR.

It’s good to know that most of CodeBuild process is actually a Linux shell and that it behaves mostly like one.

Setting up the bricks with Terraform

Again this is not an introduction to Terraform, we are considering that you have a base knowledge at least of it and the above listed pre requisites prepared.

CodeCommit repository
resource "aws_codecommit_repository" "service-base" {
  repository_name = "service-base"
  description     = "service code base"
}
IAM role and policies

We need an IAM role for Codebuild which will allow it to assume a role within our infrastructure. Attached to it we need an IAM policy with rights to handle logs, some ec2 actions related to network, actions to pull code from CodeCommit and actions to push images to ECR. Those could be a bit tighter but we do want to be able to reuse that same policy for other projects in the same infrastructure.

/* code build */
resource "aws_iam_role" "codebuild_service-base" {
  name = "codebuild_service-base"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "codebuild.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_policy" "codebuild_service-base" {
  name        = "codebuild_service-base"
  path        = "/service-role/"
  description = "Policy used in trust relationship with CodeBuild"

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Resource": [
        "*"
      ], 
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "ec2:CreateNetworkInterface",
        "ec2:DescribeDhcpOptions",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVpcs"
      ]
    },
    {
        "Effect": "Allow",
        "Action": [
            "ec2:CreateNetworkInterfacePermission"
        ],
		  "Resource": "arn:aws:ec2:{{region}}:{{account-id}}:network-interface/*",var.aws_account["id"])}",
        "Condition": {
            "StringEquals": {
                "ec2:Subnet": [
                   "arn:aws:ec2:{{region}}:{{account-id}}:subnet/[[subnets]]"
                ],
                "ec2:AuthorizedService": "codebuild.amazonaws.com"
            }
        }
    },
    
    {
      "Effect": "Allow",
      "Resource": [
        "${aws_codecommit_repository.service-base.arn}"
      ],
      "Action": [
        "CodeCommit:ListBranches",
        "CodeCommit:ListPullRequests",
        "CodeCommit:ListRepositories",
        "CodeCommit:BatchGetPullRequests",
        "CodeCommit:BatchGetRepositories",
        "CodeCommit:CancelUploadArchive",
        "CodeCommit:DescribePullRequestEvents",
        "CodeCommit:GetBlob",
        "CodeCommit:GetBranch",
        "CodeCommit:GetComment",
        "CodeCommit:GetCommentsForComparedCommit",
        "CodeCommit:GetCommentsForPullRequest",
        "CodeCommit:GetCommit",
        "CodeCommit:GetCommitHistory",
        "CodeCommit:GetCommitsFromMergeBase",
        "CodeCommit:GetDifferences",
        "CodeCommit:GetMergeConflicts",
        "CodeCommit:GetObjectIdentifier",
        "CodeCommit:GetPullRequest",
        "CodeCommit:GetReferences",
        "CodeCommit:GetRepository",
        "CodeCommit:GetRepositoryTriggers",
        "CodeCommit:GetTree",
        "CodeCommit:GetUploadArchiveStatus",
        "CodeCommit:GitPull"
      ]
    },
    {
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:CompleteLayerUpload",
        "ecr:GetAuthorizationToken",
        "ecr:InitiateLayerUpload",
        "ecr:PutImage",
        "ecr:UploadLayerPart"
      ],
      "Resource": "*",
      "Effect": "Allow"
    }
  ]
}
POLICY
}

resource "aws_iam_policy_attachment" "codebuild" {
  name       = "codebuild"
  policy_arn = "${aws_iam_policy.codebuild_service-base.arn}"
  roles      = ["${aws_iam_role.codebuild_service-base.id}"]
}

/* lambda roles */
resource "aws_iam_role" "dev_lambda" {
  name = "dev_lambda"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "lambda_logs" {
    role       = "${aws_iam_role.dev_lambda.name}"
    policy_arn = "${aws_iam_policy.lambda_logs.arn}"
}

resource "aws_iam_role_policy_attachment" "lambda_builds" {
    role       = "${aws_iam_role.dev_lambda.name}"
    policy_arn = "${aws_iam_policy.lambda_builds.arn}"
}
Lambda definition
/*
trigger_build
    service-base-lambdas
    python-lambda/trigger_build.zip
    vpc needed
*/
resource "aws_lambda_function" "trigger_build" {
  function_name    = "app_trigger_build"
  role             = "${aws_iam_role.dev_lambda.arn}"
  handler          = "trigger_build.lambda_handler"
  runtime          = "python3.6"
  s3_bucket        = "${aws_s3_bucket.lambdas.id}"
  s3_key           = "python-lambda/app_trigger_build.zip"

  environment {
    variables = {
      ENVIRONMENT = "production"
    }
  }
}

resource "aws_lambda_permission" "allow_codecommit_service-base" {
  statement_id   = "AllowExecutionFromCodeCommit"
  action         = "lambda:InvokeFunction"
  function_name  = "${aws_lambda_function.trigger_build.function_name}"
  principal      = "codecommit.amazonaws.com"
  source_arn     = "${aws_codecommit_repository.service-base.arn}"
}

Here we find the lambda function definition and a permissions for it related to code commit so that CodeCommit can actually invoke the function.

You can see we chose to host our Lambda release in an AWS S3 bucket. We will share below how we do that but any private bucket will do.

Bucket to store Lambda releases
resource "aws_s3_bucket" "lambdas" {
  bucket = "app-lambdas-zip"
  acl    = "private"

  versioning {
    enabled = true
  }
}

It’s quite handy to have versioning activated for such buckets so we activate it.

Security groups
resource "aws_security_group" "codebuild_sg" {
  vpc_id      = "${aws_vpc.app_vpc.id}"
  name        = "app codebuild sec group"
  description = "codebuild rules sec group"

 egress {
    protocol    = "-1"
    from_port   = "0"
    to_port     = "0"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
ECR repository
resource "aws_ecr_repository" "service-base" {
  name = "service-base"
}

resource "aws_ecr_lifecycle_policy" "service-base" {
  repository = "${aws_ecr_repository.service-base.name}"

  policy = <<EOF
{
    "rules": [
        {
            "rulePriority": 1,
            "description": "Expire images older than 14 days",
            "selection": {
                "tagStatus": "untagged",
                "countType": "sinceImagePushed",
                "countUnit": "days",
                "countNumber": 14
            },
            "action": {
                "type": "expire"
            }
        },
        {
            "rulePriority": 2,
            "description": "Expire tagged images older than 30 days",
            "selection": {
                "tagStatus": "tagged",
                "countType": "sinceImagePushed",
                "countUnit": "days",
                "countNumber": 30
            },
            "action": {
                "type": "expire"
            }
        }
    ]
}
EOF
}

Very simple. The lifecycle policy is important as it will insure the ECR repository doesn’t get bloated with too many images. You can obviously tailor to your needs but remember you want to have a good balance between production needs, forensic needs and cost.

CodeBuild project
resource "aws_codebuild_project" "service-base" {
  name          = "service-base"
  description   = "codebuild service-base"
  build_timeout = "5"
  service_role  = "${aws_iam_role.codebuild_service-base.arn}"

  artifacts {
    type = "NO_ARTIFACTS"
  }

  environment {
    compute_type    = "BUILD_GENERAL1_SMALL"
    image           = "aws/codebuild/ruby:2.5.1"
    type            = "LINUX_CONTAINER"
    privileged_mode = "true"

    environment_variable {
      "name"  = "RACK_ENV"
      "value" = "production"
    }

    environment_variable {
      "name"  = "AWS_DEFAULT_REGION"
      "value" = "${var.aws_region["name"]}"
    }

    environment_variable {
      "name"  = "AWS_ACCOUNT_ID"
      "value" = "${var.aws_account["id"]}"
    }

    environment_variable {
      "name"  = "SERVICE_IMAGE_REPO_NAME"
      "value" = "${aws_ecr_repository.service-base.name}"
    }

    environment_variable {
      "name"  = "IMAGE_TAG"
      "value" = "latest"
    }

    environment_variable {
      "name"  = "SERVICE_SRC_DIR"
      "value" = "code/hello_world"
    }
  }

  source {
    type     = "CODECOMMIT"
    location = "${aws_codecommit_repository.service-base.clone_url_http}"
  }

  vpc_config {
    vpc_id = "${aws_vpc.app_vpc.id}"

    subnets = [
      "${aws_subnet.priv-subnets.0.id}",
      "${aws_subnet.priv-subnets.1.id}",
      "${aws_subnet.priv-subnets.2.id}",
    ]

    security_group_ids = [
      "${aws_security_group.codebuild_sg.id}"
    ]
  }
CodeCommit trigger

This one should be with the code commit repository definition, but it requires the lambda function and the code build project to be defined too.

resource "aws_codecommit_trigger" "service-base" {
  depends_on      = ["aws_codecommit_repository.service-base"]
  repository_name = "service-base"

  trigger {
    name            = "build"
    branches        = ["master"]
    events          = ["updateReference", "createReference"]
    destination_arn = "${aws_lambda_function.trigger_build.arn}"
    custom_data     = "${aws_codebuild_project.service-base.name}"
  }
}
The actual Lambda code
import boto3

def lambda_handler(event, context):
  print('Starting a new build ...')
  cb = boto3.client('codebuild')
  build = {
    'projectName': event['Records'][0]['customData'],
    'sourceVersion': event['Records'][0]['codecommit']['references'][0]['commit']
  }
  print('Starting build for project {0} from commit ID {1}'.format(build['projectName'], build['sourceVersion']))
  cb.start_build(**build)
  print('Successfully launched a new CodeBuild project build!')

It’s in python but one can translate that to another language if need be. It’s fairly straight forward anyway as long as there is an equivalent of boto3.
The goal is to pass on to CodeBuild the necessary information to start the build : the project name and the commit id (source version).

As we described earlier we rely on an AWS S3 bucket to store our lambda releases. To prepare them we build the release zip and upload it using good old Make.

build: clean
	zip ../trigger_build.zip trigger_build.py
	cd virtual-env/lib/python3.6/site-packages && zip -r ../../../../../trigger_build.zip .

upload: ../trigger_build.zip
	aws s3 cp ../trigger_build.zip s3://ourbucket-lambdas-zip/python-lambda/service-hosting_trigger_build.zip --profile <our-profile>

update_lambda:
	aws lambda update-function-code --function-name trigger_build --s3-bucket ourbucket-lambdas-zip --s3-key python-lambda/service-hosting_trigger_build.zip --profile <our-profile>

clean:
	rm ../trigger_build.zip || echo "Already clean"

Note that we use AWS profiles locally on development hosts. Once the upload is done we can call terraform to create all the resources we have described.

Conclusion

Once all that is setup with terraform, you “just” have to push your code to the CodeCommit repository to trigger the build in CodeBuild.
Granted this lacks some of the niceties a commercial or even most OpenSource CI provide out of the box yet I do like the idea that all is contained in AWS and that, at least when starting a new project or a complete infrastructure we can rely on such basics to setup all that’s need for the team.

Where to go from there ?

Actually running tests within that setup is totally possible, just keep in mind that you will want to see logs of the test runs and possibly the exceptions if any.
Also, building and releasing the lambdas from CodeBuild would be a nice thing to do and since one can rollback to previous lambda versions in case of issue … it’s quite safe.

Subscribe to Imfiny

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.
jamie@example.com
Subscribe