Ephemeral containers for ssh

When I setup docker based infrastructures for projects I have usually a few use cases that need to be solved. Usually it starts with starting http services containers, background workers containers and console ones.

Let us see how to handle the last case.

The reason to have those

A consequence to move hosting of services into containers and considering servers hosting them as cattle is that you basically don't connect to them anymore.

This has a major impact on two regular use cases for engineering teams : running database migrations and looking into the data through an application console.

One thing many teams should adopt early is the habit to handle database migrations separately to the code changes requiring them. Adding new tables, new columns should be done before the code using it is deployed. Removing tables and columns should be done after the code using them is removed from the code base.

And then, in production cases, there can be many instances for which software engineers need to access an application live console to inspect production data and work with it. There are many ways to do this and some teams handle that through a proper back office interface. Others have to use a terminal console.

How to make them happen

Once you cannot login onto the server running the application since they are containers you need to launch such consoles through docker itself. As it's necessary to be able to start containers with specific images, specific builds, it's necessary to be able to launch a specific container with the specific image, of the specific tag.

This is possible thanks to OpenSSH and docker directly. The first tool is the command option in Ssh authorized_keys file. This allows to specify what to start when an ssh connection happens. A very simple one to start bash shell within a debian based container would be this. If you create a user in a Linux instance (let's name it 'docker-shell') you can add one or more ssh public keys in the ~/.ssh/authorized_keys file with the command part defined before the public key itself.

command="docker run -ti --rm debian --entrypoint /bin/bash" <public_key>

Notice the -ti this is made to give a TTY back to you and keep the connection open into the container. The --rm part is there to insure the container will be removed once the connection is ended.

This is enough for some use cases but it's not enough to start a specific image of a container. The solution is to craft a custom script that will be started as command.

Here is an example of such a script. We will need to pass a couple of options (the image tag for one) when connecting to the host : ssh -l docker-shell shellbox.local image-name tag. In a Ssh script these options will need to be grabbed within the SSH_ORIGINAL_COMMAND environment variable.
While it's possible to use sed or awk to split up the content of this environment variable it's handier to use read to grab such simple parameters as my friend Paul pointed out.

One last thing to know and use it the SSH_TTY environment variable. It will hold something if a proper TTY is requested for this ssh connection. The downside of calling a script within the command trick of the ssh authorized_keys is that one needs to specifically request a TTY to get one. So in the following script we start by checking the content of SSH_TTY and request the user to add the -t option to the ssh command. This can also be done through the ssh config file for the entry related to the host with the RequestTTY option set to force.

From there the script handles splitting the SSH_ORIGINAL_COMMAND into two parts : DOCKER_REPOSITORY and DOCKER_TAG_TMP. We check if the DOCKER_REPOSITORY value is among a set of authorized ones (to avoid having one member of the team start any container from a public repository). We can then check if DOCKER_TAG_TMP has a value or if we need to set a default value.

Finally we can start the requested container with the requested image, or the default one.

#!/bin/bash
if [[ "${SSH_TTY}" = "" ]]; then
  echo "No TTY requested, add -t to ssh command"
  exit 1
fi

AUTHORIZED_IMAGES=("debian" "ruby")
read DOCKER_REPOSITORY DOCKER_TAG_TMP <<<"$SSH_ORIGINAL_COMMAND"

if [[ ! " ${AUTHORIZED_IMAGES[@]} " =~ " ${DOCKER_REPOSITORY} " ]]; then
  echo "Image ${DOCKER_REPOSITORY} not authorized"
  exit 1
fi

DOCKER_TAG=${DOCKER_TAG_TMP:-stable}

echo "Starting ${DOCKER_REPOSITORY}:${DOCKER_TAG} ..."
unset SSH_ORIGINAL_COMMAND
docker run -ti --rm --entrypoint /bin/bash $DOCKER_REPOSITORY:$DOCKER_TAG

To connect you then need to use ssh -t -l docker-shell shellbox.local image-name tag.

Possible improvements

It's totally possible to not use the --rm option and work a way to ensure one can connect into the same container over and over if that's a requisite. Another thing might be to use bundle exec rails c to start a rails console instead of a simple bash one.
Or you could rely on a more regular ssh session and tmux to start containers that need to run for some time.

One thing missing from this script is also how to pass the necessary environment variables to the container so that it can start as needed and that depends upon how this kind of data (secrets and so on) is handled in your environment.

Conclusion

Ssh is often used without knowing of many powerful features it has. The command one can be used for many things. Many years ago I found out it's one way to handle git repositories, this one is very similar but funnier.
Many I share this with are actually surprised of such usage of ssh and docker. But a few could come up with additional tips and tricks such as my friend Paul.

Need help ?

We specialise in helping small and medium teams transform the way they build, manage and maintain their Internet based services.

With more than 10 years of experience in running Ruby based web and network applications, and 6 years running products servicing from 2000 to 100000 users daily we bring skills and insights to your teams.

Wether you have a small team looking for insights to quickly get into the right gear to support a massive usage growth, or a medium sized one trying to tackle growth pains between software engineers and infrastructure : we can help.

We are based in France, EU and especially happy to respond to customers from Denmark, Estonia, Finland, France, Italy, Netherlands, Norway, Spain, and Sweden.

We can provide training, general consulting on infrastructure and design, and software engineering remotely or in house depending on location and length of contract.

Contact us to talk about what we can do : sales@imfiny.com.