By Thomas Riboulet in basics — May 13, 2020

Howto : Private git hosting

In this article we will see how to setup a set of : - a git repositories server and service - a web interface to manager users, groups, and repositories - a load balancer to handle SSL termination in front of the git hosting service

When git was first released most of us relied on either CVS or Subversion (SVN) two centralized version control softwares. For a short while we wondered if it was worth moving to something else, especially a distributed version control software. Most of us weren't used to using a DVCS.

After some time Github appeared and grew to be a replacement for Sourceforge and Freshmeat, which were, at the time, the main places to find the source code and builds for the vast majority of OpenSource and Free Software.

Some decried that the crowd flocked in such large numbers towards Github and nowadays we can see some level of migrations towards Gitlab SaaS and self hosted Gitlab as a result of Microsoft acquisition of Github. Yet, the reliance on a single service to host most of our git repositories is still going counter to the DVCS concept.

In this article we will see how to setup a set of :

a git repositories server and service
a web interface to manager users, groups, and repositories
a load balancer to handle SSL termination in front of the git hosting service

We will consider docker and docker-compose to be known.

UPDATED : 2022/05/08 : SSHing Shim update following 1.16.x releases (https://docs.gitea.io/en-us/install-with-docker/#sshing-shim-with-author)

Mapping our options

There are a few FOSS tools to handle git repositories hosting. You can first rely on a plain SSH server on any GNU or BSD flavored system. Or you can go with a few other options such as the community edition of Gitlab, Gogs or Gitea (to name a very few).

Gitea is becoming a massive project that spans not only git repository management but also CI and CD layers. We won't consider that option for this article. It's still a totally good option if you want something that does more than just repopsitory management, but .. that's not the point here. We just want to handle git repositories, users, groups, access and potentially code reviews and issues.

Instead we will focus on Gitea as previous experiences have shown it's relatively easy to setup using Docker. Gitea supports private setups while still allowing to expose some public repositories easily. The interface is close to Github's, Gitlab's, Bitbucket's and others.

The plan

At the end we will have a way to manage one's personal, or a team's repositories through a web interface with the usual layer of security (SSH for the git access, 2FA for the web access). The setup should be easily reproducible and support a cold restart by relying on regular system and databases backups.

hosting

Hosting Gitea is possible on a Raspberry Pi, so most smallish VPSs out there should be enough for a personal setup, and biggers ones will be good for professional needs. Digital Ocean and Vultr cheapest VPSs should only set you back about $5 per month for this. Some additional cost is to be included for backups but I doubt that even for a small to medium team you'd be spending more than $40 per month.

Hosting with a bare metal offering such as Packet is also possible but would be overkill for the vast majority of us. And, if we were to go that way, we would have to seriously consider load balancing, repositories' data volume and reliability. Still one of the cheapest option with Packet would be to use one c1.small.x86 instance which uses a pair of SSD drivers setup with RAID-1, 32GB of RAM and 4 Cores of an Intel E3-1240 v5. This little beast comes at about $300 per month (less than that if you reserve it or use the spot market).
Scaleway also has dedicated bare metal offering which has cheaper possibilities. I am sure AWS and GCP also offer similar dedicated VPS.

To ease our mind and limit risks we should aim for :

1 to 4GB of RAM
2 to 8 cores CPU
RAID-1 SSDs for the system and the data
automatic system backups

bricks

We will rely on :

gitea : repositories management, users, groups and organisations management, pull requests, issues.
postgreSQL : direct dependency of gitea of course. One could choose another engine.
HAProxy with Let's Encrypt : we don't want to leave our http connections to gitea open and visible to everyone so we will use HAProxy, Certbot and Let's Encrypt to automatically setup SSL.

We will rely on Docker to start and run Gitea, HAProxy and PostgreSQL. To improve reliability and resiliency we could, instead, rely on hosted versions of PostgreSQL either from the IaaS provider (AWS RDS, DigitalOcean, Scaleway) or some dedicated third party (such as Heroku's). This is definitely something that should be considered and factored in as it would increase the whole setup's resilience. Of course this would increase the cost of the setup by some factor but pricing options should still allow you to keep everything under $40 per month.

getting started

We will setup a VPS with Docker, Docker Compose and a few other utilities. Then, roughly following what's described by Gitea Docker setup we will setup a pair of Gitea and PostgreSQL containers. Finally we will setup HAProxy with certbot within another container. There are a few steps that need to be manual but most of it is actually pretty straight forward.

Some choices were made as to how to name the users, where to store data and so on. You can change those but there might be some unforeseen consequences. So be prepared for that.
The main things to pay attention to is the UID and GID of theuser used by Gitea internally to store data. It has UID and GID of 1000 so the VPS user used to store data and handle ssh connections should have the same. That's directly set when creating user and group in the following.

In case you don't know, the following will cause some charges on a credit card. So, if you do follow the instructions beware of this and don't forget to destroy the resource created once you are done with it.

the server

We could use Terraform for this, yet, we will stick to just a user_data script and a few files to copy to the VPS once it's running.

This has been tested with Digital Ocean but should work, almost identically, with other VPS providers out there as long as you go with an Ubuntu release such as 20.04 LTS.

Remember to ensure you will have a way to connect to the server using SSH once it's up. Most VPS providers give a way to seed the server with the public key you want.

#!/bin/bash
apt-get update -y
apt-get install -y git docker docker-compose net-tools

addgroup --gid 1000 hosting
adduser --ingroup hosting -u 1000 git
adduser --system --ingroup hosting --ingroup docker git-docker

mkdir /home/git/.ssh
chown -R git:hosting /home/git/.ssh
mkdir -p /var/hosting/git/gitea
mkdir -p /var/hosting/git/postgres
mkdir -p /var/hosting/git/gitea/git/.ssh/
mkdir -p /app/gitea
mkdir -p /var/hosting/git/haproxy/config
mkdir -p /var/hosting/git/haproxy/certs.d
mkdir -p /var/hosting/git/letsencrypt

This basically finishes the setup of the server itself. We get two users and one group :

the git user will be used to handle the ssh connections and storage. It's important to use UID 1000 here and to have a regular user, it will need a shell.
the git-docker user will be used to start and handle containers so as to avoid using root for this part
the hosting group, for which it's also important to use GID 1000

containers

Now we merely need to start a few containers using docker-compose and this docker-compose.yml file.
Do note that you need to change the environment variables related to the database password (DB_PASSWD and POSTGRES_PASSWORD, those should be the same). The environment variables related to your domain name (CERTS) and email address (EMAIL) should also be taken care of.

version: "2"

networks:
  gitea:
    external: false

services:
  gitea:
    container_name: gitea
    image: gitea/gitea:latest
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - DB_TYPE=postgres
      - DB_HOST=db:5432
      - DB_NAME=gitea
      - DB_USER=gitea
      - DB_PASSWD=<CHANGE_ME>
    restart: always
    networks:
      - gitea
    volumes:
      - /var/hosting/git/gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "172.17.0.1:8080:3000"
      - "127.0.0.1:2222:22"
    depends_on:
      - db

  db:
    container_name: db
    image: postgres:9.6
    restart: always
    environment:
      - POSTGRES_USER=gitea
      - POSTGRES_PASSWORD=<CHANGE_ME>
      - POSTGRES_DB=gitea
    networks:
      - gitea
    volumes:
      - /var/hosting/git/postgres:/var/lib/postgresql/data

  haproxy:
      container_name: lb
      image: 'tomdess/haproxy-certbot:latest'
      environment:
          - CERTS=your.domain.name
          - EMAIL=you@your.domain.name
      volumes:
          - '/var/hosting/git/letsencrypt:/etc/letsencrypt'
          - '/var/hosting/git/haproxy/haproxy.cfg:/etc/haproxy/haproxy.cfg'
      ports:
          - '80:80'
          - '443:443'
      depends_on:
        - gitea

if one was to use a different set of GID and UID for the group and user then this can be changed through the USER_GID and USER_UID environment variables. The user_data script previously shown should be altered to match accordingly

note that ports exposed on the gitea service are tied to the docker bridge and the localhost ip addresses

This docker-compose.yml file should be placed within the git-docker user home on the host. If we ssh into the VPS as root we can use sudo to drop to the git-docker user : sudo -s -u git-docker.

As pointed out in Gitea's documentation it's necessary to setup a little script, add an SSH key and prepare the authorized key file.

First we need a simple script to handle SSH incoming commands. It should be placed in /app/gitea/gitea and rendered executable (chmod +x /app/gitea/gitea).

#!/bin/sh
ssh -p 2222 -o StrictHostKeyChecking=no git@127.0.0.1 "SSH_ORIGINAL_COMMAND=\"$SSH_ORIGINAL_COMMAND\" $0 $@"

This path is not random and cannot be changed as it's a gitea internal thing. It will be added automatically as command within each entry of the authorized_keys file that gitea manages.

As pointed out in Gitea's documentation it's necessary to setup a little script, add an SSH key and prepare the authorized key file.

First we need a simple script to handle SSH incoming commands. It should be placed in /usr/local/bin/gitea and rendered executable (chmod +x /usr/local/bin/gitea).

#!/bin/sh
ssh -p 2222 -o StrictHostKeyChecking=no git@127.0.0.1 "SSH_ORIGINAL_COMMAND=\"$SSH_ORIGINAL_COMMAND\" $0 $@"

This path is not random and cannot be changed as it's a gitea internal thing. It will be added automatically as command within each entry of the authorized_keys file that gitea manages.

Handling SSH keys

Secondly we need to create a SSH key.

sudo -u git ssh-keygen -t rsa -b 4096 -C "Gitea Host Key"

No need for a passphrase here : this key will only be used for the connection between the host and the container.

Thirdly we will need to create a link to point the git user's ssh authorized file to the place where it will be used by the gitea container.

ln -s /var/hosting/git/gitea/git/.ssh/authorized_keys /home/git/.ssh/authorized_keys

And then we will need to insert an initial line with the SSH public key we just created so that our host git user can ssh into the container's.

echo "no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty $(cat /home/git/.ssh/id_rsa.pub)" >> /var/hosting/git/gitea/git/.ssh/authorized_keys

To avoid any issue we will use chown on all those files to fix any possible owning issues.

#> chown -R git:hosting /home/git/.ssh
#> chown -R git:hosting /var/hosting/git/gitea/git/.ssh/authorized_keys

This part has configured an SSH passthrough so that the host system is passing any SSH connection for user git to the ssh server within the container. That way there is no need for a special port to be used in the ssh git urls.

tailoring haproxy setup

HAProxy will allow this setup to handle SSL without a hitch through Let's Encrypt. For this here is a configuration file inherited from the documentation of the haproxy container image used. Here is the file to put in /var/hosting/git/haproxy/haproxy.cfg.

global
    maxconn 20480
    ############# IMPORTANT #################################
    ## DO NOT SET CHROOT OTHERWISE YOU HAVE TO CHANGE THE  ##
    ## acme-http01-webroot.lua file                        ##
    # chroot /jail                                         ##
    #########################################################
    lua-load /etc/haproxy/acme-http01-webroot.lua
    # SSL options
    ssl-default-bind-ciphers AES256+EECDH:AES256+EDH:!aNULL;
    tune.ssl.default-dh-param 4096

# DNS runt-time resolution on backend hosts
resolvers docker
    nameserver dns "127.0.0.11:53"

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option forwardfor
    option http-server-close

    # never fail on address resolution
    default-server init-addr last,libc,none

frontend http
    bind *:80
    mode http
    acl url_acme_http01 path_beg /.well-known/acme-challenge/
    http-request use-service lua.acme-http01 if METH_GET url_acme_http01
    redirect scheme https code 301 if !{ ssl_fc }

frontend https
    bind *:443 ssl crt /etc/haproxy/certs/ no-sslv3 no-tls-tickets no-tlsv10 no-tlsv11
    http-response set-header Strict-Transport-Security "max-age=16000000; includeSubDomains; preload;"
    default_backend www

backend www
    server server1 172.17.0.1:8080 check port 8080
    http-request add-header X-Forwarded-Proto https if { ssl_fc }

All in this file is ready to go with the setup done before.

host and domain name

The VPS we have created should have a public ip v4, this ip address should be linked with the public hostname chosen for the git repository host. This should be an A record in the relevant DNS glue records. If not setup then haproxy and certbot won't be able to setup the certificates properly.

If there is no need for an https setup then one can avoid both the haproxy container and the DNS glue records change.

tailoring gitea

This is the time where we can start the containers as the git-docker user.

$> docker-compose up -d

Three containers should start:

gitea which is gitea itself
db which is the postgreSQL database server
lb which is the haproxy and certbot

If, after one minute or so, one of them isn't started have a look at the logs of the related one using docker log <container_name>.

Then we can directly open the hostname wanted in a web browser (let's say 'https://git.example.com') and we should be directed to the installation wizard for Gitea (you might have to click one item in the top menu, doesn't matter which). A few things will be inherited from the environment variables defined in the docker-compose.yml file.

One should read carefully the description of the different fields, and check the ones specific to the server at the bottom of the form.

Starting from the top of the form :

the database section should be ignored, the environment variables have taken care of it
general settings section :
- the site title can be altered
- the repository root path should stay as is (it relies on one of the volumes mounted)
- the Git LFS root path should stay as is
- the "run as username" should stay as 'git' unless you have altered the setup
- the ssh server domain should be the hostname you have chosen for your server
- the ssh port should be left as 22, this is the one on the container
- the gitea http listen port should be left as is (3000), this is the one on the container
- the gitea base url should be altered to use https and the hostname you have chosen for your server. No need to specify an extra port either (so something like https://git.example.org).
- the log path should stay as is
option settings section :
- alter the email settings to your liking or just disregard
- server and third party service settings :
  - you can enable local mode if you want
  - you can disable the federated avatars, and OpenID Sign-in
  - you should disable self registration except if you want it
  - you should not allow registration through external services except if you want it
  - you should disable OpenID Self registration except if you want it
  - do what you want for Captcha, view pages Email addresses visibility and organisation creation
- especially if you have deactivated the self registration you need to create an admin account through the "administrator account settings" part

using gitea

Once you have finished the install you will be directed to the login page. You can use the administrator login and password you just defined to go in.

There, to verify everything works you should :

add a SSH public key to your user by going into the user settings and the SSH section
create a repository, and copy the git/ssh string for the remote
add the remote to an existing repository of yours locally
do a git push to the remote of the master branch
profit

You can then go on adding more users and repositories to fit your needs.

Further considerations

Additional gitea configuration

Once the server is working as expected it's possible to add more users, create organisations and repositories. It's also possible to setup authentication sources such as OAuth ones (google, github, gitlab, ...) to ease adoption by other users.

But why ?

Github, Gitlab, Bitbucket, AWS, and so many others offer git hosting possibilities, even free ones. Thus, why would we setup one on our own ?

For this article we will only consider that we either want to be independent from the main providers or rely on a more decentralized web of hosting source code. After all Git is decentralized, isn't it not ?

As with private access to our private repositories then it will require other users to get accounts on this server and then be given access to the relevant organisations or repositories.

Backups

We have not seen ways to backup our setup which would require :

backing up database data (see PostgreSQL backup topics)
backing up the git data (see the VPS provider for this or rely on your usual sysadmin tools)

It's worth noting here that if those two points are properly cared for and you have a way to handle all the "manual" steps of this article through an automation tool of some sort then your Git server can be brought back to life in very little time.

CI / CD

As Gitea doesn't include CI and CD bricks it'll be upon you to either pick a SaaS provider for that matter or, in a similar fashion as we did here, pick a FOSS project and set it up within your infrastructure of choice.

As pointers for possible follow ups here are two OpenSource CI/CD tools :

Drone CI and their Github account
Concourse CI and their Github account

Conclusion

The main point of this article is to show that it's not difficult to get started hosting privately one or many git repositories without having to handle lots of costs.
As we have seen it's mostly a matter of handling a couple of containers and a tiny bit of DNS configuration.

This can also be extended to bigger needs as described early in this article and relying on separate resources to handle the database hosting in a safe and more resilient manner.

Sometimes we keep hearing that setting up such or such service is complex and thus it's easier to rely on a third party SaaS. Yet, in this case, it could be argued that it's certainly not difficult to have a working setup.
Furthermore, by relying on such cheap servers and with the required security in place we can actually rely on multiple such private servers and decentralize project hosting for a team, a project and a whole company.

Have fun.