How to create a Docker image for PostgreSQL and persist data

Posted on Sat 21 February 2015 in Development • Tagged with containers, devops, docker, fig, Linux, OSX, postgresql

Before I start, let me confirm to you that official Docker images for PostgreSQL already exist and are available here: https://registry.hub.docker.com/_/postgres/ so this howto wants to be a guide to explain how to create these images and talk about some of the Docker features.

I will assume that you have already installed Docker on your machine. I have tested these instructions both on Ubuntu Linux and OSX (OSX users will need to install boot2docker, instructions are not available in this guide).

Dockerfile

To create a Docker image we need to create a text file named Dockerfile and use the available commands and syntax to declare how the image will be built. At the beginning of the file we need to specify the base image we are going to use and our contact informations:

FROM ubuntu:14.04
MAINTAINER Andrea Grandi <[email protected]>

In our case we are using Ubuntu 14.04 as base image. After these instructions we need to add PostgreSQL package repository and GnuPG public key:

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" > /etc/apt/sources.list.d/pgdg.list

then we need to update the packages available in Ubuntu and install PostgreSQL:

RUN apt-get update && apt-get -y -q install python-software-properties software-properties-common   
  && apt-get -y -q install postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3

We are installing version 9.3 of PostgreSQL, instructions would be very similar for any other version of the database.

Note: it's important to have apt-get update and apt-get install commands in the same RUN line, else they would be considered two different layers by Docker and in case an updated package is available it won't be installed when the image is rebuilt.

At this point we switch to postgres user to execute the next commands:

USER postgres
RUN /etc/init.d/postgresql start   
  && psql --command "CREATE USER pguser WITH SUPERUSER PASSWORD 'pguser';"   
  && createdb -O pguser pgdb

We switch to root user and we complete the configuration:

USER root
RUN echo "host all  all    0.0.0.0/0  md5" >> /etc/postgresql/9.3/main/pg_hba.conf
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.3/main/postgresql.conf

We expose the port where PostgreSQL will listen to:

EXPOSE 5432

We setup the data and shared folders that we will use later:

RUN mkdir -p /var/run/postgresql && chown -R postgres /var/run/postgresql
VOLUME  ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]

Finally we switch again to the postgres user and we define the entry command for this image:

USER postgres
CMD ["/usr/lib/postgresql/9.3/bin/postgres", "-D", "/var/lib/postgresql/9.3/main", "-c", "config_file=/etc/postgresql/9.3/main/postgresql.conf"]

The full Dockerfile is available here https://github.com/andreagrandi/postgresql-docker/blob/master/Dockerfile

Building Docker image

Once the Dockerfile is ready, we need to build the image before running it in a container. Please customize the tag name using your own docker.io hub account (or you won't be able to push it to the hub):

docker build --rm=true -t andreagrandi/postgresql:9.3 .

Running the PostgreSQL Docker container

To run the container, once the image is built, you just need to use this command:

docker run -i -t -p 5432:5432 andreagrandi/postgresql:9.3

Testing the running PostgreSQL

To test the running container we can use any client, even the commandline one:

psql -h localhost -p 5432 -U pguser -W pgdb

When you are prompted for password, type: pguser
Please note that localhost is only valid if you are running Docker on Ubuntu. If you are an OSX user, you need to discover the correct ip using: boot2docker ip

Persisting data

You may have noticed that once you stop the container, if you previously wrote some data on the DB, that data is lost. This is because by default Docker containers are not persistent. We can resolve this problem using a data container. My only suggestion is not to do it manually and use a tool like fig to orchestrate this. Fig is a tool to orchestrate containers and its features are being rewritten in Go language and integrated into Docker itself. So if you prepare a fig.yml configuration file now, you will be able, hopefully, to reuse it once this feature will be integrated into Docker. Please refer to fig website for the instructions to install it (briefly: under Ubuntu you can use pip install fig and under OSX you can use brew install fig).

dbdata:
  image: andreagrandi/postgresql:9.3
  volumes:
    - /var/lib/postgresql
  command: true

db:
  image: andreagrandi/postgresql:9.3
  volumes_from:
    - dbdata
  ports:
    - "5432:5432"

Save this file as fig.yml in the same folder of the Dockerfile and spin up the container using this command: fig up

andreas-air:postgresql-docker andrea [master] $ fig up
Recreating postgresqldocker_dbdata_1...
Recreating postgresqldocker_db_1...
Attaching to postgresqldocker_db_1
db_1 | 2015-02-21 19:01:07 UTC [6-1] LOG:  database system was interrupted; last known up at 2015-02-21 17:46:10 UTC
db_1 | 2015-02-21 19:01:07 UTC [6-2] LOG:  database system was not properly shut down; automatic recovery in progress
db_1 | 2015-02-21 19:01:07 UTC [6-3] LOG:  redo starts at 0/1782F68
db_1 | 2015-02-21 19:01:07 UTC [6-4] LOG:  record with zero length at 0/1782FA8
db_1 | 2015-02-21 19:01:07 UTC [6-5] LOG:  redo done at 0/1782F68
db_1 | 2015-02-21 19:01:07 UTC [6-6] LOG:  last completed transaction was at log time 2015-02-21 17:46:10.61746+00
db_1 | 2015-02-21 19:01:07 UTC [1-1] LOG:  database system is ready to accept connections
db_1 | 2015-02-21 19:01:07 UTC [10-1] LOG:  autovacuum launcher started

If you try to write some data on the database and then you stop (CTRL+C) the running containers and spin up them again, you will see that your data is still there.

Conclusion

This is just an example of how to prepare a Docker container for a specific service. The difficoult part is when you have to spin up multiple services (for example a Django web application using PostgreSQL, RabbitMQ, MongoDB etc...), connect them all together and orchestrate the solution. I will maybe talk about this in one of the next posts. You can find the full source code of my PostgreSQL Docker image, including the fig.yml file in this repository https://github.com/andreagrandi/postgresql-docker


Automatically pull updated Docker images and restart containers with docker-puller

Posted on Sat 25 October 2014 in HowTo • Tagged with containers, docker, docker.io, Flask, Python, howto, Linux

If you use docker.io (or any similar service) to build your Docker containers, it may be possible that, once the new image is generated, you want your Docker host to automatically pull it and restart the container.

Docker.io gives you the possibility to set a web hook after a successful build. Basically it does a POST on a defined URL and send some informations in JSON format.

docker-puller listens to these web hooks and can be configured to run a particular script, given a specific hook. It's a very simple service I wrote using Python/Flask. It's also my first Flask application, so if you want to improve it, feel free to send me a pull request on GitHub.

Note: this is not the only existing service that is able to do this task. I took inspiration from this article http://nathanleclaire.com/blog/2014/08/17/automagical-deploys-from-docker-hub/ and I really tried to customize https://github.com/cpuguy83/dockerhub-webhook-listener for my own needs, but the problem is that dockerhub-webhook-listener is not ready to be used as is (you have to customize it) and I'm not very good with Golang (yet) to be able to do it in little time. This is why I rewrote the service in Python (that is my daily language). I want to thank Brian Goff for the idea and all the people in #docker @ FreeNode for the support.

How to use docker-puller

Setting up the service should be quite easy. After you clone the repository from https://github.com/glowdigitalmedia/docker-puller there is a config.json file where you define the host, port, a token and a list of hooks you want to react to. For example:

{
    "host": "localhost",
    "port": 8000,
    "token": "abc123",
    "hooks": {
        "hello": "scripts/hello.sh"
    }
}

Create a bash script (in this case it was called hello.sh) and put it under script folder and write the instructions to be executed to pull the new image and restart the container (example):

docker pull andreagrandi/test:latest
docker stop test
docker rm test
docker run --name test -d -p 8000:80 andreagrandi/test:latest

Once configured, I suggest you to setup a Nginx entry (instructions not covered here) that for example redirect yourhost.com/dockerpuller to localhost:8000 (I would advise to enable SSL too, or people could be able to sniff your token). The service can be started with: "python app.py" (or you can setup a Supervisor script).

At this point docker-puller is up and running. Go to docker.io automatic build settings and setup a webhook like this: http://yourhost.com/dockerpuller?token=abc123&hook=hello

Every time docker.io finishes building and pushing your image to the docker registry, it will POST on that URL. docker-puller will catch the POST, check for a valid token, get the hook name and will execute the relative script.

That's all! I hope this very simple service can be useful to other people and once again, if you want to improve it, I will be glad to accept your pull requests on GitHub.


How to configure Edimax EW-7811UN Wifi dongle on Raspbian

Posted on Tue 02 September 2014 in HowTo • Tagged with howto, Linux, RaspberryPI, WIFI, networking, Debian

If you want to connect your RaspberryPi to your home network and you want to avoid cables, I suggest you to use the Edimax wifi adapter. This device is quite cheap (around £8 on Amazon) and it's very easy to configure on Raspbian (I assume you are using a recent version of Raspbian. I'm using the one released on 20/06/2014).

edimax-pi3

Configure the wifi adapter

Edit /etc/network/interfaces and insert these configuration values:

auto lo
iface lo inet loopback
iface eth0 inet dhcp

allow-hotplug wlan0
auto wlan0

iface wlan0 inet dhcp
wpa-ssid YOURESSID
wpa-psk YOURWPAPASSWORD

Power management issue

There is a known "issue" with this adapter default configuration that makes it to turn off if the wlan interface is not in use for some minutes. To avoid this you have to customize the parameters used to load the kernel module. First check that your adapter is using 8192cu module:

sudo lsmod | grep 8192
8192cu 551136 0

Create the file /etc/modprobe.d/8192cu.conf and insert the following lines inside:

# prevent power down of wireless when idle
options 8192cu rtw_power_mgnt=0 rtw_enusbss=0

I also suggest to create a little entry in crontab to make the RaspberryPi ping your router every minute. This will ensure that your wifi connection will stay alive. To edit crontab just type (from pi user, you don't need to be root):

crontab -e

and insert this line at the end:

*/1 * * * * ping -c 1 192.168.0.1

where 192.168.0.1 is the IP of your router (of course substitute this value with the ip of your router).

Keep Alive Script

I created a further script to keep my WIFI alive. This script will ping the router (change the IP using the one of your router) every 5 minutes and if the ping fails it brings down the wlan0 interface, the kernel module for the wifi and bring them up again.

Just put this script in /root/wifi_recover.sh and then execute from root user:

chmod +x wifi_recover.sh
crontab -e

Insert this line inside the crontab editor:

*/5 * * * * /root/wifi_recover.sh

Conclusion

The configuration is done. Just reboot your RaspberryPi and enjoy your wifi connection.


Configuring ddclient to update your dynamic DNS at noip.com

Posted on Tue 02 September 2014 in HowTo • Tagged with howto, linux, dns

noip.com is one of the few dynamic DNS free services that are reliable to use. If you have, like in my situation, a RaspberryPi connected to your home DSL and you want it to be always reachable without knowing the current IP address (the IP could change if you have a normal DSL service at home), you need a dynamic DNS service.

To update the noip.com one you just need ddclient a tool that is available in Raspbian/Debian repository. You can install it with this command:

sudo apt-get install ddclient

then you just need to edit /etc/ddclient.conf

protocol=dyndns2
use=web, web=checkip.dyndns.com/, web-skip='IP Address'
server=dynupdate.no-ip.com
login=yourusername
password=yourpassword
yourhostname.no-ip.org

and restart the client:

sudo /etc/init.d/ddclient restart

That's all! Please remember that noip.com free accounts have a limitation: they need to be confirmed every 30 days (you will receive an email and you need to click on the link contained to update your DNS).