Skip to main content
Asked a question 5 years ago

How do I use Docker With OpenStack?

Where am I?

In Bright Computing, Inc. you can ask and answer questions and share your experience with others!

Can I use Docker as a hypervisor for OpenStack (Juno)?

There are several possible ways of using Docker with OpenStack. The one described here is based on OpenStack's Nova Docker Driver. That is, it means talking to a container using the Nova Compute API.

Not all Docker features are supported by the Nova Docker Driver.

Features that are either not supported, or have a non-standard implementation are:

  • Attaching volumes to Docker containers is not supported
  • Disk usage is limited to 10 GB per container regardless of flavor settings
  • Compute resources on a host node are distributed according to a CPU share constraint value which depends upon the OpenStack flavor used. Each container will have a CPU share constraint value of 1024 * VCPUs. Thus a 2 VCPU flavor result in a CPU shares value of 2048. More about CPU constraints can be read here: Link38

Note: Mixing a libvirt (KVM/QEMU) hypervisor with a Docker hypervisor on the same host is not recommended. 

If libvirt (KVM/QEMU) hypervisors are to be mixed with Docker hypervisors on different hosts, but still as part of the same OpenStack deployment, then it might be necessary to let OpenStack know which compute hosts can support Docker. This can be done, for example, by creating a special flavor and restricting it (via a shared custom extra specs key=value pair) to a particular Host Aggregate, thus aggregating the docker hypervisor hosts. Another way of restricting it would be by setting a special image property and relying on the 'ImagePropertiesFilter' property of Nova Scheduler to schedule the image properly.


A working deployment of Bright OpenStack 7.1 (Juno) is needed.


Code snippets used in this article are meant to be executed in the environment indicated by the shell prompts:

master#Shell on the head node
node001#Shell on the compute node
docker#Shell inside of running container
docker-image#Shell inside of software image chroot
cmsh#CMSH on the head node

Docker Nova driver installation

1. Preparing software image

First of all, the existing OpenStack software image is cloned for use in Docker:

cmsh# softwareimage cmsh# clone openstack-image docker-image cmsh# commit

After the image is cloned, a chroot is opened to this image:

cmsh# rshell docker-image

Required packages are installed:

docker-image# yum install docker git python-pip python-pbr python-oslo-utils python-pbr python-babel python-six python-oslo-serialization python-oslo-utils python-oslo-config 

Additional python modules are installed:

docker-image# pip install "oslo.concurrency<=0.2.0" "docker-py>=0.5.1"

A nova rootwrap file /etc/nova/rootwrap.d/docker.filter is created, with a directory created if needed. The file contains:

# nova-rootwrap command filters for setting up network in the docker driver # This file should be owned by (and only-writeable by) the root user [Filters] # nova/virt/docker/driver.py38: 'ln', '-sf', '/var/run/netns/.*' ln: CommandFilter, /bin/ln, root

 The Nova Docker driver is downloaded, and some patches are applied in order to make it work with Bright 7.1. The driver is then installed:

docker-image# git clone -b stable/juno docker-image# cd nova-docker docker-image# git show 04059f9c540bf4531a7a3294c09807f275208af4 | git apply docker-image# sed -i '/oslo.serialization/d' requirements.txt docker-image# python setup.py31 install

Docker is set up to use to use a tcp socket instead of unix,  in order to allow the Nova service to connect to it:

docker-image# echo "DOCKER_NETWORK_OPTIONS=-H tcp://" > /etc/sysconfig/docker-network

A bash alias is added for convenience:

docker-image# echo 'alias docker="docker -H tcp://"' >> /root/.bashrc

The chroot environment is exited:

docker-image# exit

2. Setup CMDaemon

A new category is created for Docker host nodes and the newly-created software image is set for that categoryt:

cmsh# category clone openstack-compute-hosts openstack-docker-compute-hosts cmsh# category use openstack-docker-compute-hosts cmsh# set softwareimage docker-image

The Docker service is added to the services:

cmsh# services cmsh# add docker cmsh# set autostart yes cmsh# set monitored yes cmsh# commit

The nova-compute service is set up to use Docker driver as a hypervisor:

cmsh# roles cmsh# use openstack::node cmsh# customizations cmsh# add nova.conf cmsh# entries cmsh# add DEFAULT:compute_driver=novadocker.virt.docker.DockerDriver cmsh# commit

/var/lib/docker is added to the excludelistupdate list:

cmsh# set excludelistupdate

using the text editor to add the line:

- /var/lib/docker

The nova-docker category is assigned to the nodes to be used as Docker hosts:

cmsh# device cmsh# set node001 category openstack-docker-compute-hosts cmsh# commit

Update nodes file system according to image:

device imageupdate -w -c openstack-docker-compute-hosts

After the operation is finished, nova-compute and Docker services are restarted:

cmsh# foreach -c openstack-docker-compute-hosts (services; restart openstack-nova-compute; restart docker)

Exit cmsh:

On the head node /etc/glance/glance-api.conf is edited and docker added to a list of supported images, e.g.:

[DEFAULT] container_formats=ami,ari,aki,bare,ovf,ova,docker

The glance-api service is restarted:

master# service openstack-glance-api restart

Now Docker should be up and running. The Docker hosts should be seen in the OpenStack Dashboard:

How do I use Docker With OpenStack?

3. Testing your docker setup

To test the newly-created setup, some images are first added to Glance. In order to do that, a running machine with Docker, and access to a master node, is needed. The newly-created docker compute hosts  (for example node001) can be used for this:

First of all, a Docker image is downloaded (e.g.

node001# docker pull redis

OpenStack credentials environment variables are set up:

node001# scp master:/root/.openstackrc /tmp/openstackrc node001# source /tmp/openstackrc node001# rm -f /tmp/openstackrc

The Docker image is added to Glance:

node001# docker save redis | glance image-create --is-public=True --container-format=docker --disk-format=raw --name redis

Note: The Glance image name must match name in Docker registry

Now an attempt should be made to launch an instance:

node001# nova boot --image "redis" --flavor m1.tiny test

It should be possible to locate the instance host node in dashboard (admin→instances):

How do I use Docker With OpenStack?

It should be possible to ssh to the host node and issue some Docker commands.

node001# docker ps

The shell can now be run inside the container

node001# CONTAINER_ID=d913b9975cc4 #your container id node001# docker exec -i -t $CONTAINER_ID  /bin/bash

4. Setting up access to metadata API

In order to be able to get instance metadata (e.g. instance id/name/flavor etc.), the instance network should be properly configured. Below is an example of such a network setup.

First a network is created, a subnet is added, and its gateway is set to some address:

How do I use Docker With OpenStack?

Next, the router is created, and its gateway set to the external network:

How do I use Docker With OpenStack?

After that, a second interface is added to the router, connected to the newly-created network, and its IP address set with a value for the subnet gateway:

How do I use Docker With OpenStack?

A new Docker instance is now launched using this network. The routes of the container should be checked:

docker# ip route list

The result should look something like this (note the default gateway value):

default via dev nsa806d1df-d7 dev nsa806d1df-d7  proto kernel  scope link  src

It should now be possible to access the metadata API from within the Docker container:

docker# curl


2012-08-10 2013-04-04 2013-10-17 latest


Most of time, if the Docker driver fails to boot the container, then the instance will fail to be created with an error "No valid host was found". The real error message is located in /var/log/nova/nova-compute.log on a host node. Below is a list of the most common nova-docker driver errors, along with ways to solve them.

1. Error: the compute node fails to spawn a new docker instance, nova-compute.log contains the following error:

RuntimeError: Cannot find any PID under container "CONTAINER_ID"

Reason: This error can occur for some images (e.g. busybox). It can happen if the container shuts down immediately after being created — this can be checked by having Docker run busybox on a host node.

Solution: Use images that have a running service as an entry point


2. Error: the compute node fails to spawn a new Docker instance. The nova-compute.log contains following error:

NovaException: Cannot load repository file: HTTPConnectionPool(host='', port=2375): Read timed out. (read timeout=10)

Reason: Several Docker operations such as transferring the Docker image from Glance require some time. The default timeout value is 10 seconds, and the nova-docker driver raises a timeout error after that.

Solution: increase the timeout value within /usr/lib/python2.7/site-packages/novadocker/virt/docker/client.py39 (search for the word "timeout")


3. Error: the compute node fails to spawn a new Docker instance. The nova-compute.log contains the error:

APIError: 404 Client Error: Not Found ("No such image: someimage")

Reason: most likely because the Glance image name differs from the Docker image name

Solution: recreate Glance image with the same name as the Docker image name


4. Error: failed to create a glance image using docker save | glance image-create. The command fails with:

[Errno 32] Broken pipe when trying to create glance image 

Reason: it takes time for Docker to create the image file. Glance can close the connection beyond a certain timeout value.

Solution: change/add timeout parameter in /etc/glance/glance-api.conf

[DEFAULT] client_socket_timeout=600

Alternatively the Docker image can be saved to a file and then supplied to a glance command:

node001# docker save your_image > your_image.img node001# cat your_image.img | glance image-create --container-format=docker --disk-format=raw --name your_image