Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker compose updates #9073

Merged
merged 9 commits into from
Nov 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion jekyll/_cci2/building-docker-images.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ NOTE: The _remote_ in "remote Docker" is a legacy term from when remote Docker u
[#introduction]
== Introduction

To build Docker images (for example, using `docker` or `docker-compose` commands) when using the Docker execution environment, you must use the `setup_remote_docker` key in your job:
To build Docker images (for example, using `docker` or `docker compose` commands) when using the Docker execution environment, you must use the `setup_remote_docker` key in your job:

{% include snippets/docker-auth.adoc %}

Expand Down
6 changes: 3 additions & 3 deletions jekyll/_cci2/concepts.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -281,9 +281,9 @@ jobs:
build2:
machine: # Specifies a machine image that uses
# an Ubuntu version 20.04 image with Docker 20.10.12
# and docker-compose 1.29.2, follow CircleCI Discuss Announcements
# and docker compose 1.29.2, follow CircleCI Discuss Announcements
# for new image releases.
image: ubuntu-2004:202201-02
image: ubuntu-2004:current
steps:
- checkout

Expand Down Expand Up @@ -410,7 +410,7 @@ See the xref:jobs-steps#[Jobs and steps] page for more information.
[#orbs]
== Orbs

Orbs are reusable snippets of code that help automate repeated processes, accelerate project setup, and make it straight forward to integrate with third-party tools.
Orbs are reusable snippets of code that help automate repeated processes, accelerate project setup, and help you to integrate with third-party tools.

The illustration in the <<configuration,Configuration>> section showing an example Java configuration could be simplified using orbs. The following illustration demonstrates a simplified configuration with link:https://circleci.com/developer/orbs/orb/circleci/maven[the Maven orb]. Here, the orb sets up a default executor that can execute steps with Maven and run a common job (`maven/test`).

Expand Down
74 changes: 36 additions & 38 deletions jekyll/_cci2/docker-compose.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,44 +6,56 @@ contentTags:
platform:
- Cloud
- Server v4.x
- Server v3.x
---

This document describes how to install and use Docker Compose, and assumes the reader has some experience using the `docker-compose` utility.
This page describes how to use Docker Compose in your CircleCI pipelines.

If you are new to Docker Compose, do consider reviewing the [official Docker Compose overview](https://docs.docker.com/compose/), or checking out the [Getting Started guide](https://docs.docker.com/compose/gettingstarted/).
If you are new to Docker Compose, you can review the [official Docker Compose overview](https://docs.docker.com/compose/), or checking out the [Getting Started guide](https://docs.docker.com/compose/gettingstarted/).

* TOC
{:toc}
## Using Docker Compose with machine executor
{: #using-docker-compose-with-machine-executor }

If you want to use Docker Compose to manage a multi-container setup with a Docker Compose file, use the `machine` key in your `.circleci/config.yml` file and use `docker compose` as you would normally (see Linux VM execution environment documentation [here]({{site.baseurl}}/using-linuxvm) for more details). That is, if you have a Docker Compose file that shares local directories with a container, this will work as expected.


## Using Docker Compose with docker executor
{: #using-docker-compose-with-docker-executor }

Using the `docker` execution environment combined with `setup_remote_docker` provides a remote engine similar to the one created with running Docker commands and the machine execution environment, but volume mounting and port forwarding do **not** work the same way in this setup. The remote docker daemon runs on a different system than the docker CLI and docker compose, so you must move data around to make this work. Mounting can usually be solved by making content available in a docker volume. It is possible to load data into a Docker volume by using `docker cp` to get the data from the CLI host into a container running on the docker remote host.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The remote docker daemon runs on a different system than the docker CLI and docker compose
This isn't strictly true now. Remote docker jobs now run on a machine executor, and that is where we spin up the jobs containers. The daemon is running on the same executor. I don't know if this can be reworded somehow to leave out such details 😖 Maybe it doesn't matter for your purposes here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I wondered about this - so is the whole explanation of moving data aropund to make it accessible outdated now?

This section is there to point out differences (that users need to be aware of) around using docker compose and the docker executor as an alternative to using docker compose and the machine executor. If this info is now irrelevant I can just remove it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose it does do that. It is perhaps a bit academic. I see the section has only moved as well. So, I'll give this a stamp

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👋 i am talking to a prospective customer and they pointed me to this paragraph as a reason for why they couldnt achieve the thing they wanted with remote docker.

i had the same reaction as you @DominicLavery which is that this wording is outdated because remote docker is basically machine now. if i open up a PR to edit the language, any guidance for what to remove Dom? definitley the pieces about "remote engine" and "remote docker daemon runs on a different system". any idea if the person still needs to do the docker cp stuff? i would assume no but the user i'm talking to said that it didnt work for them

"The issue I was referring to about docker is that as part of the deploy we run a command in a docker container (which has encapsulated some generic infrastructure setup) that requires as input a configuration file accessed via a bind mounted volume. Something like:

docker run -v ./config.yaml:/app/config.yaml ... image

This did not work"

also let me know if you want me to post and tag ask-execution for this question instead. the user did manage to get things working using docker cp so maybe that is still needed for some reason.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RE: docker cp
It doesn't really change anything around where the data lives. They still run in a primary docker container and so still might need to copy data around. A volume mount would occur on the hosts file system, not the primary containers so the config file or other data they would be after likely wouldn't be there

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#9170 lmk what you think


## Install Docker Compose

The `docker compose` utility is [pre-installed in the CircleCI convenience
The Docker Compose utility is [pre-installed in the CircleCI convenience
images]({{ site.baseurl }}/circleci-images/#pre-installed-tools) and machine executor images.

If you are using the Docker executor and **are not** using a convenience image, you can install Docker Compose into your [primary container]({{ site.baseurl }}/glossary/#primary-container) during the job execution with the Remote Docker Environment activated by adding the following to your [`.circleci/config.yml`]({{ site.baseurl }}/configuration-reference/) file:
If you are using the Docker executor and **are not** using a convenience image, you can install Docker Compose into your [primary container]({{ site.baseurl }}/glossary/#primary-container) during the job execution and use the xref:building-docker-images#[Remote Docker Environment], as follows:

```yml
- run:
name: Install Docker Compose
environment:
COMPOSE_VERSION: 'v2.25.0'
command: |
curl -sSL "https://github.com/docker/compose/releases/download/${COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m)" -o ~/docker-compose
mkdir -p ~/.docker/cli-plugins
chmod +x ~/docker-compose
mv ~/docker-compose ~/.docker/cli-plugins/docker-compose
# Add Docker's official GPG key:
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install docker-ce-cli docker-buildx-plugin docker-compose-plugin

- setup_remote_docker #activate the remote docker environment
```

The above code example assumes that you will also have `curl` available in your
executor. If you are constructing your own docker images, consider reading the
[custom docker images document]({{site.baseurl}}/custom-images/).

Then, to activate the Remote Docker Environment, add the `setup_remote_docker` step:

```yml
- setup_remote_docker
```
If you are constructing your own Docker images, consider reading the
[custom docker images page]({{site.baseurl}}/custom-images/).

This step enables you to run `docker compose` commands to build images:
Once you have Docker Compose installed, you can use it to build images and run containers:

```yml
- run:
Expand All @@ -63,7 +75,7 @@ Or to also verify if a service is running for example:

```yml
- run:
name: Start docker compose and verify service(s)
name: Start Docker Compose and verify service(s)
command: |
# Setting the Docker Compose project name to "circleci-demo-docker" means
# the names of our services' containers would be prefixed with "circleci-demo-docker".
Expand All @@ -85,19 +97,6 @@ See the [Example docker-compose Project](https://github.com/circleci/cci-demo-do

**Note**: The primary container runs in a separate environment from Remote Docker and the two cannot communicate directly. To interact with a running service, run a container in the service's network.

## Using Docker Compose with machine executor
{: #using-docker-compose-with-machine-executor }

If you want to use Docker Compose to manage a multi-container setup with a Docker Compose file, use the `machine` key in your `.circleci/config.yml` file and use `docker compose` as you would normally (see Linux VM execution environment documentation [here]({{site.baseurl}}/using-linuxvm) for more details). That is, if you have a Docker Compose file that shares local directories with a container, this will work as expected. Refer to Docker's documentation of [Your first docker-compose.yml file](https://docs.docker.com/get-started/part3/#your-first-docker-composeyml-file) for details.


## Using Docker Compose with docker executor
{: #using-docker-compose-with-docker-executor }

Using `docker` combined with `setup_remote_docker` provides a remote engine similar to the one created with docker-machine, but volume mounting and port forwarding do **not** work the same way in this setup. The remote docker daemon runs on a different system than the docker CLI and docker compose, so you must move data around to make this work. Mounting can usually be solved by making content available in a docker volume. It is possible to load data into a docker volume by using `docker cp` to get the data from the CLI host into a container running on the docker remote host.

This combination is required if you want to build docker images for deployment.

## Limitations
{: #limitations }

Expand All @@ -107,5 +106,4 @@ See [our support article for more information](https://support.circleci.com/hc/e
## See also
{: #see-also }


[Running Docker Commands]({{site.baseurl}}/building-docker-images/)
4 changes: 2 additions & 2 deletions jekyll/_cci2/private-images.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ contentTags:
platform:
- Cloud
---
= Using Docker Authenticated Pulls
= Using Docker authenticated pulls
:page-layout: classic-docs
:page-liquid:
:page-description: This document describes how to authenticate with your Docker registry provider to pull images.
Expand Down Expand Up @@ -116,7 +116,7 @@ jobs:
resource_class: large
working_directory: ~/my_app
steps:
# Docker is preinstalled, along with docker-compose
# Docker is preinstalled, along with docker compose
- checkout

# start proprietary DB using private Docker image
Expand Down
2 changes: 1 addition & 1 deletion jekyll/_cci2/using-docker.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ Docker also has built-in image caching and enables you to build, run, and publis
* Your application is self-sufficient.
* Your application requires additional services to be tested.
* Your application is distributed as a Docker image (requires using xref:building-docker-images#[Remote Docker]).
* You want to use `docker-compose` (requires using xref:building-docker-images#[Remote Docker]).
* You want to use `docker compose` (requires using xref:building-docker-images#[Remote Docker]).

Choosing Docker limits your runs to what is possible from within a Docker container (including our xref:building-docker-images#[Remote Docker] feature). For instance, if you require low-level access to the network or need to mount external volumes, consider using `machine`.

Expand Down
2 changes: 2 additions & 0 deletions styles/config/vocabularies/Docs/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ autogenerated
[Au]toscal[e|er|ing]
AWS
AWS Certificate Manager
AWS\sECR
Azure
backoff
[Bb]anlist?
Expand Down Expand Up @@ -99,6 +100,7 @@ Django
DLC
DNS
Docker
Docker\sHub
Dockerfiles?
[Dd]otfile
[Ee]nablement
Expand Down