A large part of the Tanzu product offering involves products and services that came with the acquisition of Pivotal.
One of the Tools that VMware has inherited from the Pivotal ecosystem, is a powerful CI/CD pipelining tool called Concourse.
Continuous Integration and Continuous Delivery (CI/CD) is a set of techniques and way of thinking, that enables developing and delivering code fast, which a high amount repeat-ability through automation.
Over the years, various tool sets that enable CI/CD have become popular, such as Jenkins, but there are many more. In fact VMware already has their own CI/CD tool, VMware Code Stream.
CI/CD tooling can now often be found as part of Git platforms such as Github and Gitlab.
While the Tanzu
Concourse CI has been the de-facto tool to use in Pivotal Cloud Foundry environments, and while Concourse itself is an open-source project, Pivotal is and, as VMware, remains much involved in its community.
Concourse is an open-source project, and its home is here:
So VMware now has two CI/CD pipeline tools in its portfolio. Asking around the community to see how people view the two tools in relation to each-other, the idea seems to be that VMware Code Stream is more on the CI (Continuous Integration) side, so more developer focused, while Concourse has always been more on the CD (continuous delivery) side, meaning its role traditionally has always been an Infrastructure automation tool, around the Pivotal products, and this will continue to be its primary use case (though it can be used for anything, technically).
As part of the VMware Tanzu offering; take a look at the official VMware page for concourse here: https://tanzu.vmware.com/concourse
It is from this page you can also get at all the (former) Pivotal resources regarding Concourse.
Concourse can be used to automate anything. It doesn’t execute logic itself, but is used to pipeline other bits of executing code, such as scripts. It executes those scripts inside containers that it spins up as needed. This execution mechanism makes it quite different from other CI/CD tools. It also makes its results highly reproducible. The concourse server itself only executes your pipelines, it doesn’t save any state itself.
If you want to learn more about how Concourse works, check out the links at the bottom of this post.
In this post I will cover the setup of a PhotonOS VM on VMware Workstation 15.5, and then the basic setup of Concourse running in docker with an example pipeline.
Setting up our Photon 3.0 VM
I am using VMware Workstation 15.5 to set up this VM. We are using the Minimal ISO image
To Execute, Concourse needs RAM, for demo purposes, 2Gb should be enough. You may find you need more. Out-of-memory errors can surface in strange ways in Concourse.. such as Docker runtime errors complaining about broken pipes.
When using the PhotonOS ‘minimal’ ISO, ssh login is not permitted to anyone by default. So we must first turn this on if we wish to use ssh.
Restart sshd to enable the ability to log in
I have unfortunately had a lot of problems with the systemd-resolved stub resolver. As you can see below, using it seems to fail in my lab causing “temporary failure in name resolution”, and I have not been able to figure out yet why this happens.
Changing my /etc/resolv.conf symlink to point to the static DNS file solves this issue.
I set IPtables to let ICMP in and out of the VM
Then we activate enable the docker daemon and turn it on
I then set up my login for dockerhub
Now lets install Git, because we want to be able to pull repositories onto our VM
We now need to install docker-compose. Grab the latest version from the official repo
curl -L https://github.com/docker/compose/releases/download/1.25.4/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose
Our VM is now ready for use. You might want to reboot it to check if everything keeps working.
Running Concourse in Docker
We are going to use the official Concourse project docker images using Docker-Compose.
If you have Kubernetes running, or you use BOSH, it is also available as a helm chart or BOSH deployment:
Concourse as a Docker deployment, consists of three containers: Postgres, the Concourse web frontend, and a worker node
Lets start by cloning the repo so we have all the files we need to work with
Now we need to run a small script in concourse-docker/keys that will generate the https keys that are needed for the 3 services to talk to each other
We must now change one thing in the docker-compose.yml
Because we are running concourse in a VM with a dedicated domain name, we must update the CONCOURSE_EXTERNAL_URL value to the name our VM can actually be resolved on.
Back in the main directory of the repo, we can now start Concourse by simply typing:
docker-compose up -d
( -d denotes that we want to run it as a daemon )
Docker will now pull the three images it needs and start the containers. It takes about 1GB of space for this.
Check the containers are running
We can now check to see if the UI is running on our VM, on port 8080
We can login using the default account ‘test’ with password ‘test’
There is not much to see here yet, as we have not yet loaded any pipelines into Concourse. We do that with the ‘fly’ command line tool.
The ‘fly’ cli tool
In order to interact with Concourse, we need to use the ‘fly’ tool. This is a single binary that you can download for mac, windows or linux, and enables us to manage our Concourse deployment and upload pipelines to it, among other things.
You can download the fly tool to our VM and run it there, or you can use it locally. In this case, I will run it from the Windows WSL Ubuntu environment.
You can download fly straight from web-interface. In fact this is quite important; the fly executable is tied to the specific version of Concourse you are using. It won’t work with an older version, its not backwards compatible.
If you find yourself having to deal with various versions of Concourse, you might want to keep various versions of the fly cli binary handly. One way to do this is to simply rename them to the various versions, like fly_580.
For working with WSL, I have a tools directory that I have added to my windows system path, that makes it available inside WSL
Fly can be used against different concourse servers. It saves these relationships as ‘targets’, that you are free to name anything.
To add a new target, and to login to current ones, we use the login command. The way concourse allows you to login is a little convoluted, it wants to do a token-exchange by having you visit a specific URL.
Copy-paste this URL into your browser.
If you are running your browser, and the fly command-line on the same machine, then Concourse will automatically send fly the token and automatically log you in.
If this mechanism does not work, you have to manually copy-paste the token from the webpage, into your terminal. In most terminals this will work ok, but I have had some trouble with putty sometimes. It may take you several attempts.
‘Teams’ are a way of dividing a Concourse server into different tenants. If you don’t specify a team, fly will log you in into the ‘main’ team.
Configuring a simple pipeline
We will now upload a simple demo pipeline to Concourse using fly. You can find it here:
https://github.com/thefluffysysop/Concourse , but it is merely a copy-paste from Stark & Wayne’s excellent Concourse Tutorial
You can clone it with this command: git clone https://github.com/thefluffysysop/Concourse.git
When setting a pipeline, fly will ask you to confirm the upload. If you are updating an existing pipeline with some changes, it will only show you the changes.
In the UI, our pipeline will appear. Visually, it contains just a single solid box. This is because our pipeline only contains one job.
We can click on our pipeline to see more details. Notice that it is paused.
We can press ‘play’ at the top right, this will ‘activate’ our pipeline, enabling any jobs we have to run.
The jobs will not run automatically however, they require specific triggers to start, or to be started manually.
We only have one job. We can start it manually by clicking into it, and pressing the ‘+’ button at the top right. This will start a new job execution, or in Concourse terminology, a ‘build’ Concourse will list the jobs that where triggered in the top bar and will number them. It will give them a color according to if they where executed successfully.
Jobs that are being run, will show in the pipeline screen as having yellow radiating lines around them, so you can easily see at which job your pipeline is.
A job consists of tasks, and our own job only had one task ‘hello world’. You can click on tasks as they are being executed to see their command line output.
When the job is complete, it turns green. In the output of the job, you can see the result of the command we specified in our task (inside our job, inside our pipeline). In this case its a just an echo, but it might as well have been a shell script or a remote ssh command. It could be anything.
Concourse is very powerful, and can be used to automate basically anything.
In the future, I will be sharing my own experiments with Concourse as I learn to use it.
Deploying the entire @pivotal #cloudfoundry stack using @concourseci as my very first pipeline, might be a bit ambitious. Something..something being thrown in deep-end. #vExpert#itqlife pic.twitter.com/jVcN1ENUaZ
— Thefluffysysop (@thefluffysysop) February 14, 2020
Since I started working with @concourseci I wanted to create a pipeline to refresh the BIND files used by my lab's #coredns , to manage DNS records in a repo and have the pipeline trigger off any changes. Finally works! My inexperience with Bash notwithstanding. #vExpert pic.twitter.com/BGJmOuLRQu
— Thefluffysysop (@thefluffysysop) March 8, 2020
If you want to learn your way around Concourse and make your own first pipeline, I very much recommend Start & Wayne’s Concourse tutorial to get your started.
The official Concourse wiki at Github, have an addition list of learning and tutorial resources
Pivotal (now VMware) maintains their own documentation at
https://docs.pivotal.io/p-concourse/v5/ , but bare in mind this is more oriented to viewing concourse as part of the larger Tanzu/Pivotal ecosystem such the Pivotal Cloud Foundry and Pivotal Container Services products.
There are some great videos on youtube that explain how Concourse works, and show demo’s
I also made a Concourse channel in the VMware Code Slack, come talk to me there!
I hope this post helps you on your way playing with Concourse for the first time. There is a good chance you will see Concourse pop up in the VMware ecosystem more in the future. Give it a try, and see how powerful it can be.