Building Go code, with and without Go modules, with Concourse

Introduction

The example code is available in directory build-golang of repository concourse-pipelines, separated in 2 branches:

branch golang-pre-modules is, well, pre-modules and requires to follow the GOPATH directory structure.
branch master is the same code, but converted to modules.

The directory structure:

build-golang/
├── README.adoc
├── ci
│   ├── build-golang-pipeline.yml
│   ├── build-task.yml
│   ├── build.sh
│   ├── unit-task.yml
│   └── unit.sh
├── cmd
│   └── cake         <1>
│       └── main.go
├── go.mod           <2>
├── go.sum           <2>
├── hello            <3>
    ├── hello.go
    └── hello_test.go

<1> An executable, that uses the hello package.
<2> Files go.mod and go.sum are added in the second example.
<3> The hello package.

Without Go modules

We begin with a minimal pipeline, nothing special (the pipeline in the sample repository has also the build job shown in the image above):

resources:
- name: concourse-pipelines
  type: git
  source:
    uri: https://github.com/marco-m/concourse-pipelines.git
    branch: golang-pre-modules
    paths: [build-golang/*]     <1>

jobs:
- name: unit
  plan:
  - get: concourse-pipelines
    trigger: true
  - task: unit
    file: concourse-pipelines/build-golang/ci/unit-task.yml

<1> The paths: directive normally is not needed. We use it because the sample repo contains multiple, independent examples and we want to trigger this specific pipeline only when something changes below the build-golang/ directory, not anywhere in the repo.

In the task configuration we use the optional task input path to create a directory structure compliant with GOPATH:

platform: linux

image_resource:
  type: registry-image
  source: {repository: golang}

inputs:
- name: concourse-pipelines
  path: gopath/src/github.com/marco-m/concourse-pipelines <1>

run:
  path: gopath/src/github.com/marco-m/concourse-pipelines/build-golang/ci/unit.sh <2>

<1> The task input path.
<2> The run script, specified according to the task input path.

Note: Concourse enforces the input path to be relative, this is why we cannot specify a path like /go/src/.... On the other hand, this is not a problem, as we will see in a moment.

Finally the run script:

export GOPATH=$PWD/gopath           <1>
export PATH=$PWD/gopath/bin:$PATH   <2>

cd gopath/src/github.com/marco-m/concourse-pipelines/build-golang <3>

echo
echo "Fetching dependencies..."
go get -v -t ./...        <4>

echo
echo "Running tests..."
go test -v ./...          <5>

<1> We set the GOPATH, using $PWD to make it absolute as Go requires.
<2> We update PATH accordingly. For this sample code this is not needed, but we show it nonetheless.
<3> We cd into the base directory of the project (one level below the task input path, as seen in the task configuration above).
<4> We explicitly fetch the dependencies.
<5> Finally we run the tests.

Optimization: task caching

As-is, the run script keeps downloading over and over the same dependencies.

We can use the task cache feature to cache the dependencies and so speed-up the build.

In the task configuration we add:

caches:
- path: depspath/
- path: gopath/pkg/

and in the run script we prepend the cache directories to the environment variables:

export GOPATH=$PWD/depspath:$PWD/gopath
export PATH=$PWD/depspath/bin:$PWD/gopath/bin:$PATH

(check the full source in the sample repo in case of doubt)

I took the trick about the 2-component GOPATH and depspath from the booklit project, which is a relatively simple Golang project built with Concourse, that I use as my reference. In case you are confused as I was the first time I saw it, depspath is not a special name, any name would do. It is simply the fact that

It is the first component of GOPATH
It is the cache

that makes it work :-)

Note also that with Go modules this 2-component path is not needed.

With Go modules

NOTE: This is not a tutorial about Go modules. Please refer to the official documentation at Go Modules.

Everything becomes simpler.

We create the Go module go.mod if needed (don’t forget to commit it to git, together with go.sum, that will be created automatically on first test/build):

$ cd build-golang
$ go mod init github.com/marco-m/concourse-pipelines/build-golang

The task file becomes a classic Concourse task file:

platform: linux

image_resource:
  type: registry-image
  source: {repository: golang}

inputs:  <1>
- name: concourse-pipelines

caches:  <2>
- path: gopath/

run:
  path: concourse-pipelines/build-golang/ci/unit.sh

<1> No more path: directive.
<2> No more depspath.

Also the run script becomes simpler:

#!/bin/bash

set -e -u -x

export GOPATH=$PWD/gopath           <1>
export PATH=$PWD/gopath/bin:$PATH

cd concourse-pipelines/build-golang <2>

echo
echo "Running tests..."             <3>
go test -v ./...

<1> Simple GOPATH.
<2> Concourse standard relative directory for the input.
<3> No more explicit fetching of the dependencies.

That’s it, happy building!

References

booklit a relatively small Golang project built by Concourse, written by the Concourse author himself! I use this project as my reference.
concourse-pipelines repository containing the full source of this example and other sample pipelines.

#concourse #cicd #golang #go-modules