I was looking at a simple way to do GitOps with Nomad for my Homelab. Unlike the Kubernetes ecosystem, there doesn’t seem to be many examples of GitOps workflows in Nomad.
My goal was to find an easy way to automatically keep my Nomad jobs up to date with my Git repo.
Ensure Docker Desktop is running and run the following command within a terminal
nomad agent -dev -bind 0.0.0.0 -network-interface=en0
Now go to http://localhost:4646 and check that the Nomad UI shows.
Hashicorp recently released an official hashicorp/nomad Docker image that contains the Nomad binary built-in, which makes it super easy to interact with a Nomad cluster from a CI/CD pipeline.
Instead of using this within a CI/CD pipeline, we can use this Docker image inside a Nomad job that can interact with our cluster.
job "gitops" {
group "gitops" {
task "gitops" {
driver = "docker"
config {
image = "hashicorp/nomad"
entrypoint = ["/bin/sh", "-c", "while true; do sleep 500; done"]
}
env {
NOMAD_ADDR = "http://${attr.nomad.advertise.address}"
}
}
}
}
We can deploy this job to the cluster using the following command
$ nomad job run gitops.nomad.hcl
==> 2023-08-13T11:44:46+10:00: Monitoring evaluation "4d713567"
2023-08-13T11:44:46+10:00: Evaluation triggered by job "gitops"
2023-08-13T11:44:46+10:00: Allocation "cdb222d8" created: node "7595b72c", group "gitops"
2023-08-13T11:44:47+10:00: Evaluation within deployment: "cbafa6b7"
2023-08-13T11:44:47+10:00: Evaluation status changed: "pending" -> "complete"
==> 2023-08-13T11:44:47+10:00: Evaluation "4d713567" finished with status "complete"
==> 2023-08-13T11:44:47+10:00: Monitoring deployment "cbafa6b7"
✓ Deployment "cbafa6b7" successful
2023-08-13T11:45:06+10:00
ID = cbafa6b7
Job ID = gitops
Job Version = 2
Status = successful
Description = Deployment completed successfully
Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
gitops 1 1 1 0 2023-08-13T11:55:04+10:00
This task won’t do anything by itself, as we’ve overridden the entrypoint to just sleep the container, but this will allow us to shell into the container and verify that it can talk to Nomad.
We can use the nomad alloc exec command to shell into the container and manually run the nomad command to verify that it can connect to Nomad.
$ nomad alloc exec -job gitops sh
/ # nomad status
ID Type Priority Status Submit Date
gitops service 50 running 2023-08-13T01:44:46Z
The command correctly printed out what was running, which shows that the it can connect to Nomad.
Next we can use Nomad’s artifact block to clone a Git repo that contains some example Nomad jobs.
Before we continue, we should stop the old job and purge it (as we plan on switching the job type to “batch”, which requires removing the old job first).
nomad job stop gitops -purge
Next we can edit the job, adding the artifact block, changing the type to batch and changing the entrypoint so it runs a example job.
job "gitops" {
type = "batch"
group "gitops" {
task "gitops" {
driver = "docker"
config {
image = "hashicorp/nomad"
args = ["job", "run", "/local/repo/jobs/redis.nomad.hcl"]
}
env {
NOMAD_ADDR = "http://${attr.nomad.advertise.address}"
}
artifact {
source = "git::https://github.com/r-portas/nomad-examples"
destination = "local/repo"
}
}
}
}
Next lets run the job and check it runs correctly
$ nomad job run jobs/gitops/gitops.nomad.hcl
==> 2023-08-13T12:27:53+10:00: Monitoring evaluation "6d0cc470"
2023-08-13T12:27:53+10:00: Evaluation triggered by job "gitops"
2023-08-13T12:27:53+10:00: Allocation "cf418091" created: node "7595b72c", group "gitops"
2023-08-13T12:27:54+10:00: Evaluation status changed: "pending" -> "complete"
==> 2023-08-13T12:27:54+10:00: Evaluation "6d0cc470" finished with status "complete"
On the last line we can check that the evaluation completed successfully, which means the example redis job from the Git repo should be running, we can check by running the following
$ nomad job status redis
ID = redis
Name = redis
Submit Date = 2023-08-13T12:25:07+10:00
Type = service
Priority = 50
Datacenters = *
Namespace = default
Node Pool = default
Status = running
Periodic = false
Parameterized = false
Summary
Task Group Queued Starting Running Failed Complete Lost Unknown
redis 0 0 1 0 0 0 0
Latest Deployment
ID = 7389275f
Status = successful
Description = Deployment completed successfully
Deployed
Task Group Desired Placed Healthy Unhealthy Progress Deadline
redis 1 1 1 0 2023-08-13T12:35:25+10:00
Allocations
ID Node ID Task Group Version Desired Status Created Modified
3a81ae69 7595b72c redis 0 run running 7m27s ago 7m8s ago
Under the Summary section we can see that the redis task is running, which means the job was successfully deployed.