Introduction

Jenkins is our preferred platform for Continuous Integration & Continuous Deployment automation. It took a little while to get set up, since we'd only just recently moved the host of the server from a physical machine to a virtual machine hosted in the cloud.  

At the start after moving to the new cloud infrastructure, our Jenkins was configured to only be ran on that cloud machine. Meaning we had one machine capable of running one build job at a time (Unreal Engine/Unreal Automation Tool doesn't like running two instances of itself on the same OS). This caused us quite some trouble in allocating when which team was permitted to use the Jenkins server to build their projects.
To solve this, I set up 2 additional machines that served as agents to Jenkins, meaning they could receive jobs from Jenkins, execute those jobs, and report the results. meaning that instead of one machine capable of running one job at a time, we now had three machines that could run three jobs simultaneously.  

Since then, I've also been able to add extra features to Jenkins, and optimize a lot of the build processes, through things such as my Jenkins Function Library and the work I've done with BuildGraph to make Jenkins easier, and more importantly, faster to work with.  

The Function Library I've prepared has also come in very useful. As it allows me to create common functions that multiple pipelines can benefit from. Just like Jenkins, it uses Groovy (Java-based) as it's language, which is a language I've grown very familiar with over the course of working with Jenkins.
For this Library, I used DavidTKate's library that we used previously as a base for my new library. Heavily modifying parts of it, and adding a lot of new functionality to it.
 

This new functionality was specifically targeted towards the custom engine distribution that we used on our Jenkins server to accelerate builds. Namely, an 'Installed Build' of Unreal Engine (similar to the one downloaded from the Epic games store, but also known under the names 'Rocket Build' or 'Precompiled Build').
I've also made sure to document the library's functions as much as possible, so that anyone who wants to use it can easily find out how to use it.  

Besides using the library to simplify the creation of pipeline scripts, It's actually also been essential in getting them to work in the first place. During the development of the pipeline for my BuildGraph, I had to figure out a way to create a pipeline script of dynamic size.
The script would take in a list of projects of indeterminate size, and generate all the steps that were needed to perform a build for each of those projects. This meant that in a sense, I had to create a script that could generate a script.  

Creating such a script is not necessarily challenging, but it causes the script to be quite large in size. Where a lot of variables have to be connected together in a certain way, and a lot of functions have to be called in correct succession. But the issue was that this Groovy pipeline had to run inside of a Java Virtual Machine, which has a maximum size of 64KB of bytecode for a single method. And since Groovy is a language that compiles to Java bytecode, this meant that I had to find a way to keep the size of the script below that 64KB limit.
Libraries however, count as a separate method, and therefore have their own 64KB limit. So by splitting the large parts of the script into library functions, I was able to keep the size of the script below the limit without sacrificing any actual functionality.  

Over the course of creating all of this functionality, I did ofcourse create quite some pipelines and scripts to test the functionality. So at the end, I was left with a few pipelines that were fully set up to build projects dynamically. Which could easily be used as templates for future projects with only having to change a few configuration settings.