WindowsPE Live-CD in Jenkins/Foreman Infrastructure

Author: Hostkey DevOps Team Leader Egor Garadzha

We have already covered the setup of the Centos/RockyLinux based LiveCD build kitchen for Linux that we use at Hostkey. It's time to talk about building a WindowsPE distribution in a Linux infrastructure, automating this process using Jenkins, and deploying MS Windows-based systems through this helper.

Traditionally, PXE-deploy Windows is implemented as part of a common Microsoft infrastructure, including DHCP / DNS / TFTP and other services. In our infrastructure, the host server for deploying the OS is Foreman, and it seemed beside the purpose to deploy the deployment environment separately for Microsoft from the very beginning.

Many years ago, we chose the path of downloading a WindowsPE wim image via iPXE. The image was manually assembled from Microsoft infrastructure. One person was watching the kitchen. As the company grew, and along with it the complexity of it tasks, we decided to automate the assembly and image management processes as much as possible. At the same time, we faced the task of supporting UEFI installations, which turned out to be not as easy as one might assume.

Automation of Image preparation

The first need was to get rid of the Microsoft infrastructure for imaging. To this end, we decided to use the old wimtools project. In this case, its support for RedHat systems is at an advanced enough level, and so we got a ready-made SPEC.

This set of console utilities allows you to work with wim images and modify them. To prepare the WindowsPE image, we need:

• Vanilla Windows distribution (for example, Windows 10);
• The mkwinpeimg utility from wimtools;
• A starting cmd script for WindowsPE.

During the build process, you can overlay and add an arbitrary directory to the set of scripts and software you need. We added to WinPE the latest version of PowerShell from Github Microsoft, a set of scripts for installing the OS, and a number of utilities that we also needed to interact with Foreman.

Build command example:

mkwinpeimg --iso --windows-dir=/mnt/windowsrepo --overlay=./PEAddons --start-script=./startpe.cmd windowspe-${BUILD_NUMBER}.iso

The path /mnt/windowsrepo is for the loop-mounted ISO image of the vanilla version of Windows 10.

After long hours of work on the image, we have formed the following assembly kitchen, more or less:

• The Jenkins task for building wimtools-rpm (we discussed earlier how you can organize basic work with packages in the infrastructure);

• The Jenkins task to build and deliver the ISO image to the WindowsPE central repository;

• The Git repository, containing the main set of scripts and small binaries for working inside the image. The same startpe.cmd and the PEAddons directory, inside which we have created a directory for PowerShell, a bin for small utilities like Go tool written by ourselves for working with “syslog” and “scripts” - for additional scripts, and so on.

• We have also placed the repository for the windows_driver_store device drivers into Git for versioning, and any problems with large binaries are solved using LFS.

• Our internal mirror with vanilla distributions, where the donor ISO is drawn from to create the PE image. The same repository had already been used while installing the OS.

Everything except the ISO mirror is stored and versioned in Git. Needless to say, it is impossible to understate the importance and convenience of this if more than one person works on the infrastructure.

If you have everything you need in Git, and a set of Linux build utilities, the task can be easily automated in Jenkins:

#!/usr/bin/env bash

ssh-agent bash -c "ssh-add $GITLAB_KEY; git clone -b $short_branch [email protected]:windows_driver_store.git PEAddons/drivers" 
wget -nv https://github.com/PowerShell/PowerShell/releases/download/v$PWSHVER/PowerShell-$PWSHVER-win-x64.zip
        
unzip -qqo PowerShell-$PWSHVER-win-x64.zip -d PEAddons/pwsh/ || true
rm -rf PowerShell-$PWSHVER-win-x64.zip PEAddons/drivers/.git
sed "1aecho Starting windowspe-${BUILD_NUMBER}" -i ./startpe.cmd

mkwinpeimg --iso --windows-dir=/mnt/windowsrepo --overlay=./PEAddons --start-script=./startpe.cmd windowspe-${BUILD_NUMBER}.iso

Scripts and linking with Foreman

An important task when performing the deployment of a system like this is the organization of the image itself, in which control over its behavior will be transferred to an external service. This kind of solution will eliminate the need to rebuild the image for each individual installation.

According to the WindowsPE standard, the start script should be startpe.cmd. This is, in fact, a regular bat file with all the limitations of the standard Microsoft shell. We use it to start PowerShell scripts.

The first and main script is the so-called getforeman.ps1 script, which is launched by startpe.cmd (code below: PowerShell).

The main problem when organizing a Windows deployment by drawing from its ISO image via iPXE is that you cannot throw any parameters into the image; that is, the image inside must receive enough information to then fetch the Foreman scripts. The scripts themselves are generated during the deployment process and are available via links such as the following:

http://foreman.example.com/unattended/provision?token=xxxxxxxx

So, we need a host token. It can be obtained from Foreman itself if you know the host name, which is provided, including to via dhcp host itself.

To get its name in the CLI, we use the dhcptest.exe utility:

$dhcphostname = ( & $env:SystemDrive\bin\dhcptest.exe --mac $netinfo.macaddress --query --print-only 12[str] --quiet )

Next, we get the host token by making a request to Foreman through a user with view permissions:

$creds = "$($user):$($foremanpass.$location)"
$uri = $foremanuri.$location

$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($creds))

$basicAuthValue = "Basic $encodedCreds"

$Headers = @{
  Authorization = $basicAuthValue
}

$foremanhostinfo = @{ $location=$(Invoke-RestMethod -Uri "$uri/api/hosts/$dhcphostname" -Headers $Headers -SkipCertificateCheck) }
$token = $foremanhostinfo.$location.token

Using the token, we can request all the necessary provision scripts directly from Foreman, and the task is complete.

Further tasks are already connected with the organization of the installation procedure on Foreman. This is also quite an extensive topic, especially considering UEFI deployment, which we hope to dedicate a separate article to.

An important question in such a system remains the issue of security. Foreman’s address, the restricted user for interacting with it, and other parameters have to be built directly into the script for accessing information and, if any of these change, the image has to be rebuilt. If the issue of convenience is resolved by assembly, then security is a subtler matter. Indeed, it was security issues that made us look towards more complex installation options that exclude WindowsPE in the first place. However, in non-public infrastructures which require Windows but don't require a large Microsoft infrastructure to deploy, performing your systems deployment through WindowsPE can be a good option.