Automating Provisioning

Updated June 2022⌗

The update included replacing the 18.04 image link with the 22.04LTS

Getting Started⌗

This is the second in a series of posts about rapid standup of a lab environment. The first can be found here. This post will cover the basics of getting GOvmomi installed, linked, and usable for standing up an Ubuntu Cloud Image on a VCSA server. This can be modified to use on a local ESX host, and used for bootstrapping the VCSA.

alt text

VMWare VCLI is going to be used for the standup leveraging the GO implementation for deployment. The end result will be a rough implementation of basic tooling for a VMWare based on prem cloud deployment without the need to build a bunch more infrastructure to get started.

The assumption in this will be that we’re working from an Ubuntu Machine, either a VM, your local machine, or Window’s Subsystem for Linux. The repository is cloned down to the /opt/ directory and that it will be the working directory.

to get started download the latest releases from the [VMWare Govmomi] page, unzip, and link into your profile. Note: that this appends it to the very end of your user profile, and not overwrite anything.

#As root or sufficient permissions to be able to deploy to /usr/local/bin
mkdir /tmp/govc
pushd /tmp/govc
curl -L https://github.com/vmware/govmomi/releases/latest/download/govc_Linux_x86_64.tar.gz --output govc.tar.gz
tar xvf /tmp/govc.tar
mv govc /usr/local/bin/govc
#This part not needed if you don't want to make it globally accessible
ln -sf /usr/local/bin/govc /bin/govc
#adding govc to the shell environment
echo "eval $(/usr/local/bin/govc shellenv)" >> ~/.profile

We’re going to go ahead and pull down the latest Ubuntu 22.04 LTS cloud image at this point as well:

curl -L -C - https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.ova  --output 22.04-cloud.ova

Modifying for your environment⌗

As is almost always the case, the files used don’t have to retain their names, it just needs to be modified to maintain the reference name when working with it.

GOVC.sh⌗

We’re going to start with the govc.sh file

export GOVC_INSECURE=1
export GOVC_URL=vcsa.contoso.com
export GOVC_USERNAME=jdoe@contoso.com
export GOVC_PASSWORD=abadpassword
export GOVC_DATASTORE=VMFS_DATA_1
export GOVC_NETWORK="vm-network"
export GOVC_RESOURCE_POOL='/Ind/host/Dev/Resources'
export GOVC_DATACENTER='Ind'
export GOVC_HOST='192.0.2.13'

Starting at the top, GOVC_INSECURE=1 is telling the local machine to ignore any errors about the SSL cert, or SSH key not being known/valid when connecting. If you have trusted certs, are running this in a secure production environment where you should fail, then change it to 0.

GOVC_URL is pretty straightforward, it’s going to be the url of the VCSA, or ESX host that you’re connecting to. Along the same lines, GOVC_USERNAME, GOVC_PASSWORD, are the credentials to connect to either the ESXi or VCSA host that is managing the setup. GOVC_DATASTORE, GOVC_NETWORK, GOVC_RESOURCE_POOL, GOVC_DATACENTER, and GOVC_HOST is all of the information around where the new VM is going to actually reside. If you’re using Distributed Resource Sharing, then these values do not need to be set.

It’s also worth mentioning that these values can all be overrode either at the command line during command execution, or later on while setting the variables for the specific VM deployment. However, if these are going to be the same, then it makes sense to just go ahead and export them now for global usage. additional reading can be found at the [GOVC] command line reference page.

Once your file is modified, you can load it for usage by calling source govc.sh and then verified with govc about

Building the Ubuntu files⌗

We’re going to use the govc command line again to create the spec file for the ova that we downloaded earlier:

govc import.spec ./22.04-cloud.ova | python -m json.tool > ubuntu.json

There’s a lot going on in this command, so let’s break it down a little more govc import.spec ./18.04-cloud.ova is going to have the govc command line read the ova, and extract the specification to import the OVA.

Next up, we are piping (|) the command into python python -m json.tool and telling python to use the json.tool module to pretty print the otherwise minified JSON.

Finally, we’re going to output the pretty printed JSON to a file called ubuntu.json > ubuntu.json

{
    "DiskProvisioning": "thin",
    "IPAllocationPolicy": "dhcpPolicy",
    "IPProtocol": "IPv4",
    "InjectOvfEnv": false,
    "MarkAsTemplate": false,
    "Name": "Rocketman",
    "NetworkMapping": [
        {
            "Name": "VM Network",
            "Network": "vm-network"
        }
    ],
    "PowerOn": false,
    "PropertyMapping": [
        {
            "Key": "instance-id",
            "Value": "id-ovf"
        },
        {
            "Key": "hostname",
            "Value": "Rocketman"
        },
        {
            "Key": "seedfrom",
            "Value": ""
        },
        {
            "Key": "public-keys",
            "Value": ""
        },
        {
            "Key": "user-data",
            "Value": ""
        },
        {
            "Key": "password",
            "Value": ""
        }
    ],
    "WaitForIP": false
}

Again, there’s a lot going on in this file if we want there to be. But there doesn’t have to be. We can set if the disks need to be thin or thick provisioned, if we’re going to set a static IP address, IPv4 or 6,What the VMWare name of the VM will be, if it’s going to be a template, network mapping (as an ordered array), should the machine be powered on immediately after creation, the hostname, public SSH keys (for the cloud user ((deafult: ubuntu)), a hashed password, and the most important, user-data.

user-data.yml⌗

The user-data.yml file is a YAML mark up file, used by the cloud-init package. This is pretty standard among the [cloud-init] based distributions, and a large part of why we’re using the cloud image instead of a standard full or minified server image. It’s set to use the cloud-init tools by default, and is expecting this file to create some basic information to get bootstrapped for the provisioner.

This file can also get really complex if we need it to following the docs. Otherwise, it can remain simple and just do a little bit of work like creating a group, user, and pushing up the SSH key.

#cloud-config
groups:
  - cloud
users:
  - name: cloud
    ssh-authorized-keys:
      - ssh-rsa <RSA KEY> cloud@contocomp
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: sudo, cloud
    shell: /bin/bash
    primary_group: cloud
    passwd: $6$rounds=4096$<HASHED PASSWORD>

Gotcha: This file must start with the #cloud-config in order to be valid.

This is again, a pretty straightforward configuration file. The groups are created, one per line. Users get created with a section per user, and some defining information about that user. This includes: their login name, authorized keys, their sudo permissions, groups, which shell, the primary group, and the password for interactive authentication.

You can see the linked documentation above if you’d like to add more.

Once the file is created, some verification can be done to make sure that it’s valid before we inject it into the ubuntu.json file. Note: you will need the cloud-init tools installed on your machine to use these commands.

cloud-init devel schema --config-file ./user-data.yml This is going to verify whether or not we have a valid syntax file. Valid cloud-config file ./user-data.yml

Next, we’re going to base64 encode the contents of this file for embedding into the ubuntu.json file. base64 ./user-data.yml If you’d like to verify the contents are the same after encoding and decoding: base64 user-data.yml | base64 -d

Once comfortable and valid, we will take that base64 encoded data and be pasting it into ubuntu.json file. Gotcha: make sure that there are no line breaks in the base64 encoded data when pasting it into the ubuntu.json file

I2Nsb3VkLWNvbmZpZwpncm91cHM6CiAgLSBjbG91ZAp1c2VyczoKICAtIG5hbWU6IGNsb3VkCiAgICBzc2gtYXV0aG9yaXplZC1rZXlzOgogICAgICAtIHNzaC1yc2EgPFJTQSBLRVk+IGNsb3VkQG9yY2hjb21wCiAgICBzdWRvOiBBTEw9KEFMTCkgTk9QQVNTV0Q6QUxMCiAgICBncm91cHM6IHN1ZG8sIGNsb3VkCiAgICBzaGVsbDogL2Jpbi9iYXNoCiAgICBwcmltYXJ5X2dyb3VwOiBjbG91ZAogICAgcGFzc3dkOiAkNiRyb3VuZHM9NDA5NiQ8cGFzc3dvcmQgaGFzaD4K

{
    "DiskProvisioning": "thin",
    "IPAllocationPolicy": "dhcpPolicy",
    "IPProtocol": "IPv4",
    "InjectOvfEnv": false,
    "MarkAsTemplate": false,
    "Name": "Rocketman",
    "NetworkMapping": [
        {
            "Name": "VM Network",
            "Network": "vm-network"
        }
    ],
    "PowerOn": false,
    "PropertyMapping": [
        {
            "Key": "instance-id",
            "Value": "id-ovf"
        },
        {
            "Key": "hostname",
            "Value": "Rocketman"
        },
        {
            "Key": "seedfrom",
            "Value": ""
        },
        {
            "Key": "public-keys",
            "Value": ""
        },
        {
            "Key": "user-data",
            "Value": "I2Nsb3VkLWNvbmZpZwpncm91cHM6CiAgLSBjbG91ZAp1c2VyczoKICAtIG5hbWU6IGNsb3VkCiAgICBzc2gtYXV0aG9yaXplZC1rZXlzOgogICAgICAtIHNzaC1yc2EgPFJTQSBLRVk+IGNsb3VkQG9yY2hjb21wCiAgICBzdWRvOiBBTEw9KEFMTCkgTk9QQVNTV0Q6QUxMCiAgICBncm91cHM6IHN1ZG8sIGNsb3VkCiAgICBzaGVsbDogL2Jpbi9iYXNoCiAgICBwcmltYXJ5X2dyb3VwOiBjbG91ZAogICAgcGFzc3dkOiAkNiRyb3VuZHM9NDA5NiQ8cGFzc3dvcmQgaGFzaD4K"
        },
        {
            "Key": "password",
            "Value": ""
        }
    ],
    "WaitForIP": false
}

Gluing the pieces together⌗

Finally, we’re ready to provision and create the vm! one command will do the work of connecting to the VCSA/ESXi server, and deploying the image and configurations. govc import.ova -options=ubuntu.json ./18.04-cloud.ova

But maybe we want to make a few changes to that basic creation, and also power the vm on after making those modification:

govc vm.change -vm rocketman -c 2 -m 2048 -e="disk.enableUUID=1"
govc vm.disk.change -vm rocketman -disk.label "Hard disk 1" -size 30G
govc vm.disk.create -vm rocketman  -name rocketman/disk2 -size 100G -eager -thick -sharing sharingMultiWriter
govc vm.power -on=true rocketman

With these commands we’re going to do basic VM maninpulation of giving the VM two cores, and two gigs of RAM, as well as enabling disk UUID of 1. govc vm.change -vm rocketman -c 2 -m 2048 -e="disk.enableUUID=1"

Then we’re going to go ahead and modify that first disk to be 30 gigs large govc vm.disk.change -vm rocketman -disk.label "Hard disk 1" -size 30G

Next, maybe our rocketman needs yet another disk for bulk storage, or to be a database host govc vm.disk.create -vm rocketman -name rocketman/disk2 -size 100G -eager -thick -sharing sharingMultiWriter we’ve also decided since this one is going to be doing heavy reads, we don’t want to wait on it being resized on the fly.

Finally, we’re ready to power it up. govc vm.power -on=true rocketman

But wait there’s more⌗

Now you’re thinking, “Great, it’s created, but now I need to go the VCSA and monitor to see what IP address the VM received from DHCP.” Not so fast, GOVC also has that ability: watch -n 10 govc vm.info rocketman

With this, we can have the Unix utility refresh every ten seconds with the vm information.

Once we know that IP address, we can go ahead and log in to it. ssh cloud@192.0.2.100 -i ~/.ssh/id_rsa

And like that, we’ve got some basic tooling to create a few more.

created the basic tooling required for our on prem VMWare based cloud deployment model. Now, some of the same tooling that might be used for Amazon Web Services, Azure, Google Cloud Platform, etc. are on prem leading to a more robust consistent experience when scaling locally.

Next up, automating machine configuration with Ansible.

References: VMWare Govmomi first post GOVC cloud-init