Opensource Configuration Management tools integration with Cloud Automation Services – Part 1

In some earlier posts I wrote about the advantages of Configuration Management tools as Puppet Enterprise for installing software and desired state configuration and how easy it is to integrate it with automation solutions as VMware vRealize Automation and Cloud Automation Services.

But what if you don’t have Puppet Enterprise or want to use opensource tools? What alternatives are there?

There are many tools out there and all have there own strengths and weaknesses. But Ansible and Puppet Bolt are two opensource Configuration Management tools many of my customers are using or looking into.

In this first blogpost I’ll explain what you can do with Cloud-Init, Puppet Bolt and how to use both with VMware Cloud Automation Services.

In a second post I’ll explain Ansible and how to setup integration with Cloud Automation Services.

Cloud-init

Looking at VMware Cloud Automation Services, Cloud-init can be used to automate the configuration or installation of software within the guest during initialization. It’s a set of scripts that are executed as soon as a deployed instance is started. Cloud-config is the language of the scripts that Cloud-init knows to execute.

Cloud-init runs on Linux workloads; for Microsoft Windows workloads, the equivalent is CloudBase-init which supports the majority of cloud-config parameters. The service on the Operating System starts early at boot, retrieves metadata that has been provided from an external provider (metadata) or by direct user data supplied by the user, such as through VMware Cloud Assembly.

For each instance that you start in Cloud Automation Services with Cloud Assembly, you have the option of passing what is called user-data. This user-data is defined in your blueprint as Infrastructure-as-Code under the cloudConfig: parameter.

My colleague Erik wrote a blogpost about it. You can read it here.

Puppet Bolt

So, what is Puppet Bolt? Bolt automates your workflow with tasks and plans. It combines the declarative Puppet language model with familiar and convenient imperative code, making it easy to learn and effective for both one-off tasks and long-term Configuration Management cross platform.

Tasks are single actions that you run on target machines in your infrastructure using SSH or WinRM. These tasks can be as simple as starting and stopping services, running scripts or as complex as deploying and configuring a complete application.

Plans are sets of tasks that can be combined with other logic. This allows you to do complex task operations, such as running multiple tasks with one command. 

Tasks can be written in any programming language that can run on the target nodes, such as Bash, Python, or Ruby. 

You can read my earlier blogpost on Puppet Bolt to learn more about Tasks and Plans.

Manifest Blocks

Bolt plans are written in the Puppet language. Within a plan, you can use Bolt to apply blocks of Puppet code (manifest blocks) to remote nodes. You can create manifest blocks that use existing content from the Forge, or mix declarative resource configuration via manifest blocks with procedural orchestration and action in a plan. 

When you run a plan that contains a manifest block, the apply_prep  function installs the packages necessary to run the Bolt apply command.

The apply_prep function identifies the nodes that do not have Puppet agents and runs the puppet_agent::install  task (from the puppet_agent module). It also copies over custom facts from the Bolt module path and runs facter on the target nodes.

Create a manifest to install NGINX on Linux

In the next steps we’ll create a manifest that sets up a web server with nginx, and then run it as a plan.

1 – Install Puppet Bolt on Linux for example Ubuntu 16.04

wget https://apt.puppet.com/puppet-tools-release-xenial.deb
sudo dpkg -i puppet-tools-release-xenial.deb
sudo apt-get update 
sudo apt-get install puppet-bolt

2 – Create a directory Boltdir

3 – Within the Boltdir directory create a site  directory and a file called Puppetfile

4 – Edit the Puppetfile and add the following code:

forge 'http://forge.puppetlabs.com'
mod 'puppetlabs-stdlib', '4.25.1'
mod 'puppetlabs-concat', '4.2.1'
mod 'crayfishx-firewalld', '3.4.0'

These are modules coming from Puppet Forge which we will be using in our manifest.

5 – Within the site directory, create a profiles directory and within the profiles directory create three directories; plans, files and templates.

6 – In the plans directory, create a manifest file called nginx_install.pp and add the following code:

plan profiles::nginx_install(
     TargetSpec $nodes,
   ) {

     # Install the puppet-agent package if Puppet is not detected.
     # Copy over custom facts from the Bolt modulepath.
     # Run the `facter` command line tool to gather node information.
     $nodes.apply_prep

     # Compile the manifest block into a catalog
     apply($nodes) {
       if($facts['os']['family'] == 'redhat') {
         package { 'epel-release':
           ensure => present,
           before => Package['nginx'],
         }
         $html_dir = '/usr/share/nginx/html'
       } else {
         $html_dir = '/var/www/html'
       }

       # install nginx
       package {'nginx':
         ensure => present,
       }

       # deploy website
       file { '/Boltdir/site/profiles/files/sample_website':
       ensure  => directory,
       owner   => 'root',
       group   => 'root',
       mode    => '0755',
       path    => $html_dir,
       source  => "file:/Boltdir/site/profiles/files/sample_website",
       recurse => true,
       }

       file {"${html_dir}/index.html":
         content => epp('profiles/index.html.epp'),
         ensure  => file,
       }

       # start nginx
       service {'nginx':
         ensure  => 'running',
         enable  => 'true',
         require => Package['nginx']
       }

       # open http - port 80
       include firewalld

       firewalld_port { 'Open port for web':
       ensure   => present,
       zone     => 'public',
       port     => '80',
       protocol => 'tcp',
       }

     }
   }

This plan will install the Puppet agent if it is not detected, sets the html directory variable, installs the nginx package, deploys website content, starts nginx and opens tcp port 80 for http traffic.

The web content is stored in the files and templates directory.

7 – In the templates directory, create a template file called index_html.epp and add the following code:

<!DOCTYPE html>
<html>

<head>
    <title>NGINX Sample Website</title>
    <link rel="stylesheet" type="text/css" href="css/main.css">
    <link rel="icon" type="image/x-icon" href="img/favicon.ico">
</head>

<body>
    <div class="container">
        <div class="blurb">
          <h1>Welcome to NGINX installed on <%= $os[family] %> with Puppet Bolt!</h1>
          <p>If you see this page, the nginx web server is successfully deployed from VMware Cloud Automation Services.</p>
        </div>
    </div>
</body>

</html>

8 – In files directory, create a sample_website directory.

9 – In the sample_website directory create two directories; css and img.

10 – In the css directory, create a file called main.css and add the following code:

body {
  margin-top: 200px;
  margin-left: 60px;
  width: 70%;
  font-family: 'Helvetica', 'Arial', 'Sans-Serif';
  background:black url(../img/bolt.png) no-repeat left top;
}

a { text-decoration: none; color: #999; }
a:hover { text-decoration: underline; }

p, ul { font-size: 1.5em; line-height: 0.5em; color: #fff; }

h1, h2, h3, h4 { color: #66ffb2 }
h1 { font-size: 2em; }
h2 { font-size: 1.7em; }
h3 { font-size: 1.5em; }
h4 { font-size: 1.3em; }

nav ul, footer ul { padding: 0px; list-style: none; font-weight: bold; }
nav ul li, footer ul li { display: inline; margin-right: 20px; }

footer { border-top: 1px solid #d5d5d5; font-size: .8em; }

11 – In the img directory, create/download a bolt.png and a favicon.ico image.

12 – Run bolt puppetfile install

13 – Run the plan on the local node:

bolt plan run profiles::nginx_install nodes=localhost

After a successful run you should see the following:

This example including the Bolt Puppetfile, Bolt plan and website content can also be downloaded from my Github Bolt repository

Create a Cloud Assembly blueprint

To use our created Bolt plan in a Cloud Assembly blueprint we first have to put our Puppet code in a public accessible repository like Github.

Then create a blueprint with Cloud-config parameters to download the Puppet code, install Puppet Bolt and run the plan after a deployment of a new instance.

1 – Login to Cloud Assembly and create a new blueprint.

2 – Drag a new Cloud Machine and Cloud Network to the canvas and edit the generated YAML code. Add the following code:

Inputs:

inputs:
  environment:
    type: string
    enum:
      - dev
      - test
      - prod
    description: The environment where to deploy the VM
    title: Environment
  size:
    type: string
    enum:
      - small
      - medium
      - large
    description: The size of the VM
    title: VM Size
  sshkey:
    type: string
    encrypted: true
    title: Enter SSH Key
    description: The SSH-Key for connectivity

Cloud Machine:

resources:
  Linux_VM:
    type: Cloud.Machine
    properties:
      image: ddeswart-Ubuntu
      flavor: 'ddeswart-${input.size}'
      constraints:
        - tag: 'environment:${input.environment}'
      cloudConfig: |
        #cloud-config
        users:
          - name: dimitri
            ssh-authorized-keys:
              - ${input.sshkey}   

        runcmd:
        - mkdir -p /Boltdir
        - /usr/bin/git clone https://github.com/ddeswart/bolt-examples /Boltdir
        - wget https://apt.puppet.com/puppet6-release-xenial.deb
        - dpkg -i puppet6-release-xenial.deb
        - apt-get update -y
        - apt-get install puppet-bolt -y
        - bolt puppetfile install
        - bolt plan run profiles::nginx_install nodes=localhost

        network:
          config: disabled
      networks:
        - name: '${resource.VM_network.name}'
          assignment: dynamic
          network: '${resource.VM_network.id}'

Cloud Network:

VM_network:
    type: Cloud.Network
    properties:
      name: App-network
      networkType: existing
      constraints:
        - tag: 'networktype:public'

Change your YAML code accordingly using your own flavor mappings, image mappings, network/storage profiles and capability tags to steer your deployment.

3 – Deploy the blueprint

4 – Check the deployment

Yes, we have success, the deployed machine is configured and software is installed using Cloud-config and Puppet Bolt. 

In this example I used Ubuntu as Linux flavor but if you want to use RedHat or CentOS instead, you only have to change the cloudConfig parameters to install Bolt on a different Linux platform.

The plan does not have to change because the manifest works cross platform. Long live Configuration Management!