Puppet BoltWhen you read my previous posts you probably know I’m a fan of Puppet. During the holidays I finally had the time to learn more about Puppet Bolt. If you don’t know what Bolt is I encourage you try it out asap. It’s an amazing new opensource tool from Puppet to automate tasks on remote systems on an as-needed basis, for example, run a script, deploying an application, checking compliancy, starting and stopping services. Bolt connects directly to remote nodes with SSH or WinRM, so you are not required to install any agent software. It’s a great way to start automating.

In this post I’ll use Puppet Bolt to automate the installation of VMware vRealize Endpoint Operations (EPOps) 7.0 agents on RHEL/CentOS systems.

Requirements:

  • VMware vRealize Operations 7.0 Advanced.
  • Endpoint Operations 7.0 agent package for RHEL/CentOS7.
  • RHEL/CentOS 7 system for using Bolt.
  • RHEL/CentOS 7 system(s) to install EPOps agent on.

Installing Bolt and Puppet Development Kit (PDK)

Bolt is very easy to install.

sudo rpm -Uvh https://yum.puppet.com/puppet6/puppet6-release-el-7.noarch.rpm
sudo yum install -y puppet-bolt

To install PDK.

sudo yum install -y pdk

PDK provides integrated testing tools and a command line interface to help you develop, validate, and test Puppet modules.

In root create a Boltdir directory. Within Boltdir create a modules directory. From modules use PDK to create a module named epops.

mkdir ~/Boltdir
cd ~/Boltdir
mkdir modules
cd modules
pdk new module epops

Puppet Bolt

Next thing to do is create a private/public keypair and use this to connect to the remote systems using SSH. You can do this manual but I found a script on the internet from Dominik Stadler to automate it. You can download it from his Github account. Before running the script, change the default FILENAME=~/.ssh/id_test into ~/.ssh/id_rsa which is the default used by Bolt.

cd ~
git clone git://github.com/centic9/generate-and-send-ssh-key
cd generate-and-send-ssh-key
./generate-and-send-ssh-key.sh --user root --host node1
Puppet Bolt Puppet Bolt Puppet Bolt

Go to Boltdir and create an inventory file to store information about your nodes and refer to them as a group.

cd ~/Boltdir
touch inventory.yaml

Edit and save the following to inventory.yaml:

---
nodes: [node1, node2, node3]
config:
  ssh:
    host-key-check: false

Check if Bolt works by running a simple command.

cd ~
bolt command run uptime --nodes all

Puppet Bolt

Building my first task

Tasks are single actions that you run on target machines in your infrastructure. They can be written in any programming language that can run on the target nodes, such as Bash, Python, or Ruby. Tasks are packaged within modules, so you can reuse, download, and share tasks on the Forge. Task metadata describes the task, validates input, and controls how the task runner executes the task.

My first task will be a bash script to check if the EPOps agent is already installed and running.

Go to ~/Boltdir/modules/epops and create a new task with PDK.

cd ~/Boltdir/modules/epops 
pdk new task ep_check
cd tasks

Puppet Bolt

This will create the tasks directory including an empty ep_check.sh script file and a metadata file ep_check.json. The metadata file should contain a short task description and any parameters the task accepts. I don’t have any parameters but we could put something like “Checks if EPOps 7.0 is installed and if the agent is running” into the metadata description field.

Edit and save the following to ep_check.json:

{
  "puppet_task_version": 1,
  "supports_noop": false,
  "description": "Checks if EPOps 7.0 is installed and if the agent is running",
  "parameters": {
  }
}

Edit and save the following to ep_check.sh:

#!/bin/bash
#
if [[ -n $(find /opt/ -type d -name "epops*") ]]
then
  echo "EPOps install directory found"
  if [[ -n $(systemctl list-units --type service --all | grep "epops-agent") ]]
  then
    if [[ -n $(service epops-agent status | grep "is not running") ]]
    then
      echo "EPOps agent is NOT running"
      echo "Trying to start agent..."
      service epops-agent start
      service epops-agent status
    else
      echo "EPOps agent is running"
      service epops-agent status
    fi  
  else
    echo "EPOps agent service is NOT installed, try to start/stop it manually from EPOps bin directory"
  fi
else
  echo "EPOps install directory NOT found"
fi

Check if the script works by running the Bolt task run command.

cd ~
bolt task run epops::ep_check --nodes all

Puppet Bolt

As you can see two nodes don’t have the EPOps agent installed, one does but the status of the agent can not be verified because there’s no service installed which concludes that EPOps is not installed through a rpm package.

The installation process

Next are the tasks needed for the installation of the EPOps agent. We can divide the installation in four stages:

1 – Download and unzip the EPOps agent installation package

2 – Delete the default agent.properties file which comes with the installation package

3 – Copy a prepared agent.properties file to the EPOps installation directory

4 – Start the EPOps agent

For stages 1 and 4 I’ll create a Bolt task.

cd ~/Boltdir/modules/epops 
pdk new task ep_install
pdk new task ep_start
cd tasks

Edit and save the following to ep_install.json:

{
  "puppet_task_version": 1,
  "supports_noop": false,
  "description": "Creates the EPOps 7.0 install directory and downloads agent software",
  "parameters": {
   "installdir": {
     "description": "Install directory of the EPOps 7.0 agent software",
     "type": "String"
     }
  }
}

Edit and save the following to ep_install.sh:

#!/bin/bash
#
echo "Stopping firewall"
systemctl stop firewalld
systemctl disable firewalld
#
echo "Create install directory and download agent software"
cd /opt/
mkdir $PT_installdir
cd $PT_installdir/
# DONT FORGET TO CHANGE IP ADDRESS, PORT, USERNAME AND PASSWORD!!!
wget -c https://<IP ADDRESS>:<PORT>/software_repo/epops-linux-7.0.tar.gz --no-check-certificate --user <USERNAME> --password <PASSWORD>
#
echo "Unpack agent software"
tar zxfp epops-linux-7.0.tar.gz

This bash script will create an installation directory using a task parameter PT_installdir, download and unpack the EPOps agent software in this directory.

The ep_install script can be tested with the command:

bolt task run epops::ep_install installdir=vmguru --nodes all

Edit and save the following to ep_start.json:

{
  "puppet_task_version": 1,
  "supports_noop": false,
  "description": "Starts the EPOps 7.0 agent",
  "parameters": {
   "installdir": {
     "description": "Install directory of the EPOps 7.0 agent software",
     "type": "String"
     }
  }
}

Edit and save the following to ep_start.sh:

#!/bin/sh
#
echo "Starting EPOps agent"
cd /opt/${PT_installdir}/epops-agent-7.0.0/bin/
./ep-agent.sh start
wait
echo "Done"

This bash script will go to the EPOps installation directory using the same parameter PT_installdir as in the previous task and run the agent start script available in the bin directory.

The ep_start script can be tested with the command:

bolt task run epops::ep_start installdir=vmguru --nodes all

For stage 2, deleting the agent.properties file, we can use Bolt command run:

bolt command run 'rm /opt/<INSTALL DIR>/epops-agent-7.0.0/conf/agent.properties --force' --nodes all

For stage 3, copying the prepared agent.properties file from the Bolt node to the remote nodes, we can use Bolt file upload.

First prepare an agent.properties file to be used in your vRealize Operations environment and place it in the files directory.

cd ~/Boltdir/modules/epops/files
touch linux-agent.properties

Edit and save the following to linux-agent.properties:

# Agent configuration file

## Use the following properties to automate agent setup
## using these values.
##
## If any properties do not have values specified, the setup
## process prompts for their values.
##
## If the value to use during automated setup is the default,
## use the string *default* as the value for the option.

## DONT FORGET TO ADD YOUR VROPS SERVER ADDRES, ADMIN USERNAME AND PASSWORD!!!
agent.setup.serverIP=<FQDN OR IP ADDRESS OF VROPS SERVER>
agent.setup.serverSSLPort=443
agent.setup.serverLogin=<USER>
agent.setup.serverPword=<PASSWORD>

##
## The server certificate thumbprint to trust, when performing silent installation.
## You can specify either SHA1 or SHA256 algorithms (in hexadecimal format).
##
## DONT FORGET TO ADD YOUR VROPS SERVER SECURITY THUMBPRINT (CAN BE FOUND ON THE VROPS ADMIN PAGE)!!!
agent.setup.serverCertificateThumbprint=<CERTIFICATE THUMBPRINT>

# Mirror /proc/net/tcp on linux
sigar.mirror.procnet=true

##
## Logging settings
##
agent.logFile=${agent.logDir}/agent.log
agent.logLevel=INFO
log4j.rootLogger=${agent.logLevel}, R
log4j.appender.R.File=${agent.logFile}
log4j.appender.R.MaxBackupIndex=1
log4j.appender.R.MaxFileSize=5000KB
log4j.appender.R.layout.ConversionPattern=%d{dd-MM-yyyy HH:mm:ss,SSS z} %-5p [%t] [%c{1}] %m%n
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R=org.apache.log4j.RollingFileAppender

##
## Redirect System.{err,out} to agent.log
## commenting out will leave System.{out,err} directed to agent.log.startup
##
agent.logLevel.SystemErr=ERROR
agent.logLevel.SystemOut=INFO

##
## Disable overly verbose logging
##
log4j.logger.org.apache.http=ERROR
log4j.logger.org.springframework.web.client.RestTemplate=ERROR
log4j.logger.org.hyperic.hq.measurement.agent.server.SenderThread=INFO
log4j.logger.org.hyperic.hq.agent.server.AgentDListProvider=INFO
log4j.logger.org.hyperic.hq.agent.server.MeasurementSchedule=INFO
log4j.logger.org.hyperic.util.units=INFO
log4j.logger.org.hyperic.hq.product.pluginxml=INFO

# Only log errors from naming context
log4j.category.org.jnp.interfaces.NamingContext=ERROR
log4j.category.org.apache.axis=ERROR

##
## Auto-inventory options
##

#autoinventory.defaultScan.interval.millis=86400000
autoinventory.syncScan.interval.millis=86400000

##
## SSL Settings
##
## The path and password to the keystore file, used to set up the SSL
## on the agent. Only required for use with your existing keystore.
## The path must be an absolute path.
## Please use the slash ("/") sign also for Windows path.
## i.e: use agent.keystore.path=c:/keystore and NOT agent.keystore.path=c:\keystore
##
# agent.keystore.path=
# agent.keystore.password=
##
## Automatically accept unverified certificates.
## Relevant only for plugins that monitor over ssl and not for the agent itself.(Agent Registration ignores it)
## For silent registration, use: agent.setup.serverCertificateThumbprint
accept.unverified.certificates=true

## Define vsphere plugin thread size (default is 1)
scheduleThread.poolsize.vsphere=2

## Should the agent adjust the metric results to account for the time offset of the server?
agent.deductServerTimeDiff=true

## Frequency for fetching commands from the server in milliseconds
agent.setup.uniPollingFrequency=60000

#Enable NFS discovery and collection by default
system.nfs.discover=trueolt file upload epops/lnx-agent.properties /opt/<INSTALL DIR>/epops-agent-7.0.0/conf/agent.properties --nodes all

Uploading the file can be tested with the command:

bolt file upload epops/linux-agent.properties /opt/<INSTALL DIR>/epops-agent-7.0.0/conf/agent.properties --nodes all

Bolting it all together

Now that we have all the necessary tasks and commands, how do we put this all together? This is were a Bolt plan comes into play. Plans allow you to run more than one task with a single command, compute values for the input to a task, process the results of tasks, or make decisions based on the result of running a task.

Plans are written in the Puppet language and given the .pp extension and should be placed in the module’s /plans directory. Plans can use any combination of Bolt functions or built-in Puppet functions. This makes a plan a very powerful tool. It’s even possible to apply Puppet manifest code using modules from Forge.

What I want to achieve with my plan is to check if the EPOps agent is installed. If this is the case I’ll skip the installation process. If no EPOps installation files are found, I’ll start the installation process.

By adding exit codes to my ep_check script I’ll be able to check the outcome and use the result to skip or continue the installation.

Go to the /epops/tasks directory, edit the ep_check.sh file and add the following exit codes; exit 0 = success, exit 1 = failure.

#!/bin/bash
#
if [[ -n $(find /opt/ -type d -name "epops*") ]]
then
  echo "EPOps install directory found"
  if [[ -n $(systemctl list-units --type service --all | grep "epops-agent") ]]
  then
    if [[ -n $(service epops-agent status | grep "is not running") ]]
    then
      echo "EPOps agent is NOT running"
      echo "Trying to start agent..."
      service epops-agent start
      service epops-agent status
      exit 0
    else
      echo "EPOps agent is running"
      service epops-agent status
      exit 0
    fi  
  else
    echo "EPOps agent service is NOT installed, try to start/stop it manually from EPOps bin directory"
    exit 0
  fi
else
  echo "EPOps install directory NOT found"
  exit 1
fi

Let’s test the exit codes.

bolt task run epops::ep_check --nodes all

Puppet Bolt

Now we’re ready to build our plan. Go to ~/Boltdir/modules/epops and create a plans directory including a new plan.

cd ~/Boltdir/modules/epops
mkdir plans
cd plans
touch epops70_linux.pp

Edit and save the following to epops70_linux.pp:

plan epops::epops70_linux(
  TargetSpec $nodes,
  String     $installdir = 'vmware',
) {
  $resultset = run_task('epops::ep_check', $nodes, '_catch_errors' => true)
  $resultset.each |$result| {
    $node = $result.target.name
    if $result.ok {
      notice("Info: Node '${node}' already contains the EPOps Agent, skipping.")
    } else {
      notice("Info: Node '${node}' requires installation of the EPOps Agent, continuing")
      run_task('epops::ep_install', $node, installdir => $installdir)
      run_command("rm /opt/${installdir}/epops-agent-7.0.0/conf/agent.properties --force", $node)
      upload_file('epops/linux-agent.properties', "/opt/${installdir}/epops-agent-7.0.0/conf/agent.properties", $node)
      run_task('epops::ep_start', $node, installdir => $installdir)
    }
  }
}

“_catch_errors => true” prevents the plan from stopping on error during the task execution of ep_check. The result of each node ($resultset.each) is then checked on success or failure ($result.ok).  Success (exit 0) means the installation is skipped.  Failure (exit 1) means the installation continues.

In a plan we don’t use the Bolt CLI commands but execution functions. See the tabel below.

Bolt CLI Plan execution function
bolt command run run_command
bolt task run run_task
bolt file upload upload_file

$installdir is the default EPOps installation directory. “vmware” will be used if the installdir option is not used in the Bolt plan run command.

Let’s test the plan by running:

bolt plan run epops::epops70_linux installdir=vmguru nodes=all

Puppet Bolt

Everything looks fine. One node has been skipped. Two nodes continued for installation. Let’s check if EPOps is really installed.

Puppet Bolt

Yes, the EPOps agent software is installed. Let’s check vRealize Operations if the agents are registered and running.

Puppet Bolt

Looks like we have a success.

I hope that with this simple example I made you curious enough about Puppet Bolt to investigate the unlimited possibilities of this tool. All code used in this blog is also available on my Github account. Thanks for reading and good luck with your automation journey!