Pineapple SSH plugin – How-to: Install Jenkins and Artifactory in a Linux Vagrant box using SSH

How-to: Install Jenkins and Artifactory in a Linux Vagrant box using SSH

Overview

This example illustrates how the SSH plugin can be used to install Jenkins and Artifactory on Linux (CentOS and Red Hat). The example integrates Jenkins with artifactory and creates a build job.

Two steps are required to install Jenkins and Artifactory:

Create a VM which serves as build server. Vagrant is used for this purpose.
Start Pineapple and execute the module.
(When the installation is completed, start the created build job at Jenkins.)

Part of the default configuration

This example named ssh-010-ssh-install-jenkins-artifactory, including all configuration files, is included in the default configuration which is created by Pineapple, so there is no need to create it by hand.

The steps

Build a Vagrant box

Install VirtualBox and Vagrant.
Create a Vagrant project directory jenkins-ci.

Copy the example Vagrant configuration file from ${user.home}/.pineapple/modules/ssh-010-ssh-install-jenkins-artifactory/vagrant/Vagrantfile directory to the Vagrant project directory:

# -*- mode: ruby -*-
# vi: set ft=ruby :

BOX_NAME = ENV.fetch("BOX_NAME", "bento/centos-7.6") 
BOX_IP = ENV.fetch("BOX_IP", "192.168.34.10") 
BOX_MEM = ENV.fetch("BOX_MEM", "2048") 
BOX_CPUS = ENV.fetch("BOX_CPUS", "2") 

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

        # virtual box provider specific settings
        config.vm.provider :virtualbox do |vb|                                  
                vb.customize ["modifyvm", :id, "--ioapic", "on"]    
                vb.customize ["modifyvm", :id, "--memory", BOX_MEM]    
                vb.customize ["modifyvm", :id, "--cpus", BOX_CPUS]      
        end     

        config.vm.define :pineapple_ci do |ci_config|
                ci_config.vm.box = BOX_NAME    
                ci_config.vm.network "private_network", ip: BOX_IP      
        end
                
end

Open a prompt, CD into the Vagrant project directory and create the box with the command: vagrant up. It will download a CentOS 6.5 image and start the VM with the IP address 192.168.34.10.

Invoke Pineapple to install Jenkins

Start your Pineapple client of choice:

Select the module named ssh-010-ssh-install-jenkins-artifactory
Select the linux-vagrant model.
Invoke the deploy-configuration operation to install Jenkins.

Start the build job

Access Jenkins at: http://192.168.34.10:8080
Select the installed job pineapple-build and build it. The build will but it is an example.

Artifactory listens on http://192.168.34.10:8081

The details

The details of the module, model and the Pineapple resources are described in the following sections.

The resources

A resource defines a entity in a IT environment which is manageable by Pineapple through some protocol. In this example, the entity is a Linux OS and the used protocol is SSH. One resource is defined for each Linux host where the YUM packages should be installed.

In this example we will use the linux-vagrant environment defined as part of the default configuration. The environment defines a network with three hosts:

Node1 with IP Address: 192.168.34.10
Node2 with IP Address: 192.168.34.11
Node3 with IP Address: 192.168.34.12

To enable SSH usage for these hosts, three resources are defined within the linux-vagrant environment in the resources file located at ${user.home}/.pineapple/conf/resources.xml:

<?xml version="1.0" encoding="UTF-8"?>
<configuration xmlns="http://pineapple.dev.java.net/ns/environment_1_0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://pineapple.dev.java.net/ns/environment_1_0
    http://pineapple.dev.java.net/ns/environment_1_0.xsd">
     
    <environments>
        <environment description="Vagrant multi-machine Linux environment" id="linux-vagrant">
            <resources>
                <resource plugin-id="com.alpha.pineapple.plugin.ssh" credential-id-ref="ssh-node1" id="ssh-node1" >
                    <property value="192.168.34.10" key="host"/>
                    <property value="22" key="port"/>
                    <property value="1000" key="timeout"/>
                </resource>
                <resource plugin-id="com.alpha.pineapple.plugin.ssh" credential-id-ref="ssh-node2" id="ssh-node2" >
                    <property value="192.168.34.11" key="host"/>
                    <property value="22" key="port"/>
                    <property value="1000" key="timeout"/>
                </resource>
                <resource plugin-id="com.alpha.pineapple.plugin.ssh" credential-id-ref="ssh-node3" id="ssh-node3" >
                    <property value="192.168.34.12" key="host"/>
                    <property value="22" key="port"/>
                    <property value="1000" key="timeout"/>
                </resource>
            </resources>
        </environment>
    </environments>
</configuration>

Each resource is defined with a different ID and IP address. All other properties are identical. The role of the attribute plugin-id is to bind the resource definition to the plugin code at runtime which implements the SSH plugin.

Please notice: Only the VM for ssh-node1 is created by Vagrant. Even though the linux-vagrant environment defines three resource, only the ssh-node1 resource is used in the example.

The credentials

Since SSH requires authentication, each resource defines a reference to a credential. A credential defines the user name and password used for authentication when a SSH session is created to a host.

Three credentials are created within the linux-vagrant environment to support authentication. The credentials are defined in the credentials file located at ${user.home}/.pineapple/conf/credentials.xml:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<configuration xmlns="http://pineapple.dev.java.net/ns/environment_1_0">
    <environments>
        <environment description="Vagrant multi-machine Linux environment" id="linux-vagrant">
            <credentials>
                <credential password="vagrant" user="vagrant" id="ssh-node1"/>
                <credential password="vagrant" user="vagrant" id="ssh-node2"/>
                <credential password="vagrant" user="vagrant" id="ssh-node3"/>
            </credentials>
        </environment>
    </environments>     
</configuration>

Please notice: Only the VM for ssh-node1 is created by Vagrant. Even though the linux-vagrant environment defines three credentials, only the ssh-node1 credential is used in the example.

The module

A module defines input to Pineapple and consists in its minimal form of a single XML file containing a model:

The directory: ${user.home}/.pineapple/modules/ssh-010-ssh-install-jenkins-artifactory contains the module. The module in this example have the structure:

ssh-010-ssh-install-jenkins-artifactory
 |
 +--- bin        
 |     +--- artifactory.config.xml
 |     +--- artifactory.repo
 |     +--- config.xml
 |     +--- hudson.tasks.Maven.xml 
 |     +--- pineapple-build.config.xml
 |     +--- settings.xml
 |     +--- weblogic-full-client-12.1.2.jar.ADD-HERE
 |
 +--- models     
 |     +--- linux-vagrant.xml
 +--- vagrant    
 |     +--- Vagrantfile

Module files

The module consists of the these files:

artifactory.config.xml is the main Artifactory configuration file. It is installed in Artifactory using a web service.
artifactory.repo is a YUM configuration file. It is installed at /etc/yum.repos.d/artifactory.repo. It defines where the Artifactory RPM packages can downloaded from.
config.xml is the main Jenkins configuration file which is installed at /var/lib/jenkins/config.xml. It defines the path to OpenJDK used for compiling build job.
hudson.tasks.Maven.xml is Jenkins Maven configuration which is installed at /var/lib/jenkins/hudson.tasks.Maven.xml. It the version of Maven to be installed and used by Jenkins for build jobs.
pineapple-build.config.xml is a build job named pineapple-build which is installed at /var/lib/jenkins/jobs/pineapple-build.
settings.xml is a Maven configuration file used to configure Maven for specific build jobs. It is installed at /var/lib/jenkins/settings.xml.
linux-vagrant.xml is the Pineapple model file for the environment linux-vagrant which installs Jenkins.
Vagrantfile is a Vagrant file which can build the VM where Jenkins is installed.

The model file

The model file named linux-vagrant.xml has the content:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<mmd:models xmlns:mmd="http://pineapple.dev.java.net/ns/module_model_1_0"     
  xmlns:shp="http://pineapple.dev.java.net/ns/plugin/ssh_1_0" >
    <mmd:variables>
      <mmd:variable key="tmp-dir" value="/tmp" />       
      <mmd:variable key="base-jdk" value="java-1.7.0-openjdk-devel" />
      <!-- continous integration server variables -->   
      <mmd:variable key="ci-usergroup" value="jenkins:jenkins" />
      <mmd:variable key="ci-home" value="/var/lib/jenkins" />
      <mmd:variable key="ci-jobs-home" value="/var/lib/jenkins/jobs" />
      <mmd:variable key="ci-config" value="config.xml" />
      <mmd:variable key="ci-maven-config" value="hudson.tasks.Maven.xml" />
      <mmd:variable key="ci-job" value="pineapple-build" />
      <mmd:variable key="ci-job-source-config" value="pineapple-build.config.xml" />
      <!-- continous integration Maven  variables -->   
      <mmd:variable key="maven-config" value="settings.xml" />
      <!-- repository manager variables -->   
      <mmd:variable key="rm-repo-config" value="artifactory.repo" />        
      <mmd:variable key="rm-config" value="artifactory.config.xml" />        
      <mmd:variable key="rm-user" value="admin" />        
      <mmd:variable key="rm-pwd" value="password" />        
      <mmd:variable key="rm-port" value="8081" />        
      <mmd:variable key="rm-ws-upload-system-config" value="/artifactory/api/system/configuration" />        
      <mmd:variable key="rm-ws-upload-artifact" value="/artifactory/libs-release-local" />        

      <!-- 3rd party libraries variables -->
      <mmd:variable key="weblogic-client-1212-jar" value="weblogic-full-client-12.1.2.jar" />
      <mmd:variable key="wbem-102-jar" value="wbemservices-1.0.2.jar" />    
    </mmd:variables>    

    <mmd:model target-resource="ssh-node1" target-operation="deploy-configuration" description="Install JVM" >
      <mmd:content>
        <shp:ssh>
          <shp:execute command="sudo sudo yum --assumeyes install ${base-jdk}" />
        </shp:ssh>      
      </mmd:content>
    </mmd:model>
    
    <mmd:model target-resource="ssh-node1" target-operation="deploy-configuration" description="Configure Artifactory package repository" >
      <mmd:content>
        <shp:ssh>
          <shp:copy-to source="modulepath:bin/${rm-repo-config}" destination="${tmp-dir}/${rm-repo-config}"/>
          <shp:execute command="sudo mv ${tmp-dir}/${rm-repo-config} /etc/yum.repos.d/${rm-repo-config}" />
        </shp:ssh>      
      </mmd:content>
    </mmd:model>    
    <mmd:model target-resource="ssh-node1" target-operation="deploy-configuration" description="Install Artifactory package" >
      <mmd:content>
        <shp:ssh>
          <shp:execute command="sudo yum --assumeyes install artifactory" />        
        </shp:ssh>      
      </mmd:content>
    </mmd:model>  
    <mmd:model target-resource="ssh-node1" target-operation="deploy-configuration" description="Start Artifactory" >
      <mmd:content>
        <shp:ssh>
          <shp:execute command="sudo service artifactory start" />            
        </shp:ssh>      
      </mmd:content>
    </mmd:model>    
    <mmd:model target-resource="ssh-node1" target-operation="deploy-configuration" description="Invoke Artifactory Web service to trigger initialization" >
      <mmd:content>
        <shp:ssh>
          <shp:execute command="curl -u ${rm-user}:${rm-pwd} -X GET http://${resource.host}:${rm-port}/api/system/configuration" />            
        </shp:ssh>      
      </mmd:content>
    </mmd:model>    
        
    <mmd:model target-resource="ssh-node1" target-operation="deploy-configuration" description="Install Jenkins packages" >
      <mmd:content>
        <shp:ssh>
          <shp:execute command="sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo" />
          <shp:execute command="sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key" />
          <shp:execute command="sudo yum --assumeyes install jenkins" />        
        </shp:ssh>      
      </mmd:content>
    </mmd:model>  
    <mmd:model target-resource="ssh-node1" target-operation="deploy-configuration" description="Install Jenkins job files" >
      <mmd:content>
        <shp:ssh>
          <!-- install Jenkins main configuration file -->
          <shp:copy-to source="modulepath:bin/${ci-config}" destination="${tmp-dir}/${ci-config}" chmod="775" />        
          <shp:execute command="sudo chown ${ci-usergroup} ${tmp-dir}/${ci-config}" />                
          <shp:execute command="sudo mv ${tmp-dir}/${ci-config} ${ci-home}/${ci-config}" />               

          <!-- install Jenkins Maven configuration file -->
          <shp:copy-to source="modulepath:bin/${ci-maven-config}" destination="${tmp-dir}/${ci-maven-config}" chmod="775" />
          <shp:execute command="sudo chown ${ci-usergroup} ${tmp-dir}/${ci-maven-config}" />                
          <shp:execute command="sudo mv ${tmp-dir}/${ci-maven-config} ${ci-home}/${ci-maven-config}" />               

          <!-- install Jenkins Maven settings.xml file for builds -->
          <shp:copy-to source="modulepath:bin/${maven-config}" destination="${tmp-dir}/${maven-config}" chmod="775" />        
          <shp:execute command="sudo chown ${ci-usergroup} ${tmp-dir}/${maven-config}" />               
          <shp:execute command="sudo mv ${tmp-dir}/${maven-config} ${ci-home}/${maven-config}" />               

          <!-- install Jenkins build job -->
          <shp:execute command="sudo mkdir -p ${ci-jobs-home}/${ci-job}" />         
          <shp:execute command="sudo chmod -R 775 ${ci-jobs-home}/${ci-job}" />       
          <shp:execute command="sudo chown -R ${ci-usergroup} ${ci-jobs-home}/${ci-job}" />                       
          <shp:copy-to source="modulepath:bin/${ci-job-source-config}" destination="${tmp-dir}/${ci-job-source-config}" chmod="775" />
          <shp:execute command="sudo chown ${ci-usergroup} ${tmp-dir}/${ci-job-source-config}" />               
          <shp:execute command="sudo mv ${tmp-dir}/${ci-job-source-config} ${ci-jobs-home}/${ci-job}/${ci-job-target-config}" />                
        </shp:ssh>      
      </mmd:content>
    </mmd:model>
    <mmd:model target-resource="ssh-node1" target-operation="deploy-configuration" description="Start Jenkins" >
      <mmd:content>
        <shp:ssh>
          <shp:execute command="sudo service jenkins start" />            
        </shp:ssh>      
      </mmd:content>
    </mmd:model>  

    <mmd:model target-resource="ssh-node1" target-operation="deploy-configuration" description="Delayed installation of Artifactory configuration and artifacts" >
      <mmd:content>
      <shp:ssh>

        <!-- install Artifactory system configuration -->
        <shp:copy-to source="modulepath:bin/${rm-config}" destination="${tmp-dir}/${rm-config}" chmod="775" />
          <shp:execute command="curl -u ${rm-user}:${rm-pwd} -X POST -H &quot;Content-type:application/xml&quot; --data-binary @${tmp-dir}/${rm-config} http://${resource.host}:${rm-port}${rm-ws-upload-system-config}" />            
          <shp:execute command="rm -f ${tmp-dir}/${rm-config}" />            

        <!-- deploy Pineapple 3rd party libraries -->
        <shp:copy-to source="modulepath:bin/${weblogic-client-1212-jar}" destination="${tmp-dir}/${weblogic-client-1212-jar}" substitute-variables="false" chmod="775" />
          <shp:execute command="curl -u ${rm-user}:${rm-pwd} -X PUT  -T ${tmp-dir}/${weblogic-client-1212-jar} http://${resource.host}:${rm-port}${rm-ws-upload-artifact}/oracle/weblogic-full-client/12.1.2/${weblogic-client-1212-jar}" />            
          <shp:execute command="rm -f ${tmp-dir}/${weblogic-client-1212-jar}" />            
        </shp:ssh>      
        </mmd:content>
    </mmd:model>    
</mmd:models>

The configuration details

Two schema are used in the model file. The http://pineapple.dev.java.net/ns/module_model_1_0 is used to define the namespace mmd which defines the general infrastructure for models. The http://pineapple.dev.java.net/ns/plugin/ssh_1_0 schema is used to define the namespace shp which is used to define the model for the SSH plugin. Since multiple schemas are used to define the model file, the elements are qualified.

Initially in the model a set of variables are defined within the variables stanza. The variables are referenced from the remaining part of the model file.

The target-resource attribute defines a reference to the resource which is targeted when the model executed. In this case, the value ssh-node1 targets the model to a single resource. The two other nodes in the linux-vagrant environment isn't used in this example.

The model file is divided into five sub-models, which performs the tasks:

Install JVM. The OpenJDK 1.7.0 package is installed. The package with the postfix "devel" is installed to get the JDK and not just the JRE. The JDK is used from build jobs in Jenkins. An alternative would be to configure Jenkins to download the JDK from Oracle. Oracle downloads require authentication which is avoided when using the OpenJDK.
Configure Artifactory package repository. The installation of Artifactory is prepared by installing the file YUM repo file /etc/yum.repos.d/artifactory.repo which defines where the Artifactory RPM packages can downloaded from. The property gpgcheck is set to zero to disable checksum validation since the Artifactory package fails checksum validation.
Install Artifactory package>. YUM is used to install Artifactory as an OS service.
Start Artifactory. Starts Artifactory as an OS service.
Invoke Artifactory web service to trigger initialization. A Artifactory web service is invoked to trigger initialization. While Artifactory is initializaing, then Jenkins is installed.
Install Jenkins packages. The Jenkins package repository is added to the local YUM configuration. YUM is used to install Jenkins as an OS service.
Install Jenkins job files. Installation of the Jenkins files consists of four steps:
- The main Jenkins configuration file config.xml is installed on the server in the Jenkins configuration directory at /var/lib/jenkins.
- The Jenkins Maven configuration file hudson.tasks.Maven.xml is installed in /var/lib/jenkins.
- The Jenkins Maven settings.xml file used by builds is installed in /var/lib/jenkins.
- A build job named build-pineapple is installed in /var/lib/jenkins/jobs.
Start Jenkins. Starts Jenkins as an OS service.
Delayed installation of Artifactory configuration and artifacts. By now Artifactory is initialized and it is configured using its REST API.

The target-operation attribute is used to restrict that the sub-models are only executed when Pineapple is invoked with the deploy-configuration operation. This allows for the definition of a set of sub-models which only responds to the undeploy-configuration which implements the inverse semantics, e.g. uninstallation of Jenkins from the target server.

Finally, the execute commands in the models is executed using sudo. The requirement for sudo depends on the privileges of the user used by the SSH plugin to connect to a given host. The users was defined in credentials file in the previous section.

Integration between Jenkins and Artifactory

Artifactory configuration

When Artifactory is initialized then it is configured using its REST API. The configuration uploads a source system configuration. The system configuration defines:

The local repository libs-release-local which can be used to store 3rd party libraries which isn't available in any of the public Maven repositories.
An example remote repository zk-ce which holds the publicly available ZK artifacts. The repository defines the URL: http://mavensync.zkoss.org/maven2.
Another example repository zk-ee which holds the ZK enterprise artifacts which requires a ZK enterprise license. The repository defines the URL: http://mavensync.zkoss.org/zk/ee-eval.

The source system configuration file artifactory.config.xml is generated by configuring Artifactory maually and the exporting the configuration.

Uploading artifacts to Artifactory

The web service /artifactory/libs-release-local/path/to/jar/ can be used with HTTP POST to install artifacts into the libs-release-local local repository.

In the model there is included an example which illustrates how an artifact can uploaded, e.g. WebLogic full client (JAR). The example is commented out to avoid run time fauilure due to the JAR is missing:

<shp:copy-to source="modulepath:bin/${weblogic-client-1212-jar}" destination="${tmp-dir}/${weblogic-client-1212-jar}" substitute-variables="false" chmod="775" />
<shp:execute command="curl -u ${rm-user}:${rm-pwd} -X PUT  -T ${tmp-dir}/${weblogic-client-1212-jar} http://${resource.host}:${rm-port}${rm-ws-upload-artifact}/oracle/weblogic-full-client/12.1.2/${weblogic-client-1212-jar}" />            
<shp:execute command="rm -f ${tmp-dir}/${weblogic-client-1212-jar}" />

Bulid integration with Artifactory

The Maven configuration file /var/lib/jenkins/settings.xml is referenced by build jobs. The file defines the repositories used by Maven to resolve dependencies:

<?xml version="1.0" encoding="UTF-8"?>
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.1.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <servers>
    <server>
      <username>admin</username>
      <id>central</id>
    </server>
    <server>
      <username>admin</username>
      <id>snapshots</id>
    </server>
  </servers>
  <mirrors>
    <mirror>
      <mirrorOf>*</mirrorOf>
      <name>repo</name>
      <url>http://192.168.34.10:8081/artifactory/repo</url>
      <id>repo</id>
    </mirror>
  </mirrors>
  <profiles>
    <profile>
      <repositories>
        <repository>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
          <id>central</id>
          <name>libs-release</name>
          <url>http://192.168.34.10:8081/artifactory/libs-release</url>
        </repository>
        <repository>
          <snapshots />
          <id>snapshots</id>
          <name>libs-snapshot</name>
          <url>http://192.168.34.10:8081/artifactory/libs-snapshot</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
          <id>central</id>
          <name>plugins-release</name>
          <url>http://192.168.34.10:8081/artifactory/plugins-release</url>
        </pluginRepository>
        <pluginRepository>
          <snapshots />
          <id>snapshots</id>
          <name>plugins-snapshot</name>
          <url>http://192.168.34.10:8081/artifactory/plugins-snapshot</url>
        </pluginRepository>
      </pluginRepositories>
      <id>artifactory</id>
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>artifactory</activeProfile>
  </activeProfiles>
</settings>

An important configuration in the file is the mirror definition which configures Maven to access Artifactory at http://192.168.34.10:8081/artifactory/repo for resolution of all dependencies. This controls the resolution of dependencies and it speeds up the build process.

The configuration file can be generated by exporting it from Artifactory.