arthurguru: Setting up a local RPM repository for change management control

Setting up a local RPM repository for change management control

Intro

Red Hat's policy on updating RPM packages are that they should be applied as soon as they are released. This is fair enough and adequate for most users but it does not fulfil the requirements of change management control.
Most commercial sites will create a new baseline by applying patches to their staging or testing environments first and letting them run for a number of days or weeks before applying the same patches (or the same baseline) to their production environments. This kind of change management ensures that both the staging, testing and production environments are the same, and that any problems can be addressed in the staging or testing environments before they are applied (or rejected) for the production environments.
Unfortunately under Red Hat's policy, new RPM updates may be released while you are undergoing testing in your staging or test environments.
All is not lost however, the solution is to create your own local RPM repository mirror and release your RPM updates to your staging, testing and production environments in a controlled manner (why they don't teach you this at ITIL certification courses other than how to spell ITIL I don't know).
The following solution is for CentOS 6.3 using the yum package manager.

Unix: Preparation

You will need a HTTP server to serve your local RPM repository (preferably Apache).
You will need a low priority, non-critical file share of about 20Gb in size.
You will need to install the RPM packages yum-plugin-priorities and yum-utils.
You will need a script that manages the local repository baseline.

Unix: Create the local repository server

Select a host to be your local RPM repository and run a HTTP service on it with a ServerRoot folder that can accommodate about 20Gb of storage, then install the RPM packages yum-plugin-priorities and yum-utils.
That's about it...all you need now is some sexy RPM re-baseline script to tie it all together.
In my real world scenario I actually salvaged an old openfiler server that was due for retirement and created a new file share on it which was accessible via HTTP and NFS protocols.
Select a host that will run the RPM re-baseline script that I will elaborate on later. This could be the same host as the storage server, or another server that has the storage NFS mounted.
For the sake of simplicity lets assume the storage server, HTTP server and RPM re-baseline script server all reside on the same host that has a hostname alias entry in DNS called myrepo.mycompany.com. Having a dedicated server is a bit of overkill as this service doesn't do much most of the time.

Unix: Set up the yum repository configurations

Install the yum-plugin-priorities and the yum-utils RPM packages on the myrepo.mycompany.com host.
Install the yum-plugin-priorities RPM package on all hosts that will use this local RPM repository.
At most sites sysadmins often include more than just the base repository, and in our setup we will also be pulling in updates from the EPEL repositories as well. To do this well and conveniently we implement the priorities feature of the yum package manager and set the local RPM repository with the highest priority and all external RPM repositories with a lower priority.
Apply the following yum repo configurations to all hosts that will install their RPM packages from the local RPM repository.
Configure the local RPM repository configuration file /etc/yum.d/my.repo.
This file should start off as a copy of your public repositories with all relevant details adjusted to point to the local RPM repository. Set a higher priority for each local RPM repository.
# Local RPM repository
#
# Arthur Gouros - 10/12/2012
#
[my_base]
name=MyCompany - CentOS-$releasever - Base ($basearch)
baseurl=http://myrepo.mycompany.com/repo_mirror/linux6/current/centos6/base/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
priority=1

[my_updates]
name=MyCompany - CentOS-$releasever - Updates ($basearch)
baseurl=http://myrepo.mycompany.com/repo_mirror/linux6/current/centos6/updates/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
priority=1

[my_extras]
name=MyCompany - CentOS-$releasever - Extras ($basearch)
baseurl=http://myrepo.mycompany.com/repo_mirror/linux6/current/centos6/extras/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
priority=1

[my_epel]
name=MyCompany - Extra Packages for Enterprise Linux 6 ($basearch)
baseurl=http://myrepo.mycompany.com/repo_mirror/linux6/current/epel6/epel/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
priority=2
Set a lower priority for the external RPM repository configuration file /etc/yum.d/CentOS-Base.repo.
E.g. for CentOS Base, Updates and Extras I have set a priority of 3.
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the 
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
priority=3

#released updates 
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
priority=3

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
priority=3

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
priority=3

#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contrib
#baseurl=http://mirror.centos.org/centos/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
priority=3
Set a lower priority for the external RPM repository configuration file /etc/yum.d/epel.repo.
E.g. for EPEL I have set a priority of 4.
[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
priority=4

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 6 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch/debug
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-6&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
gpgcheck=1
priority=4

[epel-source]
name=Extra Packages for Enterprise Linux 6 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/6/SRPMS
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-6&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6
gpgcheck=1
priority=4
Controlling access to your RPM repositories using priorities gives a sysadmin much more flexibility in managing their Linux hosts.

Unix: Set up the RPM re-baseline script

Here it is folks, I usually store this script on the myrepo.mycompany.com host as /usr/local/sbin/yum_rebase_linux6.sh and either call it from cron once a week under the guise of some IT Policy and Procedures Change Management process, or run it by hand whenever you want to re-baseline your RPM packages in your local repository.
The first time you run this script it will take a while as you will be downloading gigabytes worth of packages to populate the repository. The next day you run the script it will be quicker because the script will make a copy (using hard linking for storage efficiency) of the previous repository and will only download the changes (or the delta in Change Management speak).
The script assumes you have set up a HTTP ServerRoot at /var/www/html.
#!/bin/sh
#
# BSD License for yum_rebase_linux6.sh 
# Copyright (c) 2013, Arthur Gouros
# All rights reserved.
# 
# Redistribution and use in source and binary forms, with or without 
# modification, are permitted provided that the following conditions are met:
# 
# - Redistributions of source code must retain the above copyright notice, 
#   this list of conditions and the following disclaimer.
# - Redistributions in binary form must reproduce the above copyright notice, 
#   this list of conditions and the following disclaimer in the documentation 
#   and/or other materials provided with the distribution.
# - Neither the name of Arthur Gouros nor the names of its contributors 
#   may be used to endorse or promote products derived from this software 
#   without specific prior written permission.
# 
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 
# POSSIBILITY OF SUCH DAMAGE.
#
#
# This script updates a file share with the latest updated packages from
# specified yum repositories. The time this script is executed forms the
# baseline of all the packages inside the file share.
#
# Client hosts should use this file share for their yum updates.
#
# A typical change management sequence may look like this:
#
#   Step 1. This script is run to create a new time based package baseline.
#   Step 2. Staging hosts (and other non-critical hosts) run 'yum update'
#           against this baseline.
#   Step 3. A delay, usually one week, transpires to ensures systems are
#           operating correctly.
#   Step 4. Production hosts run 'yum update' against the same baseline
#           that was used for staging hosts.
#   Step 5. These steps repeat themselves over a regular interval.
#
# This script will hold onto a fixed number of old baselines to allow
# for a roll back safeguard.
#
#
# Author: Arthur Gouros - 10/12/2012

#
# Global configs
#
BASELINE_DIR="/var/www/html/repo_mirror/linux6"
OLD_BASELINES_TO_KEEP=4
EPOCH="`date '+%Y%m%d'`"
SYMLINK_NAME="current"

#
# Runtime configs
#
NEW_BASELINE="reposync_${EPOCH}"

if test -L "${BASELINE_DIR}/${SYMLINK_NAME}"
then
  CUR_BASELINE="`ls -ld ${BASELINE_DIR}/${SYMLINK_NAME} | awk -F'>' '{ print $NF }'`"
  CUR_BASELINE="`basename ${CUR_BASELINE}`"
else
  CUR_BASELINE="__none__"
fi


prepare_new_baseline_directory()
{
  #
  # Create the new baseline directory and make it ready for repository syncing.
  #
  if test ! -d "${BASELINE_DIR}/${NEW_BASELINE}"
  then
    if test -d "${BASELINE_DIR}/${CUR_BASELINE}"
    then
      # Copy using hardlinks for space and speed efficiency
      cp -rpl "${BASELINE_DIR}/${CUR_BASELINE}" "${BASELINE_DIR}/${NEW_BASELINE}"
      err=$?
      if test $err -ne 0
      then
        echo "ERROR: Repo clone failed with error ${err}: cp -rpl ${BASELINE_DIR}/${CUR_BASELINE} ${BASELINE_DIR}/${NEW_BASELINE}"
        exit 1
      fi
    else
      mkdir -p "${BASELINE_DIR}/${NEW_BASELINE}"
      err=$?
      if test $err -ne 0
      then
        echo "ERROR: Mkdir failed with error ${err}: mkdir -p ${BASELINE_DIR}/${NEW_BASELINE}"
        exit 1
      fi
    fi
  else
    echo "ERROR: New baseline already exists, please rename/remove: ${BASELINE_DIR}/${NEW_BASELINE}"
    echo "       Hint: You could rename ${NEW_BASELINE} to yesterday and relocate the symlink to it"
    echo "             then run this script again, or try again tomorrow."
    exit 1
  fi
}


sync_repositories()
{
  #
  # CentOS 6 repositories
  #
  mkdir -p "${BASELINE_DIR}/${NEW_BASELINE}/centos6"
  /usr/bin/reposync --quiet --newest-only --repoid=base --repoid=updates --repoid=extras --download_path="${BASELINE_DIR}/${NEW_BASELINE}/centos6"
  err=$?
  if test $err -ne 0
  then
    echo "ERROR: Reposync failed with error ${err} while downloading to ${BASELINE_DIR}/${NEW_BASELINE}"
    exit 1
  fi
  /usr/bin/createrepo -q "${BASELINE_DIR}/${NEW_BASELINE}/centos6/base"
  /usr/bin/createrepo -q "${BASELINE_DIR}/${NEW_BASELINE}/centos6/updates"
  /usr/bin/createrepo -q "${BASELINE_DIR}/${NEW_BASELINE}/centos6/extras"

  #
  # Fedora EPEL 6 repositories
  #
  mkdir -p "${BASELINE_DIR}/${NEW_BASELINE}/epel6"
  /usr/bin/reposync --quiet --newest-only --repoid=epel --download_path="${BASELINE_DIR}/${NEW_BASELINE}/epel6"
  err=$?
  if test $err -ne 0
  then
    echo "ERROR: Reposync failed with error ${err} while downloading to ${BASELINE_DIR}/${NEW_BASELINE}"
    exit 1
  fi
  /usr/bin/createrepo -q "${BASELINE_DIR}/${NEW_BASELINE}/epel6/epel"

  #
  # update the baseline symlink
  #
  rm -f "${BASELINE_DIR}/${SYMLINK_NAME}"
  ln -s "${NEW_BASELINE}" "${BASELINE_DIR}/${SYMLINK_NAME}"
}


apply_repolist_stamp()
{
  #
  # This is just to capture what repos were available on the server when this script ran.
  #
  /usr/bin/yum repolist enabled > "${BASELINE_DIR}/${NEW_BASELINE}/.repolist" 2>&1
}


remove_old_baselines()
{
  # to keep the nth add 1
  n_line="`expr ${OLD_BASELINES_TO_KEEP} + 1`"
  # List the baseline folders alphabetically and in reverse.
  for d in `ls -1dr ${BASELINE_DIR}/reposync_* 2>/dev/null | sed -n ${n_line}',$p'`
  do
    d2="`basename ${d}`"

    # Do not remove any directory with the repo symlink.
    if test "${d2}" != "${NEW_BASELINE}"
    then
      rm -fr "${BASELINE_DIR}/${d2:=none}"
    fi
  done
}


############################
# Main
############################
prepare_new_baseline_directory
apply_repolist_stamp
sync_repositories
remove_old_baselines

exit 0