UMATLAS yum repository

NOTE: As of January, 2018, sysprov02, an SL7 VM, has replaced sysprov01, and sysprov01 has been shut down. All refs to sysprov01 below have been changed to sysprov02 as of Feb 13, 2019.

The umatlas repo primary mirror is sysprov02. It is aliased as http://mirror.aglt2.org/pub/umatlas or http://mirror.local/pub/umatlas. Rsync is enabled on this machine so it can be mirrored to http://mirror.msulocal/mirror/umatlas (http://msut3-rx6.aglt2.org/mirror/umatlas). The /etc/rsync.conf file is managed by cobber and re-written during a cobbler sync. The rsync URL is rsync://sysprov02.local/cobbler-repos/umatlas. The actual directory is /var/www/cobbler/repo_mirror/umatlas.

In cobbler we reference it as a repository that is not mirrored locally. In our URL we use a variable that will be replaced by the IP of the cobbler server used in the kickstart. If kickstarted via private IP it'll be the server's private IP, same for public IP. Example: http://@@http_server@@/pub/umatlas/sl5/x86. We could also use the unaliased path at /cobbler/repo_mirror/umatlas/sl5 (or sl6). The /pub aliases are setup via /etc/httpd/conf.d/cobbler-aliases.conf. We use AliasMatch directives so the $releasever variable can be used in client yum configs (ie, umatlas/sl6.5 or RHEL 6Server all point at umatlas/sl6). Cfengine manages that file.

The repository is arranged by distribution, then arch. Noarch packages are symlinked by a script into both architectures. There are also some packages applicable to all distributions kept in an "all" directory and symlinked into each distribution by arch. The script that does all this is /usr/local/RepoTools/createrepo (cfengine installs it).

For example: Packages from umatlas/all/x86_64 ----> umatlas/sl6/x86_64 and umatlas/sl5/x86_64.

For another example: Packages in umatlas/sl6/noarch ----> umatlas/sl6/x86_64 and umatlas/sl6/i386

Here are a few examples of where packages go in the hierarchy.

***** IMPORTANT * Do not put packages directly in those paths. Work in an SVN working directory and use the sync-repo.sh script to put packages into the repository. Read more in the next section.

Applicable systems	Location in SVN working dir
All x86_64 dists	packages/all/x86_64
All i386 dists	packages/all/i386
All dists, any arch	packages/all/noarch
SL6, any arch	packages/sl6/noarch
SL6, x86_64	packages/sl6/x86_64

....and so on, presumably you get the idea. Reading /usr/local/RepoTools/createrepo will provide more detail.

Adding or Removing Packages

Login to sysprov02 as yourself and create your own directory under /home/packagers to work in. All work in this regard should happen in an SVN working directory. Packages added directly to /var/www/cobbler/repo_mirror/umatlas will be overwritten by this workflow.

Let's imagine we're adding a new java package to our repo and removing an out of date one. There's lots of ways to get it there but often I just wget it from wherever it is hosted. Always delete packages you don't need anymore - we can always very easily get them back from SVN (if not from a public mirror). There's no reason to ever keep a package around in the repository.

1. First checkout the repository. The space /home/packagers is provided as a working space for this purpose on sysprov02. Everyone in the "packagers" group can create a working directory here. The following command will checkout the repository into a directory named "umatlas" in your local directory. You could also end the command with a . (a period) to dump the contents of umatlas directly in your local dir.

   kinit (to use kerberos credentials to authenticate, ssh key will also work) svn co svn+ssh://ndt.aglt2.org/repos/yum/umatlas 

2. cd umatlas/packages/sl6/x86_64 (remove package we are updating - easier to do with wildcards if we do it before downloading anything, but be sure to be safe with your wildcards!) svn delete java-1.7.0* wget http://mirror.mcs.anl.gov/pub/scientific-linux/6x/x86_64/updates/security/java-1.7.0-openjdk-1.7.0.45-2.4.3.2.el6_4.x86_64.rpm wget http://mirror.mcs.anl.gov/pub/scientific-linux/6x/x86_64/updates/security/java-1.7.0-openjdk-devel-1.7.0.45-2.4.3.2.el6_4.x86_64.rpm svn add java-1.7.0* 

3. Now we can sync to the working repository. The sync-repo.sh script is at the top level of the svn working dir so you will want to cd there from wherever you were adding packages.

   ./sync-repo.sh --yes 

   You'll be asked for your sudo password (required to set group ownership and permissions). To do a trial run just leave off the "--yes"
4. Once again, be at the top level of the working directory to catch all changes you may have made. Check your changes back into svn:

    svn ci -m "Update java-openjdk to 1.7.0.45" 

***** IMPORTANT * If this is an urgent change that needs to go into effect immediately, you need to manually force the other mirror to update. Otherwise it will do so on a nightly schedule. Please see how in the next section.

Propagating to mirrors

Yum repo configuration references both available mirrors. If the mirror at MSU is not updated then your packages may not end up being installed. It is automatically updated nightly but sometimes you may need it faster.

Update MSU mirror on msu3. (OBSOLETE AS OF FEBRUARY 1, 2018)

Log into msu4.msulocal

/exports/vmware/mirror/mirrorumatlas.sh

Update the MSU mirror on msu-prov02 (NEW AS OF FEBRUARY 1, 2018)

The same process as is used on sysprov02 at UM should now be used on msu-prov02, the SL7 Cobbler system at MSU, new as of January, 2018. So, once the changes are checked in to svn from sysprov02, log in to msu-prov02, move to your packagers directory, svn update the umatlas repo, and run the same sync-repo.sh script.

Symmetry between MSU and UM Cobbler servers.

This process is reversible, ie, initial changes can be made on msu-prov02, checked in to svn, then checked out and synced on sysprov02.

-- BenMeekhof - 22 Nov 2013