Manuals

Cobbler manual:

http://www.cobblerd.org/manuals/

For information on the Cheetah template language used in kickstart templates:

http://www.cheetahtemplate.org/docs/users_guide_html/

(you only need to read the parts detailing the language since it's already embedded in cobbler)

Plumbing

System Information

Definitions for profiles, systems, distros, kickstarts, snippets, etc are stored in /var/lib/cobbler. This entire directory tree is a git repo. Any time you do a 'cobbler sync' a new commit is made to the repo. If a mistake is made or useful definitions are lost you can pull them back out of the repository. An example of restoring a missing system definition:

cd /var/lib/cobbler/config/systems.d

git log

commit 53d91a2e557335fb536ab5460f0421cab17c8e5b
Author: cobbler 
Date:   Tue Dec 22 11:40:44 2015 -0500

    API update
....

In git log output, the last commit will be the one where the file is deleted (because after that the file didn't exist to be logged). So you could do something like:

git checkout 53d91a~1 -- filename

Note the '~1' at the end of the commit id (you can abbreviate to 5 chars as long as it is not-ambiguous with any other id). It says to checkout the commit 1 commit before.

If you are not sure which file, you can use a ' . ' to checkout the whole directory or wildcards to match filenames from a given commit but be careful with that.

Another tip to find commits where files were deleted (and only those commits): git log --diff-filter=D --summary

Diff two commits to find out where things went wrong by putting in the two commit IDs to diff. With no arguments git diff behaves like svn diff and shows working vs repository: git diff 53d91a 234fgd

After restoring a missing file restart cobblerd and it should show up again in the interface. Sync cobbler to commit your fixes.

Git is well documented and a google query for almost any operation will find you some examples on stackoverflow. We have a git book floating around too.

Yum repositories

All stored in /var/www/cobbler/repo_mirror. We don't use the ks_mirror directory though cobbler might put distro packages in there if you import one from DVD. Better is to define the repo and them mirror it over the network or copy files from media into the directory in repo_mirror. By 'better' I just mean 'more consistent' and 'less likely to be confusing later' and 'more likely to work'.

A nightly mirror is run from /etc/cron.daily/reposync.sh which in turn runs 'cobbler reposync'.

Profiles

Profiles define the repositories used to build and set some Kickstart Metadata (See General => Kickstart Metadata in web interface, or as below in CLI). Here's a quick reference.

Any profile setting can be over-ridden by per-system kickstart metadata. The list of variables is below, or else read the 'flexible-select-partition' snippet.

NOTE: All profiles create 200M /boot partition.

NOTE: The default in the 'flexible-select-partition' cobbler snippet is 2GB swap and everything else in /. The profiles over-ride from there.

NOTE: On reboot, Dell systems will attempt to update BIOS/firmware and then they will reboot if necessary. The first run of cf-agent will happen within an hour or can be initiated manually: 'cf-agent -Kf failsafe.cf ; cf-agent -K

server-*	Generic build, does nothing with partitions except the default.
server-small*	Swap: 2GB, /tmp 1GB, /var/cache/openafs: 2GB, /var: 4GB or grow, root 7G or grow (depends on whether you select growvar or growroot variant). Assuming 7GB root in either case the minimum disk is 16GB. No cvmfs or home partition. Should be ok for most small task-specific VMs.
server-afs-fileserver	Inherits from server-small-growroot. /var/cache/openafs not created. /var becomes 3.5GB, /tmp is 512MB.
server-large-*	Makes everything bigger: Swap 4GB, /tmp 10GB, /var/cache/openafs 4GB, /var 10GB or grow, / 30GB or grow (depends on growvar or growroot variant) . Use for ~60GB disk or larger (though it can squeeze into less it doesn't make much sense then).
server-dcache	Inherits from server-large-growroot. /var becomes 100GB, everything else the same.
server-gatekeeper	Inherits from server-large-growroot. /var becomes 30GB, add 25GB cvmfs partition. Everything else the same.
desktop	Root: 30GB, Swap: 4GB, /tmp 10GB, /var/cache/openafs: 4GB, /var 10GB. Separate home takes all unused disk space.
rescue-mode-privnet (or -pubnet)	Setting a system to use this profile and PXE booting it will boot a rescue image. Use pubnet or privnet depending what interfaces might be available. You can export this profile to a bootable iso as described later in this doc.

Variable Reference

You have the following to work with in the "ksmeta" setting of profiles,sub-profiles, and systems. In the web interface the field is "Kickstart Metadata". Any variable below can be set to a different value by specifying it in "ksmeta" with a new value. In web interface look for General -> Kickstart Metadata.

Partition "snippet" is in Snippets -> flexible_select_partition (or /var/lib/cobbler/snippets/flexible_select_partition). Most decisions are not hard-coded in the snippet so any arrangement can be made at the profile or system level.

Snippet cluster_compute_partitions has some defaults and partition handling that are different (no LVM, multiple temp partitions, default 10GB root instead of default --grow). This choice is made in the file default.ks by checking the value of the $profile variable. This cluster-compute profile doesn't exist anymore because we never switched to building compute nodes from cobbler. The partition snippet hasn't been updated in a while but it is left for reference.

The default.ks file is editable in web interface under "Kickstart Templates" or at /var/lib/cobbler/kickstarts/default.ks. All profiles use this kickstart, and the Cheetah template language is used to make it specific to different profiles or kickstart meta-data. Snippets are just bits included in the file.

Partition/volume sizes are specified in MB. All except swap and / default to 0 in Cobbler 'flexible-select-partition'. Any can be set to 0 in kickstart meta-data to disable creation, or set to some value other than the profile you are using.

interactive	If 'interactive=True' is set in the kickstart metadata then an interactive install will be performed with defaults from the cobbler kickstart.
pvsize	How large to size the physical volume that holds the various system volumes. Defaults to use entire disk.
rootdisk	Which drive to be used for install. Used in a grep of fdisk output. Default value is 'rootdisk=sd\	vd' to grep match first "sd" or "vd" (KVM virtual disks use vd). Example: "rootdisk=sdb".
maxdisk	The partition template ignores disks larger than 5TB as a safety feature to avoid over-writing important storage. Override this by setting 'maxdisk' in kickstart metadata. It is set in GB: maxdisk=5000
swapsize	How big is swap (defaults to 2GB if nothing specified).
afscachesize	Size of /var/cache/openafs.
rootsize	Size of root filesystem. Can accept 'grow' as value (this is the default if nothing is specified).
tmpsize	Size of /tmp. Defaults to 2048 (2GB). There's nothing built-in to put /tmp on different disks but have a look at cluster-compute-partitions Snippet for example.
varsize	Size of /var. Can accept 'grow' as value.
cvmfssize	Set size of cvmfs.
homesize	Set size of home. Can accept 'grow' as value.
fstype	Filesystem type. Defaults to ext4 for SL6, xfs for SL7 (same as normal defaults used by installer).

How to create a system?

First, determine what profile matches your needs, or if this class of system should have a new sub-profile. Create any needed sub-profile first, and then use it for the --profile setting.

cobbler system add --name=shoe --hostname=shoe.aglt2.org --gateway=192.41.230.1 --profile=server-sl7 \ 
--interface=eth0 --dns-name=shoe.local --mac=macaddress --ip-address=10.x.x.x --subnet=255.255.252.0 --status=production

Status is not required. The web interface is an equally good way to create systems. Be sure you fill in all the fields. https://sysprov02.aglt2.org/cobbler_web

Only one network interface can be added at a time. Use "cobbler system edit" to add more (see below). --name is the name in the cobbler database. --dns-name is the name that cobbler configures in DNS. Of course you don't need dns name for aglt2.org or other public interfaces since we don't configure those zones. --hostname is the system fully qualified hostname in /etc/sysconfig/network.

How to create a subprofile with some different settings?

cobbler profile add --name=server-gatekeeper --parent=server-large-growroot-sl6 --ksmeta="varsize=30768 cvmfssize=25600" 

More information on available variables and defaults is in the table in the next section.

How to setup just my one-off server with some different settings?

Same, just use --ksmeta (or "Kickstart Metadata" in web GUI). Over-ride whatever you need using variables as listed in previous section.

How to edit a system (for example, to add more network interfaces or change the profile)

cobbler system edit --name=bl-1-10  --interface=em2 --mac=00:26:B9:26:B0:94 --ip-address=192.41.230.210 --subnet=255.255.254.0 --static=1

How to make a system without a certain partition? For example, I want one with one large ' / ' partition only?

Set the appropriate variable to 0. The server-* base profiles will create a system with no partitions except ' / ' and 2GB swap (variable rootsize defaults to 'grow', swapsize to 2048). The rootsize,varsize,and homesize variables all respect a setting of 'grow' (but not all at once!). Others don't simply because it never made any sense to put the logic in - to do so would be a simple modification of flexible-select-partition.

Cobbler web interface

https://sysprov02.aglt2.org/cobbler_web

Use your kerberos password to login. Only certain users named in /etc/cobbler/users.conf can login (note that cfengine manages the file, do not directly edit)

Post-build configuration policy

IMPORTANT: Don't have cobbler configure anything if not necessary for basic bootstrap. IE, if the host can get far enough to contact the cfengine policy server, it is time for your config to happen in cfengine policy. One exception to this is "func" which does get a basic config from cobbler setup (but is also managed by cfengine after install). Another is bonded network interfaces. Technically, we had to build over a non-bonded nic and could use it to pull the first policy run and configure bonded interfaces. Cobbler has provisions for configuration of bonded/slaved nics and has to know about network interfaces regardless so we do it there during install steps.

CfenginePolicyWorkflow

First boot following build

Experience with Cobbler says to watch out for these items in the first boot of a system following the first build. These are best checked by going to single user mode when it reboots following the build.

• If you used a NIC to build with, that will not be used during actual running, edit the corresponding ifcfg file and set ONBOOT=no
• Make sure you have both public and private NICs working. Start the network service and ping the gateways. If this fails, fix it before going on.
• Make sure there is a valid /etc/resolv.conf file
• Make sure the date/time are correct

Once these are all checked, it is safe to exit from single user mode and let the system continue onwards.

Server setup and replication

Putting a machine in cfengine class "CobblerServers" will cause the following:

• All necessary cobbler and cobbler web packages installed
• Cobbler web service setup (https://hostname/cobbler_web).
• DNS and DHCP server templates setup in /etc/cobbler/ (contents will vary by site of cobbler server, see cobbler.cf for values)
• /etc/cobbler/settings configured appropriate to site (some DNS and DHCP info also here)
• Firewall ports opened for DNS (other services only needed on trusted networks) NOTE: Need to open some ports for "offline" installs in physics nets
• Installs our dynamic kickstart in /var/lib/cobbler/default.ks

One would still need to manually add repositories, distros, etc. Much of this can be done by using cobbler's replicate function to replicate an existing server.

Cfengine will automatically "cobbler sync" to sync DNS, DHCP, and Rsync config files (including DNS zone files). This is similar to a "rocks sync" in some actions.

Replicate configuration

http://cobbler.github.io/manuals/2.6.0/3/2/8_-_Replication.html

The 'cobbler replicate' command can be used to sync profiles, distros, kickstarts, snippets, and (if desired) systems from one cobbler machine to another. We typically want everything but systems synced from the UM system to the MSU system, and may want to sync the other way as needed (ie, if updates/changes are made on the MSU machine we want at UM too). Rsync may need to be enabled on the master (I forget, but you need it anyways to sync umatlas repo from UM -> MSU so either way...)

Sample:

cobbler replicate --master sysprov01.aglt2.org 

By default this does only profile and distro metadata are synced. There are flags to get more.

Mirror repositories

cobbler repo add --mirror=rsync://mirror.mcs.anl.gov/scientific-linux/6x/x86_64/os --name=sl-current \ 
--keep-updated=Y --arch=x86_64 --mirror-locally=Y --breed=rsync

cobbler repo add --mirror=rsync://rsync.scientificlinux.org/scientific/6x/x86_64/updates/security --name=sl-current-updates \ 
--keep-updated=Y --arch=x86_64 --mirror-locally=Y --breed=rsync

If mirror-locally is N, then installs will point to the remote repository (but it must be an ftp or http url, not rsync). Otherwise the repository is mirrored to /var/www/cobbler/repo_mirror

You can mirror any yum repository, it doesn't have to be rsync enabled. For example, the umatlas repo is mirrored from linat05.

To update the mirror manually:

cobbler reposync --only=repo-name

A nightly cron in /etc/cron.repos.d runs this as well.

Create distros

cobbler distro add --name=sl-current --kernel=/var/www/cobbler/repo_mirror/sl-current/images/pxeboot/vmlinuz \ 
--initrd=/var/www/cobbler/repo_mirror/sl-current/images/pxeboot/initrd.img --arch=x86_64 --breed=redhat --os-version=rhel6  --ksmeta="tree=http://@@http_server@@/cobbler/repo_mirror/sl-current "

os-version is generally irrelevant unless provisioning a virtual instance with "koan" (at this time I don't know anything about that). If you use "cobbler import" it deals with all the details of creating a distro for you (didn't work for me to import Scientific Linux 6 but I didn't try very hard).

Some manual sequences, to avoid the Web GUI

Set a system to netboot

• cobbler system edit --name=<system_name> --netboot-enabled=1
• cobbler sync

Building systems

IMPORTANT: Cobbler Sync

When you make changes in cobbler to profiles or hosts, you generally have to click "Sync" in the web interface or run "cobbler sync" on the command line. Changes to kickstart meta-data do not require a sync, but things like MAC addresses, IP, hostnames, interfaces do. When in doubt sync - it doesn't take long.

Create a profile

If you have some special needs for a class of systems you might create a new profile descended from another or all new.

cobbler profile add --name=server-generic --distro=sl-current --repos=sl-current,sl-current-updates,umatlas-sl6

By default profiles use /var/lib/cobbler/default.ks. This template is setup to be usable for any profile using Cheetah template language constructs to choose appropriate partitions, packages, and more for a given profile.

Try "cobbler profile add --help" for more options.

Create a sub-profile descended from one of the top-level profiles with some differences

You would use this profile in --profile of your server. For example, perhaps you'd like to create a "server-gatekeeper" sub-profile which is basically identical to a generic server (same repositories and many common partitions) but has some small changes.

cobbler profile add --name=server-gatekeeper --parent=server-generic --ksmeta="varsize=30768 cvmfssize=25600" 

Create a system

It is not strictly necessary to create a system. If no system is created and you boot any machine in our private network you will get a menu of available profiles. This will install a system with a single private-net (4010) dhcp configured interface at whatever IP address might be available. The system will be added to the list of cobbler systems with a numeric ID (I think it might be unix timestamp of when the system auto-registered itself. We could modify this auto registration string to whatever we can come up within the context of either the templating language or kickstart post install script). When cobbler auto-registers the system it will have the IP used filled in so the system should always get that IP and you could mark it "static" to write into network config files at next build.

cobbler system add --name=shoe --hostname=shoe.aglt2.org --gateway=192.41.230.1 --profile=server-generic \ 
--interface=eth0 --mac=macaddress --ip-address=ipaddress --subnet=subnet --status=production

cobbler system edit --name=shoe --interface=eth1 --mac=macaddress --ip-address=ipaddress --subnet=subnet --netboot-enabled=Y --status=production

The system will inherit other info from the profile (which kickstart to use, repositories). If no IP is specified it will be setup as a dhcp system with some address out of the pool configure by the cobbler dhcpd template.

You don't need --dns-name because cobbler doesn't manage any DNS for us.

System is ready to boot at this point.

Setup a system with some one-off differences

This would make a server actually look more like a desktop (strictly for example, you should probably use the desktop profile to do that)

cobbler system add --name=shoe --hostname=shoe.aglt2.org --gateway=192.41.230.1 --profile=server-generic --interface=eth0 --dns-name=shoe.local --mac=macaddress --ip-address=ipaddress --subnet=subnet \ 
 --ksmeta="rootsize=30768 tmpsize=30768 varsize=20576 cvmfssize=30768 homesize=102400'"

Create a typical dcache system with bonded nics

You can only add 1 interface with "cobbler system add". Then further interfaces are added with "edit".

 cobbler system add --name=umfs09 --hostname=umfs09.aglt2.org --profile=server-dcache --interface=em1 --mac=90:B1:1C:10:EB:25 

cobbler system edit --name=umfs09 --interface=bond0 --interface-type=bond   --bonding-opts="mode=4 xmit_hash_policy=1 max_bonds=2 arp_interval=250 arp_ip_target=192.41.230.2,192.41.230.1"  --ip-address=192.41.230.29 --netmask=255.255.254.0 --static=1 

cobbler system edit --name=umfs09 --interface=bond0.4010 --dns-name=umfs09.local --interface-type=bond --ip-address=10.10.1.29 --netmask=255.255.252.0 --static=1  
cobbler system edit --name=umfs09 --interface=p2p1 --mac=A0:36:9F:10:0D:1C --interface-type=bond_slave --interface-master=bond0 

cobbler system edit --name=umfs09 --interface=p2p2 --mac=A0:36:9F:10:0D:1E --interface-type=bond_slave --interface-master=bond0

This should result in a system with a proper bonded 20 Gbps configuration. We don't assign an address to em1, but the mac is there in cobbler's database so we can boot PXE from it and build the right profile and system name. As of yet this is untested (other than the commands are valid and have been input on our cobbler server).

In the final kickstart, there will be three "network" lines with em1,bond0,bond0.4010. Then in post-install the slave interfaces will be configured. You can view the ultimate product at https://sysprov02.aglt2.org/cblr/svc/op/ks/system/umfs09

Edit a system

Everything you can do with "cobbler system add" you can edit with "cobbler system edit".

Build a desktop or other system outside our network

Sometimes we have to build systems on networks we don't control. In particular the "Tier 3" desktops used by ATLAS users in our group reside on the physics network. Cobbler can generate an ISO for us to use which has the necessary setup to automatically install the system using cobbler repositories and kickstarts. It can also have the cobbler repositories built in so no network at all is required.

When you boot one of the images below, it will come up to a grub menu where you can choose from the systems and profiles you've included in the image.

If building on the physics network, we can "borrow" their dhcp server to get an address at boot. Using that connectivity the system can grab the cobbler kickstart from the cobbler server as defined in the ISO boot entry.

Option 1 - Use a CD/USB key and build system over network

1. Log into sysprov01 and build the image. We are going to limit what is on our image because processing every system (the default) takes longer and the resulting boot menu is very long. You can build the image with one system or as many as you want if you might re-use it later. If you do not give it "--systems" or "--profiles" then every defined profile and system is included and it will take a couple minutes to create (which is fine if you want to do it that way).

   cobbler buildiso --systems="t3pers05 t3pers12 t3pers13 t3pers15 t3pers18" --profiles="desktop rescue-mode-pubnet rescue-mode-privnet" 

   NOTE: I've also include the rescue mode profiles in my image. They will boot into SL rescue mode using the network to download the necessary pieces from our cobbler repository. There are two different rescue mode kickstarts depending on whether we have access to a public network or the private (local/msulocal) network. NOTE: If you boot a system to the "desktop" profile it will install using dhcp to get an address and be registered to cobbler with a temporary hostname.
2. If you need to use a CD, you can stop here and burn the resulting "generated.iso". However a USB key is probably the easiest option.
3. Convert the file "generated.iso" produced by buildiso into one that will boot from a USB key:

    isohybrid generated.iso 

4. That's it! Copy the generated.iso file someplace where you can insert a USB key and write to the key. On a windows system, this utility makes it easy: http://sourceforge.net/projects/win32diskimager/
   On a linux system, do "dd if=generated.iso of=/dev/sdXXXXXX" where XXXXX is your USB device and NOT your working hard drive (I hope, for your sake).

Option 2 - Use a CD/USB key and copy repositories to it (network not required)

I have not tested this option, but maybe a unique situation comes up requiring it.

1. The buildiso command is very similar to the network install with the addition of paramaters to indicate a standalone installation and what distro to include.

   cobbler buildiso --standalone --distro="sl-current" --systems="t3pers05 t3pers12 t3pers13 t3pers15 t3pers18" --profiles="desktop rescue-mode-pubnet rescue-mode-privnet" 

2. The remaining steps are the same as a network install.

Regarding dhcp/net installs: If building on a network without any dhcp server we'll need to configure an IP address on the kernel command line so there is some way for the system to reach our cobbler server (or, presumably, one could build a "standalone" ISO instead). You'll see there is a field in the web interface for kernel options (there is one also for options you want to be present in the installed system but you don't need to put an IP there). You put an IP there matching the desired system IP. Or on the command line it is the same option string:

cobbler system edit --name=name --kopts="ip=1.2.3.4"

(or "cobbler system add" if this is a new system)  

You only have to do this if there is no way for the system to get a DHCP address. That means that never would you need to do it for installing an ATLAS T3 desktop in the physics network, and you don't need to do it in our networks where we have the cobbler dhcp server (among others).

Option 3 -- Building on private net for use on public net

Bring the machine to 2268, put it on the RED local network cable. In Cobbler, set up the build with an unused private IP address, eg, 10.10.0.54 (be sure to remove that IP from Cobbler post-build). PXE build.

Individual steps to then follow:

• Catch it on boot and go to single user mode
• Change to the YELLOW physics network cat5e cable
• Change the root password to the desktop standard one
• edit /etc/sysconfig/network to change the gateway IP address,
  • probably to 141.211.100.1
• edit /etc/sysconfig/network-scripts/ifcfg-eth0 to put in the correct IP, netmask, etc., eg
  • 141.211.100.100 and 255.255.255.0 (those are for t3pers14)
• delete /etc/sysconfig/network-scripts/route*
• Check /etc/resolv.conf to make sure that 10.10.10.10 is first in the list
• Run "service network start" and make sure that you can ping the gateway
• Fix /var/cfengine/policy_server.dat to have only one line, ie,
  • 192.41.231.51
• Now, exit from the single user shell and cf-agent should run (once) successfully
• Log in
• Run (again) cf-agent -Kf failsafe.cf; cf-agent -K
• Edit /etc/inittab to set the run level to 5 from the default 3

Reboot to make sure it is totally happy.

Everything should now be good to go.

Cobbler kickstart templates

The snippets and templates in /var/lib/cobbler are automatically versioned in a local git repository whenever a "cobbler sync" is run. TODO is to replicate that repository so it can be cloned easily elsewhere (which I believe is also a function that can be performed by "cobbler replicate" to easily duplicate to some other cobbler server).

Here are just a few relevant parts of our kickstart template demonstrating how it is dynamic based on system profiles.

$tree and $yum_repo_stanza are pre-defined by cobbler based on the distro and repos configured in system profile. $SNIPPET picks up a snippet out of some other file in /var/lib/cobbler/snippets (editable from web interface).

#set global $profile = $getVar('$profile', 'none')
#set global $parent = $getVar('$parent', 'none')

# Use network installation
url --url=$tree
# If any cobbler repo definitions were referenced in the kickstart profile, include them here.
$yum_repo_stanza
# Network information
$SNIPPET('network_config')

#if $profile != 't3-sl6-desktop'
# Do not configure the X Window System
skipx
#end if

# Use /tmp/partinfo for partitions
$SNIPPET('flexible_select_partition')

As you can see, different text can be included with if statements checking on the profile. There are of course many more possibilities with the templating language than this.

Our snippet "flexible_select_partition" has logic built-in to select the right partitioning scheme for different systems. Some of that is done with the templating system, some is via bash script. If a "disk" variable is set it will use that, otherwise it will use the smallest disk available on the system. No matter what, it will not setup partitions on any disk greater than 5TB. Variables are set in distros, profiles, or systems with something like '--ksmeta="disk=sdc" ' on the command line for create or edit commands.

#raw and #end raw encapsulate portions that will not be interpreted by the templating engine. Otherwise it will attempt to substitute variables in our bash code.

For more information read the most current version of flexible_select_partitions in the cobbler web interface or /var/lib/cobbler/snippets on a cobbler server.

Upgrading Cobbler to SL7

sysprov02 was PXE built as SL7 from sysprov01 with a newer Cobbler rpm version, 2.8.2 vs 2.6.11. The VM itself was a clone of sysprov01, with one exception; a single 240GB disk was used, instead of 2 disks, one 40GB, the second 200GB.

Following the bare installation, and before anything was replicated from sysprov01, a comparison was made of snippets and kickstarts. Well, only one kickstart, default.ks, and with 2.8.2 that came empty, so simply replicate this from sysprov01 along with everything else that was replicated, as shown below. As for snippets used in that default.ks, we find the following comparison of useful versions, and what should be kept and what should be replaced.

[root@sysprov02 ~]# grep -i snippet cobbler_sysprov01/kickstarts/default.ks
$SNIPPET('network_config')   SAME
$SNIPPET('cluster-compute-partitions')   OLD ONLY
$SNIPPET('flexible_select_partition')  OLD ONLY
$SNIPPET('log_ks_pre')    SAME
$SNIPPET('kickstart_start')    LIKELY VERSION DOES NOT MATTER, SO, KEEP NEWEST
$SNIPPET('pre_install_network_config')   SAME
$SNIPPET('pre_anamon')   MUST USE NEW AS DIRECTORY HAS CHANGED
$SNIPPET('log_ks_post')   SAME
$SNIPPET('post_install_kernel_options')     SAME
$SNIPPET('post_install_network_config')    KEEP OLD VERSION
$SNIPPET('func_register_if_enabled')    SAME
$SNIPPET('download_config_files')   SAME
$SNIPPET('koan_environment')    SAME
$SNIPPET('cobbler_register')    SAME
$SNIPPET('post_anamon')    MUST USE NEW AS DIRECTORY HAS CHANGED
$SNIPPET('post-run-cf-agent')   OLD ONLY
## $SNIPPET('post-update-bios-firmware')   NO LONGER IN USE
$SNIPPET('kickstart_done')    USE NEW
$SNIPPET('interactive_package_groups')   OLD ONLY
$SNIPPET('desktop_package_groups')   OLD ONLY
$SNIPPET('mlnx_packages')   OLD ONLY
$SNIPPET('func_install_if_enabled')    SAME

Using "cobbler replicate"

Cobbler can transfer mirrored trees, packages, snippets, kickstart templates, and triggers as well. 
To do this, just use the appropriate flags with cobbler replicate.

Options:
  -h, --help            show this help message and exit
  --master=MASTER       Cobbler server to replicate from.
  --distros=PATTERN     pattern of distros  to replicate
  --profiles=PATTERN    pattern of profiles to replicate
  --systems=PATTERN     pattern of systems to replicate
  --repos=PATTERN       pattern of repos to replicate
  --image=PATTERN       pattern of images to replicate  (there are no images)
  --omit-data           do not rsync data

The chosen command is "cobbler replicate --master=sysprov01.aglt2.org --sync-all", which takes a long time as it also replicate some 110GB of repo files. It also took a few iterations as the first time through it failed on a missing profile or 3. Note that the "--sync-all" parameter is not indicated in the help above, that came from a web page. However, "cobbler replicate -h" does indeed show it.

Other Information

need to start cobbler, httpd, dhcpd, tftp. All but httpd can/should be started prior to any cobbler replication.

cobbler check -- This actually shows a removed but still referenced repo that remains to be fixed
What is cobbler get-loaders?
Yup, need to run that, as it copies loaders such as pxelinux from the Cobbler site.

The dhcpd.conf template is stored in svn as dhcp.template, and gets copied to /etc/cobbler, but svn is set to NOT sync that file. It must be copied in before dhcpd can successfully start.

systemctl restart cobblerd
cobbler sync
This pair should be run after everything else syncs up.

At UM, the umatlas repo must be synced from the svn source before sysprov02 became useful. This was accomplished by making the /home/packagers directory tree, then logging in as myself and checking out the entire umatlas svn tree and then doing the sync.

As of the creation of msu-prov02, this umatlas repo situation will become the default at MSU as well. See GLPI ticket 4820 for details on this, and on other items (such as the following CNAME detail) that will be changed to more fully synchronize the way in which the MSU and UM Cobbler instances work.

Finally, for the UM instance, the 2 CNAMEs, mirror.local and mirror.aglt2.org, had to be changed to point to sysprov02 instead of to sysprov01, and new CNAMEs were created for the Dell repos, that will continue to point to umrocks6 and msut3-rx6.

-- BenMeekhof - 22 May 2013