PuppetInfrastructure < AGLT2

You are here: Foswiki>AGLT2 Web>PuppetInfrastructure (09 May 2013, BenMeekhof)Edit Attach

Index of other pages

ForemanPuppetInitialSetup - unorganized notes from initial setup. Mostly you won't need these.

HOWTO: Build new host with foreman

Mostly self-explanatory through web interface. Create new host, choose host group (at least base/linux/sl probably). Some things to note:

If creating on the VMware resource, foreman will create the host for you but you'll have to go fix the network interfaces before it will build. Network interfaces need to be in the right network on vSwitch. There may be other tweaks needed, at the time of writing I don't recall
Look at steps here to see how to associate templates: http://projects.theforeman.org/wiki/foreman/Unattended_installations
Look under More->Provisioning in web interface for partition tables, OS trees, etc. More->Configuration for host groups and puppet classes applied

(More information and step-by step coming someday. For now you have to read the terrible foreman docs like I did. Google will help a lot, there is more information on mailing lists)

HOWTO: Bootstrap puppet onto existing host

Install puppet repo:

http://yum.puppetlabs.com/el/5/products/x86_64/puppetlabs-release-5-7.noarch.rpm

http://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-7.noarch.rpm

Then just "yum install puppet"

Without setting up host in foreman

This minimal /etc/puppet/puppet.conf will be enough that you can run "puppet agent --test". Then return to the puppet master host and run "puppet sign " or "puppet cert list" to see outstanding if you're not sure what hostname got registered. After the first run, the host will show up in the foreman dashboard and you can put it in a hostgroup to get some puppet classes applied to it.


[main]
vardir = /var/lib/puppet
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl


[agent]
pluginsync      = true
report          = true
ignoreschedules = true
daemon          = false
ca_server       = syscfg01.aglt2.org
certname        = hostname.aglt2.org
environment     = production
server          = syscfg01.aglt2.org

Set up host in foreman first

I think then you can skip the certificate signing step because foreman will do it all for you.

HOWTO: Administer gitolite

Making configuration changes

Gitolite changes are made by cloning the gitolite-admin repository and pushing changes back to the master. Here is a sample of how to do it:

git clone gitolite@git.aglt2.org:gitolite-admin (creates local dir "gitolite-admin")
...change stuff... (more details below on what you might change)
If you added any new files, such as new ssh public keys: git add filename
git commit -a (-a means stage all files modified, and you'll be asked to put in a log message)
git push (push commits to the source repository)

Enabling access for a user

Clone the admin repository and put the user's ssh public key into keydir as username.pub Edit conf/gitolite.conf to give them access to a repo, or put into group with access. More information on managing access rights is on the website for gitolite

Creating a new repo

Add a line for the repo and some sort of access specification. It will be created when you push the config back to gitolite.

After making changes, remember you have to push the commits back to gitolite.

HOWTO: Create new git branch with auto-created puppet environment

NOTE: You will probably figure out that you can shortcut around this and simply "cp -rf" into a new environment directly on syscfg01. Maybe this will make sense for very small changes or if you just want to experiment with an alternative environment but don't need to track any changes it also means we have no track of where or who made changes to the production environment should you start putting things into it from your experiment.

Step 1: Clone the repositories for puppet and hiera.

You can do this on any host.

git clone gitolite@ndt.aglt2.org:puppet
Cloning into 'puppet'...
...
git clone gitolite@ndt.aglt2.org:hiera
Cloning into 'hiera'...
...

(uses ssh key for auth)

Step 2: Create a branch and make it an available environment.

cd puppet
git checkout -b testing
Switched to a new branch 'testing'
cd hiera
git checkout -b testing
Switched to a new branch 'testing'

You are now working on a new branch. You can at anytime push your branch back to the origin to set it up on the puppetmaster(s). You could do it right away, or do it after you make some changes. Let's say we made a few changes already so we need to possibly add or commit

 git add 
 git commit -a -m "my commit log"

(push the branch to the remote repository)
 git push origin testing

Total 0 (delta 0), reused 0 (delta 0)
remote: Updating remote branch /etc/puppet/environments/testing
remote: /bin/sh: line 1: cd: /etc/puppet/environments/testing: No such file or directory
remote: Initialized empty Git repository in /etc/puppet/environments/testing/.git/
remote: Scientific Linux SL release 5.7 (Boron)
remote: Branch testing set up to track remote branch testing from origin.
remote: Switched to a new branch 'testing'
To gitolite@ndt.aglt2.org:puppet
 * [new branch]      testing -> testing

Repeat the sequence above to create the environment in hiera also.

The remote commands are gitolite creating the new environment on our puppetmaster. Any changes you push from your branch will be synchronized.

Step 3: Push changes and test puppet with your branch

Commit our changes on the branch:

git add 
 git commit -a -m "my commit log"
git push

Git push automatically causes your updates to be checked out by the puppetmaster.

To test, on the Foreman host syscfg01.aglt2.org, go to "Hosts" and edit the host you want to test. Change environment to "testing". Then on the host you can run "puppet agent --test". Specifying environment on the command line won't work, it will use what the server says to use.

Step 4: Merge branch into production

So it went great, you're awesome. Time to merge back into production.

Merge them into the production branch:

git checkout production
git merge testing
git push

Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 377 bytes, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Updating remote branch /etc/puppet/environments/production
remote: Scientific Linux SL release 5.7 (Boron)
remote: From ndt.aglt2.org:puppet
remote:  * branch            production -> FETCH_HEAD
remote: Updating f39c4a3..b3428b6
remote: Fast-forward
remote:  0 files changed, 0 insertions(+), 0 deletions(-)
remote:  create mode 100644 extras/testingfile
remote: Updating remote branch /etc/puppet/environments/testing
remote: Scientific Linux SL release 5.7 (Boron)
remote: From ndt.aglt2.org:puppet
remote:  * branch            testing    -> FETCH_HEAD
remote: Updating f39c4a3..b3428b6
remote: Fast-forward
remote:  0 files changed, 0 insertions(+), 0 deletions(-)
remote:  create mode 100644 extras/testingfile
To gitolite@ndt.aglt2.org:puppet
   f39c4a3..b3428b6  production -> production
   f39c4a3..b3428b6  testing -> testing

You'll notice in my example it also updated testing because I had made the change in that branch but had not pushed the branch yet. That's because I was writing this doc and I didn't really push my change to test it first...I just did it all at once.

Step 5: Kill your branch

It's up to you whether to destroy your local branch, but generally you would want to if you're done with it.

git branch -d testing

However, it still exists on the origin repository. Notice it's going to entirely delete the environment on the puppetmaster when we do this:

 git push origin :testing

remote: Deleting remote branch /etc/puppet/environments/testing
To gitolite@ndt.aglt2.org:puppet
 - [deleted]         testing

Remember, there is also a hiera branch and environment to remove.

HOWTO: Manage users in Hiera and use create_resources

Users and groups are defined in our hiera data. By default, every machine picks up the accounts in /etc/hiera/production/accounts.yaml (replace "production" with current puppet environment in effect). However you could define accounts in other, more specific, places in the hierarchy such as um.yaml, or somehostname.yaml.

Example of defining users. This structure is turned into a hash that is used to instantiate a new puppet type. You only specify the top-level of the hash once, as shown below.

IT IS VERY IMPORTANT THAT YOU PUT AT LEAST ONE SPACE OF INDENT TO INDICATE THE STRUCTURE OF THE DATA OR IT WILL NOT PARSE AND HIERA WILL FAIL WITH AN UNHELPFUL RUBY TRACEBACK. PUPPET WILL GIVE NO INDICATION OF FAILURE DURING RUNS, IT JUST WON'T HAVE ANY VALUE WHERE YOU EXPECT ONE. DO NOT USE TABS, THEY WILL BREAK THE PARSING TOO

accounts::users:
 bmeekhof:
  uid: 96843
  gid: 614
  realname: 'Benjeman J. Meekhof'
  groups:
   - 'wheel'
   - 'apache'
   - 'packagers'
   - 'gasmon'
   - 'aglt2'
   - 'svn'

 pnatasha:
  uid: 223504
  gid: 614
  realname: ''
  groups:
   - 'umsusy'

You could put a similar structure in some other yaml file in the hierarchy. Hiera will merge them when it produces the final hash structure.

Groups are similar:

accounts::groups:
 ligo:
  gid: 55672
 bzhou:
  gid: 530
 umatlas:
        gid: 614
 umsusy:
  gid: 55685
 grase:
        gid: 55663

How does this tie together with puppet code? Let's look at our "accounts" module. By including the "accounts" class on a node (as it is included in the "base" host group) you initiate the code below. Note that all module file locations are structured similarly.

/etc/puppet/environments/production/modules/accounts/manifests/init.pp

class accounts ($users,$groups,$homeprefix='/home') {

create_resources('accounts::user',$users)
create_resources('accounts::group',$groups)

}

This class takes three parameters. Looking back at our hiera code, you can see we specified "accounts::users" and "accounts::groups" (accounts::homeprefix also happens to be set in accounts.yaml, or else it would default to /home as specified). These values are automatically read in when we include the class on a node. There is no need to explicitly use hiera calls in your manifests.

create_resources reads in the hashed structure that hiera will output and creates the resource types specified. Where is that resource defined? In the user.pp file in the accounts module. It could also be defined in the init.pp file and given a simple method name like "createuser" but I like this structure better.

/etc/puppet/environments/production/modules/accounts/manifests/user.pp


define accounts::user($uid,$gid,$realname,$groups,$home="$accounts::homeprefix/$title",$shell="/bin/bash") {

  user { $title:
    ensure            =>  'present',
    uid               =>  $uid,
    gid               =>  $gid,
    groups            =>  $groups,
    shell             =>  $shell,
    home              =>  $home,
    comment           =>  $realname,
    managehome        =>  false,
  }

}

Understanding hiera hierarchy

Here is our /etc/puppet/hiera.yaml. If you change this file, restart httpd or puppet won't be using the modified version.

---
:backends:
  - yaml
:hierarchy:
  - %{hostname}
  - %{location}
  - %{virtual}
  - common
  - accounts

:yaml:

  :datadir:  '/etc/hiera/%{environment}'

Note that the hiera datadir follows the puppet environment.

You ask hiera for a value in a manifest with one of the "hiera" functions. More often (always?), you will probably want to not explicitly ask and instead rely on puppets automatic lookup of class parameters. This means that someone not having hiera can still include your class and specify parameters directly if they want.

$var = hiera('keyname');

It will search the hierarchy starting with hostname until it finds a value and then return that value. If it is a "deep merge" hash lookup, such as with "create_resources", it looks through the whole hierarchy and produces a hash that is a recursive merge of all hash values found. A conventional "hiera_hash" in puppet returns a merge of the top level keys. An array lookup with hiera_array in puppet will return a merged and flattened array of all values for a key. It is all explained in much better detail here: http://docs.puppetlabs.com/hiera/1/lookup_types.html#array-merge

So, for example, let's say I have some value for the mysql class that usuall I want to be 'localhost'. So I put it into common.yaml as such:

mysql::mysqld_bind_address: 'localhost'

On the command line, we can test the lookup like this:

 hiera mysql::mysqld_bind_address environment=production
localhost

Let's say that for one host, I want it on the public interface. So I edit syscfg01.yaml and put in the same line:

mysql::mysqld_bind_address: '192.41.231.84'

Now see how we get different values depending on the asking context:

 hiera mysql::mysqld_bind_address environment=production
localhost

 hiera mysql::mysqld_bind_address environment=production hostname=syscfg01
192.41.231.84

When puppet asks, it will specify the environment and hostname. It will also specify location and virtual. It gets these facts from a combination of "facter" and also by using Foreman as an External Node Classifier (ENC). You can run facter on the command line on any puppet host, and you can see the foreman ENC facts by running a command like this on syscfg01:

/etc/puppet/node.rb syscfg01.aglt2.org

--- 
  parameters: 
    owner_email: "bmeekhof@umich.edu"
    puppet_ca: syscfg01.aglt2.org
    root_pw: "1234"
    Location: UM
    location: UM
    hostgroup: base/linux/sl/syscfg
    owner_name: "Admin User"
    domainname: ""
    foreman_env: production
    organization: AGLT2
    puppetmaster: syscfg01.aglt2.org
    Organization: AGLT2
  classes: 
    "concat::setup": 
    hostcert: 
    "puppetdb::master::routes": 
    stdlib: 
    rack: 
    foreman_proxy: 
    "hostcert::osgtools": 
    firewall: 
    "mysql::server": 
    puppetdb: 
    mysql: 
    accounts: 
  environment: production

Any of these facts could be used as a step in the hiera hierarchy. The external classifier is specified on the puppet master in puppet.conf.

Topic revision: r8 - 09 May 2013, BenMeekhof

AGLT2

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback