In ROCKS5 whole new scheme for user customization of partitioning. The annoyances with getting custom partitioning done in ROCKS4 seem to be gone we no longer need multiple scripts to achieve the desired partitioning. The new way is much easier to understand.

Custom partitioning layout is defined using normal kickstart commands written to file /tmp/user_partition_info within a pre script. You no longer use the weird pseudo kickstart xml tags put in the main blocks of node xml scripts. The ROCKS installer determines if the host partitioning should be changed, and if so, the contents of user_partition_info are copied to /tmp/partition-info which is pulled into the kickstart file (ks.cfg) with an %include. If the installer decides that the host partitioning should not be changed, the user_partition_info is not used and ROCKS builds the partition-info file based on the info from the ROCKS DB.

The conditions to change host partitioning are:

• no partition info in DB
• no .rocks-release file on first partition of a disk

References

• https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2009-January/036626.html ROCKS Email
• https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2008-December/035532.html ROCKS Email
• http://www.rocksclusters.org/roll-documentation/base/5.0/customization-partitioning.html ROCKS 5 Docs
• https://wiki.rocksclusters.org/wiki/index.php/Partitioning ROCKS Wiki
• http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Installation_Guide/s1-kickstart2-options.html RHEL Kickstart Options
• rocks-dist/x86_64/build/include/installclass/do_partition.py ROCKS source code

A Bottom-Up View

DB Info

If a host has partition information in the DB, that info is given to the installer, which parses it and generates the kickstarts commands for partitioning.

ROCKS5 creates the file /tmp/db_partition_info.py (in the installer) like this:

cat > /tmp/db_partition_info.py << 'EOF'
RocksVersion = '5.2'
RocksRelease = 'Chimichanga'
KickstartHost = '10.10.2.3'

dbpartinfo =  {'sda': [('sda1', '32.3kB', '16.8GB', '', 'ext3', 'boot', '', '/'),
                      ('sda2', '16.8GB', '4195MB', '', 'ext3', '', '', '/var'),
                      ('sda3', '21.0GB', '1045MB', '', 'linux-swap', '', '', 'swap'),
                      ('sda4', '22.0GB', '10.2GB', '', '', '', '', ''),
                      ('sda5', '22.0GB', '10.2GB', '', 'ext3', '', '', '/state/partition1')]}

EOF

Tests

Have installed system. Partition table is correctly in ROCKS DB.

[root@compute-100-0 ~]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
msuroxi.aglt2.o msuroxi.local   255.255.255.255 UGH   0      0        0 eth0
255.255.255.255 *               255.255.255.255 UH    0      0        0 eth0
224.0.0.0       *               255.255.255.0   U     0      0        0 eth0
10.10.2.0       *               255.255.254.0   U     0      0        0 eth0
169.254.0.0     *               255.255.0.0     U     0      0        0 eth0
default         msuroxi.local   0.0.0.0         UG    0      0        0 eth0

[root@msuroxi nodes]# rocks list host partition compute-100-0
DEVICE MOUNTPOINT        START  SIZE   ID TYPE       FLAGS FORMATFLAGS
sda1   /                 32.3kB 16.8GB -- ext3       boot  -----------
sda2   /var              16.8GB 4195MB -- ext3       ----- -----------
sda3   swap              21.0GB 1045MB -- linux-swap ----- -----------
sda4   ----------------- 22.0GB 10.2GB -- ---------- ----- -----------
sda5   /state/partition1 22.0GB 10.2GB -- ext3       ----- -----------

partitioning files from normal reinstall of a customized node. Note that here ROCKS doesn't copy commands from user_partition_info to partition-info.

-bash-3.2# cat user_partition_info 
clearpart --all --initlabel --drives=sda
part / --size 8000 --ondisk sda
part swap --size 2000 --ondisk sda
part /var --size 5000 --ondisk sda
part /tmp --size 1 --grow --ondisk sda

-bash-3.2# cat partition-info 
part / --fstype ext3 --onpart sda1
part /var --fstype ext3 --onpart sda2
part swap --noformat --onpart sda3
part /tmp --noformat --onpart sda5

On reinstall, for above node, / and /var are reformatted, but /state/partition1 is not.

Touch /var/.rocks-release, reinstall. /var is reformed.

if the ROCKS DB is wiped of above,

A Top-Down View

For example, in Rocks 4 we used the following.

   Dell-Compute-JBOD )
       echo "<part>/ --size=20000 --ondisk=sda</part>"
       echo "<part>swap --size=16000 --ondisk=sda</part>"
       echo "<part>/var --size=10000 --ondisk=sda</part>"
       echo "<part>/tmp --size=1 --grow  --ondisk=sdb</part>"
       ;;

A simple translation of the above to ROCKS 5 is:

partfile="/tmp/user_partition_info"
...
   Dell-Compute-JBOD )
        cat  <<-EOT
       echo "clearpart --all --initlabel --drives=sda,sdb" > $partfile
       echo "part / --size 20000 --ondisk sda" >> $partfile
       echo "part swap --size 16000 --ondisk sda" >> $partfile
       echo "part /var --size 10000 --ondisk sda" >> $partfile
       echo "part /tmp --size 1 --grow  --ondisk sdb" >> $partfile
EOT
       ;;

In Rocks 4 we used two shell scripts (located in /home/install/extras/bin) to help control the partitioning. One, "wipe_disks.sh" contained "dd" commands to clear each disk of its partition table. This was used in conjunction with wiping the Rocks DB information concerning the disk partitioning (rocks remove host partition node_name) to establish the desired disk partitioning upon rebuilding the node.

In testing Rocks 5 disk partioning, the "clearpart" command can play a role, that apparently obviates the need to use the wipe_disk.sh script, thus likely allowing us to remove it from use. The following tests were run on dc2-2-21 at UM in establishing this usage.

• wipe disks using shell script and remove partition info from Rocks DB (ROCKS uses user_partition_info in this case)
  • Does not matter if clearpart is present, formats /, /var and /tmp
• remove .rocks-release files only, clearpart not present (ROCKS does not use user_partition_info in this case)
  • formats only / and /var
  • .rocks-release files are back following rebuild
• remove .rocks-release files, remove partition table info from Rocks DB, clearpart not present (ROCKS uses user_partition_info in this case)
  • node will not build (reason is that partition table still exists)
• Add back clearpart, conditions otherwise the same (Not clear what this test was?)
  • /, /tmp and /var all reformat during the rebuild

From this we conclude that partitions on a node can be change by

• the clearpart command should be present in set_partitions.sh
  • The presence of "clearpart" has no effect in other situations we've examined. A reboot is clean, and a rebuild formats / and /var only, leaving other partitions untouched.
• remove all .rocks-release files on the worker node
• clear the Rocks DB partition information for the node
• force a PXE boot.

A Cleaner Way To Write Partitioning Info

Simply create the /tmp/user_partition_info file from a pre script in appliance node. This avoids the replace-partition.xml node (which prevents doing plain installs as it can't be disabled in the graph) and is much simpler than echoing text from a bash script.

<pre>

<file name="/tmp/user_partition_info">
clearpart --all --initlabel --drives=sda
part / --size 8000 --ondisk sda
part swap --size 2000 --ondisk sda
part /var --size 5000 --ondisk sda
part /tmp --size 1 --grow --ondisk sda
</file>

</pre>

If branching is required for different nodes within one appliance, suggest implement an eval in the above block. One could also hang a new node from the appliance node and activate it conditionally with an attribute.

Currently, ignoring file servers, we have one-disk and two-disk Dells, with different disk sizes on the two-disk versions. This could be merged to simply having one and two disk systems, but that still leaves at least one required decision branch. Following up on this for future development seems a good path to follow as time allows.

Programmatic Partitioning

The ROCKS developers give an example of programmable partitioning. The idea is to use a python script that runs on the installing node. This script can inspect the node and automatically handle different situations. Using python you can include helper routines from ROCKS and anaconda code. We have not investigated this much, but it seems useful for some situations. For instance, instead of relying on clearpart, disks to be wiped could be selected based on parameters other than simply the device name (which isn't always reliable on multi-disk systems).

Outstanding Questions

If you want to repartition a single disk on a multidisk system, how to safely do it.

Will the issues with failed fscks of /tmp arise (need to review this issue).

Migrating a node from ROCKS4 to ROCKS5 frontend. Can we keep the existing partitioning?
Answer: yes. During testing, specific partitioning was not applied, other than establishing the NIC and other needed information to PXE boot the node without using insert-ethers. Existing Rocks4 partitions were retained, and the Rocks DB seemed to contain up to date information following the build.

Some interesting scripts dealing with the transfer of hosts from Rocks4 to Rocks5 can be found on umrocks.aglt2.org at:
file: /home/install/manually_add_host.sh
file: /home/install/manually_add_appliance.sh
directories: /home/install/Rocks-Scripts

-- TomRockwell - 08 Oct 2009