General steps to follow on Migrating data from one OST to another

1. Set the source OST in read only status from mds server, if you do not want the files to be migrated to certain OSTs, set them to readonly too.

[root@umdist11 ~]# lctl get_param osp.t3test-OST0001*.max_create_count
osp.t3test-OST0000-osc-MDT0000.max_create_count=20000
[root@umdist11 ~]# lctl set_param osp.t3test-OST0001*.max_create_count=0
osp.t3test-OST0001-osc-MDT0000.max_create_count=0

2. List the files from the source OST (from lustre client), execute the command on a client with 10Gbps network

[root@c-16-11 t3test]# lfs find --ost t3test-OST0001 /luste/t3test/

3. Use lfs_migrate to migrate the list of files to other OSTs (any other writable OSTs)

#lfs_migrate -y file1 file2 file3

Or

#lfs find --ost t3test-OST0001 /luste/t3test/|lfs_migrate -y 

The speed of migrating varies from client to client, bl-2-1 70MB/s, c-16-39 180MB/s

Files can be split into different list, and let each client to migrate a list of files in parallel

Our practice on retiring multiple OSTs

Steps to retire one or multiple OSS/OST

1) Set the targeted OSTs or OSSs to be readonly on mdtmgs serverretire multiple OSSes[root@mdtmgs Lustre]# sh /root/tools/LustreOSTRW.sh ro umdist01 umdist02 ....retire multiple OSTs[root@mdtmgs Lustre]# sh /root/tools/LustreOSTRW.sh ro OST0001 OST0002

2) Scan the files on the targeted OSTs or OSSs,choose one work node(with 10Gbps link) to run the "lfs find" command, running multiple instances of "lfs find" does not gain performance due to the bottleneck on the mdtmgs database

For OSTs, do WN_10G

#lfs find /lustre/umt3 --ost OST1 OST2

For OSSs,first generate the command line to scan all the OSTs in the OSSs.

#/atlas/data08/manage/cluster/LustreMigration/OSTScanGen.sh umdist01 umdist02
lfs find /lustre/umt3 --ost umt3B-OST000c --ost umt3B-OST000d --ost umt3B-OST000e --ost umt3B-OST0010 --ost umt3B-OST0011 --ost umt3B-OST0012 --ost umt3B-OST0013 --ost umt3B-OST0014 --ost umt3B-OST0015 --ost umt3B-OST0016 --ost umt3B-OST0017 --ost umt3B-OST003f >/atlas/data08/manage/cluster/LustreMigration/FileList/ostflist_umdist01_umdist02

Then choose a 10Gpbs work node, to run the above command, this command typically last 2-3 days depending on the size of the lustre file system, because it will scan the database on mdtmgs, and pick out the files stored on the specified OSTs.

3) From WNMover (a list of work nodes with 10Gpbs links), choose X nodes to run the "lfs migration" in parallel. (retire condor on these work nodes), and put the X work nodes(local name) in the file FileList/WNMover

4) Split the filelist generated in 2) to X files, run this command on a SL7 node due to the differences of split#cd /atlas/data08/manage/cluster/LustreMigration/FileList/;split -d -l Y ostflist_umdist01_umdist02Y is number of lines for each file.

5) On the selected work node, run#/atlas/data08/manage/cluster/LustreMigration/lustre_migrate.sh /atlas/data08/manage/cluster/LustreMigration/FileList&This script looks for files such as x00,x01... as input files for migration, and output the log files as log_x00, log_x01

6) There might be files failed at migration on the targeted OST or OSS, repeat step 2)-5) to move the leftover files

Please note: we originally use X=10 work nodes to do the migration, but this could cause the lustre system to hang, and one has to restart mdtmgs and all OSS servers to recover from it. So maybe X=5 is a safe value.

-- WenjingWu - 07 Feb 2020