Tonneson DokuWiki

Lustre

Architecture

Type	Description
Metadata Server (MDS)	The MDS server makes metadata stored in one or more MDTs available to Lustre clients. Each MDS manages the names and directories in the Lustre file system(s) and provides network request handling for one or more local MDTs
Metadata Target (MDT)	The MDT stores metadata (such as filenames, directories, permissions and file layout) on an MDS. Each file system has one MDT. An MDT on a shared storage target can be available to many MDSs, although only one should actually use it. If an active MDS fails, a passive MDS can serve the MDT and make it available to clients. This is referred to as MDS failover.
Object Storage Servers (OSS)	The OSS provides file I/O service, and network request handling for one or more local OSTs. Typically, an OSS serves between 2 and 8 OSTs, up to 8 TB each. The MDT, OSTs and Lustre clients can run concurrently (in any mixture) on a single node. However, a typical configuration is an MDT on a dedicated node, two or more OSTs on each OSS node, and a client on each of a large number of compute nodes.
Object Storage Target (OST)	The OST stores file data (chunks of user files) as data objects on one or more OSSs. A single Lustre file system can have multiple OSTs, each serving a subset of file data. There is not necessarily a 1:1 correspondence between a file and an OST. To optimize performance, a file may be spread over many OSTs. A Logical Object Volume (LOV), manages file striping across many OSTs.
Lustre clients	Lustre clients are computational, visualization or desktop nodes that run Lustre software that allows them to mount the Lustre file system.

How-Tos

Generate a list of devices and determine the OST’s device number

Run at the MGS or OSS:

$ lctl dl

The lctl dl command output lists the device name and number, along with the device UUID and the number of references on the device.

Deactivate the OST (on the OSS)

Run at the MDS:

$ lctl --device <OST device name or number> deactivate

The OST device number or device name is generated by the lctl dl command. The deactivate command prevents clients from creating new objects on the specified OST, although you can still access the OST for reading.

Check general health

Run at any (MGS/MDS/OSS/client):

$ cat /proc/fs/lustre/health_check

Run at client:

$ lfs check servers

Add new filesystem

For MDT:

Create VG/LV for new MDT on Dom0 servers
Attach to MGS and MDS using xm block-attach
Edit Xen config file to permanently attach new block device
On MGS add to /etc/ha.d/haresources (for LinuxHA only)
On MDT add to /etc/ha.d/haresources (for LinuxHA only)
On MDT format new filesystem using mkfs.lustre
On MDT mount new filesystem

For OSS:

Fdisk on Dom0 servers
Attach to OSS servers using xm block-attach
Edit Xen config file to permanently attach new block device
On each OSS add to /etc/ha.d/haresources (for LinuxHA only)
On each OSS format new filesystem using mkfs.lustre
On each OSS mount new filesystem

For clients:

Install supported kernel and patchless client components (lustre and lustre-modules)
vi /etc/modprobe.conf and add 'options lnet networks=“tcp0(eth0)'
Reboot to new kernel
vi /etc/fstab and add a similar line '(MDS IP addr)@tcp:(MGS IP addr)@tcp:/(oss_mount) /(local_mount) lustre defaults 0 0'
mount -a