Mdadm and LVM
- Category: 電腦相關
- Last Updated: Wednesday, 25 January 2017 22:50
- Published: Wednesday, 25 January 2017 15:22
- Written by sam
拆開VG，組成Raid(mdadm)
root@wantgooA:/# vgdisplay
--- Volume group ---
VG Name sam_new_elk
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 2
Act PV 2
VG Size 2.00 TiB
PE Size 4.00 MiB
Total PE 523774
Alloc PE / Size 523774 / 2.00 TiB
Free PE / Size 0 / 0
VG UUID QLDsV9-m7Y0-Bwwm-hZ2g-7QJJ-jcXo-nNjpe6root@wantgooA:/# vgs
VG #PV #LV #SN Attr VSize VFree
sam_new_elk 2 1 0 wz--n- 2.00t 0這個是目前的PV配置
root@wantgooA:/# pvdisplay
--- Physical volume ---
PV Name /dev/sdd1
VG Name sam_new_elk
PV Size 1023.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 261887
Free PE 0
Allocated PE 261887
PV UUID dBXJos-XqAq-bO3p-502D-9Gz7-UsC1-MiaScJ
--- Physical volume ---
PV Name /dev/sde1
VG Name sam_new_elk
PV Size 1023.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 261887
Free PE 0
Allocated PE 261887
PV UUID cbSve4-pRZ0-1qig-Hxdv-ALXY-Es09-5qn3yq
root@wantgooA:/# pvs
PV VG Fmt Attr PSize PFree
/dev/sdd1 sam_new_elk lvm2 a-- 1023.00g 0
/dev/sde1 sam_new_elk lvm2 a-- 1023.00g 0
root@wantgooA:/# umount /dev/mapper/sam_new_elk-lvol0當初也沒有預留，所以PVMOVE也沒得玩了(當然要新加入新的DISK就行)
root@wantgooA:/# pvmove /dev/sdd1
No extents available for allocation
root@wantgooA:/# pvmove /dev/sde1
No extents available for allocation再來直接殺了
root@wantgooA:/# vgremove sam_new_elk
Do you really want to remove volume group "sam_new_elk" containing 1 logical volumes? [y/n]: y
Logical volume "lvol0" successfully removed
Volume group "sam_new_elk" successfully removed準備四個 dev/disk 兩兩作成raid0
root@wantgooA:~# mdadm --create --verbose /dev/md0 --level raid0 --raid-devices=2 /dev/sde1 /dev/sdd1
root@wantgooA:~# mdadm --create --verbose /dev/md1 --level raid0 --raid-devices=2 /dev/sdf1 /dev/sdg1
root@wantgooA:~# pvs
PV VG Fmt Attr PSize PFree
/dev/md0 lvm2 --- 2.00t 2.00t
/dev/md1 lvm2 --- 2.00t 2.00t
root@wantgooA:~# vgcreate vg_group /dev/md0 /dev/md1
Volume group "vg_group" successfully created
root@wantgooA:~# lvcreate -l 100%free -n for_elk vg_group
Logical volume "for_elk" created
root@wantgooA:~# vgdisplay
--- Volume group ---
VG Name vg_group
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 2
Act PV 2
VG Size 4.00 TiB
PE Size 4.00 MiB
Total PE 1047422
Alloc PE / Size 1047422 / 4.00 TiB
Free PE / Size 0 / 0
VG UUID A9H0rT-5tE8-0pOh-45f3-ei9y-qnft-D0Dac7
root@wantgooA:~# lvdisplay
--- Logical volume ---
LV Path /dev/vg_group/for_elk
LV Name for_elk
VG Name vg_group
LV UUID kWKNJh-bqfV-22rJ-Vbb6-YBnQ-cjq1-71bSKW
LV Write Access read/write
LV Creation host, time wantgooA, 2017-01-25 15:18:36 +0800
LV Status available
# open 0
LV Size 4.00 TiB
Current LE 1047422
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 4096
Block device 253:0
root@wantgooA:/# mkfs.ext4 /dev/vg_group/for_elk (format ext4)
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 1072560128 4k blocks and 268140544 inodes
Filesystem UUID: 01a30176-01f0-41a0-b935-d1c048d58013
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
root@wantgooA:/# blkid (check uuid)
/dev/sdd1: UUID="21e169de-7171-333f-df49-40f60f3288ce" UUID_SUB="ff5be51b-715e-60d6-08b0-301f90552015" LABEL="wantgooA:0" TYPE="linux_raid_member" PARTUUID="1a337eae-01"
/dev/sde1: UUID="21e169de-7171-333f-df49-40f60f3288ce" UUID_SUB="dd17abab-4462-8aff-9656-835daf6faa24" LABEL="wantgooA:0" TYPE="linux_raid_member" PARTUUID="565908cc-01"
/dev/sdb1: UUID="92eb7671-c309-4627-b487-e2c6259df51d" TYPE="ext4" PARTUUID="0798be1f-01"
/dev/sdg1: UUID="121ff2ee-18de-ff77-4384-fadeca135985" UUID_SUB="462218be-bb08-37aa-0c55-8541c6572dec" LABEL="wantgooA:1" TYPE="linux_raid_member" PARTUUID="20428fe5-01"
/dev/sdf1: UUID="121ff2ee-18de-ff77-4384-fadeca135985" UUID_SUB="ff5c3f94-803f-e37a-be9e-cdcaaa96f448" LABEL="wantgooA:1" TYPE="linux_raid_member" PARTUUID="d5569cfd-01"
/dev/sdc1: UUID="e1749b6d-94e1-4519-8faf-e6824a67f248" TYPE="ext4" PARTUUID="2523aa78-01"
/dev/sda1: UUID="4ccfcb68-12a9-4bad-a03b-b67438fe573c" TYPE="ext4"
/dev/md0: UUID="l61q4c-MAtP-DM6F-cSRI-5MDU-hAX4-w3PNVS" TYPE="LVM2_member"
/dev/md1: UUID="M30uFP-XyF4-yl5F-yXRi-La1U-wYeD-yNUcVX" TYPE="LVM2_member"
/dev/mapper/vg_group-for_elk: UUID="01a30176-01f0-41a0-b935-d1c048d58013" TYPE="ext4"
root@wantgooA:/#vi /etc/fstab (for auto mount, notice UUID)各2個磁碟組RAID0，共MD0、MD1，再組進LVM，cp sdc to vg
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 494.00 0.00 63232.00 256.00 134.89 278.04 0.00 278.04 2.02 100.00
sde 0.00 0.00 0.00 465.00 0.00 59520.00 256.00 78.68 167.14 0.00 167.14 2.15 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 326.00 0.00 41728.00 0.00 256.00 3.32 10.18 10.18 0.00 3.04 99.20
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 960.00 0.00 122880.00 256.00 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 960.00 0.00 122880.00 256.00 214.60 225.27 0.00 225.27 1.04 100.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 558.00 0.00 71316.00 255.61 94.76 193.00 0.00 193.00 1.79 100.00
sde 0.00 0.00 0.00 465.00 0.00 59520.00 256.00 138.05 278.86 0.00 278.86 2.15 100.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 285.00 0.00 36480.00 0.00 256.00 3.04 10.67 10.67 0.00 3.47 98.80
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 929.00 0.00 118816.00 255.79 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 929.00 0.00 118816.00 255.79 233.94 256.62 0.00 256.62 1.08 100.00
4個磁碟組RAID0，共MD0，再組進LVM，cp sdc to vg
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 12.00 0.00 19.00 0.00 2104.00 221.47 0.19 10.11 0.00 10.11 3.58 6.80
sde 0.00 0.00 0.00 17.00 0.00 2048.00 240.94 0.22 12.71 0.00 12.71 3.53 6.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 17.00 0.00 2048.00 240.94 0.20 11.53 0.00 11.53 3.06 5.20
sdf 0.00 0.00 0.00 17.00 0.00 2048.00 240.94 0.20 12.00 0.00 12.00 3.06 5.20
sdc 0.00 0.00 386.00 0.00 49408.00 0.00 256.00 3.34 8.64 8.64 0.00 2.56 98.80
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 79.00 0.00 8248.00 208.81 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 78.00 0.00 8248.00 211.49 0.95 12.21 0.00 12.21 1.03 8.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 11.00 0.00 322.00 0.00 40144.00 249.34 36.72 89.93 0.00 89.93 1.09 35.20
sde 0.00 0.00 0.00 322.00 0.00 41216.00 256.00 36.30 90.26 0.00 90.26 1.06 34.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdg 0.00 0.00 0.00 321.00 0.00 41016.00 255.55 33.00 81.89 0.00 81.89 1.10 35.20
sdf 0.00 0.00 0.00 300.00 0.00 38400.00 256.00 34.10 83.81 0.00 83.81 1.13 34.00
sdc 0.00 0.00 192.00 0.00 24576.00 0.00 256.00 3.04 14.98 14.98 0.00 5.25 100.80
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 1755.00 0.00 222088.00 253.09 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 1755.00 0.00 222088.00 253.09 140.60 62.57 0.00 62.57 0.21 36.00
raid group
root@wantgooA:~# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Wed Jan 25 16:34:33 2017
Raid Level : raid0
Array Size : 4290244608 (4091.50 GiB 4393.21 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Wed Jan 25 16:34:33 2017
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Chunk Size : 512K
Name : wantgooA:0 (local to host wantgooA)
UUID : 83ea23da:6277d6b3:d4862b1e:768147ae
Events : 0
Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 8 65 1 active sync /dev/sde1
2 8 81 2 active sync /dev/sdf1
3 8 97 3 active sync /dev/sdg1root@wantgooA:/tmp# fio -filename=/dev/md0 -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=2G -numjobs=4 -runtime=1000 -group_reporting -name=readtest
readtest: (g=0): rw=read, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.11
Starting 4 threads
Jobs: 4 (f=4): [R(4)] [100.0% done] [11156KB/0KB/0KB /s] [697/0/0 iops] [eta 00m:00s]
readtest: (groupid=0, jobs=4): err= 0: pid=3133: Wed Jan 25 22:36:37 2017
read : io=8192.0MB, bw=14612KB/s, iops=913, runt=574073msec
root@wantgooA:/tmp# fio -filename=/dev/md0 -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=4 -runtime=1000 -group_reporting -name=writest
writest: (g=0): rw=write, bs=16K-16K/16K-16K/16K-16K, ioengine=psync, iodepth=1
...
fio-2.1.11
Starting 4 threads
Jobs: 3 (f=3): [_(1),W(3)] [100.0% done] [0KB/10917KB/0KB /s] [0/682/0 iops] [eta 00m:00s]
writest: (groupid=0, jobs=4): err= 0: pid=3151: Wed Jan 25 22:49:59 2017
write: io=8192.0MB, bw=13290KB/s, iops=830, runt=631176msec還是蠻差的