Sunday, July 01, 2007

Veritas problems and solutions.

I found 2 solutions to this problem. The difference is minor. They are Veritas commands.

We have a machine running Solaris and VxVM, attached to a Clarion box. The SAN link was down, and Storage guy managed to bring the link back after a while. We had a situation where:

1. OS detects all the SAN disks.
2. VxVM had not able to recognize back the original disks, though the device paths are the same.
3. All fs mounted on those disks have IO error. Though a format read test is passed.
4. It seems that this is purely a VxVM problem now.


# vxprint -g dg01 -th <== Showed the following...
dg dg01 2 2 123000 1021305687.1295.obp1
dm appsdisk c0t1d0s2 sliced 11555 71112735 -
dm appsmirror c1t1d0s2 sliced 11555 71112735 -
dm homedisk c2t0d0s2 sliced 14135 35349424 -
dm homemirror c3t0d0s2 sliced 14135 35349424 -


v home - DISABLED ACTIVE 16896000 SELECT - fsgen
pl home-01 home DISABLED RECOVER 16897232 CONCAT - RW
sd homedisk-01 home-01 homedisk 0 16897232 0 c2t0d0 RLOC
pl home-02 home DISABLED RECOVER 16897232 CONCAT - RW
sd h omemirror-01 home-02 homemirror 0 16897232 0 c3t0d0 ENA


Solution 1:
Part 1:
1. vxdctl enable
2. vxdisk list <--- you will see plenty of failed disk message because of broken SAN link.
3. /etc/vx/bin/vxreattach
4. vxdisk list
Part 2:
1. vxmend -g dg01 -o force off home-01
2. vxmend -g dg01 on home-01
3. vxmend -g dg01 fix clean home-01
4. vxvol -g dg01 start home
5. fsck -F vxfs /dev/vx/rdsk/dg01/home
6. mount -F vxfs /dev/vx/dsk/dg01/home /export/home

You only need to fix on 1 plex, the clean plex will resync the other plex which has problem.

Solution 2:
Part 1:
1. vxdctl enable
2. vxdisk list <--- you will see plenty of failed disk message because of broken SAN link.
3. /etc/vx/bin/vxreattach
4. vxdisk list

Part 2:
1. vxmend -g dg01 fix stale home-01
2. vxmend -g dg01 fix clean home-01
3. vxvol -g dg01 start home
4. fsck -F vxfs /dev/vx/rdsk/dg01/home
5. mount -F vxfs /dev/vx/dsk/dg01/home /export/home

Again, you only need to fix 1 plex. It will automatically resync the other plex.

No comments: