Veritas Storage Checkpoint

create
# fsckptadm -v create thu_7pm /checkpt1

Mount
# mount -F vxfs -o ckpt=thu_7pm /dev/vx/dsk/datadg/vol1:thu_7pm /checkpt1

remount
# mount -F vxfs -o ckpt=may_23,remount,rw /dev/vx/dsk/datadg/vol1:may_23 /fsvol_may_23

umount
# umount /fsvol_may_23

#umount /dev/vx/dsk/datadg/vol1:may_23


Convert data to no data chkpt
# fsckptadm -s set nodata thu_7pm /checkpt1 (convert data to nodata)

-s option means synchronous

this nodata checkpoint only contains a collection of markers

# fsckptadm set nomount thu_7pm /checkpt (convert mountable to non-mount)


# fsckptadm set remove thu_7pm /checkpt (convert non-removeable to removeable)

set quotas
# fsckptadm setquotalimit /checkpt1 1024 512

1024 - no.of hard limit blocks
512 - no.of soft limit blocks

remove
# fsckptadm remove thu_7pm /checkpt1

list
# fsckptadm -clv list /mnt0

-c : resets statiscal counters
-l : displays statistical info on the fs, in addition to its checkpoints
-v: verbose mode, displays extensive statistical info

restoring a file system

# fsckpt_restore -l /dev/vx/dsk/datadg/vol01

Full-sized instant volume snapshots - method 2 : using empty volume

Full-sized instant volume snapshots - method 2 : using empty volume

1. find the required size for the snapshot volume.

#LEN=`vxprint -g datadg -F%len vol01`

2. find the name of the dco volume

#DCONAME=`vxprint -g datadg -F%dco_name vol01`

3. find the dco region size.

#RSZ=`vxprint -g datadg -F%regionsz $DCONAME`

4. Create a volume of the required size and redundancy

#vxassist -g datadg make snapvol01 $LEN init=active

5. To prepare the volume for snapshot operation
(to enable the fast resync using dco)

#vxsnap -g datadg prepare snapvol regionsize=$RSZ

6. create the snapshot using the empty volume.

#vxsnap -g datadg make source=vol01/snapvol=snapvol01

7. split the snapshot volume into a separate disk group

#vxdg split datadg offhostdg snapvol01

or

#vxdg -g datadg -o expand split datadg offhostdg snapvol01 snapvol02

8. deport the disk group

#vxdg deport offhostdg

9. import the disk group in remote host

(remote host)#vxdg import offhostdg

10. recover the disk group

(remote host)#vxdg -g offhostdg -sb

11. XXX Perform off-host processing XXX
fsck and mount, etc.....

12. deport the disk group

(remote host)#vxdg deport offhostdg

13. import back on the local host.

#vxdg import offhostdg

14. join the disk group

#vxdg join offhostdg datadg

15. Recover the disk group.

#vxrecover -sb

if the volume still in disabled state, then start it.

#vxvol -g datadg start vol01

16. update the snapshot volume
(refreshing the volume will sync the changed block from vol01 to
snapvol01)


#vxsnap -g datadg refresh snapvol01

or

#vxsnap -g datadg refresh snapvol01 source=vol01

17. If full resync of the volume is needed, then a reattach is required.

#vxsnap -g datadg reattach snapvol01 source=vol01

18. remove the snapshot volume if not required anymore.

#vxedit -g datadg -rf rm snapvol01

Full-Sized Instant Volume Snapshots - method 1: addmir to have 1 plex to attach to the volume.

1. A mirrored-volume created with 2 plexes and fastresync turned on
====================================================================

pw250(root)::vxsnap -g tdg prepare v8

pw250(root)::vxprint -htg tdg v8
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
EX NAME ASSOC VC PERMS MODE STATE
SR NAME KSTATE

v v8 - ENABLED ACTIVE 409600 SELECT - fsgen
pl v8-01 v8 ENABLED ACTIVE 409600 CONCAT - RW
sd disk1-01 v8-01 disk1 0 409600 0 c2t8d18 ENA
pl v8-02 v8 ENABLED ACTIVE 409600 CONCAT - RW
sd disk2-01 v8-02 disk2 0 409600 0 c2t8d19 ENA
dc v8_dco v8 v8_dcl
v v8_dcl - ENABLED ACTIVE 544 SELECT - gen
pl v8_dcl-01 v8_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk6-01 v8_dcl-01 disk6 0 544 0 c2t8d17 ENA
pl v8_dcl-02 v8_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk3-01 v8_dcl-02 disk3 0 544 0 c2t8d20 ENA



2.Additional snap plex has been added to the volume.

pw250(root)::vxsnap -g tdg addmir v8 plex=v8snap-v3 alloc=disk4
====================================================================
v v8 - ENABLED ACTIVE 409600 SELECT - fsgen
pl v8-01 v8 ENABLED ACTIVE 409600 CONCAT - RW
sd disk1-01 v8-01 disk1 0 409600 0 c2t8d18 ENA
pl v8-02 v8 ENABLED ACTIVE 409600 CONCAT - RW
sd disk2-01 v8-02 disk2 0 409600 0 c2t8d19 ENA
pl v8-03 v8 ENABLED SNAPDONE 409600 CONCAT - WO
sd disk4-01 v8-03 disk4 0 409600 0 c2t8d21 ENA
dc v8_dco v8 v8_dcl
v v8_dcl - ENABLED ACTIVE 544 SELECT - gen
pl v8_dcl-01 v8_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk6-01 v8_dcl-01 disk6 0 544 0 c2t8d17 ENA
pl v8_dcl-02 v8_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk3-01 v8_dcl-02 disk3 0 544 0 c2t8d20 ENA
pl v8_dcl-03 v8_dcl DISABLED DCOSNP 544 CONCAT - RW
sd disk4-02 v8_dcl-03 disk4 409600 544 0 c2t8d21 ENA




3. After break off the plex, it is used to create new snapshot volume

pw250(root)::vxsnap -g tdg make source=v8/newvol=v8snapvol/plex=v8-03
pw250(root)::
================================================================

v v8 - ENABLED ACTIVE 409600 SELECT - fsgen
pl v8-01 v8 ENABLED ACTIVE 409600 CONCAT - RW
sd disk1-01 v8-01 disk1 0 409600 0 c2t8d18 ENA
pl v8-02 v8 ENABLED ACTIVE 409600 CONCAT - RW
sd disk2-01 v8-02 disk2 0 409600 0 c2t8d19 ENA
dc v8_dco v8 v8_dcl
v v8_dcl - ENABLED ACTIVE 544 SELECT - gen
pl v8_dcl-01 v8_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk6-01 v8_dcl-01 disk6 0 544 0 c2t8d17 ENA
pl v8_dcl-02 v8_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk3-01 v8_dcl-02 disk3 0 544 0 c2t8d20 ENA
sp v8snapvol_snp v8 v8_dco

v v8snapvol - ENABLED ACTIVE 409600 ROUND - fsgen
pl v8-03 v8snapvol ENABLED ACTIVE 409600 CONCAT - RW
sd disk4-01 v8-03 disk4 0 409600 0 c2t8d21 ENA
dc v8snapvol_dco v8snapvol v8snapvol_dcl
v v8snapvol_dcl - ENABLED ACTIVE 544 ROUND - gen
pl v8_dcl-03 v8snapvol_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk4-02 v8_dcl-03 disk4 409600 544 0 c2t8d21 ENA
sp v8_snp v8snapvol v8snapvol_dco
pw250(root)::




4. Split the diskgroup to create a new snap diskgroup
============================================================

pw250(root)::vxdg split tdg snapdg v8snapvol
pw250(root)::
pw250(root)::vxprint -htg snapdg
............
..........

dg snapdg default default 46000 1207136413.39.pw250

dm disk4 c2t8d21s2 auto 65536 10141440 -

v v8snapvol - DISABLED ACTIVE 409600 ROUND - fsgen
pl v8-03 v8snapvol DISABLED ACTIVE 409600 CONCAT - RW
sd disk4-01 v8-03 disk4 0 409600 0 c2t8d21 ENA
dc v8snapvol_dco v8snapvol v8snapvol_dcl
v v8snapvol_dcl - DISABLED ACTIVE 544 ROUND - gen
pl v8_dcl-03 v8snapvol_dcl DISABLED ACTIVE 544 CONCAT - RW
sd disk4-02 v8_dcl-03 disk4 409600 544 0 c2t8d21 ENA
sp v8_snp v8snapvol v8snapvol_dco


pw250(root)::vxprint -htg tdg
............
.....

v v8 - ENABLED ACTIVE 409600 SELECT - fsgen
pl v8-01 v8 ENABLED ACTIVE 409600 CONCAT - RW
sd disk1-01 v8-01 disk1 0 409600 0 c2t8d18 ENA
pl v8-02 v8 ENABLED ACTIVE 409600 CONCAT - RW
sd disk2-01 v8-02 disk2 0 409600 0 c2t8d19 ENA
dc v8_dco v8 v8_dcl
v v8_dcl - ENABLED ACTIVE 544 SELECT - gen
pl v8_dcl-01 v8_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk6-01 v8_dcl-01 disk6 0 544 0 c2t8d17 ENA
pl v8_dcl-02 v8_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk3-01 v8_dcl-02 disk3 0 544 0 c2t8d20 ENA
sp v8snapvol_snp v8 v8_dco


5. EXport the diskgroup
=========================
pw250(root)::vxdg deport snapdg
pw250(root)::vxdg list
NAME STATE ID
oracledg enabled,cds 1205751433.22.pw450
tdg enabled,cds 1207125877.35.pw250




6. Import the diskgroup on the remote host
===========================================

pw450(root)-->vxdg import snapdg
pw450(root)-->vxrecover -g snapdg -sb
pw450(root)-->
pw450(root)-->vxprint -htg snapdg
.........
..........

dg snapdg default default 46000 1207136413.39.pw250

dm disk4 c3t18d21s2 auto 65536 10141440 -

v v8snapvol - ENABLED ACTIVE 409600 ROUND - fsgen
pl v8-03 v8snapvol ENABLED ACTIVE 409600 CONCAT - RW
sd disk4-01 v8-03 disk4 0 409600 0 c3t18d21 ENA
dc v8snapvol_dco v8snapvol v8snapvol_dcl
v v8snapvol_dcl - ENABLED ACTIVE 544 ROUND - gen
pl v8_dcl-03 v8snapvol_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk4-02 v8_dcl-03 disk4 409600 544 0 c3t18d21 ENA
sp v8_snp v8snapvol v8snapvol_dco
pw450(root)-->
pw450(root)-->
pw450(root)-->mount -F vxfs /dev/vx/dsk/snapdg/v8snapvol /vsnap
pw450(root)-->
pw450(root)-->df -h /vsnap
Filesystem size used avail capacity Mounted on
/dev/vx/dsk/snapdg/v8snapvol
200M 2.1M 186M 2% /vsnap



7. Deport from remote host and import on original host
==========================================================

pw450(root)-->umount /vsnap
pw450(root)-->
pw450(root)-->vxdg deport snapdg

pw250(root)::vxdg import snapdg
pw250(root)::
pw250(root)::vxrecover -g snapdg -sb
pw250(root)::


8. Join the snap disk group into original disk group
=========================================================
pw250(root)::vxdg join snapdg tdg
pw250(root)::vxprint -htg tdg
..........
...

dg tdg default default 15000 1207125877.35.pw250

dm disk1 c2t8d18s2 auto 65536 10141440 -
dm disk2 c2t8d19s2 auto 65536 10141440 -
dm disk3 c2t8d20s2 auto 65536 10141440 -
dm disk4 c2t8d21s2 auto 65536 10141440 -
dm disk5 c2t8d22s2 auto 65536 10141440 -
dm disk6 c2t8d17s2 auto 65536 20381440 -

v v8 - ENABLED ACTIVE 409600 SELECT - fsgen
pl v8-01 v8 ENABLED ACTIVE 409600 CONCAT - RW
sd disk1-01 v8-01 disk1 0 409600 0 c2t8d18 ENA
pl v8-02 v8 ENABLED ACTIVE 409600 CONCAT - RW
sd disk2-01 v8-02 disk2 0 409600 0 c2t8d19 ENA
dc v8_dco v8 v8_dcl
v v8_dcl - ENABLED ACTIVE 544 SELECT - gen
pl v8_dcl-01 v8_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk6-01 v8_dcl-01 disk6 0 544 0 c2t8d17 ENA
pl v8_dcl-02 v8_dcl ENABLED ACTIVE 544 CONCAT - RW
sd disk3-01 v8_dcl-02 disk3 0 544 0 c2t8d20 ENA
sp v8snapvol_snp v8 v8_dco

v v8snapvol - DISABLED ACTIVE 409600 ROUND - fsgen
pl v8-03 v8snapvol DISABLED ACTIVE 409600 CONCAT - RW
sd disk4-01 v8-03 disk4 0 409600 0 c2t8d21 ENA
dc v8snapvol_dco v8snapvol v8snapvol_dcl
v v8snapvol_dcl - DISABLED ACTIVE 544 ROUND - gen
pl v8_dcl-03 v8snapvol_dcl DISABLED ACTIVE 544 CONCAT - RW
sd disk4-02 v8_dcl-03 disk4 409600 544 0 c2t8d21 ENA
sp v8_snp v8snapvol v8snapvol_dco



9. Start the snap volume as it is in disabled state

Refresh the snap volume with start the resync process with the orignial volume.
========================================================
pw250(root)::vxvol -g tdg start v8snapvol
pw250(root)::vxsnap -g tdg refresh v8snapvol

Veritas Traditional Volume Snapshot.

Traditional Volume Snapshots
==============================

make pitc
----------
1. vxvol set fastresync=on

2. vxassist -g xxxdg snapstart datavol

3. vxassist -g xxxdg snapshot datavol snapvol

use pitc
---------
4. vxdg split
5. vxdg join

update pitc
------------
6.vxassist -g xxxdg snapback snapvol

or

7. vxassist -g xxxdg -o resyncfromreplica snapback snapvol


display pitc info
------------------
8. vxassist -g xxxdg snapprint datavol


abort the snapshot
--------------------
To remove a snapshot mirror that has not been detached and moved to a snapshot volume.

9. vxassist -g xxxdg snapabort datavol



destroy pitc
-------------
10.destroy pitc

Xeon-toasted Chicken

This is a funny one.

In the bus....

Visit to mom's hometown

Linux VIO Considerations

Here's the support info provided by IBM:

1. For AIX 5L V5.3 VIO clients, only the APV VIOS is supported by IBM.
2. Linux VIO clients may use the APV VIOS or the Linux VIO server.


So it seems to be that having Linux VIO client is a better choice.
But do i want to run linux on such expensive IBM hardware??!

I can achieve better performance, availability and serviceability by running linux on x64/x86 server.

Conclusion is Linux VIO client / server is only meant for rich and affordable customers.

shares

silverstone - 4000 - PN17
amsteel - 4000 - PN17
mbfcorp - 8000 - PN17
twscorp - 1000
pasdec - 3000
scomi - 1000
GHL - 231
mobif - 4000
pmhldg - 1000
faber - 1000
bolton - 1000
dialog - 1000
sapcres - 1000
leader - 6000
rcecap - 3500
bimb - 2000
jland - 1000
ramunia - 2000
oilcorp - 1000
ta -1000

Play at school

............................

First they demand for an public apology from this young man. Only when he did so, then they wanted to take action against him. That's the strategy, i am totally disgusted by the way they handled this matter. This sick government!!

Photo on 15/8/2006

it was taken about a year ago. The big brother looks the same, whereas the young brother has grown up noticeably.

Veritas problems and solutions.

I found 2 solutions to this problem. The difference is minor. They are Veritas commands.

We have a machine running Solaris and VxVM, attached to a Clarion box. The SAN link was down, and Storage guy managed to bring the link back after a while. We had a situation where:

1. OS detects all the SAN disks.
2. VxVM had not able to recognize back the original disks, though the device paths are the same.
3. All fs mounted on those disks have IO error. Though a format read test is passed.
4. It seems that this is purely a VxVM problem now.


# vxprint -g dg01 -th <== Showed the following...
dg dg01 2 2 123000 1021305687.1295.obp1
dm appsdisk c0t1d0s2 sliced 11555 71112735 -
dm appsmirror c1t1d0s2 sliced 11555 71112735 -
dm homedisk c2t0d0s2 sliced 14135 35349424 -
dm homemirror c3t0d0s2 sliced 14135 35349424 -


v home - DISABLED ACTIVE 16896000 SELECT - fsgen
pl home-01 home DISABLED RECOVER 16897232 CONCAT - RW
sd homedisk-01 home-01 homedisk 0 16897232 0 c2t0d0 RLOC
pl home-02 home DISABLED RECOVER 16897232 CONCAT - RW
sd h omemirror-01 home-02 homemirror 0 16897232 0 c3t0d0 ENA


Solution 1:
Part 1:
1. vxdctl enable
2. vxdisk list <--- you will see plenty of failed disk message because of broken SAN link.
3. /etc/vx/bin/vxreattach
4. vxdisk list
Part 2:
1. vxmend -g dg01 -o force off home-01
2. vxmend -g dg01 on home-01
3. vxmend -g dg01 fix clean home-01
4. vxvol -g dg01 start home
5. fsck -F vxfs /dev/vx/rdsk/dg01/home
6. mount -F vxfs /dev/vx/dsk/dg01/home /export/home

You only need to fix on 1 plex, the clean plex will resync the other plex which has problem.

Solution 2:
Part 1:
1. vxdctl enable
2. vxdisk list <--- you will see plenty of failed disk message because of broken SAN link.
3. /etc/vx/bin/vxreattach
4. vxdisk list

Part 2:
1. vxmend -g dg01 fix stale home-01
2. vxmend -g dg01 fix clean home-01
3. vxvol -g dg01 start home
4. fsck -F vxfs /dev/vx/rdsk/dg01/home
5. mount -F vxfs /dev/vx/dsk/dg01/home /export/home

Again, you only need to fix 1 plex. It will automatically resync the other plex.

Yummmm

Wisdom Tooth

I had the x-ray done this morning. The x-ray clearly shows that i had all 4 wisdom teeth either decaying or laying nicely wating for the right moment. The 2 decaying one are killing me for the past 2 nights.
The one at upper-left started Saturday night, and really gave me a difficult time to sleep. The worst one came at evening Sunday, it just too painfull to be born. I almost fainted.
Go see a dentist, this is the serv 1 incident for me personally.

Sungai Kerayong site visit

It's me having some fun at the site. Though we not supposed to go near, but it was a sunny Sunday afternoon, and who cares!

What a sunshine day

This is 3pm in the afternoon. You know this is Kuala Lumpur, 3pm on a shinny day is very hot. They like the school environment. The big brother likes the flag so much, the little one just picks up walking 'skill', simply like to walk only. Their mom didn't like the idea to play at 3pm under open air. I have a different philosophy, 3pm is the time to stay at home for her, while for me it is good time to go out and have some fun.

Smart Tunnel and etc

This is from Director-general of Deparment of Irrigation and Drainage Malaysia. He was responding to the Smart tunnel which has not completed its water tunnel work. He says the government has come up with a ruling that future development projects include good drainage systems and ponds. So what has happened to the current devleopment projects? Is good drainage systems and ponds optional?? I guess you have the answer

Explanation

This happens to a disk which has a inode error on the OS disk.
The SA wants to do a fsck but afraid the it might cause unpredictable error on the disks. (Well,
i dont think this is really necessary, but the user requested for it)
This machine has the boot disks encapsulated and mirrored. First we want to know the mirrored disk's plex. The mirrored disk is c2t1d0. You can see the purpose is to break mirror and boot up onmirrored disk to perform a fsck on mirrored disk. When it's done, boot up from root disk and
reattach the mirror.

You figure out first what's this..

# vxprint -thg rootdg | grep pl | grep 02

pl export-02 export ENABLED ACTIVE 8389656 CONCAT - RW

pl local-02 local ENABLED ACTIVE 8389656 CONCAT - RW

pl pstore-02 pstore ENABLED ACTIVE 205119 CONCAT - RW

pl rootdisk1vol-02 rootdisk1vol ENABLED ACTIVE 4194828 CONCAT - RW

pl rootvol-02 rootvol ENABLED ACTIVE 29360907 CONCAT - RW

pl var-02 var ENABLED ACTIVE 12584484 CONCAT - RW

1. Remove volume pstore.

# vxassist –g rootdg remove pstore

2. Set up system boot information on mirror disk

# /etc/vx/bin/vxbootsetup rootmirr

3. Break rootmirror (c2t1d0)

# vxplex –g rootdg dis rootvol-02

# vxplex –g rootdg dis rootdisk1vol-02

4. FSCK mirror disk (c2t1d0).

# fsck –F ufs /dev/rdsk/c2t1d0s0

5. Mount root file system in mirror disk

# mount /dev/dsk/c2t1d0s0 /mnt

6. Change /etc/system in mirror disk

# cd /mnt/etc

# cp system system.ddmmyy

# cp system.rfc263171 system

7. Change /etc/vfstab in mirror disk.

# cd /mnt/etc

# cp vfstab vfstab.ddmmyy

# cp vfstab.rfc263171 vfstab

The entries from / and /swap will be changed as shown:

/dev/dsk/c2t1d0s1 - - swap - no -

/dev/dsk/c2t1d0s0 /dev/dsk/c2t1d0s0 / ufs 1 no logging

8. Un-mount file system.

# cd /

# umount /mnt

9. Shutdown system. Reboot system,

# init 0

ok boot vx-rootmirror -sw

Enter root password to enter maintenance mode

Perform FSCK on /

# fsck –F ufs /dev/rdsk/c2t0d0s0

When done, bring down system

# init 0

10. Reboot system with vx-rootdisk,

ok boot vx-rootdisk

IF boot is OK, go to step 11.

IF boot failed, go to step 12.

11. Re-attach mirror,

# vxplex –g rootdg att rootvol rootvol-02

# vxplex –g rootdg att rootdisk1vol rootdisk1vol-02

Job done. Use “vxtask list” to check progress of synchronization.

12. Boot the system off the detached mirror disk,

# init 0

ok boot vx-rootmirror –s

13. Check to ensure system is booted with disk slices rather than volume devices.

# df –k /

Filesystem kbytes used avail capacity Mounted on

/dev/dsk/c2t1d0s0 14451472 4468338 9838620 32% /

14. Stop all volumes in rootdg, re-attach those detached plexes,

# vxvol –g rootdg stopall

Attach detached plexes,

# vxplex –g rootdg att rootvol rootvol-02

# vxplex –g rootdg att rootdisk1vol rootdisk1vol-02

Make the newly attached plexes active,

# vxmend –g rootdg –o force off rootvol-01

# vxmend –g rootdg –o force off rootdisk1vol-01

# vxmend –g rootdg fix clean rootvol-02

# vxmend –g rootdg fix clean rootdisk1vol-02

# vxvol –g rootdg startall

15. Revert /etc/system and /etc/vfstab from backup files created earlier,

# cp /etc/system.ddmmyy /etc/system

# cp /etc/vfstab.ddmmyy /etc/vfstab

16. Shutdown system and reboot with mirror disk again,

# init 0

Ok boot vx-rootmirror

System should boot up using volume device, “df –k /” to verify,

# df –k /

Filesystem kbytes used avail capacity Mounted on

/dev/vx/dsk/bootdg/rootvol

14451472 4468384 9838574 32% /

17. Online the disabled plexes,

# vxmend –g rootdg on rootvol-01

# vxmend –g rootdg on rootdisk1vol-01

Synchronization will starts, use “vxtask list” to check progress.