As part of NDMP restore validation I found a strange and a weird issue.
The details are given below:
Netbackup Master/Media all are on 7.0.1 Version
Policy Type: NDMP
Ontap: 8.1.1 7-mode
We have taken full backup of a volume that is successfully completed with status 0. when we validated actual data with backedup data we found that
chunk of files got missed in the backup and found extra folders/files which are not part of actual data.
The total count of the missed files are 51. Out of which 49 files are part of 1 folder.
The folder name is pretty complex(It is a long name along with the special characters like “_” ,spaces)
Verifications Performed:
1)Where as in NetApp, volume snapshot is taken at the beginning of NDMP backup session and data is actually read
from this snapshot. So “open file” concept will not applicable here for the missed files.
2)Are the missed files/folders really exists during the backup- Yes , got the confirmation from user that these files
are already exists from long and the data is static.
3)What are the extra files got backedup and why these files/folders assumed as strange ?
During the restore validation we found that extra chunk files got backedup and these files are never part of
actual data and the name of the folder looks very starnge and started with -:60RFN00 and it contains all of the
missed files of one folder.but the actual complex folder name changed as :60RFN00 and we found 2 other similar
folders in the backup restore.
Questions:
Why it got renamed?
Is it really renamed during restore or Netbackup itself communicating these folders with strange name or Netapp
changing it? Who would be the culprit?
One of our expert Marianne suggested me to verify the possibility of share-name vs actual name?
http://www.wafl.co.uk/tag/ls/
Its an excellent suggestions however, it will not be applicable here because filer team set only the share names not
folder name.
When we tried to take the backup up of this missed folder exclusively the backup got failed with generic error code 99 .
Problem 1:
1/24/2014 7:50:28 AM - begin writing
1/24/2014 7:50:29 AM - Error ndmpagent(pid=2556) NDMP_LOG_ERROR 1 DATA: Backup terminated: backup configuration failure (2) on RPC error:
ERROR: BAD ENV OP (for PATH)
1/24/2014 7:50:29 AM - Error ndmpagent(pid=2556) NDMP backup failed, path =PATH1/24/2014 7:50:29 AM - Error bptm(pid=12244) none of the NDMP
backups for client CLIENTNAME completed successfully
1/24/2014 7:50:34 AM - end writing; write time: 00:00:06
NDMP backup failure(99)
Errors Noticed:
1)Error ndmpagent(pid=5916) MOVER_HALTED unexpected reason = 3 (NDMP_MOVER_HALT_INTERNAL_ERROR)
1/21/2014 3:17:36 PM - Error ndmpagent(pid=5916) NDMP backup failed, path = /vol/Prestwick/.snapshot/nightly.0
1/21/2014 3:17:37 PM - Error bptm(pid=2500) io_ioctl_ndmp (MTBSF) failed on media id 0082L5, drive index 0, return code 10
(NDMP_NO_TAPE_LOADED_ERR) (bptm.c.8569)
1/21/2014 3:17:43 PM - end writing; write time: 00:29:13
NDMP backup failure(99)
2)- Error bptm(pid=8496) io_ioctl_ndmp (MTFSF) failed on media id 0027L5, drive index 1, return code 18 (NDMP_XDR_DECODE_ERR) (bptm.c.6912)
1/21/2014 3:13:03 PM - Error ndmpagent(pid=10144) connection 0000000001A5BAB0 ndmp_message_process_one failed, status = 18
(NDMP_XDR_DECODE_ERR)
1/21/2014 3:13:04 PM - Error ndmpagent(pid=10144) NDMP backup failed, path = UNKNOWN
Found no much details/clue on the ndmpd and ndmpagent logs
I have tried increasing the unified logging on media server which then encountered error below:
PROBLEM 2:
Running vxlogcfg -a -p 51216 -o 117 -s DiagnosticLevel=5 -s DebugLevel=5 gives error message.
Error:V-1-8-0 invalid setting / value passed
Cause:Netbackup Processes seem to be hung
Solution:
1: Stop the netbackup proccess on the master server
2: Run vxlogcfg -a -p 51216 -o 117 -s DiagnosticLevel=5 -s DebugLevel=5
3: Start the netbackup proccess
4: Run vxlogcfg -a -p 51216 -o 117 -s DiagnosticLevel=0 -s DebugLevel=0
Ref TN: http://www.symantec.com/business/support/index?pag...
To fix the problem1 I have done below tests:
1)Tried taking the backup of this strange name( :60RFN00 ) which is never part of Volume
Result: Backup is completed successfully.
2)I have asked Netapp Admin to take the snapshot of this volume to verify whether Netapp changing the folder or not
verified the snapshot.The folder name dint get changed in the snapshot but when tried to take the backup of this backup again the complex folder
renamed as :60RFN00.
3)Trying the backup by removing the spaces between special characters on the name of folder.
Worked out. Strange name dint enter this time in the backedup data. Validated using restore window.
Resolution:
Removing the space between special characters for the complex folder name resolved the issue.
Recommendation:
1)Upgrade Netbackup Version to 7.5/7.6 which is latest
2)Advise the user to avoid creating complex folder/file names for NAS.
Thanks
Sri Vani