Troubleshoot NetBackup Status Code: 99

During regular (standard) NDMP backups, avoid potential NDMP communications failures between NetBackup media servers and the Network Attached Storage (NAS) host.

The following troubleshooting steps may help isolate the root cause of NDMP backup issues. If they do not, proceed with logging instructions in Step 8.

NOTE:  Terminology for the types of NDMP backups can be confusing:

• Remote NDMP:  The backup is performed on the media server, using the device paths defined to the media server.
• Local:  The backup is performed using the device paths attached/zoned to the NAS host.
• 3-Way:  The backup of a NAS host is performed using the device paths attached/zoned to another NAS host

To isolate the root cause of a NDMP backup failure:

1. On the media server, test the connection to the NAS device on the NDMP port (10000) by typing the telnet command.  Try both the hostname and its IP address.  For example: telnet ndmp_host 10000

2. To test connectivity and credentials of the NDMP host and its devices:

•  Type tpautoconf -verify [NAS hostname]
•  Type tpautoconf -probe [NAS hostname] to  probe the scsi bus for devices (local or 3-Way only).
•  On another master or media server, type the above two commands to check for another route or network path.
• Compare the probe results with the native NAS commands:

          NetApp:  sysconfig -t
          EMC Celerra:  server_devconfig [datamover name] -l -s -n  (list, scsi, non-disk)
         Other:  See system documentation

3. On the NAS, and type the command ndmpd status to verify that the NDMP daemon is running.  If it is not, enable the ndmp daemon and verify again by typing the command ndmpd status.  For information on starting the ndmp daemon, see the NAS system documentation.

4. For EMC Celerra devices, ensure that the SnapSure feature is enabled.  See TECH170310 for information on SnapSure and other considerations.

5. On the media or master server, type the telnet and tpautoconf command tests again from Step 1 to test connectivity, credentials, and devices.

6.  If Local or 3-Way backup, do the following to verify that the volume can be accessed and backed up:

• First, mount a scratch tape in the drive using robtest or some other means and perform a backup.
• On the NAS device, type the appropriate native command:

Network Appliance (NetApp): dump 0uf nrst0a /<file system> (Substitute the proper non-rewind device path if it is not nrst0a as in this example).

This command will write data from the specified file system on the Network Appliance server directly to tape. If this dump does not complete successfully, then the problem is most likely a NetWork Appliance problem. To resolve the issue, open a call with Network Appliance technical support. If the dump does complete successfully, open a call with NetBackup support.

For EMC Celerra, refer to the server_archive command in the system documentation.

NOTE:  The Backup Path specified in the NetBackup policy must be the exact case-sensitive path as the filer sees it.

7.  For performance issues, try backing up the path in the policy to a null device and note the timing:

NetApp:  dump 0f null /vol/volname
EMC Celerra:  server_archive <datamover name> -w -f /dev/null/ -J <file path>

8. Based on your findings, proceed with any required configuration changes or, if there are no changes, begin backups or restores.

9.  If none of the above troublesooting steps resolve the issue, enable debug logging on the NetBackup media server and on the NAS device.

To enable debug logging:

1. Depending on your media server platform, do the following:

For Windows media server: 

• On the NetBackup console, go to Host Properties --> Media Server and double-click the media server that services this backup to open the Properties dialog box. If this media server is the same as the master server, go to Master Server instead.
• Go to the Logging section and change the global logging level to 5 (maximum).
• Click OK to exit.

For UNIX media server: 

• Add VERBOSE to the /usr/openv/netbackup/bp.conf file in the following format:  VERBOSE = 5

2. Make sure that the following legacy log directories exist on the media server that services the backup:

For Windows media server:
\NetBackup\logs\bpbrm
\NetBackup\logs\bptm
\NetBackup\logs\ndmpagent

For Unix media server:
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bptm
/usr/openv/netbackup/logs/ndmpagent

3. Run the following commands on the media server that services this backup to increase the unified logging levels:

For Windows media server:
\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DebugLevel=6
\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DebugLevel=6
\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=6
\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=6

For Unix media server:
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DebugLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DebugLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=6

4. Enable ndmpd debug logging on the NAS device:

• For Network Appliance NAS, type the following to enable ndmpd debug logging:  ndmpd debug 70
• EMC Celerra:  Enable debug logging on the datamover as described in Article TECH150646
• For other NAS devices, see the vendor documentation.

5. Reproduce the error.

6.  Disable logging:

• For Network Appliance NAS, type the following to disable ndmpd debug logging:  ndmpd debug 0
• EMC Celerra, see the disable commands in TECH150646.
• For other NAS devices, see the vendor documentation.

7. Retrieve the bpbrm, bptm, ndmpagent logs from the NetBackup media server for today's date. 

8. Do one of the following to retrieve logs from the NAS device:

• Pull the datamover server_log from the EMC Celerra:  server_log server_2 -a -s > /output.file  (assumes ‘server_2’ is the datamover name)
• Pull the system log and the ndmpdlog from the Network Appliance NAS:
  • /vol/etc/messages
  • <volume>/etc/log/ndmpdlog (/vol/etc/log/mlog in DataOnTap 8)

9. Type the commands on the media server to disable logging:

For Windows media server:

<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DebugLevel=0
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DebugLevel=0
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=0
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=0

For Unix media server:

/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DebugLevel=0
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DebugLevel=0
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DiagnosticLevel=0
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DiagnosticLevel=0

10. Create a temp directory.
 For example: C:\temp\vxul or /tmp/vxul

11. Run the following command to retrieve all unified logs for today's date:

For Windows media server: <install path>\NetBackup\bin\vxlogmgr -c -t 1 -f c:\temp\vxul

For Unix meda server: /usr/openv/netbackup/bin/vxlogmgr -c -t 1 -f /tmp/vxul

Where the '-t 1' value will retrieve the unified logs from the last hour.  Adjust the value as appropriate to go back past the beginning of the job.

12. Compress the temp directory (Zip for Windows, tar for Unix) and send to Symantec NetBackup Support along with the legacy logs and the NAS logs.

13. To create a long listing of the policy, run the following comand on the NetBackup master server: 

Windows master server: <install path>\NetBackup\bin\admincmd\bppllist <name of policy> -L

Unix master server: /usr/openv/netbackup/bin/admincmd/bppllist <name of policy> -L

14. On the media server, on the command line, run the nbsu tool as follows to create the output file:

Windows media server: 

• In a command window, change the directory to <install path>\NetBackup\bin\support
•  Type the following command:  nbsu -c -t -nozip

Unix media server: 

• Type the following command: /usr/openv/netbackup/bin/support/nbsu -c -t

The script will create an output file and tell you where it was stored.

15. Cut and paste the Job Details for the job in the Activity Monitor during the backup attempt.  Send that to Symantec Support along with the nbsu output file and the bppllist output.

Related Articles: