Mar 1, 2021

VERSA SNMP SERVICE TROUBLESHOOT

SNMP common problems of v2c and v3:

Unable to send traps
Unable to reply SNMP poll
Desired SNMP version not enabled
SNMP agent ip address is not 127.0.0.1
SNMP manager not listening to the right UDP port and not sending out with correct port.
Default SNMP port changed.
By default, the SNMP agent is disabled on flexvnf.
Community mismatch.
snmp-server unreachable
Using wrong mib to poll info. There are versa specific MIBs.
Target-source not properly configured.

Depending on the kind of issue the customer faces, troubleshooting steps vary. In general, the below troubleshooting steps can be followed.

Bird view troubleshooting steps:

Ensure snmp-server is reachable.
Ensure vnf-manager configuration has an SNMP exit interface and the server IP address. If this part is not configured, the SNMP daemon does nothing.
Ensure desired SNMP version is configured.
Ensure snmp-server IP address is configured properly for (v2c).
Be sure to check the authentication configuration.

Detail troubleshooting steps:

Ensure snmp agent is enabled

   set snmp agent enable

Ensure agent ip address is 127.0.0.1

   set snmp agent ip 127.0.0.1

Ensure snmp is listening to port 161. Snmp-server sends snmp poll with destination IP address 161.

   set snmp agent udp-port 161

Ensure desired SNMP version is running. V2c and v3 have commonly used SNMP versions. Depending on the version enabled, need to troubleshoot according.

   set snmp agent version v2c
   set snmp agent version v3

Ensure snmp community is configured. SNMP community is nothing but a password used between snmp-agent and snmp-manager.

    set snmp community versa sec-name versa123
    Note:
    versa    = community name
    versa123 = community password.

Be sure to configure the SNMP server IP address (v2c).
Ensure that target-source is configured, may not always be necessary.
Ensure that the SNMP access method is no-auth-no-priv. This is applicable to v2c. For v3, need to take care of the username and password.
Ensure views are enabled.
Check that server IP address is reachable and available in the routing table.
Ensure that vnf-manager is properly configured. This is important. You need to include the interface name in the vnf-interface list and server ip address in vnf-managent-ip.

For example:

    admin@cpe7-cli> show configuration system vnf-manager
    ip-addresses [ 10.100.1.1/32 10.100.1.10/32 210.1.1.2/32 ];

    <<<<<remplace 210.1.1.2/32 with 10.163.40.228/32 (server ip)
    vnf-mgmt-interfaces [ tvi-0/7.0 vni-0/1.0 ]; <<<< replace vni-0/1.0 
    with lan port (vni-0/4.0)

Use tcmpdump whenever necessary filtering with ports 161 and 162. For example: look for Getrequest and GetReply.

    admin@cpe7-cli> tcpdump vni-0/1 filter "port 161 or port 162"

    Starting capture on vni-0/1

    tcpdump: verbose output suppressed, use -v or -vv for full protocol

    decode
    listening on vs_trc_vni_0_1, link-type EN10MB (Ethernet), capture

    size 262144 bytes
    03:50:05.813321 52:54:00:a2:a6:58 > 52:54:00:65:3f:af, ethertype

    IPv4 (0x0800), length 78: 210.1.1.2.58884 > 210.1.1.1.161:

    GetRequest(21)  .1.3
    03:50:05.817306 52:54:00:65:3f:af > 52:54:00:a2:a6:58, ethertype

    IPv4 (0x0800), length 78: 210.1.1.1.161 > 210.1.1.2.58884:

    GetResponse(21)  .1.3=[noSuchObject

Use below debugging tools to collect developer-level messages. Run tail command before initiating SNMP poll/trap. It shows important information about what's going on; authentication failed, succeed, etc.

    set confdConfig logs developerLogLevel trace
    set confdConfig Logs snmpLogLevel info

    [admin@cpe7: confd] $ cd /var/log/versa/confd/
    [admin@cpe7: confd] $ sudo tail -f devel.log
    <INFO> 29-Nov-2018::06:50:31.294 cpe7 confd[5961]: devel-snmpa auth 
    init check failed: noGroupName

You should be able to conclude something by using the above steps. It's common to see configuration mistakes in customer setup.
Keep the working config ready for reference. You can use below:

snmpv3 and v2c working config example:

    set snmp agent enabled
    set snmp agent ip 127.0.0.1
    set snmp agent udp-port 161
    set snmp agent version v2c
    set snmp agent version v3
    set snmp agent engine-id
    set snmp agent engine-id enterprise-number 42359
    set snmp agent engine-id from-ip 210.1.1.2 <<< replace with 
    10.163.40.228 or server ip.
    set snmp agent max-message-size 2000
    set snmp system contact ""
    set snmp system name CPCPE00000142TGG-PRU-PH-01                  
    set snmp system location "7F UPTOWN PLACE TOWER 1, 1 EAST 11TH     
    DRIVE, UPTOWN BONIFACIO,TAGUIG CITY,METRO MANILA, PHILIPPINES 1634"
    set snmp community public sec-name public
    set snmp target v2_10.163.40.228 ip 210.1.1.2 <<< replace with     
    10.163.40.228 or server ip.
    set snmp target v2_10.163.40.228 udp-port 162
    set snmp target v2_10.163.40.228 tag [ std_v2_inform std_v2_trap ]
    set snmp target v2_10.163.40.228 timeout 1500
    set snmp target v2_10.163.40.228 retries 3
    set snmp target v2_10.163.40.228 v2c sec-name public
    set snmp notify std_v1_trap tag std_v1_trap
    set snmp notify std_v1_trap type trap
    set snmp notify std_v3_inform tag std_v3_inform
    set snmp notify std_v3_inform type inform
    set snmp notify std_v3_trap tag std_v3_trap
    set snmp notify std_v3_trap type trap
    set snmp target-source 210.1.1.1 <<< replace 210.1.1.1 with vni- 
    0/4.0 ip address. 
    set snmp vacm group All-Rights member netvendor sec-model [ usm ]
    set snmp vacm group All-Rights member public sec-model [ v2c ]
    set snmp vacm group All-Rights access any no-auth-no-priv
    set snmp vacm group All-Rights access v2c no-auth-no-priv read-view     
    internet
    set snmp vacm group All-Rights access v2c no-auth-no-priv write-     
    view internet
    set snmp vacm group All-Rights access v2c no-auth-no-priv notify- 
    view internet
    set snmp vacm group All-Rights access usm auth-priv read-view 
    internet
    set snmp vacm group All-Rights access usm auth-priv write-view 
    internet
    set snmp vacm group All-Rights access usm auth-priv notify-view 
    internet
    set snmp vacm view internet subtree 1.2 included
    set snmp vacm view internet subtree 1.3 included
    set snmp vacm view internet subtree 1.3.6.1 included
    set snmp usm local user netvendor auth
    set snmp usm local user netvendor auth sha
    set snmp usm local user netvendor auth sha password G5n5nmp3509
    set snmp usm local user netvendor priv
    set snmp usm local user netvendor priv aes
    set snmp usm local user netvendor priv aes password G5n5nmp3509

    admin@cpe7-cli> show configuration system vnf-manager

    ip-addresses [ 10.100.1.1/32 10.100.1.10/32 210.1.1.2/32 ]; <<<<<     
    remplace 210.1.1.2/32 with 10.163.40.228/32 (server ip)
    vnf-mgmt-interfaces [ tvi-0/7.0 vni-0/1.0 ]; <<<< replace vni-0/1.0 
    with lan port (vni-0/4.0)

These tools and commands will come in handy in times of need:

Show route routing-instance

    ping <snmp server-ip> routing-instance <>
    show configuration snmp | display set
    show configuration vnf-management
    tcpdump vni-0/<> filter “port 161 or port 162”

You can use free software SNMP server: reasoning mib browser(no v3 support) and ManageEngine MIBBrowser(v3 support).
Be familiar yourself with snmpwalk commands.

For example,

/opt/entuity/Entuity/lib/tools/snmpwalk -v 3 -E 8000a577010aa328e4 -n "" -u vuser -l authPriv -a sha -A versa123 -x AES -X versa123 -m ALL -M ./versa-mibs 10.172.18.78 .1.3.6.1.2.1.1

snmpengineID (-E) = 8000a577010aa328e4 <<< show confdCnofig snmpAgent snmpEngine snmpEngineID

Basic Config steps:

1. Configure snmp agent:

a. Enable snmp agent.

b. Configure agent ip (127.0.0.1)

c. Configure version (v2c/v3)

2. Configure snmp system:

a. Contact

b. Location

c. Name

3. Configure snmp trap:

4. Configure community name and sec-name (password):

5. Configure snmp target: a. Destination port

b. Server ip address.

c. Associate Community sec-name to the target.

6. Configure target source if required.

7. Configure vacm group name and associate v2c and usm(v3) to it.

8. Specify v2c and usm authentication method on vacm group for read/write/notify access.

9. Specify view type.

10. Specify usm username and password for v3 authentication.

SNMP Default Config:

snmp agent is disabled by default.
SNMP traps are enabled by default
OIDs are configured.

Cheatsheet:

Snmp enable or not
Agent ip address 127.0.0.1
Udp-port
Version
Community name and password
Manager/target details:

Server ip

Port number

Tag

Timeout

Retries

Associate community to the sender

Trap/notify
Snmp source ip
Associate community to vacm group
Associate access right to the vacm group

Read/write/nofity

Specify view detail in vacm

Configuring snmp agent:

Enable snmp agent

   set snmp agent enable

Configure snmp server(manager):

Configure the server ip address.

  set snmp target v2c_config ip 10.144.1.19

2. Configure IP address that will be used as the source address forSNMPp traffic

  set snmp target-source 210.1.1.1 <<< not required

Working snmp configuration when the snmp manager is located on the LAN side. I have only tested SNMP v2, not v3. For v3 below config will not work.

set snmp agent enabled
set snmp agent ip 127.0.0.1
set snmp agent udp-port 161
set snmp agent extra-listen ::1 161
set snmp agent version v2c
set snmp agent max-message-size 50000
set snmp system contact ""
set snmp system name cpe7
set snmp system location India
set snmp community versa123 sec-name versa123
set snmp target v2_210.1.1.2 ip 210.1.1.2
set snmp target v2_210.1.1.2 udp-port 162
set snmp target v2_210.1.1.2 tag [ std_v2_inform std_v2_trap ]
set snmp target v2_210.1.1.2 timeout 1500
set snmp target v2_210.1.1.2 retries 3
set snmp target v2_210.1.1.2 v2c sec-name versa123
set snmp notify std_v1_trap tag std_v1_trap
set snmp notify std_v1_trap type trap
set snmp notify std_v2_inform tag std_v2_inform
set snmp notify std_v2_inform type inform
set snmp notify std_v2_trap tag std_v2_trap
set snmp notify std_v2_trap type trap
set snmp notify std_v3_inform tag std_v3_inform
set snmp notify std_v3_inform type inform
set snmp notify std_v3_trap tag std_v3_trap
set snmp notify std_v3_trap type trap
set snmp target-source 210.1.1.1
set snmp vacm group access-v2c-public-internet member versa123 sec-model [ v2c ]
set snmp vacm group access-v2c-public-internet access v2c no-auth-no-priv read-view internet
set snmp vacm group access-v2c-public-internet access v2c no-auth-no-priv notify-view internet
set snmp vacm view internet subtree 1.2 included
set snmp vacm view internet subtree 1.3 included
set snmp vacm view internet subtree 1.3.6.1 included

Working snmp configuration when the snmp manager/server is located on the remote LAN side. I have tested only SNMP v2, not v3. So, snmp v3 will not work with the below configuration.

snmp polling works fine, with below config:
set snmp agent enabled
set snmp agent ip 127.0.0.1
set snmp agent udp-port 161
set snmp agent extra-listen ::1 161
set snmp agent version v2c
set snmp agent max-message-size 50000
set snmp system contact ""
set snmp system name cpe7
set snmp system location India
set snmp community versa123 sec-name versa123
set snmp target v2_200.1.1.2 ip 200.1.1.2
set snmp target v2_200.1.1.2 udp-port 162
set snmp target v2_200.1.1.2 tag [ std_v2_inform std_v2_trap ]
set snmp target v2_200.1.1.2 timeout 1500
set snmp target v2_200.1.1.2 retries 3
set snmp target v2_200.1.1.2 v2c sec-name versa123
set snmp notify std_v1_trap tag std_v1_trap
set snmp notify std_v1_trap type trap
set snmp notify std_v2_inform tag std_v2_inform
set snmp notify std_v2_inform type inform
set snmp notify std_v2_trap tag std_v2_trap
set snmp notify std_v2_trap type trap
set snmp notify std_v3_inform tag std_v3_inform
set snmp notify std_v3_inform type inform
set snmp notify std_v3_trap tag std_v3_trap
set snmp notify std_v3_trap type trap
set snmp vacm group access-v2c-public-internet member versa123 sec-model [ v2c ]
set snmp vacm group access-v2c-public-internet access v2c no-auth-no-priv read-view internet
set snmp vacm group access-v2c-public-internet access v2c no-auth-no-priv notify-view internet
set snmp vacm view internet subtree 1.2 included
set snmp vacm view internet subtree 1.3 included
set snmp vacm view internet subtree 1.3.6.1 included

Thanks for Reading!

NETWORKTCPIP