Nagios XI WMI agent-less monitoring for Azure Stack HUB Windows Virtual Machine, Step by Step.

It was a bit struggle to set up an WMI agentless monitoring for Windows VMs in Microsoft Azure Stack HUB (MASH) , so here is the memo I've been through.

 

-System environment

    Monitoring Server:  Linux VM in MASH (Ubuntu 18.01.1) 

    Monitoring Server Size: Standard DS2 v2 (2 vcpus, 7 GB memory)

    Moniroting Software: Nagios XI (5.6.9)

    

    Monitored Server: Windows VM in MASH (Windows 2019 DC)

    Monitored Server Size: Standard DS2 v2 (2 vcpus, 7 GB memory)

 

Notes: 

1. To simplify the network environment, They are in the same IP subnet and there is no dedicated FWs in between other than Netwrok Security Group (NSG) setting for each Virtual Machines which is provided by MASH, and Windows firewall on the Windows VM. (Later steps shows how to set them up.)

 

2. You shoud try the below reference documnet first, and see if Nagios XI's WMI agentless monitoring works in your envioroment.  If it doesn't work, try my steps.

Reference document: 

https://assets.nagios.com/downloads/nagiosxi/docs/Monitoring-Windows-Using-WMI-and-Nagios-XI.pdf

 

3. Maybe you can consider to set up an agent based monitoring, such as NCPA, if these WMI monitoring settings didn't work for you.  

 

- Nagios XI WMI agent-less monitoring set up -

-- On WIndows Virtual Machine:

1. Create an WMI service accout.  

     net user <service account name> <password> /add

     e.g. 

     net user wmiagent wmiaget /add

     # ! You shoud use more complex pw in yours, of course! 

 

 2. Adding Remote Activation Priviladge to WIndows DCOM.

    Start > Run > DCOMcnfg.exe

    

f:id:htr1036:20200108034928p:plain

Launch DCOM

     Navigate "Component Service > Computers and properties of My Computer".

     And Select "COM Security > Edit Item" under the "Launch and Activation Permissions".

 

     

f:id:htr1036:20200108035415p:plain

Edit COM Security

    Add the created user "wmiagent".

    

f:id:htr1036:20200108035640p:plain

Add wmiuser

    Select these permissions below for the user "wmiagent".

     

f:id:htr1036:20200108035858p:plain

Add permissions

    And click "OK" to save.

 

 

 3. Adding Remote WMI Access.

    Start > RUN > WMImgmt.msc

     

f:id:htr1036:20200108040252p:plain

WMImgmt.msc

      Right click on "WMI Control (Local)" and select "Properties".

      And Select Security Tab > CIMV2 and click "Security".

    

f:id:htr1036:20200108040555p:plain

CIMV2 Security

    Click "Add...." button and add "wmiagent". Then select the follwoing permissions below for wmi user.

    

f:id:htr1036:20200108040812p:plain

Add wmi sec permissions

    Click OK to save.

 

  4. Windows Firewall settings

     Server Manager > Tools > Windows Firewall with Advanced Security

     Select "Inbound Rules" > "Filter by Group" and select " Filter by Windows Management Instrumentation (WMI)".

    

f:id:htr1036:20200108041226p:plain

Windows Firewall with Advanced Security

    Make sure these two inbound rules are enabled like below:

 

    

f:id:htr1036:20200108041408p:plain

FW Inbound rules for WMI

 

 

 

 5. Set WMI user to the "Performance Log Users" Group

     Server Manager > Tools > Computer Management

     Add the user "wmiagent" to the "Performance Log Users Group" like below:

    

f:id:htr1036:20200108041843p:plain

Performance Log Users Group

    Click OK to save.

 

 6. Set Additional Permissions

     On the Windows machine, Check SID for the WMI user account (wmiagent) by running the following command;

        wmic useraccount where name='wmiagent' get sid

     The output will be something like this:

        SID
        S-1-5-21-xxxxxxxx-xxxxxxxxx-xxxxxxxxx-xxxx


    Keep this SID somewhere, and check the current security descriptor (SD) for SCMANAGER.

        sc sdshow SCMANAGER

    The output will be like this:

        D:(A;;CC;;;AU)(A;;CCLCRPRC;;;IU)(A;;CCLCRPRC;;;SU)(A;;CCLCRPWPRC;;;SY)(A;;KA;;;BA)(A;;CC;;;AC)(A;;CC;;;S-1-15-3-xxxx-528118966-xxxxxxxxxx-xxxxxxxxx-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxx-xxxxxxxxxx)S:(AU;FA;KA;;;WD)(AU;OIIOFA;GA;;;WD)

    You need to add an entry to the SD that contains your SID,

         (A;;CCLCRPRC;;;<SID>)


    For example using the SID above:

         (A;;CCLCRPRC;;; S-1-5-21-xxxxxxxx-xxxxxxxxx-xxxxxxxxx-xxxx)


    This needs to be inserted to the beginning of the SD after the D:, using the example above it looks like:

         sc sdset SCMANAGER D:(A;;CCLCRPRC;;;S-1-5-21-xxxxxxxx-xxxxxxxxx-xxxxxxxxx-xxxx)(A;;CC;;;AU)(A;;CCLCRPRC;;;IU)(A;;CCLCRPRC;;;SU)(A;;CCLCRPWPRC;;;SY)(A;;KA;;;BA)(A;;CC;;;AC)(A;;CC;;;S-1-15-3-xxxx-528118966-xxxxxxxxxx-xxxxxxxxx-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxx-xxxxxxxxxx)S:(AU;FA;KA;;;WD)(AU;OIIOFA;GA;;;WD)

 

    Run 'sc sdshow SCMANAGER' again to check the new SD has been set.

 

 

 7. Set Fixed WMI Port (135 and 24158)  

    Since Azure Stack HUB's NSG allows connection through its port specification, WMI which uses dynamic ports by default have to be changed to the fixed port setting.

    On the Windows machine, run the following command (from admin CMD);

         winmgmt -standalonehost

    Restart WMI service:

          net stop winmgmt

          net start winmgmt


    Set a firewall rule for the fixed WMI port:

         netsh firewall add portopening TCP 24158 WMIFixedPort


    Check if Fixed port is set on the windows machine (Optional)
    On the Windows DCOM console, navigate to " Computers > My Computer > DCOM Config > Windows Management & Instrumentation"

    And check its properties:

    

f:id:htr1036:20200108043635p:plain

Check WMI port setting

     You should see the "Use static endpoint" is selected and shows "24158" as a default port number.

    Note:  You will have to set up a new NSG rule for the fixed WMI ports (135 and 24158 both) for both Virtual Machines in Azure Stack HUB.

  

Ref: https://support.quest.com/kb/114559/how-to-set-a-static-fixed-wmi-port-in-windows

       https://docs.microsoft.com/en-us/windows/win32/wmisdk/setting-up-a-fixed-port-for-wmi

       https://social.msdn.microsoft.com/Forums/en-US/0a7f66b4-aef5-4697-8d1f-397bb554d15a/remote-wmi-to-azure-vm-the-object-exporter-specified-was-not-found

 

 

 -- On Nagios XI server: 

 1.  Set up hosts for WMI monitored servers.

         sudo vi /etc/hosts

    Add host's entries of monitoring target hosts using below format:
         .
         .
         # WMI monitoring
         <IP Address> <FQDN> <hostname>

    Write and quit (wq!) from the vi.


    Check if wmic command can now reached to wmi monitoring hosts or not:

    CPU check using FQDN:


        $ /usr/local/nagios/libexec/check_wmi_plus.pl -H <target's FQDN> -u 'wmiagent' -p '<PW>' -m checkcpu -w '60' -c '70'
OK (Sample Period 74 sec) - Average CPU Utilisation 0.30%|'Avg CPU Utilisation'=0.30%;60;70;


    CPU check using IP address:


         $ /usr/local/nagios/libexec/check_wmi_plus.pl -H <target host's IP> -u 'wmiagent' -p '<PW>' -m checkcpu -w '60' -c '70'
OK (Sample Period 1 sec) - Average CPU Utilisation 2.15%|'Avg CPU Utilisation'=2.15%;60;70;

 

 2. Set up Authentication file for WMI auth.  

   Create an authentication file (as a hidden file):

         sudo vi /usr/local/nagios/etc/.wmi_auth


         username=wmiagent
         password=<PW>

 

     Write and Quit (wq!) from the vi. 

     Check auth file's current permission. (It's 644 and belong to root group.)


         $ sudo ls -ltr /usr/local/nagios/etc/.wmi_auth
         -rw-r--r-- 1 root root 38 Jan 3 14:47 /usr/local/nagios/etc/.wmi_auth

 

     Change its mode from 644 to 640 and set it to "nagios" group so that only users belong to nagios group can read:


         $ sudo chmod 640 /usr/local/nagios/etc/.wmi_auth

         $ sudo chgrp nagios /usr/local/nagios/etc/.wmi_auth

 

    Check if both have been changed:


         $ ls -ltr /usr/local/nagios/etc/.wmi_auth
         -rw-r----- 1 root nagios 38 Jan 3 14:47 /usr/local/nagios/etc/.wmi_auth

 

 

-- On Nagios XI Web GUI:

 Set up WMI monitoring using Auth file.

     Navigate to "Configure > Configuration Wizards".

     

f:id:htr1036:20200108045936p:plain

Naigos XI Config Wizard

    ...and select Windows WMI wizard:

    

f:id:htr1036:20200108050039p:plain

WMI Wizard

     Enter FQDN or IP address of your target host and specify the Auth file you've created.

     

f:id:htr1036:20200108050227p:plain

Set WMI monitor with Auth file

    Also increase the Trancate output Length to 65536.

    

   Set up discovered monitors as required.

f:id:htr1036:20200111082112p:plain


    

    Note: MASH doesn't allow ICMPs'  traffic as of now (2020 Jan) so the ICMP ping monitor won't work (always shows as an unreachable alert). So here, it shoud be checked off like above. To monitor node's up/down, maybe the other ping monitors, such as telnet, URL check or other port monitoring check would be considered.  

 

    Change intervals and notifications  as required.

      

f:id:htr1036:20200108051037p:plain

Monitoring intervals

    

f:id:htr1036:20200108051155p:plain

Notification settings

    Click Finish to save. 

 

    Check if now Naigos XI is polling setup monitors:

   

f:id:htr1036:20200108051449p:plain

WMI monitoring Service Status

 

 

-- Trouble shooting:

    

f:id:htr1036:20200108051752p:plain

"UNKNOWN - Permission denied when trying to store the state data. "

If you see this kind of message on the Nagios Xi Web Console, run below:

 

    Check if a state file exists under the /tmp on Nagios xi server:

 

          $ sudo ls -ltr   /tmp/*.state

          -rw-rw-r-- 1 nagiosxi01admin nagiosxi01admin 202 Jan  3 15:26 /tmp/cwpss_checkcpu__xxxxxxxxxx___.state

 

 

    Remove all state files: 

 

         $ sudo rm -f  /tmp/*.state

  

    Check it again, and should see this message:

 

          $ sudo ls -ltr   /tmp/*.state

          ls: cannot access '/tmp/*.state': No such file or directory

 

     Check if there is no more "UNKNOWN - Permission denied when trying to store the state data." message on the Nagios XI.

 

--End