Cisco Amp For Endpoints Linux Connector ClamAV Definition Update Troubleshooting

I recently deployed Cisco Amp For Endpoints to a few CentOS 6 and 7 servers. After installation, I noticed that the dashboard showed no ClamAV updates. I let each one sit overnight, but returned to the same status.

To solve “Definitions Last Updated: None,” assuming there is no connectivity issue to the ClamAV servers, restart the connector services as shown below. However, be sure to verify ping, DNS resolution, and port 80/443 tests to the server listed in the dashboard.

Utilize logs in /var/log/cisco and tools in /opt/cisco/amp/bin to help troubleshoot and resolve the issue.

Step 1

Identify if there’s an issue. Go to the dashboard and find the servers. You’ll probably see an orange alert that definitions haven’t been updated.

20181101-ClamFailure

The AMP For Endpoints dashboard shows that ClamAV definitions have never been updated.

Step 2

There are several log files in /var/log/cisco that can provide additional service information.

CentOS 7

[root@ops01~]# systemctl list-unit-files | grep amp
cisco-amp.service                      
      enabled
cisco-ampmon.service                         
      disabled
cisco-ampupdater.service                     
      disabled

[root@ops01~]# systemctl restart cisco-ampupdater.service
[root@ops01~]# systemctl restart cisco-amp.service
[root@ops01~]# cd /opt/cisco/amp/bin
[root@ops01 bin]# ls
ampcli ampmon ampservice ampsupport cisco-amp-helper libclammspack.so.0 libclamunrar_iface.so libclamunrar_iface.so.7.1.1 libclamunrar.so.7 libmnl.so libmnl.so.0.2.0
ampdaemon ampscansvc ampsigncheck ampupdater libclammspack.so libclammspack.so.0.1.0 libclamunrar_iface.so.7 libclamunrar.so libclamunrar.so.7.1.1 libmnl.so.0 modules

[root@ops01 bin]# ./ampupdater
[root@ops01 bin]# 

CentOS 6

[root@ops01]# less ampdaemon.log

Oct 22 15:51:41 NETFLOW ampdaemon[4427] [ui]:[notice]-[ui_comms.c@1476]:[139769633826560]: UI connection on socket 29 allows privileged commands
Oct 22 16:00:30 NETFLOW ampdaemon[4427] [air_libs]:[error]-[fa_cloud_async.c@670]:[139769602356992]: _fa_async_handle_response(): received stale or superfluous response: sessid=0 txid=55

[root@ops01]# less ampupdater.log

Oct 22 15:51:31 NETFLOW [4619] [config]:[info]-[config.c@944]:[140238088169440]: xpath /config/updater/lastmod does not exist or is empty
Oct 22 15:51:31 NETFLOW [4619] [config]:[info]-[config.c@944]:[140238088169440]: xpath /config/updater/server does not exist or is empty
Oct 22 15:51:31 NETFLOW [4619] [updater]:[info]-[updater.c@130]:[140238088169440]: no update server present

[root@NETFLOW bin]# cd /etc/init/
[root@NETFLOW init]# ls | grep amp
cisco-amp.conf
cisco-ampmon.conf
cisco-ampupdater.conf

[root@NETFLOW init.d]# initctl stop cisco-amp
cisco-amp stop/waiting

[root@NETFLOW init.d]# initctl start cisco-amp
cisco-amp start/running, process 25542

[root@NETFLOW init.d]# initctl status cisco-amp
cisco-amp start/running, process 25542

[root@NETFLOW init]# initctl status cisco-ampupdater
cisco-ampupdater stop/waiting

[root@NETFLOW init]# initctl start cisco-ampupdater
cisco-ampupdater stop/waiting

Step 3

Wait 5-10 minutes, then check/force sync policies.

[root@ops01 bin]# ./ampcli
Trying to connect...
Connected.
ampcli> help
    scan         Initiate/pause/stop a scan
                  * See 'scan help' for more.
    status       Get ampdaemon status
    sync         Sync policy
    policy       Show policy
    exclusions   List custom exclusions
    history      Show event history
                  * See 'history help' for more.
    quarantine   List/restore quarantined file(s)
                  * See 'quarantine help' for more.
    about        About AMP for Endpoints Connector
    notify       Toggle notifications
    verbose      Toggle verbose mode
    q            Quit ampcli interactive mode

ampcli> history list
Date                   Event               Details
=================================================================================
| 2018-10-05 10:37 AM | Definition Update | ID:128 known viruses:6670463
|                     |                   | version:25009 cloud:0 local:4
|                     |                   | official:4 type:CVD (Update succeeded)=================================================================================
[ Page 1 of 1 (total results: 5) ]

ampcli> sync
Request sent to sync policy to latest version

ampcli> policy
Quarantine Behavior:
  Do not quarantine malicious files.
Protection:
  Monitor program install.
  Monitor program start.
  Passive on-execute mode.
Proxy: NONE
Notifications: Do not display cloud notifications.
Policy: Audit Policy for Linux (#149)
Last Updated: 2018-10-05 11:06 AM

Step 4

Wait a few more minutes, then check the AMP dashboard.

20181101-ClamSuccess

The AMP For Endpoints dashboard shows that ClamAV definitions are now successful.

Step 5

If needed, check the logs in /var/log/cisco.

If you see errors such as:

Oct 5 10:35:40 ops01....vls ampdaemon[16018] [cvd_update]:[error]-[hb_clam.c@2064]:[139886856197888]: Unable to access file so using default time of 0 for /opt/cisco/amp/etc/clamav/main.cvd: No such file or directory

These should have been resolved by the above steps.

If you’re still having issues, open a support case and generate troubleshooting info.

[root@ops01 bin]# ./ampsupport
  [logger] Set minimum reported log level to notice
  date
  uname -a
  hostname
  …
  …
  semodule -l
  total 3768359, actual 3768359
  Saved the support package to: /root/Desktop/AMP_Support_2018_10_05_15_15_07.zip
  [logger] Shutdown file logger for module:

[root@ops01 bin]#

Conclusion

There might be a more succinct way to troubleshoot and resolve this issue, but this is the process that worked reliably for me. I haven’t exactly pinned down why a simple service restart is necessary on first install, but it’s an easy fix.

 

ASA 982-28 Undocumented Change of Functionality?

We have recently begun upgrading ASA 5500X models from version 9.8.2-20-smp to 9.8.2-28-smp. This is a very minor interim update. However, I caught the following startup error from console while watching the ASA boot.

Reading from flash...
!!
no mac-address auto
               ^
ERROR: % Invalid input detected at '^' marker.
*** Output from config line 14, "no mac-address auto"

I opened a TAC case to request more information on the possible impact of the removal of this command. Our ASAs are running in single-context mode, so from reading the documentation I gathered that we should see no loss of functionality. This was confirmed by the case engineer. https://www.cisco.com/c/en/us/td/docs/security/asa/asa82/command/reference/cmd_ref/m.html#wp2043127

mac-address auto: Auto-generates MAC addresses (active and standby) for shared interfaces in multiple context mode.

The engineer was not able to provide details on why this was removed, and was also unable to locate documentation supporting the removal. The interim release notes can be found here: https://www.cisco.com/web/software/280775065/139997/ASA-982-Interim-Release-Notes.html

In summary, ASAs running in single-context mode should be unaffected. If you are running in multiple-context mode, I would suggest opening a case or carefully reviewing above documentation to assess the impact this could have on your firewall configuration.

Quick Note on Firepower Security Intelligence Feeds – HTTP/S Only

I spent some time today attempting to get a Firepower Security Intelligence feed to update from a network file share. I thought it would be an easy task since it IS possible to upload a Security Intelligence list from a network share. However, this is not the case for a feed, as I later confirmed in a TAC case with Cisco.

This environment is on Firepower Services version 6.1.0.1 running on ASA 5525s managed by a virtual FMC.

I tried three options:

\\192.168.1.99\Firepower\Blacklist\Blacklist_Feed.txt 
//192.168.1.99/Firepower/Blacklist/Blacklist_Feed.txt 
smb://192.168.1.99/Firepower/Blacklist/Blacklist_Feed.txt

I had hope at first, primarily due to the fact that our backup profile is directed toward a share. The destination URL is “smb://192.168.1.99/Firepower/Backups/” so I know the permissions in that directory would have allowed Firepower to access the file.

The first attempt generated the below logs. The whole string, beginning with the backslashes, was being converted to a hostname and a DNS lookup was performed. As expected, no record was found.

root@firepower:/var/log# grep -Ri "Blacklist_Feed" * 
process_stderr.log:2018-05-29 18:17:22 CloudAgent[4263]: * Rebuilt URL to: \\192.168.1.99\Firepower\Blacklist\Blacklist_Feed.txt/ 
process_stderr.log:2018-05-29 18:17:22 CloudAgent[4263]: * getaddrinfo(3) failed for \\192.168.1.99\Firepower\Blacklist\Blacklist_Feed.txt:80 
process_stderr.log:2018-05-29 18:17:22 CloudAgent[4263]: * Couldn't resolve host '\\192.168.1.99\Firepower\Blacklist\Blacklist_Feed.txt'

Later attempts did show connections being established to the file server, even on the proper port. Unfortunately, Firepower was evidently not able to go any further than this.

root@firepower:/var/log# tail -f process_stderr.log
2018-05-29 18:47:37 CloudAgent[4263]: * Trying 192.168.1.191... 
2018-05-29 18:47:37 CloudAgent[4263]: * Connected to 192.168.1.99 (192.168.1.99) port 445 (#0) 
2018-05-29 18:47:37 CloudAgent[4263]: * Closing connection 0

Although it was a disappointing end to the day, I will put in a feature request with Cisco to allow Security Intelligence feeds via SMB.

Update 6/10/18

It doesn’t appear that feeds via SMB will happen any time soon. I reached out to our Cisco technical contact, as well as the Cisco Security team on Twitter. Neither indicated that this would be a feature they’re interested in pursuing.

20180610-CiscoTweet

Cisco declines to pursue feed updates via SMB.

 

Firepower Threat Defense Installation Troubleshooting

Like it or not, Cisco’s vision is to facilitate device configuration primarily through graphical user interfaces. Making advanced features more easily configurable will be a blessing to some, but challenging to many.

As we’re seeing in the new Firepower Threat Defense line of code, a unified ASA and Firepower Services image, command-line access is restricted to troubleshooting only with no traditional CLI configuration options available. This post documents issues I encountered while setting up an ASA 5515-X, migrating from ASA 9.1.7 to FTD 6.2.0.

Summary

This ASA was for code demoing, thus I had confreg 0x41 set to bypass initial configs. This caused issues accessing the FTD web management interface. The solution was to reset the confreg to 0x1 and reissue a command to enable the web service.

  1. Boot to rommon
  2. Set confreg to 0x1 and boot
  3. > configure network management-interface enable management0

Before Updating Confreg

#Here is the network setup. Everything looks fine. 
> show network
===============[ System Information ]===============
Hostname : firepower
Management port : 8305
IPv4 Default route
Gateway : 192.168.45.1
======================[ br1 ]=======================
State : Enabled
Channels : Management & Events
Mode : Non-Autonegotiation
MDI/MDIX : Auto/MDIX
MTU : 1500
MAC Address : FC:5B:39:A9:6E:74
----------------------[ IPv4 ]----------------------
Configuration : Manual
Address : 192.168.45.10
Netmask : 255.255.255.0
Broadcast : 192.168.45.255
----------------------[ IPv6 ]----------------------
Configuration : Disabled
===============[ Proxy Information ]================
State : Disabled
#Notice that the Management 0/0 interface is administratively down.
> show running-config
interface GigabitEthernet0/0
 nameif outside
 cts manual
  propagate sgt preserve-untag
  policy static sgt disabled trusted
 security-level 0
 ip address dhcp setroute
!
interface GigabitEthernet0/1
 nameif inside
 cts manual
  propagate sgt preserve-untag
  policy static sgt disabled trusted
 security-level 0
 ip address 192.168.1.1 255.255.255.0
!
interface Management0/0
 management-only
 shutdown
 nameif diagnostic
  cts manual
  propagate sgt preserve-untag
 policy static sgt disabled trusted
 security-level 0
 no ip address

What made this issue tough to troubleshoot was the lack of indications of error. If I had not stumbled upon a small post made on a support forum, I would have been spinning my wheels for much longer. Once I identified that the configuration-register may be the culprit, I was able to quickly resolve the problems.

Troubleshooting and Resolution

> reboot
This command will reboot the system. Continue?
Please enter 'YES' or 'NO': YES
[…]
Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.
Boot interrupted.
[rommon starts here]
Management0/0
Link is UP
MAC Address: fc5b.39a9.6e75
Use ? for help.
#Notice that my confreg is set to 0x41. Let's change that to 0x1.
rommon #0> confreg
Current Configuration Register: 0x00000041
Configuration Summary:
 boot default image from Flash
 ignore system configuration
Do you wish to change this configuration? y/n [n]: n
rommon #1> confreg ?
HEX argument must have a "0x" prefix!
rommon #1> confreg 0x1
Update Config Register (0x1) in NVRAM...
rommon #2> confreg
Current Configuration Register: 0x00000001
Configuration Summary:
boot default image from Flash
Do you wish to change this configuration? y/n [n]: n
rommon #3> boot

At this point, the ASA with FTD boots. I still had a few troubleshooting steps to complete before confirming that the web management interface was working.

After changing confreg but before rebooting again:

> show managers

Managed locally.
#Notice that the Management interface is no longer shutdown. 
> show running-config
interface GigabitEthernet0/0
 nameif outside
 cts manual
  propagate sgt preserve-untag
  policy static sgt disabled trusted
 security-level 0
 ip address dhcp setroute
!
interface GigabitEthernet0/1
 nameif inside
 cts manual
  propagate sgt preserve-untag
  policy static sgt disabled trusted
 security-level 0
 ip address 192.168.1.1 255.255.255.0
!
interface Management0/0
 management-only
 nameif diagnostic
 cts manual
  propagate sgt preserve-untag
  policy static sgt disabled trusted
 security-level 0
 no ip address

Thinking the issue was resolved, I visited https://192.168.45.10:

Service Unavailable. 
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. 
Please try again later.

I wasn’t quite sure what to make of this error so I decided to find the web server service and reboot it. I’m not sure if this is the proper method, but it didn’t seem to hurt anything.

> expert

admin@firepower:~$ sudo su
Password:

root@firepower:/home/admin# ps aux | grep http
root 10104 0.0 0.0 40848 5884 ? S 15:20 0:00 /ngfw/usr/bin/httpsd -D FOREGROUND
www 10107 10.3 7.4 2415436 609796 ? Sl 15:20 1:25 /ngfw/var…start
www 10223 0.0 0.0 1771748 6908 ? Sl 15:20 0:00 /ngfw/usr/bin/httpsd -D FOREGROUND
www 18499 0.0 0.0 1116244 6208 ? Sl 15:27 0:00 /ngfw/usr/bin/httpsd -D FOREGROUND
root 18872 0.0 0.0 4424 648 ttyS0 S+ 15:34 0:00 grep http

root@firepower:/home/admin# pmtool restartbyid httpsd

I visited https://192.168.45.10 again:

"The Firepower Device Manager application cannot be opened."

Clearly that didn’t work as intended so I rebooted the FTD device again. When the system was back up, I tried visiting the web interface.

Service Unavailable.
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. 
Please try again later.

When turning it off and on again doesn’t work…. turn more stuff off and on.

> configure manager delete
> configure manager local
> show running-config
#Interface config looks the same as the above output.

The next command is one that I felt shouldn’t have needed to be issued. When visiting the web interface, I was seeing something, so there was clearly a service answering the phone. It may be the case that in one of the above steps, the management interface was switched to something other than Management0.

Interestingly, we can see below that this command does not utilize the name of the physical ASA interface which is Management0/0. Instead, it uses what might be a logical management interface.

> configure network management-interface enable management0/0

Invalid interface supplied.

> configure network management-interface enable management0

I visited https://192.168.45.10 again and the FTD manager now works as intended.


Final Thoughts

This issue was quite frustrating due to the fact that I could have easily fixed it (temporarily) if traditional ASA CLI configuration was available. Abstracting away tasks through menus and buttons reduces the user’s flexibility and control over their own systems.

For better or worse, this is the direction of the future. A system designed on rails will lower the barrier of entry for lesser-skilled people to use, but can be much more difficult to get back on track when something breaks. Fortunately, Firepower Threat Defense brings forth convenience and features that are welcomed additions to the all-too-commonly over-worked IT administrator.

Cisco Firepower Blocking Mozilla Firefox Cloudfront DNS Queries – Or How I Found NORMANDY

This is a tale of how chasing curiosity can expose the undercover intricacies of everyday technology. What started with a perplexing occurrence led to the exploration of a system architecture that silently interacts with one of the world’s most popular internet browsers.


I’ve deployed Cisco Firepower at several client sites. In every instance, I’ve seen many Security Intelligence events in the DNS Malware, DNS DGA, and DNS CnC categories. Most of these are lookups to Cloudfront.net, Amazon’s Content Delivery Network. Unable to pinpoint the source of these requests, I’ve had no choice but to block the traffic and brace myself for the scream test.

Fortunately, the screams never came. From my perspective, all systems and users were 200 OK. However, it’s been causing me to wonder at night; “what’s broke?”

One day when spinning up a new Firepower deployment, I noticed those familiar DNS requests and traced the source IP back to a coworker who happened to be sitting next to me. I asked him what sites he had been to and if he’d be willing to perform additional testing with me. He agreed, then proceeded to open his browser and ask me what sites I wanted him to connect to. Before going any further, I saw several new events.

These requests typically follow the same hostname format, .cloudfront[dot]net. In this case, I saw:

d6wjo2hisqfy2.cloudfront[dot]net

But this time I noticed something new.

normandy-cloudfront.cdn.mozilla[dot]net

Finally, I’ve caught the culprit red-handed: Mozilla Firefox. The pieces are starting to come together, but there are still questions.

NORMANDY

The first Cloudfront link that I posted resolves to a site that states in large letters “NORMANDY.” Without the second link, this did not make sense to me. Now, I’ve begun the hunt to determine its purpose.

I soon came across the Normandy Github page. It appears to be a server architecture for a whole ecosystem of content delivery and metric gathering. Per the description: “Normandy is a collection of projects that provide a fast and accurate way to interact with unbiased subsets of Firefox users.” These projects are (from GitHub):

  • recipe-server – Django server that manages and delivers recipes efficiently and securely
  • recipe-client-addon – Firefox system add-on to fetch, verify, and execute recipes
  • lints – Shared tools and configs for linting
  • compose – Scripts and configurations to create multi-server environments for development and testing

Side note, “linting” is the process of automating bug-finding by searching for code that is likely to be non-portable or otherwise problematic.

The ReadtheDocs documentation site offers a more complete description: “Normandy is a component in a larger system called SHIELD. The end goal of SHIELD is to allow Firefox to perform a variety of actions to aid the user and gather feedback […]. To do this, we need a way to instruct Firefox to perform certain actions without having to download an update or wait for a new release. SHIELD is split into three main concepts: Actions, Recipes, and the runtime.”

So far, we’ve established that Normandy is a sever architecture that leverages other projects to facilitate interaction with Firefox users. Once contacted, components of SHIELD deliver recipes to solve problems or gather data.

Now I am intrigued on three fronts: What is the scope of this larger system, SHIELD; specifically, what are the tasks of recipes; and how does Firefox determine when to contact Normandy?

SHIELD

Up to this point, the actual function of aforementioned “runtime,” “actions,” and “recipes” have been quite vague. Let’s start by defining what is meant by “runtime.” Per ReadtheDocs, it appears to be a small sandbox environment that can affect a running instance of Firefox. Shortly after Firefox boots, the runtime contacts Normandy, identifies recipes that match the running Firefox environment, and downloads actions that implement the recipes.

The previous sentence alludes to the difference between an “action” and a “recipe.” Simply put, an action is a piece of JavaScript that provides the framework for interaction with a Firefox component. Actions themselves have no direct access to the browser, except through a driver object. A driver object explicitly grants permissions for an action to interact with a feature outside the runtime. The action can then execute, for example to determine whether or not the user should be prompted for a survey.

After the requisite conditions have been established by the action, the action then invokes a recipe that contains the data, parameters, or configuration that needs to be delivered. Interestingly, the ReadtheDocs page contains an example that displays an “implementation_url” variable that closely resembles the suspicious DNS query from the beginning of this post. This URL is:

https://normandy.cdn.mozilla[dot]net/api/v1/action/console-log/implementation/8ee8e7621fc08574f854972ee77be2a5280fb546/

You will notice that the implementation (AKA recipe) is being applied to the console-log action.

Additional information on SHIELD can be found at the Mozilla wiki.

My investigation is proving to be very successful. I now know the nature of the requests coming through Firepower, I understand the components at play, and I’ve learned many interesting capabilities of Firefox that I’m certain will translate to other programs I encounter.

Control

Knowing your environment is valuable, but controlling it is critical. As mentioned earlier, SHIELD is intended to automatically gather telemetry data from users. Many people strive to maintain absolute control over the information that they produce, whether it be in the form of emails, text messages, or metadata.

Fortunately, Mozilla provides users with this guide that outlines how to disable the various types of automatic connections that Firefox makes. It took some hunting, but Mozilla also has a Wiki page that provides detailed documentation on the collected Telemetry data. You can view the data that your browser has collected by inserting

about:telemetry

into your Firefox address bar.

Additionally, if you have read the Firefox Browser Privacy Notice (yeah, right) you would have come across the Firefox Health reports. These reports contain statistics similar to what you’d see in about:telemetry, but in a more user friendly form. Even if you don’t intend to disable Health Reports or Telemetry, it’s still worth checking these out. To view your Health Report, click the Firefox Menu button (the three horizontal lines), then the question mark in the lower right corner, the Firefox Health Report.

Cause and Effect

As with many aspects of life, the choices we make when defending our network can cause unintended results. In this case, the implementation of Cisco Firepower (which, by the way, has been awesome so far) into my networks also brought the enforcement of what Cisco Talos calls “security intelligence categories.”

The categories in particular that are blocking the legitimate traffic described in this post are:

  • Command and Control (CnC)- Sites known to host botnet control traffic.
  • Malware- Sites known to distribute malware.
  • Domain Generation Algorithm (DGA)- Automatically generated domain names that often host malware such as Cerber.

Additional details can be found in Cisco’s documentation.

When we consider the URL that is being blocked, it makes sense why Firepower may have mistakenly categorized the Firefox connection as malicious.

d6wjo2hisqfy2.cloudfront[dot]net

Cloudfront is a well-known server hosting platform. Still, this does not stop some criminals from purchasing servers, often with stolen credentials or financial information, from spinning up new servers to distribute malware. Similarly, the criminals may also use hacked Cloudfront servers to issue commands to swarms of botnets.

Even more suspicious is the hostname of this server, “d6wjo2hisqfy2.” It very closely resembles a domain that could be generated by Domain Generation Algorithm. Cisco Umbrella has a good breakdown on the effectiveness of this technique. However, the blocking of this site by Firepower is strange behavior because the hostname is a subdomain of Cloudfront, not its own registered domain. Therefore, although I can’t confirm this, my guess is that Firepower is flagging this traffic based on the nonsensical hostname alone.

Strangely, not all requests to Cloudfront are being consistently blocked. Specifically, I noticed that requests to

d2pj9rkatqbt38.cloudfront[dot]net

were not being blocked. In this particular case, the URL above redirects to a the website of a compliance and risk management company.

Remediation

And with that, my journey down the rabbit hole draws near to conclusion. The final task remaining is to decide how to handle the blocked traffic.

Due to the fact that Cloudfront may still host compromised content, I cannot add a domain-wide whitelist. Additionally, I also cannot whitelist the specific URL that we’ve dissected above, as it is not always consistent. Thus, I will be leaving Firepower to continue blocking connections to Normandy.

Cisco is among the world’s largest networking vendors, and Mozilla Firefox is among the world’s most popular browsers. I felt that I should do due diligence and report my findings to Mozilla via their bug tracker. At the time of posting, I did not fully understand the nature of the requests that were being blocked. Still, Mozilla contributors showed a disappointing lack of interest in discussing these issues.

My submission on Mozilla’s bug tracker can be found here. You will notice some confusion regarding my intent in submitting the issue, so this may be worth revisiting at a later date.


Final Thoughts

Throughout my research, every new page seemed to contain information on topics that I had absolutely zero exposure to. Simply following curiosity once again proved to be a great method of expanding my limit of understanding. Next time you come across something that piques your interest, keep in mind that exploration is often worthwhile.

As always, thanks for reading.

-Dan