Firepower Threat Defense Installation Troubleshooting

Like it or not, Cisco’s vision is to facilitate device configuration primarily through graphical user interfaces. Making advanced features more easily configurable will be a blessing to some, but challenging to many.

As we’re seeing in the new Firepower Threat Defense line of code, a unified ASA and Firepower Services image, command-line access is restricted to troubleshooting only with no traditional CLI configuration options available. This post documents issues I encountered while setting up an ASA 5515-X, migrating from ASA 9.1.7 to FTD 6.2.0.

Summary

This ASA was for code demoing, thus I had confreg 0x41 set to bypass initial configs. This caused issues accessing the FTD web management interface. The solution was to reset the confreg to 0x1 and reissue a command to enable the web service.

  1. Boot to rommon
  2. Set confreg to 0x1 and boot
  3. > configure network management-interface enable management0

Before Updating Confreg

#Here is the network setup. Everything looks fine. 
> show network
===============[ System Information ]===============
Hostname : firepower
Management port : 8305
IPv4 Default route
Gateway : 192.168.45.1
======================[ br1 ]=======================
State : Enabled
Channels : Management & Events
Mode : Non-Autonegotiation
MDI/MDIX : Auto/MDIX
MTU : 1500
MAC Address : FC:5B:39:A9:6E:74
----------------------[ IPv4 ]----------------------
Configuration : Manual
Address : 192.168.45.10
Netmask : 255.255.255.0
Broadcast : 192.168.45.255
----------------------[ IPv6 ]----------------------
Configuration : Disabled
===============[ Proxy Information ]================
State : Disabled
#Notice that the Management 0/0 interface is administratively down.
> show running-config
interface GigabitEthernet0/0
 nameif outside
 cts manual
  propagate sgt preserve-untag
  policy static sgt disabled trusted
 security-level 0
 ip address dhcp setroute
!
interface GigabitEthernet0/1
 nameif inside
 cts manual
  propagate sgt preserve-untag
  policy static sgt disabled trusted
 security-level 0
 ip address 192.168.1.1 255.255.255.0
!
interface Management0/0
 management-only
 shutdown
 nameif diagnostic
  cts manual
  propagate sgt preserve-untag
 policy static sgt disabled trusted
 security-level 0
 no ip address

What made this issue tough to troubleshoot was the lack of indications of error. If I had not stumbled upon a small post made on a support forum, I would have been spinning my wheels for much longer. Once I identified that the configuration-register may be the culprit, I was able to quickly resolve the problems.

Troubleshooting and Resolution

> reboot
This command will reboot the system. Continue?
Please enter 'YES' or 'NO': YES
[…]
Use BREAK or ESC to interrupt boot.
Use SPACE to begin boot immediately.
Boot interrupted.
[rommon starts here]
Management0/0
Link is UP
MAC Address: fc5b.39a9.6e75
Use ? for help.
#Notice that my confreg is set to 0x41. Let's change that to 0x1.
rommon #0> confreg
Current Configuration Register: 0x00000041
Configuration Summary:
 boot default image from Flash
 ignore system configuration
Do you wish to change this configuration? y/n [n]: n
rommon #1> confreg ?
HEX argument must have a "0x" prefix!
rommon #1> confreg 0x1
Update Config Register (0x1) in NVRAM...
rommon #2> confreg
Current Configuration Register: 0x00000001
Configuration Summary:
boot default image from Flash
Do you wish to change this configuration? y/n [n]: n
rommon #3> boot

At this point, the ASA with FTD boots. I still had a few troubleshooting steps to complete before confirming that the web management interface was working.

After changing confreg but before rebooting again:

> show managers

Managed locally.
#Notice that the Management interface is no longer shutdown. 
> show running-config
interface GigabitEthernet0/0
 nameif outside
 cts manual
  propagate sgt preserve-untag
  policy static sgt disabled trusted
 security-level 0
 ip address dhcp setroute
!
interface GigabitEthernet0/1
 nameif inside
 cts manual
  propagate sgt preserve-untag
  policy static sgt disabled trusted
 security-level 0
 ip address 192.168.1.1 255.255.255.0
!
interface Management0/0
 management-only
 nameif diagnostic
 cts manual
  propagate sgt preserve-untag
  policy static sgt disabled trusted
 security-level 0
 no ip address

Thinking the issue was resolved, I visited https://192.168.45.10:

Service Unavailable. 
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. 
Please try again later.

I wasn’t quite sure what to make of this error so I decided to find the web server service and reboot it. I’m not sure if this is the proper method, but it didn’t seem to hurt anything.

> expert

admin@firepower:~$ sudo su
Password:

root@firepower:/home/admin# ps aux | grep http
root 10104 0.0 0.0 40848 5884 ? S 15:20 0:00 /ngfw/usr/bin/httpsd -D FOREGROUND
www 10107 10.3 7.4 2415436 609796 ? Sl 15:20 1:25 /ngfw/var…start
www 10223 0.0 0.0 1771748 6908 ? Sl 15:20 0:00 /ngfw/usr/bin/httpsd -D FOREGROUND
www 18499 0.0 0.0 1116244 6208 ? Sl 15:27 0:00 /ngfw/usr/bin/httpsd -D FOREGROUND
root 18872 0.0 0.0 4424 648 ttyS0 S+ 15:34 0:00 grep http

root@firepower:/home/admin# pmtool restartbyid httpsd

I visited https://192.168.45.10 again:

"The Firepower Device Manager application cannot be opened."

Clearly that didn’t work as intended so I rebooted the FTD device again. When the system was back up, I tried visiting the web interface.

Service Unavailable.
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. 
Please try again later.

When turning it off and on again doesn’t work…. turn more stuff off and on.

> configure manager delete
> configure manager local
> show running-config
#Interface config looks the same as the above output.

The next command is one that I felt shouldn’t have needed to be issued. When visiting the web interface, I was seeing something, so there was clearly a service answering the phone. It may be the case that in one of the above steps, the management interface was switched to something other than Management0.

Interestingly, we can see below that this command does not utilize the name of the physical ASA interface which is Management0/0. Instead, it uses what might be a logical management interface.

> configure network management-interface enable management0/0

Invalid interface supplied.

> configure network management-interface enable management0

I visited https://192.168.45.10 again and the FTD manager now works as intended.


Final Thoughts

This issue was quite frustrating due to the fact that I could have easily fixed it (temporarily) if traditional ASA CLI configuration was available. Abstracting away tasks through menus and buttons reduces the user’s flexibility and control over their own systems.

For better or worse, this is the direction of the future. A system designed on rails will lower the barrier of entry for lesser-skilled people to use, but can be much more difficult to get back on track when something breaks. Fortunately, Firepower Threat Defense brings forth convenience and features that are welcomed additions to the all-too-commonly over-worked IT administrator.

Advertisements

Cisco Firepower Blocking Mozilla Firefox Cloudfront DNS Queries – Or How I Found NORMANDY

This is a tale of how chasing curiosity can expose the undercover intricacies of everyday technology. What started with a perplexing occurrence led to the exploration of a system architecture that silently interacts with one of the world’s most popular internet browsers.


I’ve deployed Cisco Firepower at several client sites. In every instance, I’ve seen many Security Intelligence events in the DNS Malware, DNS DGA, and DNS CnC categories. Most of these are lookups to Cloudfront.net, Amazon’s Content Delivery Network. Unable to pinpoint the source of these requests, I’ve had no choice but to block the traffic and brace myself for the scream test.

Fortunately, the screams never came. From my perspective, all systems and users were 200 OK. However, it’s been causing me to wonder at night; “what’s broke?”

One day when spinning up a new Firepower deployment, I noticed those familiar DNS requests and traced the source IP back to a coworker who happened to be sitting next to me. I asked him what sites he had been to and if he’d be willing to perform additional testing with me. He agreed, then proceeded to open his browser and ask me what sites I wanted him to connect to. Before going any further, I saw several new events.

These requests typically follow the same hostname format, .cloudfront[dot]net. In this case, I saw:

d6wjo2hisqfy2.cloudfront[dot]net

But this time I noticed something new.

normandy-cloudfront.cdn.mozilla[dot]net

Finally, I’ve caught the culprit red-handed: Mozilla Firefox. The pieces are starting to come together, but there are still questions.

NORMANDY

The first Cloudfront link that I posted resolves to a site that states in large letters “NORMANDY.” Without the second link, this did not make sense to me. Now, I’ve begun the hunt to determine its purpose.

I soon came across the Normandy Github page. It appears to be a server architecture for a whole ecosystem of content delivery and metric gathering. Per the description: “Normandy is a collection of projects that provide a fast and accurate way to interact with unbiased subsets of Firefox users.” These projects are (from GitHub):

  • recipe-server – Django server that manages and delivers recipes efficiently and securely
  • recipe-client-addon – Firefox system add-on to fetch, verify, and execute recipes
  • lints – Shared tools and configs for linting
  • compose – Scripts and configurations to create multi-server environments for development and testing

Side note, “linting” is the process of automating bug-finding by searching for code that is likely to be non-portable or otherwise problematic.

The ReadtheDocs documentation site offers a more complete description: “Normandy is a component in a larger system called SHIELD. The end goal of SHIELD is to allow Firefox to perform a variety of actions to aid the user and gather feedback […]. To do this, we need a way to instruct Firefox to perform certain actions without having to download an update or wait for a new release. SHIELD is split into three main concepts: Actions, Recipes, and the runtime.”

So far, we’ve established that Normandy is a sever architecture that leverages other projects to facilitate interaction with Firefox users. Once contacted, components of SHIELD deliver recipes to solve problems or gather data.

Now I am intrigued on three fronts: What is the scope of this larger system, SHIELD; specifically, what are the tasks of recipes; and how does Firefox determine when to contact Normandy?

SHIELD

Up to this point, the actual function of aforementioned “runtime,” “actions,” and “recipes” have been quite vague. Let’s start by defining what is meant by “runtime.” Per ReadtheDocs, it appears to be a small sandbox environment that can affect a running instance of Firefox. Shortly after Firefox boots, the runtime contacts Normandy, identifies recipes that match the running Firefox environment, and downloads actions that implement the recipes.

The previous sentence alludes to the difference between an “action” and a “recipe.” Simply put, an action is a piece of JavaScript that provides the framework for interaction with a Firefox component. Actions themselves have no direct access to the browser, except through a driver object. A driver object explicitly grants permissions for an action to interact with a feature outside the runtime. The action can then execute, for example to determine whether or not the user should be prompted for a survey.

After the requisite conditions have been established by the action, the action then invokes a recipe that contains the data, parameters, or configuration that needs to be delivered. Interestingly, the ReadtheDocs page contains an example that displays an “implementation_url” variable that closely resembles the suspicious DNS query from the beginning of this post. This URL is:

https://normandy.cdn.mozilla[dot]net/api/v1/action/console-log/implementation/8ee8e7621fc08574f854972ee77be2a5280fb546/

You will notice that the implementation (AKA recipe) is being applied to the console-log action.

Additional information on SHIELD can be found at the Mozilla wiki.

My investigation is proving to be very successful. I now know the nature of the requests coming through Firepower, I understand the components at play, and I’ve learned many interesting capabilities of Firefox that I’m certain will translate to other programs I encounter.

Control

Knowing your environment is valuable, but controlling it is critical. As mentioned earlier, SHIELD is intended to automatically gather telemetry data from users. Many people strive to maintain absolute control over the information that they produce, whether it be in the form of emails, text messages, or metadata.

Fortunately, Mozilla provides users with this guide that outlines how to disable the various types of automatic connections that Firefox makes. It took some hunting, but Mozilla also has a Wiki page that provides detailed documentation on the collected Telemetry data. You can view the data that your browser has collected by inserting

about:telemetry

into your Firefox address bar.

Additionally, if you have read the Firefox Browser Privacy Notice (yeah, right) you would have come across the Firefox Health reports. These reports contain statistics similar to what you’d see in about:telemetry, but in a more user friendly form. Even if you don’t intend to disable Health Reports or Telemetry, it’s still worth checking these out. To view your Health Report, click the Firefox Menu button (the three horizontal lines), then the question mark in the lower right corner, the Firefox Health Report.

Cause and Effect

As with many aspects of life, the choices we make when defending our network can cause unintended results. In this case, the implementation of Cisco Firepower (which, by the way, has been awesome so far) into my networks also brought the enforcement of what Cisco Talos calls “security intelligence categories.”

The categories in particular that are blocking the legitimate traffic described in this post are:

  • Command and Control (CnC)- Sites known to host botnet control traffic.
  • Malware- Sites known to distribute malware.
  • Domain Generation Algorithm (DGA)- Automatically generated domain names that often host malware such as Cerber.

Additional details can be found in Cisco’s documentation.

When we consider the URL that is being blocked, it makes sense why Firepower may have mistakenly categorized the Firefox connection as malicious.

d6wjo2hisqfy2.cloudfront[dot]net

Cloudfront is a well-known server hosting platform. Still, this does not stop some criminals from purchasing servers, often with stolen credentials or financial information, from spinning up new servers to distribute malware. Similarly, the criminals may also use hacked Cloudfront servers to issue commands to swarms of botnets.

Even more suspicious is the hostname of this server, “d6wjo2hisqfy2.” It very closely resembles a domain that could be generated by Domain Generation Algorithm. Cisco Umbrella has a good breakdown on the effectiveness of this technique. However, the blocking of this site by Firepower is strange behavior because the hostname is a subdomain of Cloudfront, not its own registered domain. Therefore, although I can’t confirm this, my guess is that Firepower is flagging this traffic based on the nonsensical hostname alone.

Strangely, not all requests to Cloudfront are being consistently blocked. Specifically, I noticed that requests to

d2pj9rkatqbt38.cloudfront[dot]net

were not being blocked. In this particular case, the URL above redirects to a the website of a compliance and risk management company.

Remediation

And with that, my journey down the rabbit hole draws near to conclusion. The final task remaining is to decide how to handle the blocked traffic.

Due to the fact that Cloudfront may still host compromised content, I cannot add a domain-wide whitelist. Additionally, I also cannot whitelist the specific URL that we’ve dissected above, as it is not always consistent. Thus, I will be leaving Firepower to continue blocking connections to Normandy.

Cisco is among the world’s largest networking vendors, and Mozilla Firefox is among the world’s most popular browsers. I felt that I should do due diligence and report my findings to Mozilla via their bug tracker. At the time of posting, I did not fully understand the nature of the requests that were being blocked. Still, Mozilla contributors showed a disappointing lack of interest in discussing these issues.

My submission on Mozilla’s bug tracker can be found here. You will notice some confusion regarding my intent in submitting the issue, so this may be worth revisiting at a later date.


Final Thoughts

Throughout my research, every new page seemed to contain information on topics that I had absolutely zero exposure to. Simply following curiosity once again proved to be a great method of expanding my limit of understanding. Next time you come across something that piques your interest, keep in mind that exploration is often worthwhile.

As always, thanks for reading.

-Dan

Cisco Upgrade Audit Report Tips

Those of you who have recurring patch cycles to keep Cisco gear in compliance may also be required to generate reports to prove you’ve done the work. Your reporting needs may vary, but hopefully this post gives you some ideas on how to be more efficient.

During the upgrade process, I dump all evidence into a text file, parse it, and use find/replace to generate headings. I use Microsoft Word and Notepad ++ for these tasks. Currently, I’ve only built parsing strings for Cisco ASAs and Catalyst switches.

My reports are typically formatted as follows:
Intro Page
–    Name, dates, quick description of changes.
Table of Contents
–    Device Name
–        Proof of signature verification
–        Current software version
–        Current ASDM version (if applicable)
–        Serial Number
Content

sample-report

A sample mitigation report.

 

This works very well for me and whomever needs to review the report. They can quickly skim the table of contents to verify that everything is at the correct version.

The only part that’s not automated is adding the device names to Heading 2. Once I figure out a reliable method of doing this, I will update this post. Update 02/07/2018- I have found a way to add headings! I have also added support for switch stacks.

Before I upgrade a firewall, I will first verify that the signature is correct which ensures the image has not become corrupt during transfer. Do this with the “verify” command and copy the output into Notepad ++. After I’ve upgraded the firewall, I use “show version,” “show inventory,” “failover exec mate show version,” and “failover exec mate show inventory.” Obviously, use whatever commands apply to your environment.

Switches handle image verification differently. For this reason, I add the “/md5” option to the “verify” command. I will manually compare the output with the known-good hash. This hash is provided by Cisco when you hover over the software name when downloading from the Cisco support site. Note: If you are upgrading a switch stack, specify the switch number after “flash” for heading creation to work properly.


Open Notepad++
Ctrl+f , replace
mode regex

--Switches--

#Create Headings
#If there are stacks that have been upgraded:
find: ^(.*)(#[ ]*verify /md5 flash1:/.*)$
replace: +@+\1+@+\n\1\2

#Create Headings
find: ^(.*)([ ]*#[ ]*verify /md5 flash:/.*)$
replace: +@+\1+@+\n\1\2

find: (^System serial.*$)
replace: +++\1+++

find: (^System image file is "flash.*$)
replace: +++\1+++

#note: compare this hash to the one provided by Cisco
find: (verify /md5 \(flash[12]*:.*)
replace: +++\1+++

#Paste into Word
#Open find/replace
#Select "use wildcards"

find: +\@+{1,1}<*>+\@+
replace: Format only, style heading 2

find: +++{1,1}System serial number*+++
replace: Format only, style heading 3

find: +++{1,1}System image file is "flash*+++
replace: Format only, style heading 3

find: +++{1,1}verify /md5 \(flash*+++
replace: Format only, style heading 3

--Firewalls--

find: ^(.*)(#[ ]*verify disk0:/asa.*)$
replace: +@+\1+@+\n\1\2

find: (^Serial Number: .*$)
replace: +++\1+++

find: (^Verified disk0.*)
replace: +++\1+++
 OR
find: (^Digital signature successfully validated.*)
replace: +++\1+++

find: (^Cisco Adaptive Security Appliance Software Version .*$)
replace: +++\1+++

find: (^Device Manager Version.*$)
replace: +++\1+++

#Paste into Word
#Open find/replace

#Don't do this one if you already did it for the switches. 
#In other words, create headings for switches and firewalls at the same time.
find: +\@+{1,1}<*>+\@+
replace: Format only, style heading 2

find: +++{1,1}Serial Number*+++
replace: Format only, style heading 3

find: +++{1,1}Verified disk0:*+++
replace: Format only, style heading 3
 OR
find: +++{1,1}Digital signature successfully validated+++
replace: Format only, style heading 3

find: +++{1,1}Cisco Adaptive Security Appliance Software Version*+++
replace: Format only, style heading 3

find: +++{1,1}Device Manager Version*+++
replace: Format only, style heading 3

---End---

find: +++
replace: no formatting (blank)

find: +\@+
replace: no formatting (blank)


Command Summary

Firewalls

  • “verify disk0:/asa917-9-k8.bin”
  • “show version”
  • “show inventory”
  • “failover exec mate show version”
  • “failover exec mate show inventory”

Switches

  • “verify /md5 flash:/c2960x-universalk9-mz.152.5.E.bin”, or
  • “verify /md5 flash1:/c2960x-universalk9-mz.152.5.E.bin”, and
  • “verify /md5 flash2:/c2960x-universalk9-mz.152.5.E.bin”
  • “show version”
  • “show inventory”

I hope you find this structure useful for your reporting requirements. As always, thanks for reading.

-Dan

Cisco Firepower Training Guide: Part Two – Initial Tasks

Once you have your SFR module installed and linked to the FireSight Management Center, you can begin bringing the system up to speed. Keep in mind that any report templates that you may have exported from other sites require the same Firepower version in order to import. For example, a report generated at version 6.1.0 cannot be imported by a system at version 6.1.0.1. After your FMC is set up and seeing data from the firewall, let Firepower run in monitor-only mode for approximately 1-2 weeks after your last change.

If your firewall is on a valid Cisco contract, it is often helpful to create a support case. However, they will typically require you to be specific with your inquiry. Once you have them on the phone, don’t be afraid to branch out slightly and ask questions. Respect that they are trying to solve your problem quickly and efficiently, but you can often receive valuable information about other topics if you’re polite.

Another note on Cisco support: I typically do not generate Firepower “troubleshoot files” as these contain sensitive information (i.e., SSH keys and IP schemes) that I do not want leaving my environments. When creating a case, be upfront about skipping this step and simply request a webex instead.

Here is a list of setup tasks that I go through before removing monitor-only mode from the firewall, or in other words, “going live.” These are not necessarily in any order, but I’ll try to put any prerequisite tasks first.

  • Link the managed devices (ASAs) to the FMC.
    • Create any groupings now.
  • Add any licenses.
  • Apply updates: FMC, SFR, VDB, URL, CRL, etc.
  • Define security zones, i.e., firewall interfaces.
  • Define $Home_Net.
  • Install the Sourcefire User Agent.
    • I usually put it on a domain controller, then point it toward itself and other domain controllers.
  • Set up the mail relay server.
  • Create yourself a unique user instead of admin and change your display preferences.
  • Set up email and syslog event notifications for impact flags, discovery events, etc.
  • Create your realm and identity policy.
    • Download users and groups.
  • Add a health policy.
  • Create a File policy.
  • Create an Intrusion policy.
  • Add your internal network to the Network Discovery policy.
    • Enable application detection.
    • Enable “capture banners.”
  • Add feed lists to the DNS policy.
  • Create an Access Control policy.
    • Add security intelligence feed lists.
    • Create an HTML block page.
  • Create or import reports.
  • Configure backup settings.
  • Add scheduled recurring tasks.
  • Optional: Enable change reconciliation reports.

As you can see, there are many tasks involved in preparing your Firepower system. In upcoming posts, I will go into more detail on each step. Until then, feel free to contact me either here in the comments, or via email at techitw.wp@gmail.com.

Thanks for reading,

Dan

Cisco Firepower Training Guide: Part One- Overview

Many of my days are spent installing Cisco Firepower either on new or existing ASAs. I receive many questions from clients on various topics during the setup process, so I’ve decided to share my tips and tricks, procedure, and insight.

These posts aren’t intended to be comprehensive step-by-step manuals. Rather, they should help guide you in the right direction. There are many good free online resources for additional training. PeteNetLive and Lab Minutes are particularly handy.


Before we get too deep, we should take a minute to understand the role Cisco Firepower plays in a network topology.

In July 2013, Cisco purchased Sourcefire, a company that specialized in intrusion detection and prevention (IDS/IPS) appliances. Sourcefire’s ability to detect malicious traffic is based on sets of rules from an open-source IDS/IPS called Snort. Cisco integrated this technology with their existing line of firewalls and rebranded the combination as “Next-Generation Firewalls,” or “NGFW.”

Cisco 5500-X model ASAs (firewalls) have the capability of running a Sourcefire, or SFR, module. This module is essentially a virtual Linux distribution running within the ASA. The “brain” of this module is the FireSight (or Firepower) Management Center (FMC). From the FMC, an administrator defines rules and actions for the SFR module to act upon. It is possible to run a limited version of Firepower managed via ASDM, but it lacks usability and several features.

When traffic flows through the firewall, all ACLs, NATs, and similar functions are performed using the ASA configuration. The traffic is then passed to the SFR module for inspection. This is shown in the image below.

cisco-firepower-diagram

Diagram of traffic flow between an external host and internal host.


This post was mostly to briefly touch on some background information. In part two, I will get into more operational content.

Thanks for reading,

Dan