This is a tale of how chasing curiosity can expose the undercover intricacies of everyday technology. What started with a perplexing occurrence led to the exploration of a system architecture that silently interacts with one of the world’s most popular internet browsers.
I’ve deployed Cisco Firepower at several client sites. In every instance, I’ve seen many Security Intelligence events in the DNS Malware, DNS DGA, and DNS CnC categories. Most of these are lookups to Cloudfront.net, Amazon’s Content Delivery Network. Unable to pinpoint the source of these requests, I’ve had no choice but to block the traffic and brace myself for the scream test.
Fortunately, the screams never came. From my perspective, all systems and users were 200 OK. However, it’s been causing me to wonder at night; “what’s broke?”
One day when spinning up a new Firepower deployment, I noticed those familiar DNS requests and traced the source IP back to a coworker who happened to be sitting next to me. I asked him what sites he had been to and if he’d be willing to perform additional testing with me. He agreed, then proceeded to open his browser and ask me what sites I wanted him to connect to. Before going any further, I saw several new events.
These requests typically follow the same hostname format, .cloudfront[dot]net. In this case, I saw:
But this time I noticed something new.
Finally, I’ve caught the culprit red-handed: Mozilla Firefox. The pieces are starting to come together, but there are still questions.
The first Cloudfront link that I posted resolves to a site that states in large letters “NORMANDY.” Without the second link, this did not make sense to me. Now, I’ve begun the hunt to determine its purpose.
I soon came across the Normandy Github page. It appears to be a server architecture for a whole ecosystem of content delivery and metric gathering. Per the description: “Normandy is a collection of projects that provide a fast and accurate way to interact with unbiased subsets of Firefox users.” These projects are (from GitHub):
- recipe-server – Django server that manages and delivers recipes efficiently and securely
- recipe-client-addon – Firefox system add-on to fetch, verify, and execute recipes
- lints – Shared tools and configs for linting
- compose – Scripts and configurations to create multi-server environments for development and testing
Side note, “linting” is the process of automating bug-finding by searching for code that is likely to be non-portable or otherwise problematic.
The ReadtheDocs documentation site offers a more complete description: “Normandy is a component in a larger system called SHIELD. The end goal of SHIELD is to allow Firefox to perform a variety of actions to aid the user and gather feedback […]. To do this, we need a way to instruct Firefox to perform certain actions without having to download an update or wait for a new release. SHIELD is split into three main concepts: Actions, Recipes, and the runtime.”
So far, we’ve established that Normandy is a sever architecture that leverages other projects to facilitate interaction with Firefox users. Once contacted, components of SHIELD deliver recipes to solve problems or gather data.
Now I am intrigued on three fronts: What is the scope of this larger system, SHIELD; specifically, what are the tasks of recipes; and how does Firefox determine when to contact Normandy?
Up to this point, the actual function of aforementioned “runtime,” “actions,” and “recipes” have been quite vague. Let’s start by defining what is meant by “runtime.” Per ReadtheDocs, it appears to be a small sandbox environment that can affect a running instance of Firefox. Shortly after Firefox boots, the runtime contacts Normandy, identifies recipes that match the running Firefox environment, and downloads actions that implement the recipes.
After the requisite conditions have been established by the action, the action then invokes a recipe that contains the data, parameters, or configuration that needs to be delivered. Interestingly, the ReadtheDocs page contains an example that displays an “implementation_url” variable that closely resembles the suspicious DNS query from the beginning of this post. This URL is:
You will notice that the implementation (AKA recipe) is being applied to the console-log action.
Additional information on SHIELD can be found at the Mozilla wiki.
My investigation is proving to be very successful. I now know the nature of the requests coming through Firepower, I understand the components at play, and I’ve learned many interesting capabilities of Firefox that I’m certain will translate to other programs I encounter.
Knowing your environment is valuable, but controlling it is critical. As mentioned earlier, SHIELD is intended to automatically gather telemetry data from users. Many people strive to maintain absolute control over the information that they produce, whether it be in the form of emails, text messages, or metadata.
Fortunately, Mozilla provides users with this guide that outlines how to disable the various types of automatic connections that Firefox makes. It took some hunting, but Mozilla also has a Wiki page that provides detailed documentation on the collected Telemetry data. You can view the data that your browser has collected by inserting
into your Firefox address bar.
Additionally, if you have read the Firefox Browser Privacy Notice (yeah, right) you would have come across the Firefox Health reports. These reports contain statistics similar to what you’d see in about:telemetry, but in a more user friendly form. Even if you don’t intend to disable Health Reports or Telemetry, it’s still worth checking these out. To view your Health Report, click the Firefox Menu button (the three horizontal lines), then the question mark in the lower right corner, the Firefox Health Report.
Cause and Effect
As with many aspects of life, the choices we make when defending our network can cause unintended results. In this case, the implementation of Cisco Firepower (which, by the way, has been awesome so far) into my networks also brought the enforcement of what Cisco Talos calls “security intelligence categories.”
The categories in particular that are blocking the legitimate traffic described in this post are:
- Command and Control (CnC)- Sites known to host botnet control traffic.
- Malware- Sites known to distribute malware.
- Domain Generation Algorithm (DGA)- Automatically generated domain names that often host malware such as Cerber.
Additional details can be found in Cisco’s documentation.
When we consider the URL that is being blocked, it makes sense why Firepower may have mistakenly categorized the Firefox connection as malicious.
were not being blocked. In this particular case, the URL above redirects to a the website of a compliance and risk management company.
And with that, my journey down the rabbit hole draws near to conclusion. The final task remaining is to decide how to handle the blocked traffic.
Due to the fact that Cloudfront may still host compromised content, I cannot add a domain-wide whitelist. Additionally, I also cannot whitelist the specific URL that we’ve dissected above, as it is not always consistent. Thus, I will be leaving Firepower to continue blocking connections to Normandy.
Cisco is among the world’s largest networking vendors, and Mozilla Firefox is among the world’s most popular browsers. I felt that I should do due diligence and report my findings to Mozilla via their bug tracker. At the time of posting, I did not fully understand the nature of the requests that were being blocked. Still, Mozilla contributors showed a disappointing lack of interest in discussing these issues.
My submission on Mozilla’s bug tracker can be found here. You will notice some confusion regarding my intent in submitting the issue, so this may be worth revisiting at a later date.
Throughout my research, every new page seemed to contain information on topics that I had absolutely zero exposure to. Simply following curiosity once again proved to be a great method of expanding my limit of understanding. Next time you come across something that piques your interest, keep in mind that exploration is often worthwhile.
As always, thanks for reading.