CF security update (March 1 2019), part 2: further details, prevention, and more
Note: This blog post is from 2019. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.This is my part 2 post which follows onto the Part 1, released the night of March 1, when the new CF updates were released as an emergency update. If you've not yet read that, do that first, to get some basic info and needed context for what follows.
And if you HAVE already read part 1, if it was before Saturday morning, do go back and reread it. I had added some important info that I thought shouldn't wait to Part 2, which I knew could take me a while. See especially the sections there, "A brief introduction to the vulnerability and the fix", "Should you be worried?", and "What if you can't apply the update immediately, and can't wait for part 2?".
And my apologies for the delay in getting part 2 out. For various reasons, including related to additional research work I'm doing on this exploit beyond CF, I was unable to post this then. Better late than never, I hope. Indeed, I had listed quite a lot in Part 1 that I hoped to cover in a part 2. I don't want to delay getting this out any later, so I will get done today what I can and post that, and carry over into a part 3 (or beyond) whatever remains. There are some natural breaks, fortunately. Thanks for your patience.
Following are what I cover here in Part 2:
- More detail about the vulnerability and what was "fixed"
- Wouldn't an antivirus package on the server detect this sort of trojan?
- How to add further protection from it (especially if you may be unable to implement the update for some reason)
- Considering running a security scan of your CFML code
- Consider implementing a web application firewall
- How to prevent execution of the files used in the attack, if they may already be on your server
- Another benefit of applying the latest updates
- What about Lucee?
As for the rest of the topics I'd listed at the end of Part 1, those will have to wait to a Part 3 (or beyond).
More details about the vulnerability and what was "fixed"
By now you should have read the technotes for the release, as well as the APSB (Adobe Product Security Bulletin, which again indicated this update as a priority 1 critical vulnerability. More specifically, the APSB reports that, "these updates resolve a?critical?vulnerability that could lead to arbitrary code execution in the context of the running ColdFusion service.??"?
I hope that what I shared in Part 1 (especially those additions Saturday morning) helps you to see what the vulnerability is about. Again, I am reluctant to offer more detail than I have, but if you have read what I wrote and see what Adobe "fixed", that should help you to see why this was a potentially significant problem.
I also discussed in part 1 who should be worried. It's not that EVERY CF SHOP necessarily IS vulnerable, but it's that you MAY have code in place that WOULD make you vulnerable, and the CF fix helps to act as a fail-safe to protect you.
The bottom line again is that you are vulnerable if someone COULD upload to your server, for example, a file that was an image file, but which was changed to have a server-executable extension (like cfm). How could that happen?
- If you have code that DOES allow file uploads (in CFML)
- which DO save to a web-accessible folder (as one may do for images or docs that you let your users upload, to then be web-accessible)
- and then if you FAIL to check for the file extensions of files being uploaded
- especially if you may rely on the strict=true feature of cffile action="upload" (and related tags and statements) to protect you in its mime type check
- and all the more if you may have relied on putting in a whitelisted list of file extensions in the ACCEPT attribute, when in fact CF was ignoring that if STRICT=true
Unless you were doing your own blacklist or whitelist check of file extensions (separate from relying any ACCEPT list values), then you would be vulnerable.
And that's why Adobe implemented the new protections.
Wouldn't an antivirus package on the server detect this sort of trojan?
You would think so. Again, that's what's so pernicious about this problem. The answer is, almost universally: no, a server-antivirus would not detect this sort of trojan file. The problem (for the antivirus tools) is that, on the surface, such a trojan is "just an image". They don't really care that it has a server-executable extension.
And lest you think I am bad-mouthing some particular tool, I will add that the specific trojan used on my client's site was uploaded to the site VirusTotal.com, which is a nifty service that then passes the file to a few dozen popular anti-virus engines and then reports if it is flagged. Only one did (Qihoo-360): not Avast or AVG, not BitDefender or Clam AV, not F-Secure or Kaspersky, nor Malwarebytes, McAfee, Microsoft, Sophos, Symantex, or the rest that it checks. And actually, Qihoo only cause an aspx of this sort. It did not detect a cfm file of this sort.
What are the new protections?
Essentially, it's that Adobe has now implemented (if you apply the March 1 updates to 11, 2016, or 2016) a new server-level setting of blacklisted file extensions, controlled as the last setting on the CF admin Server Settings page. And you can override that at the application level, and you can of course manage the list (in the Admin level) via the Admin API. We'll discuss each of these a bit more in a moment. Then there's also the new honoring of any whitelisted extensions in the ACCEPT attribute, even if STRICT is true.
First, I'll point out that on the first day the updates were released, you wouldn't have known what those were unless you followed the offered link to the CF Admin docs on Server Settings page and paged down to the bottom of the table with help on all those values. I raised that concern and they changed the technotes (for each update) to list all the extensions. In case you didn't go back to the technote for the CF version you're running, here they are:
As for that Admin-level setting, here is a screenshot of the new admin page setting:
Again you can override that list at the Application level (application.cfc or cfm), such as if you wanted to ALLOW, say, a .cfm to be uploaded. That could be very dangerous, if it's uploaded to a web-accessible folder. But if you somehow NEED to allow it, let's say, and if you take the precautions (such as NOT to put it in a web-accessible folder), then the point is that at least the feature allows you to override the Admin setting.
As for the code to do that, see the link they offer to the docs, which shows an example of doing that application-level override. (Note the option also to "*" (asterisk) to block all files or to use "" (the empty string) to allow all extensions.)
As for code to call to the Admin API (such as if you wanted to script overriding the settings at the Admin-level), that is offered in each update's technote.
But I do have a couple of quibbles with their examples (both of the application-level override and using the Admin API) which I've raised but have not yet been addressed.
First, they show only an abbreviated list of extensions. Since this overwrite would be setting a NEW list, I have argued that they should show in the example ALL the extensions, minus whichever one/s should be allowed. As it is, if someone literally used their example (showing only a few extensions), then they would be exposed again to all the OTHER server extensions now NOT black-listed by that override.
Second, I have also requested that they change the order of the discussion of these 3 features in the update technotes. It should logically discuss the admin setting first, and then the app-level override. Currently it's the reverse. I'd argue the the admin API should remain 3rd in order, since that's llikely to be the least-used feature of the 3.
Finally, though the docs don't mention it, I have found in testing that the new feature ALSO causes CF to honor as a whitelist any extensions you may offer in the ACCEPT attribute (of the CFFILE tag, or the ACCEPT argument of the script equivalent to the tag).
How to add further protection from it (especially if you may be unable to implement the update for some reason)
I stated to this in my update to part 1, but I will repeat it here for those who may need to hear this now: if you can't apply the update immediately, what can you do to protect yourselves in the meantime, if you DO have code that is vulnerable to this exploit? This applies also, of course, to those who may be on CF10 or earlier, as Adobe no longer supports those (indeed, beware that support for CF11 ends in April 2019).
Well, again, you could add your own protections on pages that DO allow file uploads via CFML:
- First find if you have any CF pages (*.cf*) which execute the CFFILE tag with an ACTION="upload" (or "uploadall", or use the script statements FileUpload or FileUploadAll), as part of a process to receive uploaded files
- then in any such pages, ensure that it checks (or you add code to check) that the file uploaded is NOT processed if it has a server-executable extension (like those listed above,) or that the extension is one that you allow
As for finding such files, you can of course use your own favored tool for finding files containing a given string (such as your editor, or an OS tool), but if you have no such tool, I will say that for those on Windows, I can't recommend enough the wonderful free tool, FileLocator Lite. I have also written about it before. While the post is old, the tool is better than ever (ever-being updated) and my enthusiasm for it has not waned a bit.
As for code to do the needed whitelisting or blacklisting, rather than elaborate here, I will point you to the coverage of the topic in the CF Developer Security Guidelines, by CF security maven Pete Freitag. While nearly 30 pages of great content on various aspects of CFML code-level security, note that it specifically offers a CFML code example for just such safer file upload processing on its page 9.
Granted, the guide has not been updated since CF11, but the content is as applicable as ever. Sadly, the PDF is locked to prevent even copying of text, which is odd, but fortunately it's not a lot of code (it's CFML, after all!). I know that Pete has wanted to do an updated guide, so hopefully this lock of the file will be lifted then, if not sooner.
Considering running a security scan of your CFML code
The Dev Sec Guide would not only help with this vulnerability, but of course it covers many others. And I wanted to separate out a new point for the sake of both those those who DO apply the update as well as those who cannot. It surely takes a lot of effort to consider, let alone implement, ALL the protections in the Developer Security Guidelines.
That's why you really should consider the idea of running a security scan against your CFML code, as another means to find any of many kinds of vulnerabilities. The good news is that there are a couple of options for that. First Adobe Security Code Analyzer, a feature introduced in CF 2016 which was provided within ColdFusion Builder 2016 (or 2018), and which works with CF 2016 or 2018 Enterprise (or the trial edition), though not the Standard or Developer editions. Big grr about that, as I and others have long argued. And others aren't a fan of needing to use CFBuilder as the interface for the tool.
Fortunately, there is another new option to consider (for scanning your CFML code for vulnerabilities). Pete Freitag (of Foundeo) is releasing a new product, called Fixinator. It's in preview now, but you can sign up there for early access.
One way or another, everyone would benefit from doing a security scan on their code.
Consider implementing a web application firewall
And yet another means to protect yourself--whether you apply this update or not--is a web application firewall (or WAF), which adds yet another layer of protection around your application, watching for incoming traffic and data (and more) which it may detect as being an exploit. Some can be implemented in CFML, others in your web server, others in your network, while still others as service providers.
Discussing such tools any further would be worthy of its own blog post, if not many. But I will point out that I keep list of such tools in a category of my CF411.com site, specifically:
Note that I have subcategories for each of those different levels (CFML code-level protection, CFML application-level protection, web server-level protection, and so on). Nearly all such tools offer options to initially log detections only (rather than rejecting them), at least until you are comfortable with how the WAF is configured.
Finally, you may notice I have there yet another related subcategory, called Runtime Application Self-Protection (RASP) tools. Consider those as a new twist on the concept of a WAF. I also list there Intrusion Detection tools. That's something I'll cover more in a later part to this series.
How to prevent execution of such trojans, until you get updated or if they may already be on your server
I plan to cover in Part 3 various means to find and remove the various hacks that may have been enabled by this vulnerability (which is not easy, but of course will be of prime importance to anyone hacked)
But in the meantime, and as we close here in Part 2, I wanted to point out that this particular trojan (a file masquerading as an image or other "expected" file type, but with a server-executable extension) is something that we can also prevent running, and that protection will help you going forward, whether until you get the CF update in place, or in case such files may already have been put on your server.
If you consider the example I have used, of a site allowing upload of images (to a web-accessible folder), you may realize that one protection you can enable is to tell your web server to simply not ALLOW execution (requesting of) any file extensions in such a folder (like an image folder).
In the case of IIS, you can do that with the IIS "Request Filtering" feature, which has as its first tab a list of file extensions that should be blacklisted (or whitelisted). And you can set that at the server level (in which case it's inherited by all sites, unless they override it), or at the site level (in which case it's inherited by all folders on the site), or (the key for us) you can set it at the folder level. So you could tell IIS that you don't want to allow execution of any of the above-listed server-executable file extensions, at least in such an "images" folder. Or maybe its your "docs" folder, or your "logs" folder. Whatever folder you allow people to upload files to. Or again you could set it to whitelist (only allow) the extensions you DO expect.
And as for Apache, you can do that same sort of control, at the server, site, or folder level, using Apache conf file directives, among other means.
For both web servers, I won't elaborate here what to do, because for one thing you could find how via easy searching on Google, etc. Better still, both features are covered in the CF Lockdown Guide (also written by Pete Frietag). While it shows using those features (in both IIS and Apache) in the context of preventing access to CF-oriented extensions where desired, the exact same concept and steps can be applied to blacklisting (and even whitelisting) extensions for the purposes we are discussing. See section 4.7 of the CF2018 version of the lockdown guide, and similar sections in earlier editions of the guide.
That said, once this exploit itself happens, often the bad actors will cause implementation of still MORE new and changed filed on your server. This again is not really new. Pretty much any vulnerability that has allowed "remote execution of arbitrary server code" has led to that sort of hijacking. We will consider that and talk about trying to find such in part 3.
Another benefit of applying the latest updates
As we wrap up part 2, I want to point out that a "side-benefit" of applying these latest (March 1) updates to CF11, 2016, and 2018 is that if you may not have applied those from Feb, or those from Sep 2018, each of them also included important OTHER security updates. You can read their respective technotes to learn more. (You can just click the URL for a technote, which has the CF version and update number in it, and decrement the number by one to see an earlier update. Or just google "coldfusion [x] updates" (like coldfusion 11 updates) to find a page for each release and a list of all its updates.
And if you may have heard of a lot of problems with the updates of Feb 12, note that those were resolved with new updates on Feb 25 (the week before this update) with corrections of the CF11 and CF2016 updates (not needed for the CF2018 Feb 12 update, called update 2).
What about Lucee?
If you may be using Lucee, the open source alternative CFML engine, I have news to report for you: I did present this problem to them over the weekend, and Igal Sapir (a committer on the project) told me that he had implemented an update today as 184.108.40.206, which will be pushed out ASAP. Keep an eye out on the Lucee site for more.
Coming in Part 3 (and/or beyond)
Before wrapping up part 2, again I had listed a few more topics that I'd meant to cover. I hope to do those soon, in a part 3 (or beyond, if warranted):
- how to find such files (that may have been uploaded), and removing them
- how to find possible shell scripts that may have been implemented with them
- how to find evidence of calls to such, per web server logs
- how to detect possibly modified code, edited once the attacker was in
- longer term mitigation considerations to protect from such vulnerabilities
Until then, comments are welcome. And again, if you need help applying the update (or resolving problems in applying them), I'm here to help. See the box below for more.
For more content like this from Charlie Arehart:
Need more help with problems?
- Signup to get his blog posts by email:
- Follow his blog RSS feed
- View the rest of his blog posts
- View his blog posts on the Adobe CF portal
- If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
- See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
The recent CF 11 Patch 18 has broken our production website. Would you happen to know if the new patch does any inspection of the URL string or URL variables and sanitizes or removes them if they look suspicious?
We have a process which encrypts the URL variables, combines the URL string, creates the encrypted target URL, redirects there, and then decrypts the URL on the subsequent page load for processing by ColdFusion. However, on some URLs which worked before the update certain URL variables are now missing, causing our application to fail. I'd greatly appreciate your thoughts on this matter.
But CF11 updates 17/16 changed many things, and yes, I am aware of an issue that sounds like yours: https://tracker.adob...
Sadly, there is no fix offered, and it's even closed by Adobe, though people are pleading there to re-open it. There is code provided that shows what's not working, and you can test it to see if you experience it--then see if what it discusses might relate to the kind of error you are having.
BTW, you say you went to "the latest", but it would help to know what update you came "from". You can look at the cfusion/hf-updates folder to see folders which indicate the numbers of previous updates attempted. (There may be jar's there which reflect updates that were downloaded but may not have been attempted.)
That link you posted is pretty much what we encountered. One of our guys found the article as well and was the impetus for our fix. The solution was to switch from URLencodedFormat to the EncodeForURL function on all of our web pages.
We did come from CF11 Patch 16, then Patch 17, but did not catch the issue in our testing. This was likely due to our PreProd and Testing environments having a smaller sub-set of data compared to our Production system. In Production, there were only certain URLs that caused the issue. They likely had a % character in them, or something similar, causing the issue. Those records did not exist in PreProd or Test so it slipped by.
And if anyone getting notifications about these is no longer interested, not there's a link in the email to unsubscribe from the notifications about this post.