ColdFusion March 2023 emergency update, and what to do about it
TLDR; For some folks, the above may be all you need to hear: you may be dropping your coffee and donuts now to get the update applied. Still others will see this "huge post" and think, "crap, I don't have time for this". For you, skip to the bottom and its "concluding key points". You can then decide what you think you do or don't "need to know" and pick and choose from the sections as you like.
Finally, for those who prefer because of the importance of all this to be led more carefully through understanding things (in a way that's worked for the many people I have helped so far this week, and is far more than either Adobe or Hackernews has shared), please do read on.
As indicated in the technotes for the update, it "addresses vulnerabilities that could lead to arbitrary code execution, arbitrary file system read , and memory leak." The first two especially should get your attention.
After pointing to and offering information on obtaining the available CF update, I'll go further to discuss:
- be careful about thinking "this vulnerability doesn't apply to us" (I've seen many people coming to incorrect conclusions)
- what the vuln enabled (a hack which is scary, that could be easily perpetrated on any CF instance from CF2021 to CF11, prior to this update)
- how might you check if you were hit with the hack?(something not mentioned in Adobe's security bulletin, update technotes, or other announcements)
- what if you find you WERE hit?
- how to protect yourself without the update, if you need time to get the update deployed--which will also help those on CF2016 and CF11, as Adobe no longer updates those but they are also vulnerable to this hack
- some tips to address trouble some have had applying the update
- additional protections you could consider to help temper such hacks
Before moving on, I do want to clarify further the first point above: the hack I discovered CANNOT be perpetrated against Lucee (as it never implemented the cfclient feature that is being leveraged), but note that it CAN be perpetrated against Commandbox (if using an adobe cfengine prior to this this update). Granted, most devs may trust that the random port Commandbox exposes by default is one that "no one can get to", but do note that if that may have be configured to be exposed otherwise (such as via a web server like IIS, Apache, or nginx), then of course anyone who COULD access that web server may be able to run CF requests against it, and thus they could run the kind of request behind this hack (more on that below).
I. About the updates for CF2021 and 2018
Before offering my elaborations on things, let's get some basic points about the updates out of the way.
First, the updates are CF2021 update 6 and CF2018 update 16, and as usual the updates should appear in your CF Admin and be downloadable/installable from there, if your server has internet access (or at least the domains for the updates are accessible: www.adobe.com and cfdownload.adobe.com). Otherwise you can obtain and apply the updates manually, as discussed in the update technotes. (Note that the manual update steps differ between CF2021 and CF2018.).
(BTW, if you obtained the jar or zip to apply the update manually to CF2021, you may find that the manual update process failed to work correctly. Adobe fixed that and issues an updated version of the jar and zip for CF2021. Just go get it again.)
For more on the updates, see the Adobe discussions in the following resources. It's great that they have taken many steps to spread the word, though they've been rather spartan and perhaps not as clear as they could be about the vulnerability (thus more to follow in this post):
- The technotes for CF2021 update 6 and CF2018 update 16. As always, the technote has information on obtaining and applying the update, including important information if you may be skipping from updates earlier than the immediately precending one (more on that topic later here)
- The APSB (Adobe Product Security Bulletin) regarding the security vulnerability (or in this case, vulnerabilities) addressed by this security update, APSB 23-25, which includes the "Priority 1" rating I mentioned above, and the CVSS scores of the vulnerabilities (the highest being 9.8 out of 10), as well as the CVE/CWE classifications of the vulns. Again, I will share more on the reason for the high score in the next section.
- Adobe's blog post about the update on the CF portal (including comments from others)
- Adobe's forum thread about the update, in the CF Community Forums (including comments from others)
- Adobe's Facebook post about the update on the CF programmer's group (with comments from others)
- Adobe's Slack channel mention of the update (with comments from others)
- No post still as of today, Mar 15, on Adobe's twitter account
II. Be careful about thinking "this vulnerability doesn't apply to us"
Before I jump into the details about the vuln (and the hack), I want to address a few points where I have seem people making incorrect conclusions about what they have read or heard regarding this vuln and the hack that can be perpetrated.
First, the technotes indicate that with the update Adobe has "disabled cfclient by default". Now, some are misreading that as being about the "client variables" feature in CF, but it has nothing AT ALL to do with that (and whether you use them or not).
Far more important, DO NOT hear that mention of cfclient and conclude, "Oh: we don't use that, so we can relax." No, no, NO! That's like saying you don't need to worry about a vulnerability in your home's window locks because you "only enter the house through the door"! It's the BAD guys who were leveraging a vuln that is merely RELATED to an aspect of the cfclient feature--whether you and your code use that feature or not! But yes, the update WILL disable that cfclient feature (and thus this vuln).
Finally, some are saying, "oh, our servers aren't open to the internet". That, too, is NOT going to protect you. If you have connected as web server (like IIS or Apache) to CF, for any app at all, then a call made into that web server that got to CF could be used to perpetrate this hack. It does NOT matter if users need to "login" to your app: this request won't even go against "your app" but against "CF", itself. And the same thing is true of the built-in web server (used for the CF Administrator, typically, such as at port 8500). Anyone who DOES have access to that port could potentially try to use the hack URL.
With that as preface, let's move on first to the vuln and the hack it enabled.
III. What the vuln enabled: seriously bad stuff
As is the nature of security bulletins, the resources above are rather spartan in their indications of what was possible if this vulnerability was perpetrated against your server. While it's true that discussion of such hacks is a very sensitive subject, and I will not offer here HOW to perpetrate the hack, I WILL at least offer more info to help you try to find out WHETHER the hack was perpetrated against you.
But let's start with better clarifying what the update technotes and APSB bulletin mean when the vuln "could lead to arbitrary code execution" and "arbitrary file system read", what that means is that a bad guy could use a specifically crafted URL to view ANY file that CF could access. Yep, that's bad.
But it gets worse: they could also then use a specifically crafted URL to cause a CFM file to be placed on your server in a web accessible folder, like the cfusion/wwwroot, while also pulling down into that file (from a remote server) CFML that offered a "shell" which then would allow the hacker to basically perform any action that CFML could perform, including querying your databases, add/change/delete files, export data or files off your server, and more. Yes, very, very bad. Now do I have your attention?
This would be classified as a zero-day--and in fact I was so wishing I could tell folks about it sooner, but I was embargoed from discussing it until the update was released. Let me add also that while I found it with a client a couple of weeks before (when I reported it to Adobe), I also found evidence on another client's server that the "bad guy URLs" causing the hack were attempted as far back as Dec 2022. (Again, I wish I could share the details of the URL, but I can't--but that's why I share below the importance of searching your logs for the appearance of the _cfclient query string value.)
What's the connection to CFClient?
And now do you see how this problem has NOTHING to do with "whether you use cfclient or not"? So what's the connection to cfclient? I'll just leave it at the fact that this "specifically crafted URL" was leveraging something that CF's "cfclient" feature enabled. It's not something YOU might ever use, but it was something the bad guys figured out THEY could leverage to perpetrate this hack. Again, I will show later how you can look for evidence indicating whether someone had perpetrated the attack.
As for this cfclient feature, which offered to enable mobile device support, it never really took off. But it was added in CF11, which is why the vulnerability extends also to CF2016 and CF11--which again are are no longer updated by Adobe, so there will be no fix offered by them for this. See below for how you can protect yourself if you are still on those old versions--which have still other significant security vulnerabilities that have been fixed only in later CF versions. (Again, it's because Lucee never implemented the cfclient feature at all that it is not vulnerable to this specific hack.)
How it is I am so confident of what I'm sharing here
And I can clarify that I know exactly what the "specifically crafted" URL was that enabled the ability to view any file and cause creation of a shell script to then allow performing any action. I found the vulnerability (while helping a client who detected the file creation, thankfully), and after gathering more evidence and information I reported it to Adobe (and to Pete Freitag, who later added some more valuable insight), and Adobe then implemented this fix.
And I can confirm that after applying the update (or the protection I'll share for those who can't apply the update), that bad guy URL could no longer view files or cause creation of the bad guy shell script file.
I should add that there is that mention of a 3rd characteristic of the vulnerability that's "fixed" by this update: a "memory leak". Note that the discussion of that in the APSB relates it to "path traversal". I know that the part of the hack I saw did also leverage "path traversal" (allowing the "specifically crafted url" to point to files in any folder it had access to, on the same drive/file system that CF was installed on). I had not myself noticed any aspect of this involving a memory leak, but folks should take this into consideration as simply another reason to apply the update.
What if you DO use cfclient?
If somehow you (or some CFML app on your server does) USE the cfclient feature, note that the update disables that by default. Adobe took this "scorched earth" approach to block the serious nature of this vulnerability. But note that the update technotes clarify that if you DO use cfclient then you would likely NEED to use the offered JVM argument to re-enable that feature (
But note that the technotes also offer yet ANOTHER JVM argument related to this,
To be clear, both these JVM arguments are NEW as of this update. They would have no impact at all (and would be simply ignored) if you try to use them with an update of CF earlier than these two from March 2023.
IV. How might you check if you were hit with the hack?
So assuming you'll either apply the update (as discussed above) or the other protection I offer below (if you can't apply the update), you may still wonder: "how can I know if the bad guys" hit my server?
To be clear, this is something not covered at all in Adobe's security bulletin, update technotes, or other announcements. Indeed, I am walking on thin ice in adding this discussion, but I think it's important because it's not enough to know you "fixed the castle door"...if the bad guys got into the castle before you fixed that door.
While I can't say I can help you prove with 100% confidence whether the hack was perpetrated on your server, I can at least offer a couple of means by which you can look for evidence that would be left behind if it happened. Think of it as looking for fingerprints.
(And of course I can't offer here the "specifically crafted URL" that would allow you to confirm THAT the hack COULD be perpetrated--valuable though that would be--because doing that would expose to the bad guys--whether out in the wild or perhaps in your own organization--who could use that to again a) display any file CF can access or b) lead to the creation of a file on your server. Instead, let's focus on a) finding any evidence and b) closing that door--whether with the update or otherwise.)
Evidence #1: Look in the CF cfclasses folder, for files following a certain pattern
As some may know, when one executes CFML (makes a request for a CFM or CFC file), CF will compile that CFML into Java, and there's a setting in the CF Admin ("save class files", on the "caching" page, which is enabled by default) to have such a compilation saved into a folder within CF, called cfclasses, located at CF's cfusion/wwwroot/WEB-INF/cfclasses folder (if you have more than one instance, that would be [instancename]/wwwroot/WEB-INF/cfclasses folder). The primary benefit of this saving of the compiled class file is so that on a CF restart, the compilations need not happen all over again.
Any time a file is created and first run, or if it's run after the CFML source is edited, the class file for that file (in that folder) would be updated. (Technically, if you have the related "trusted cache" setting turned on, then editing a file would not cause recompilation until a CF restart or clearing of the template cache...but this is getting into the weeds relative to the point of this post.)
Anyway, the pattern of file names in that cfclasses folder is that they all start with "cf", then they have the filename and extension but with a 2e for the ".", so that an index.cfm would be cfindex2ecfm, then that's followed by a number which represents a form of a hash of the original CF file's location. (I have a post from 20 years ago with more on that.)
Well, if the vulnerable "specifically crafted URL" is executed, causing display of a log file or xml file, you would find in that cfclasses folder a file with the pattern not of *2ecfm* or *2ecfc* but instead
You can search this folder using either the simple search feature in Windows File Explorer, or the Linux Find command, or any other tool that facilitates searching for files by name. (For Windows, I highly recommend the free tools File Locator Lite and Ultrsearch, both of which I've blogged about before.)
And while I'm trying to share in this section enough info for someone to at least TRY to check if any such file appears there, let me warn that before you rely on this completely, there some caveats to consider:
- If you find you don't have ANY files in the cfclasses folder, then someone has unchecked the "save class files" feature in the CF Admin. In that case, you can't use this folder to find this "evidence"
- Similarly, if you DO have files in that folder but none are recent, then there's a good chance someone turned that setting off in the past--so that the files you see are leftovers from when it WAS enabled. If that's it IS disabled, then it means again that you can't use the information in this folder as the "evidence" I propose. If the "save class files" IS enabled, this lack of recent files could just mean you've not had any new or recently edited CFM files.
- Do be careful not to over-react if you DO find some file matching the *2elog* pattern: if what follows the 2elog in the file name is NOT numbers (again, representing a form of a hash of the location of the file), then that's NOT evidence of the representation of a .log file having ended up here. More specifically, if you see some filename with a pattern like something2elogin2ecfm followed by numbers, that merely means the name of the file was something.login.cfm, and the ".log" within that is not representing a file EXTENSION of .log--which again would be followed by numbers in the pattern of files here
- Finally, this "evidence" I refer to is related to that "specifically crafted URL" which I found had been used to perpetrate the hack on clients (yes, more than one) who I helped in finding and researching this hack. I can't know for sure if there are other evidences of other aspects of the vuln(s) fixed by this
Even if somehow you can't rely on the evidence above, there is still other evidence you can look for.
Evidence #2: Web server access logs (tracking each request coming in)
Another place to look for evidence of this hack having been perpetrated is in your web server's access logs, which should be logging every incoming request (for static files as well as CF requests). Both IIS and Apache (the primary web servers supported by ColdFusion) do log requests by default.
Again, I can't give out here the details of the "specifically crafted URL" (to have you look for or try that), but I feel I can safely disclose just a tiny tidbit that at least was distinctive in that URL--and we can then look for that in our web server access logs. And it's related clearly to the feature that the update disables by default: cfclient.
More specifically, if you look in your access logs for any requests having been made with a querystring including the phrase:
Let me note at this point also that below I talk about also how/why you may want to search for the uuencoded variant of that _cfclient string as:
(And it may help some readers to know that the "query string" in a URL is the portion that follows a filename and then ?. In CFML, we regard those as "url variables". And in web server logs, the query string is always logged by default, after the file name--typically separated with a space, FWIW, and NOT with a ?.)
As I noted in the last section, you can use tools to make it easy to search for that string (which do the job far better than Windows search, for instance, or even better than the file search features of popular editors like VS Code, Sublime, and Notepad++, as well as traditional CF editors like CFBuilder, Dreamweaver, HomeSite+, and so on.) But, if you do use any such tool to search the IIS or Apache logs on Windows, be sure to "run it as admin", so that it can see all the log files. Linux users will of course tend to favor grep as their go-to tool for searching files by content. On Windows, I favor again File Locator--and I'll note that its "Pro"/paid version has the ability to search through files that are zips, which may be important if your web server logs might get zipped up after some period of time.
(And if you may use the tool, GrepWin, as I found some of my clients using, be sure to change it from doing a "regex" to a "text" search, as well as change its "limit search" feature from the default of only searching files where "size is less than 2000k", changing that to search files of "all sizes".)
There's indeed much more I could say on this topic of searching your web server access logs. At some point I may do another post with more on things such as the following:
- How to find where your IIS access logs are (including making sure you look at the folders for the logs for EVERY site)
- How to find where your Apache access logs are
- Tools that make searching through such logs easier, from built-in ones, to ones you can easily add, to saas-based logging tools you might leverage
- Why it's important in IIS to run any such local tool "as admin" to make sure you are looking into folders that you might be blocked from, even if you ARE an admin
- Why it's important to beware, notice, and deal with the fact that something may have been enabled on your server to zip up the web server access logs after some period of time, which could make them harder to search
- Why it's important to make sure all sites ARE logged, and that they reflect current traffic being tracked (again, so as to make sure you can trust your search results)
- Why I would propose it's important to search first to see/confirm how many days of log files you have, relative to the number of sites you have, so you can trust the sites ARE being logged
- Why I would also propose you do a sanity check to make sure that whatever search tool or feature you use DOES find some string that you know DOES exist in those web server logs
- How the cf built-in web server and undertow web server within Commandbox (if you are using that with CF) do not by default log requests, and how to enable them if you wanted to
While I know that some reading this post may "need" that additional info now, the post is already long and taking very long to write. People want the key details and are awaiting my delivering that. So these topics will have to wait for another post.
Again, the previous section's "evidence" has so far been a more accurate predictor of whether the hack was perpetrated on a given CF server. (Indeed, as for that last bullet point in the list above, that discussion would ALSO apply if a request was made with that "specifically crafted url" against either the CF built-in web server or the Commandbox undertow server running with CF.)
Beware encodingThat said, I will note again one last thing: some may want to point out that bad guys will often encode a querystring or its values. Indeed, when I first encountered this hack with a client, much of the querystring WAS encoded: specifically URL-Encoded UTF8 Unicode, so lots of it was %'s and two-digit number. (There are useful tools to allow you to encode/decode such strings, such as this and this and this.)
But a key point here is that the web server access logs (for IIS and Apache) will indeed log the URL exactly as it came in, so it's true that if this string were encoded, it would be encoded in the logs and you'd need to consider looking for that. (This is the sort of stuff that can make talking about all this so difficult, as there are so many angles to consider.)
But I will say this: I never saw the _cfclient phrase encoded in the cases where the hack had been successful (per the other evidence above). But if you may want to look for that _cfclient as a url-encoded uft-8 unicode string, the value is: %5f%63%66%63%6c%69%65%6e%74.
But let's move on to other matters.
V. What if you find you WERE hit?
This is tough news to share, but I've told you how bad the vuln is. If you DO find evidence that the bad-guy URL was used on your server, you will be able to see readily in the web server access logs what this "specifically crafted URL" was.
And as for the first part of the hack, where they could "see any file", at the end of the URL you may find you would be able to see the path and name of the file that they attempted to view. (If it's encoded, use that urldecoder.org web site that I'd just mentioned, which will decode the string into clear text for you.) If they file/path they requested is valid, they were able to see that file. That's bad enough.
The worse news is if you find they ran a variant of the URL that did MORE than just view files. Again, I can't offer an example, but if you find that any of the requests (with that _cfclient query string and more) refers to cfhttp or cfexecute, for example, that's part of how they were able to perhaps cause a file to be placed on your server: typically that shell script that then allows them to call that page to do about anything.
So first, you may wonder, "where should I look for the file they may have placed there?" And the fact is that they could put it ANYWHERE that CF could write to, but in my experience helping people they were putting one or more files in the cfusion/wwwroot folder (which normally is served only by requests to the built-in web server. But there's a little-known behavior CF has where it WILL look into the cfusion/wwwroot (or your [CF]/[instancename]/wwwroot if running multiple CF instances in CF Enterprise or Developer) for a requested file, if it's not found in the root for your site in your web server (IIS or Apache, typically). So again that's where they would write such a file (this has been a characteristic of the past few similar remote file execution vulns in CF). But again it could be in ANY folder CF could access, and that was web-accessible so that they bad guy could access it.
Second you may wonder, "how can I know if they DID put a file somewhere?". Again, that's hard to know. The URL may give you a clue, but once they put one file in place they could then call that to put still other files in place: indeed, they could edit your OWN files, modifying them in some nefarious way. And this is of course the nightmare scenario: you'd understandably be panicked, wondering "goodness, what all could they have modified"? So this is why I focused above on helping you find evidence of their having even perpetrated the hack, if that's possible to find. At least if it's not, you can worry LESS (though not completely).
Third, you could use the search tools (discussed elsewhere in this post) to find files on the server with a modified date/time that is on or since the date that you found the hack attempt, looking both in any site webroot or in the cf wwwroot (and subfolders).
Fourth, do you use version control/git/etc? If so, you could compare a last known good version of the code from BEFORE the date you found the hack was perpetrated. That may help you readily detect unexpected changes on the source files in your server. Even if you do not, do you perhaps have a local copy of the code that you only "push up to the server"? If so, you could compare THOSE two codebases. For that comparison task I will note that I love the tool BeyondCompare for this job (and which works on Windows, Mac, and Linux--and has a free trial that's 30 days of USE rather than 30 days from install). It can even do a compare over FTP if you may still be working that way. It just does a better job comparing folders, files, and simply text than any other tool I've ever used. I realize everyone has their favorites. For more on it, see a post I did in the past.
(I will mention toward the end of the post how there are also file change detection tools that one could put in place to help warn or even BLOCK such unexpected file writes, so that you know of the problem BEFORE it happens. That is how the first client I helped did detect the problem, which led to our digging in and then my identifying how it had been perpetrated, which I then reported to Adobe.)
Finally, as for the bad-guy shell script, if that got put on your server, again you may see calls made to that file (whose name would be indicated in that second kind of hack URL, but which could be random). Sadly, the calls to that shell script (CFM page) will be form post's, and nothing logs what gets passed to form posts: not web server logs, not CF logs, not even FusionReactor (the popular monitor for CF). I do realize that if you find many post calls to such a shell script, there will be nothing you can do but lament terribly about what may have been done. And I realize that for some, they may feel this situation might warrant their rebuilding the machine from scratch, perhaps taking stronger security measures going forward. Again, I talk about some of those you might consider, toward the end below.
Let's move on now to how folks can protect themselves, especially those who have NOT yet been hit with the hack (but those who have may want to consider this also).
VI. How to protect yourself if you need time to get the update deployed, or are on CF2016 or 11?
While applying the update is the best "medicine" against this hack, I do realize that some folks may still "need time" before they can arrange to perform this update: maybe they can't restart CF during the day (despite the urgency of this update), or perhaps they wisely want to test against dev/test/qa/staging first (which is always smart). If they're reading this now and won't be able to do the CF update for hours (or even days), must they remain vulnerable? No.
Recall that I discussed above one of the evidences of the hack was an incoming URL with a querystring having the value of "_cfclient" (without the quotes). There was additional info beyond that, of course. But this string was a key part of the "pick" that broke the "lock" of CF and allowed the bad-guy URL to do its deed (showing any file or causing one with a bad-guy shell to be put on your server).
Well, you can tell your web server to block any requests that use that phrase in the querystring. That's what I'll cover in this section.
This applies also to those who are running CF2016 or CF11: again, they are vulnerable to this hack, but Adobe will not be offering updates to those. And of COURSE this should be a clarion call to get off such old CF servers. (I did a post that discussed that topic for another serious vuln a couple of years ago.) Beware also that CF2018 is also approaching its end of life and will last get updates in July 2023. Finally, I have a webinar on Migrating apps to ColdFusion 2021 from earlier versions to help anticipate/overcome the challenges.
Check before blocking
Before implementing something to BLOCK requests that have the _cfclient query string, is it possible that you have some legit use of that? Maybe, but not likely. Certainly if you don't use that cfclient (mobile) feature introduced in CF11, then CF code won't cause URLs to have that. And since _cfclient is a pretty unusual string, it's not likely any of your OWN code will cause it to be in URLs.
Still, you can use the same techniques I discussed above about looking for evidence, to see if there are ANY uses of that _cfclient query string in any of your web server access logs. And don't forget the tips about considering search tools, and running them "as admin" on Windows.
Blocking the _cfclient query string in IIS
Some readers may have learned over time that IIS has a "request filtering" feature that is enabled by default since IIS 7 and it already does some blocking of things Microsoft deems to be unsafe. The great news for us is that you can use the feature to add blocks for still more things. Whether you want to control things at the server level (the top level in the IIS UI's left navigation tree) or at a site level, the request filtering feature works the same (And if you block something at the server level, it's blocked at the underlying site levels--unless you override something at the site level.)
Again, this post is not where I can go much deeper into the topic than that. You can do web searching about the feature and find many docs, blogs, and discussions that include screenshots and more detail. I can only give you a push in the right direction here.
(Update on 3/18/23: FWIW, James Moberg offered a helpful tip, replying on the CF portal, for configuring IIS for this block via web.config xml entries instead. Indeed, I could have mentioned also making the same entries via applicationhost.config, or using appcmd. Again I was torn about going to such depth. Still, James makes a great point how the web.config approach can help those who can't access the IIS Ul. Now, back to what I'd offered originally, for those who can.)
So if you visit the "request filtering" feature at the server level, across the top are several tabs. Note that the last one is "query strings". Click on that, and on the right you will see a link for "deny query strings". Click on that, and enter _cfclient. As soon as you submit that, the change takes effect. If a client makes a request that includes that query string, the client will get a 404 status code and error message. (This is a bit of an oddity. Most web servers use a 403 status to reflect a rejected request.)
One more point: some will know that this "request filter" feature also offers a "url" tab in its options: but beware, despite its name that ONLY controls checking for a string in the filename or path of the request, NOT the query string.
Before moving on/declaring victory, you should go ahead and issue a test request to some page on your site, using this _cfclient querystring, before and after adding this block, to confirm that it's doing it's job. If you have a URL to page called your.cfm, simply change it to your.cfm?_cfclient. Or if you have a querystring, you can add it using &, as in your.cfm?yourquerystring&_cfclient. (It's not even necessary to "set" the _cfclient to any "value".) If you run that URL before implementing this block your web app might simply ignore it (as it might any querystring it wasn't designed to recognize: and to be clear, the _cfclient string without any content does nothing in terms of CF page processing itself.) If you try the request AFTER implementing the block, do you get a 404? If so, you're now "protected". Of course, check to make sure as well that nothing about your normal request processing has any problem. Since you made only the one change above, it would be easy to "undo".
Finally, I can confirm that even if the _cfclient string is presented as urlencoded unicode (%5f%63%66%63%6c%69%65%6e%74), it IS still properly blocked via this block on the query string _cfclient.
Blocking the _cfclient query string in Apache
For most Apache users, the most common way to perform blocking of a query string would be to leverage the standard Apache Rewrite module, which is at least implemented (readily available to enable) in all modern Apache deployments, though it's not necessarily enabled by default.
And if you already know all about it, you can skip to the specific rule I will suggest. But as I've helped people this week, many who are stuck with the responsibility of dealing with this issue are NOT that familiar with Apache, or at least not necessarily with using rewrite rules in general, let along blocking on the query string in particular. So I'm sharing here just a little more detail than I did about IIS, to help such folks.
First, to enable the rewrite module in Apache, you should find you can simple uncomment this line you should find already existing in your httpd.conf file:
Of course, before changing this file it would be wise for you to make a copy of the original, so that you can recover if you make some mistake in this or the steps to follow.
(And if you are not familiar with finding and modifying your httpd.conf file, there are details which can vary depending on your OS. Indeed, in some implementations there may be other .conf files included within the main httpd.conf. Here again, I have to shortcut the discussion and can't go into the depth which some may prefer. But this is again a topic you will find discussed amply online, as using url rewriting in some way is almost universal when working with Apache. See the last section here for how to reach out to me for direct help.)
Second, one has to then specifically enable the rewrite engine, which can be done via another directive in the httpd.conf file. (I should note that Apache is very flexible, and this directive like others can be set in this file, within a "virtual host" in this or included .conf files, at the directory level in such a .conf file, or even in a separate .htaccess file placed in the webroot of your site.) I'll just say that if you don't find it already existing already in those files, you could enable it in the httpd.conf file with this line:
(These changes will take effect when Apache is restarted or if you know how to tell it to reload its configuration. I will note also that it's when you do this that you may get errors if you've made any mistakes. Recall that I had you take a backup of the httpd.conf before you started changing it, so that you might restore it if needed.)
Third and finally, to add a block of requests that have this _cfclient string in their query string, add these lines (such as in your httpd.conf file, after the line above that enabled the engine):
RewriteRule .* - [F,L]
I can say that in a stock httpd.conf file which had been modified otherwise only to enable the CF web server connector, I simply added the 3 lines above to the bottom of the file and all worked as expected. Now restart Apache (or reload it's config if you're aware of how to do that for your OS and Apache version.) If somehow it fails to start, recall you made a backup of the httpd.conf file before changing it.
Again, you can find ample online resources discussing more about Apache rewrites, how and where to set them (which again can be at different "levels" within the .conf file/s, or in an .htaccess file.
Before moving on, do see the end of the previous section on IIS for my discussion on how you can run a test request using the _cfclient querystring, both before and after making the change. If you now get an error message (perhaps saying only "Forbidden") when you use this querystring, you're now "protected".
Finally, like with IIS, I can confirm that even if the _cfclient string is presented as urlencoded unicode (%5f%63%66%63%6c%69%65%6e%74), it IS still properly blocked via this Apache block on the query string _cfclient.
Blocking the _cfclient query string in CF's built-in web server
Some may wonder why I bring this up: the built-in web server is that one you use to access the CF Admin (typically, and by default since CF2016), such as via port 8500. Well, even though that port is protected from "outside access" by default in all firewalls, there's always the possibility that someone WITHIN YOUR NETWORK could learn of and try to perpetrate the bad-guy URL. It's possible.
And remember this whole section on blocking the _cfclient query string has been addressing those who either would not yet or could not apply the update (especially those on CF2016 or CF11). I'm simply pointing out that if you want to be complete in such protection (without the CF update), this is something to at least be aware of.
Sadly, it does not seem possible to add a block of a query string in the CF built-in web server. It's technically the Tomcat web server (and while Tomcat is an Apache project, the web server within Tomcat is NOT the Apache web server but instead formally it's known as Coyote--not to be confused with Catlina, which is the name of the Tomcat servlet container. But enough!) The issue is that the Tomcat/Coyote web server does not seem to offer a way to block requests via a query string.
If anyone knows or learns that it IS possible, please do let us know in the comments here (or reach out to me directly. My contact info is offered via a link in the side nav bar.)
VII. Some tips to address trouble some have had applying the update
We're nearing the end of the race. There are just a few tips I want to offer related to applying this update.
First, note that if you obtained the update within the first 24 hours of its release and had trouble applying it manually, re-download the zip or jar and try applying the update again. There was a slight problem in the internal structure of the folder within the zip and jar that caused that manual update process to fail.
Second, another matter about manual update (which is common to all CF updates) is that you may notice the update technote indicating you must "use the JRE bundled with ColdFusion" when manually updating CF (when using the java -jar command). Please don't over-think that, if you have changed CF to use some other JVM (as you should, keeping updated to the latest version of Java that your CF supports, which for CF2021 and 2018 is Java 11, and the latest version of that at this writing is 11.0.18, as I wrote about in January.) They are NOT really saying you MUST use the JRE that was bundled with CF. They mean simply that you would not want to use a LATER (or earlier) Java version that CF does NOT support, which is ANY version OTHER than Java 11, as I write. If you may have installed Java 17 or 19, for instance, for use with other tools, they would NOT want you to use THAT Java version in manually installing the CF update. What the wording SHOULD say is "be sure to use the java command within the JVM that ColdFusion is configured to use".
Finally, as with ANY CF update, do beware that if you are skipping any updates, meaning you are NOT on the immediately prior update (5 for CF2021, 15 for CF2018), then you should keep in mind that in applying this update, you will be pulling in also the changes from the update(s) you are skipping. That is just all the more important to consider if you may not have a test server (and process) on which to make first that there are no application issues in doing the update. Some CF updates (like this one) are very simple, but others may introduce breaking changes or additional installation steps. It's incumbent on your to read the technotes of the updates you are skipping, and I'll note that besides update 5 of CF2021, another "major" update was update 2. And as for CF2018, besides update 15, other "major" updates were 12, 8, and 4. The update technotes for CF2018 even specifically mention particular things to keep in mind if skipping updates 8 or 4.
Let's move on to the last section of things to consider, related to how you might improve your security posture against such hacks.
VIII. Additional protections you could consider to help temper such hacks
Here again, this is a topic that could warrant its own entire post, if not multiple posts. As much as I would love to elaborate on them here, it's just not the right place. I'm sure many are weary by the time they get to this point in the post--if indeed they do. I will say first that the Adobe CF Lockdown Guide (written by Pete Freitag) does go into several of these topic in more detail, as may the CF docs on some points, and as may broader security resources.
But at least in brief points, here are additional protections you could consider to help with mitigate the impact of such vulns and hacks.
First, you could limit what CF as a process is able to do, so that if a bad guy could perpetrate a hack they may at least be more constrained in what they could do. This could include:
- changing the default (on Windows) of the CF service running as "local system", changing to run as a newly created user with more limited access (not hard, but challenging for some to identify what folders that user SHOULD be able to read or especially WRITE to
- changing various folder permissions to limit them, such as not letting the cfusion/wwwroot be written to--but that's challenging because there are some folders underneath that, like the WEB-INF/cfclasses folder discussed above, where CF DOES need write access to that
- enabling the CF "sandbox security" feature, which enables even ore fine-grained control of what folders, datasources, and even external sites can be called from CFML code--even with different permissions for different applications. It can even limit certain tags/functions from being able to be executed, like cfexecute, cfhttp, cfdbinfo, and more. But it can be challenging to implement sandbox security for various reasons. I can help.
- implementing firewall restrictions that limit what sites/domains the CF process is able to call, so that it can only call out to what it is expected to call out to. Such a restriction would prevent hacks like this being able to call out to other servers either to pull down bad-guy shell scripts or export/exfiltrate data from your server to a hacker's server
- implementing a file change detection tool, to notify you or even BLOCK any attempt by the CF process or CF code to WRITE to an unexpected folder
- blocking the Windows certutil command , which is one way that a bad-guy URL could have unexpectedly caused execution of this tool (via cfexecute) and download files onto your server
Besides the CF lockdown guide, I will note also that the CF Lockdown Tool (available in CF2018 and above) can implement many of these and indeed ALL measures recommended in the Lockdown Guide. For some shops. A hack like this may be the "last straw" that pushes some orgs to consider using that tool. Some find it unduly onerous in its protections (perhaps breaking your apps or at least hampering your server administrators if either tries to do things that would now be restricted).
Finally, as for finding the CF Lockdown Guide, as well as finding other security resources and tools (including web app firewalls, security code scanners, security monitoring tools, file change detection tools, and more), note that I keep a list of them as a category of my CF411 site (of tools and resources related to CFML), specifically Security Resources and Security/Protection Tools.
I am trying to be careful not to go too far down the rabbit hole of security. The focus of this post is primarily this one vuln and hack and the updates--but it's such a potentially grave one that people either ARE more worried about it than other recent updates and vulns, or perhaps soon/now will be, so I felt I needed to offer the various elaborations that I have.
That said, it's certainly been a long post (perhaps my longest), and I want to wrap it up with some summary reminders and key points.
IX. Concluding key points
In summary, here's what folks should be doing about this hack and its resolution:
- If on CF2021 or 2018, apply the update released this week (updates 6 and 16, respectively. If you're seeing this in the future and there are newer updates, those would incorporate this fix.)
- If you can't do the update right away, or if you are on CF2016 or 11 which will get no update, you can add the block that I discuss above, for any request that has within its querystring the phrase "_cfclient", which is a key part of the "specifically crafted" hack url I have been discussing
- Before blocking it, you should look at your web server access logs (carefully), hoping you find NO prior use of that--neither legitimate (if something in your application may use that in its querystring) nor illegitimate (if the bad guys tried to use that to perpetrate the hack. I cannot tell you here any more detail about that URL used to perpetrate the hack, as that would expose it to be exploited
- You can look for other evidence that the hack may have been perpetrated, by looking in your cfclasses folder to see if there are any files there containing the pattern 2elog and then some numbers, or 2exml and then some numbers. If you do not (but you do see other files in that folder that are recent), then it's a strong sign that the hack was not performed on your server, as it would have left that fingerprint. Sure, the bad guys COULD try to delete that, but then you have the web server logs above to check. And sure, they COULD delete those or try to edit out certain lines. Sometimes we can't know things with 100% certainty in the ever-evolving spy-vs-spy world of hacking. I'm offering the best info I can given the situation.
Indeed, it should be stated that I offer this as educational information. There's no way I can know about the details of every server or accept responsibility for your performing these actions. You consider all this at your own risk. Again, I am putting forth a good faith effort to offer the best level of detail I can.
Beyond that, let me conclude with repeating a few other key points:
- I think this is an urgent update, on par with a zero-day, that everyone should apply. That said, there can always be challenges applying updates, so it's best to test in a non-prod env. Be careful, especially, if skipping to this update from some update prior to the immediately preceding one, as that means more is changing with this update than JUST the security fix it introduces
- If you were hit, the bad guys could at least look at any file that CF could access. In the worst case, they were able by the hack to put a file on your server with shell code that allowed them to then do about anything (modify files, query databases, offload data from the server).
- If you obtained the update within the first 24 hours of its release and had trouble applying it manually, re-download the zip or jar and try applying the update again
- Do NOT fall into the trap of thinking "it's about CFClient and we don't use it". It's that a cfclient-related capability if what was used to effect the hack. The update DISABLES that cfclient feature, so those who don't use it are ESPECIALLY the ones who should apply the update. If you DO use cfclient, there is a JVM arg for the update that can re-enable that feature
- There are additional protections you could consider to help with such hacks, such as limiting what operations CF can do, what files it can access, what servers it can call out to, detecting unexpected file changes, and more
- If you are still on CF2016, CF11, or earlier, beware that they no longer get any updates and you should absolutely be looking to get off of those. I have presentations that can help with that, or I can assist directly.
- Be aware that CF2018 will itself get its last updates in July of this year, 2023. See more in a post I did on that topic.
Sorry it took until a couple of days after the release of the update to get this out. It's been a fluid situation (helping others, both clients and in the community), and I've had to work hard here (and over the past couple of days) writing this up with a balance of "just enough" while "not too much" info on the updates, vuln, hack, and ramifications.
I'm sure I've disappointed some, whether with "still not enough" or "way too much" info (14 pages, if printed). It's the nature of the beast. I welcome feedback, of course, and especially if I have made any mistakes. When you write something over multiple days it's easy to lose some continuity, and even to plum forget something you very much meant to include.
Keeping notified of Adobe updates and security bulleting
Update I am adding this a few days after the post, as something I should have mentioned. Some would wonder, "how can I be notified when Adobe has security updates? or any CF update?" There are a few options:
- First, you can sign up to be notified by Adobe when there are SECURITY updates. FWIW, that is something offered from the general Adobe security page which is linked to from the bottom of all ASPBs, but you could be forgiven never thinking to click that link)
- As far as CF updates, those do appear in your CF Admin, but if you don't want to wait to "notice" that, you can configure the CF Admin also to have an email sent from your server to notify you of an update. See the "settings" tab at the top of the CF Admin page on updates, and the option, "If updates are available, send email notification to". Sadly, there's no verification button there to confirm things work. Make sure your CF Admin "mail" page is configured and verified to send email at all.
- Finally, still another option is the paid service from Pete Freitag of Foundeo, hackmycf.com. Despite the scary sounding name, it's a service I recommend all CF shops leverage, as it is configured to run from INSIDE your server to make sure you're aware of updates and issues about CF, the JVM, and other key server configuration matters
X. If you'd like direct assistance
Finally, if you want help applying the update, or dealing with any trouble you hit, or considering any of the above, I am available for remote troubleshooting consulting, and that page offers more on my rates, approach, satisfaction guarantee, and online calendar from which to find and select slots from as little as 15 mins to 2 hours.
While I can certainly apply the update to any CF instance in less than 5 minutes (with a CF restart), there can of course be reasons why it can take longer (but not much), in this case I'd also recommend we look for evidence of the hack which can also be done in as little as a few more minutes.
For more content like this from Charlie Arehart:
Need more help with problems?
- Signup to get his blog posts by email:
- Follow his blog RSS feed
- View the rest of his blog posts
- View his blog posts on the Adobe CF portal
- If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
- See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
I know ideally everyone would be on current versions of ColdFusion, but we work with multiple different clients that have different upgrade cycles , and some are waiting for 2023 to come out, so I am looking to support those as much as possible.
If it is not this code, and is someplace else in ColdFusion:
1) Does -Dcoldfusion.cfclient.enable=false work on CF 2016? I am guessing not, but just curious
2) Is there a way that if we are not using CFClient to just remove it from old servers?
Question...the CF Admin has an option "Enable mobile's server workflow" to turn on/off. If this is turned off, is the server still vunerable?
Matt, no, removing cf_scripts is NOT the solution to this problem and it would not help. And yes, I mean even if you or someone you know found evidence that that was used to perpetrate that "specifically crafted URL" I hinted at.
Let me be clear: it's NOT the files in that folder that ARE allowing the hack. So no, the bad guys could use other files. I will have to leave it at that.
Bottom line, it's the _cfclient querystring that is the key to the problem, not the files used in referring to it. I know I'm not giving enough detail for folks to figure things out: and that's the point.
As for the jvm arg, no. As I note in the post, that is NOT something you can "just do" in CF2016. It will mean NOTHING to CF 2016 or 11, or to CF2021 or 2018 before the update is applied. The jvm arg is for someone who APPLIES this update and yet DOES want to be able to use cfclient.
Finally, as for you asking how to remove it from old servers, I'm not aware of any way to. Again, this problem is inherent in how CF itself processed that _cfclient querystring, not some page that was called.
So someone on CF2016 (or 11) should use the other approach I offered, of simply blocking that _cfclient query string. I made that clear. I think you're trying to consider some potential corner cases or alternatives that I might have missed. I get it.
But please let me know if you now feel that you have all the info you need to proceed safely.
That said, nothing about the alternatives I offer should require an alternative. It would seem this would lead to the same result. I'm about to enter another client call so can't test it now.
I can only say this: of all of the products listed as having updates that day, only AEM is "like" CF (in being a Java-based application server). And what it lists are not the same cve's.
I suspect instead this was Adobe's version of a "patch Tuesday", where they released at once many updates for many products.
If somehow time or insights shared here might show that there's indeed some connection among them all, that will be very interesting.
...but they don't connect it readily to this one. Even so, I am fairly confident that what they are finding is the RESULTING bad-guy file that I discussed in this post as the second aspect of the vuln (the "arbitrary code execution", or their ability to cause placement of a bad-guy file on your server.)
Sadly, the Adobe technote and PSIRT/APSB bulletin as well as Mitre CVE's don't give enough detail to readily allow Rapid7 and others to connect the dots to this vuln and the update from last week. And they likely are unaware of my post here.
But I wanted to point this out in case others finding or following this post may "wonder" if that is some "new hack". I really don't think so. Once those folks dig further they should find that the root cause was this _cfclient querystring and the rest of the "specifically crafted url" (which I can't share, as it would allow perpetration of the hack).
Bottom line: get your CF2021 and 2018 updated to the March updates (6 and 16, respectively), or get the block of the _cfclient querystring var in place, all as discussed above.