[Looking for Charlie's main web site?]

A third Priority 1 CF security update has been released, Jul 19 2023

Just days after two P1 CF security updates were released on Jul 11 and 14, Adobe has released yet another on Jul 19.

Yes, this is shocking. Yes, unless there's a good explanation, I can understand how many would feel "someone on the CF team should be flogged". Don't shoot me: I'm just the messenger. I don't work for Adobe.

But I will add that in this post, besides just sharing news about the update (and more than JUST pointing to the update), I also offer an ADDITIONAL "fix" some will want to consider, to go BEYOND what this update addresses. See the discussion on "blocking the _cfclient query string".

Read on for more, where I cover:

  • Finding more info on this update
  • A suggestion on blocking the _cfclient query string
  • News for those doing manual offline installs: this update DOES have a zip
  • As for doing a Java update along with this update
  • CF2018 WAS indeed also updated

Finding more info on this update

As with the last two update (which I blogged about first on Tuesday and then on Friday), you can find the links to the specific updates (CF2023 update 3, CF2021 update 9, and CF2018 update 19) offered in the security bulletin, APSB23-47, as well as in a forum post from Adobe.

Of course, you can also see the update in your CF Admin, if it's set to "check for updates" or you click the button there for that.

Like the last two, this one also solves " critical? and moderate vulnerabilities?that could lead to?arbitrary code execution and security feature bypass.". And the bulletin adds that "Adobe is aware that CVE-2023-38205 has been exploited in the wild in limited attacks targeting Adobe ColdFusion.." There no further detail that I know of.

Though I will add that there have been various articles in security press about these recent CF vulnerabilities. One is this, from Rapid7, from the 17th (two days ago, so befoer the update today): Active Exploitation of Multiple Adobe ColdFusion Vulnerabilities. It was updated today to indicate that, "Adobe released a fix for the patch bypass of CVE-2023-29298 on July 19 and assigned it CVE-2023-38205. Rapid7 has confirmed the new patch works.".

A suggestion on blocking the _cfclient query string

Before leaving it at that, though (regarding "the vulnerability being fixed", one thing to note in that Rapid7 article is that they show the exploit being perpetrated using the same _cfclient querystring that had been at the root of the March 2023 CF Security update (update 6 for CF2021 and update 16 for CF2018--cf2023 had not come out yet, but does incorporate that update.)

Anyway, in my own blog post elaborating on that March vuln (and update), I noted how this _cfclient querystring arg is NOT something that most CF shops need to support.

To be clear, that has NOTHING TO DO WITH CF CLIENT VARIABLES. Instead, this use of a _cfclient querystring value was something added in support of (and intended to be used only by) the CF Mobile functionality (like the tag, cfclient) added in CF11. Virtually no one ever implemented that or is still using it. As such, there is for most people NO LEGITIMATE reason to let CF even PROCESS A REQUEST that includes that _cfclient querystring. (There is much more that can be added on that querystring, or sent with a POST, which is where the vulns are happening.)

Could CF just block it? Yes, they could but they do not. Perhaps soon they will consider that. Until then, could you? Should you? I would argue that yes, if you don't want to leave it to waiting for Adobe to find and fix each little vuln that is perpetrated using this as its starting point, you should just block it. (If you may have legit uses of the cfmobile feature, then no, you cannot just block it.)

In my post from March, I discussed this in more detail, in a section of the post I labelled"How to protect yourself if you need time to get the update deployed, or are on CF2016 or 11?". And there I discuss both how to search your web server logs for any uses of that querystring (such as to determine if it's used regularly by your app, or only in attempts to break in), as well as how to block use of that querystring in the common web servers CF supports, IIS and Apache.

I can help directly with this, of course.

News for those doing manual offline installs: this update DOES have a zip

For those doing a manual offline install, those will want to hear that UNLIKE the last two (the ones for CF2021 and 2023, specifically), Adobe is offering in this update the zip (holding a complete repo for use by the update mechanism and/or CFPM tool), where the previous two updates only offered the jar.

Some people were encountering challenges doing the past two updates when they were running their CF on a machine that had no internet connection. Things SHOULD go more smoothly with this update. See the update technotes for info on handling such a manual offline update, for all 3 CF versions (but again this zip approach applies only to CF2021 and CF2023, not CF2018.)

As for doing a Java update along with this update

Two points to be made here, one new since the last two updates...

First, of interest to some readers: note that this newer APSB security bulletin (like the last one) does change its wording (compared to the first one last week) about the need of a Java update. It (correctly) no LONGER suggests Java 17 (which as I discussed in my first post was INAPPROPRIATE for those on CF2021 or 2018). It now reads, "Adobe recommends updating your ColdFusion JDK/JRE LTS version to the latest update release.". It then goes on to say "Check the ColdFusion support matrix below for your supported JDK version", and offers links to those.

I discuss more about this matter in my post on the first update last week,

Second, note also that since then a new JVM update has come out, July 18, and I posted on that last night. In there, I also clarify what JVM versions are supported by what CF versions. (And yes, crazy that we have to deal with a Java update as well. At least those are scheduled and quarterly.)

CF2018 WAS indeed also updated

Finally, as for CF2018, while the first update last week was to be its LAST update (as its end-of-life date was in fact July 13), Adobe did kindly offer a CF2018 update for both the update on Friday the 14th as well as today, the 19th. Unless there are still more in coming days, we really should expect THIS update to now due to be the last for CF2018 (and folks should be getting OFF of CF2018. It's not new information. That last blog post I shared was from January., and Adobe publishes the end of life date, at a page offered as a link in that post.

Conclusion and other thoughts

Again, if you are finding this update (and my post) and you had NOT yet read the post I did on Tuesday the 11th, I recommend you do. It will give you some useful context. Same with the shorter post on Friday the 14th. Check out also the comments there, though I do try to keep the posts updated with any critical info.

As with those first two, I'm afraid there's no further information I can offer than what's offered in Adobe's resources on this update. I'm just posting this for the sake of readers of my blog. (Some people may not log into the admin or setup any of various ways to be otherwise notified when an update is released. I am planning to do a post on helping you with other options there, if I could come up for air!))

But like the last ones, if I learn anything interesting, I will update this post (or add comments). And as always, if you would like help preparing for, implementing, or dealing with problems after performing these updates, I am available for online remote consulting. More on my rates, approach, satisfaction guarantee, online calendar, and more at carehart.org/consulting.

For more content like this from Charlie Arehart: Need more help with problems?
  • If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
  • See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
Comments
Is this going to be a daily occurrence going forward from Adobe...? Goodness, this is rather embarrassing. :(
# Posted By Tom Woo | 7/19/23 1:14 PM
See my second paragraph, for my sentiment. I share your dismay.
FYI, the link to the zip file for Update 3 for ColdFusion 2023 is broken.

It should be:

https://cfdownload.a...

If anyone has a faster method of letting Adobe know than I do, feel free to reach out to them & let them know.

Cheers,

Danny
# Posted By Danny Gunter | 7/19/23 1:31 PM
Danny, I assume you mean the link in the cf2023 update 3 technote (as I have no link to it here). Just clarifying for others.

And note that while the link in step 1 of the offline install has it wrong :

https://cfdownload.a...

The link in step 2 is right (has 2023 vs 2021 in the url):

https://cfdownload.a...

I've never understood why they offer the link in both steps there.

Anyway, you should report it either in the Adobe forum post I'd linked to above, or at tracker.adobe.com, or by emailing [email protected] or [email protected].

This is me "teaching to fish", so I'll leave it to you, given your sincere (and correct) concern about the matter. Trust me, I have no inside track, and often paying customers get better response to some things than I do.
Charlie,

That is correct. I was referencing the "link" to download. I didn't even notice the "Unzip" one LOL.

I emailed CF support per your suggestion.

Thanks for teaching me to fish!

Cheers,

Danny
# Posted By Danny Gunter | 7/19/23 2:06 PM
And thank YOU for being open to learning.

Sadly, some folks really don't take kindly to even a sincere effort to teach them, if they've not asked--and that could be a root of some wider societal problems, I think.

Anyway, such people's heads must explode with my posts (and replies in the community), so they probably don't follow my blog. I write for everyone else. :-)
I'm all about security updates and resolving issues. It would be embarrassing if they did nothing about it.
Jakob, could you please clarify whose comment (or what part of my post) you are referring to? It's just not clear from what you said. Is the "they" Adobe, I assume? What is the "it", if they did nothing about it? Sorry if you (or anyone else may) think it's obvious.
Charlie - Yes, sorry, I only saw the first comment when I was reading a bit ago. I was replying to that comment, referring to Adobe taking the initiative to get fixes out fast being a good thing. Some people see it is a cup half empty, I was implying it's half full.
Ah, OK. Thanks. And agreed. :-) Though some may still reasonably question why these have been so numerous and within days. Seems like some more fundamental problem could maybe be solved. Sadly, we never get enough detail to know.
I have issues with the install via the website (file permissions with windows, failures with copying and removing files). as a workaround I do the JAR file install. since the servers are connected to the internet -- the zip file download should be unnecessary right? the packagesurl in the neo_updates.xml file still points to the adobe download for the packages. I am hoping that works correctly.
# Posted By MRC | 7/19/23 8:27 PM
You have a couple of choices.

First, you COULD extract the zip, and find the jar in its bundles/updateinstallers folder, then run that via java -jar.

Or what you regard as file permissions issues may instead be simply that you set the cf service to run as a limited user (like "cfuser", rather than the default of local system), but then the update process (launched from the admin) can't stop cf...so the update fails.

I've been meaning to do a post about this, though it's not new. The solution is to get the free Service Security Editor tool (from Core Technologies) and use it to tell Windows that this cfuser CAN stop/start all the cf services. Then run the update.

Look for a post to come with more details. Hope those help in the meantime.
it seems to work - the jar file update runs without any issues. the cfadmin shows the new core version as 09 and the packages don't show any updates are needed. it would be nice to know whether that was in fact correct. what might be helpful is instructions on how to edit the neo_updates.xml file to point to a downloaded file (from the zip) instead of using a URL for those cases when the server can't have an outside connection.
# Posted By MRC | 7/19/23 9:27 PM
Adobe doesn't post any information (that i can find) on how to edit the neo_updates.xml file for the case of doing the install as I described (entirely without any connection). the least that they could do would be to post an example of how to make those changes.
# Posted By MRC | 7/19/23 9:32 PM
Your result is correct. These most recent security updates (9, 8, 7, 6--and the same for CF2023 updates 3, 2, 1) have no package updates. The install log tell you this ("All the packages are already installed") vs updates like cf2021 update 5, which would report (for any package updated) that the "package and its dependencies have been downloaded successfully". And then it's the next launch of CF when the packages ARE updated.

As for your last point, well, that's what the update technotes DO say (at least for today's updates, like u9 of CF2021), saying "Update "packagesurl" in cfusion/lib/neo_updates.xml of cfusion and all its child instances to point to"..."bundlesdependency.json present inside the downloaded folder."
would it just be a file path?

default value
<defaultpackagesurl>https://www.adobe.co...</defaultpackagesurl>

local file
<defaultpackagesurl>c:\downloads\bundles\bundlesdependency.json</defaultpackagesurl>

would that be correct? or does the XML tag need to change as well?
# Posted By MRC | 7/19/23 9:58 PM
MRC, yes, but please, rather than ask me about this, just check out the docs I mentioned. More specifically, see "INSTALL THE UPDATE IN OFFLINE MODE MANUALLY" section at https://helpx.adobe....

But I would agree those docs could be improved, in many ways, some of which I've mentioned in past posts.
It's already too late for these updates? I'm seeing VERY suspicious activity in IIS logs, even before CF2021 Update 7 was released.

I didn't see anything suspicious before Update 6, so naturally I assumed that this issue was fixed in Update 6.

Now there's no way to tell if our servers have been compromised?
# Posted By Jakke | 7/20/23 10:27 AM
Charlie, the problem is that the offline instructions don't really show an example. it's not much to go by.

INSTALL THE UPDATE IN OFFLINE MODE MANUALLY

Download the hotfix installer from the link.
Unzip the repository to a place where all ColdFusion server instances can access it.

Update "packagesurl" in cfusion/lib/neo_updates.xml of cfusion and all its child instances to point to <InstallerReposityUnzippedPath>/bundles/bundlesdependency.json present inside the downloaded folder.
# Posted By MRC | 7/20/23 1:13 PM
Hello,

Has anyone experienced issues with CF2018 Update 19 and using JVM 11.0.20? I tried upgrading from CF 2018 Update 13 and JVM 11.0.13...

When I tried the manual .jar install using 11.0.20, I received the this error:

Error: An unexpected error occurred while trying to open file hotfix-019-330149.jar

Using the 11.0.13 JVM worked.

Once installed (with no errors), now my sites are just showing blank pages, however the CF Admins work fine.

Thanks in advance,
Shaun
# Posted By Shaun Smalldridge | 7/20/23 2:13 PM
Yes, others are having the problem. (I've been too busy with helping folks about other things today so have not been able to test it out or find a good solution.)

To be clear, it worked fine with 11.0.19, so no need to go back to 11.0.13. And if that's where you WERE, go up to 11.0.19 and you'll at least be on what was the latest as of MONDAY of this week. :-) And again, it WILL work to download the CF updates.

Hope to have news soon on solving this. I've found and documented similar problems in past JVM updates, such as 11.0.11 and 11.0.17. I'm sure there's a similar explanation and solution. When I find it, I'll mention it here and in my post on the new JVM update from Tuesday.
This wording by Adobe is vague "We have also updated the lockdown installer for the version in this update. This addresses a security issue. To apply this fix, you should download and install the lockdown installers from Download ColdFusion." Does anyone know if we need to reinstall the lockdown tool if it was already installed with the prior version???
# Posted By Roberto A. | 7/20/23 4:38 PM
Roberto, no one "knows" for sure, if or why one would have to reinstall the auto lockdown tool. I asked this in the first post on the first update last week. No one from Adobe ever replied.

What I can say I do know is this: when Adobe has been asked (directly, by clients of mine), the answer has been "yes, reinstall it". But of course that's a PITA, and it's unclear if doing that "improves" anything. Would it cause the new CF (and Java) update to be applied if it was not already? Sure. But one could (and some would) install that separately, even if they had applied the autohotfix tool.

So one could ask Adobe to consider two shops. Shop 1 had installed the most recent prior autolockdown tool and chooses not to reinstall with the updated version), so they ONLY apply the latest CF and Java update. Shop 2 goes ahead and reinstalls the new autolockdown tool. How will the two differ? What will Shop 2 get by their effort that Shop 1 does not?

If anyone may do testing, folks would love to hear.
I assume we can just apply update 19 since it is cumulative and we do't have to apply update 17 or 18 first, correct?
# Posted By JD | 7/21/23 8:54 AM
JD, yes. They're always cumulative and you can do just the latest.

The few times in the last 10 years that Adobe said otherwise, it was merely because of technical challenges downloading/installing in the cf admin ui. None of these have that, to be clear.
The public exploits for the recent WDDX deserialization vulnerabilities (CVE-2023-29300, CVE-2023-38203, CVE-2023-38204) *have* included the _cfclient string in the URL parameters, but this is not actually required for exploitation. So while blocking requests with "_cfclient" will likely block lots of automated scanning, it would still be possible to exploit an unpatched system. Blocking "_cfclient" still may be a good idea to stop a lot of automated junk, but it's not complete protection here.

All CF versions that deserialize WDDX are probably vulnerable -- so that would include something very old and unsupported like CF10. Any accessible CFC with a remote method is a pathway to exploitation of the WDDX deserialization vulnerabilities, since all you need to do is reach the GetArgumentCollection() call in coldfusion.filter.FilterUtils.class. I've done some testing and code review, and haven't found any other user-accessible ways to hit GetArgumentCollection() and control the WDDX object to be deserialized, such as calling a UDF with a user-controlled variable from a CFM or a via cfinvoke. Not a guarantee, as I certainly could have missed something.

Exploiting CVE-2023-26360 (from APSB23-25/March 2023) *does* require the _cfclient parameter, as the vulnerable code path is only reachable by a very flow that does require CF Client functionality. When creating the exploit for CVE-2023-29300 (CVE-2023-38203, really) ProjectDiscovery based some of their work on Rapid7's analysis of CVE-2023-26360 -- which is why I think the PoC for CVE-2023-29300 included _cfclient.
# Posted By Brian | 8/1/23 6:13 AM
Brian, thanks for the elaborations. Sadly, there's been precious little offered on all this, especially with respect to what people can do on cf2016 or earlier. So more light is welcome.

And to your point here, I'll add first that I've tried to clarify (where I mentioned the _cfclient querystring) that the references to that (from resources other than Adobe) may relate to only SOME specific one/s of the many vulns addressed by the 3 July CF updates.

It's a tough situation. You helpfully mention the relationship of one or some of the vulns to wddx deserialization, and that may seem tempting to folks, who'd wonder "how can we just turn that off?" Well, no, sadly we can't. There's not even a means in the cf security sandbox feature (which is used by a scant percentage of folks).

And even if we could, the entire cf admin and other features rely on reading and writing cf config files which are xml files saved in wddx format (an xml schema created by Adobe 25 years ago). All the neo*.xml files in the cfusion/lib folder are an example of this.

So again, we can't just "turn off wddx", but then it's not clear if we would have to: as you note, it's exploitation via remote method calls that's at the heart of what's happening. Can we somehow block just that? That's not yet clear.

I will add that I've been given some additional information (from a source asking to remain anonymous) that MAY be at least one MORE thing that folks on cf2016 and earlier could do, related to this matter of wddx deserialization. Again, it's not yet clear what might be "broken" by adding this new block. (Even if it may too wide-impacting for some, others may accept it as their "only choice".) That's why I've not yet shared word of it. I want to be careful how I present it, and I'm doing more testing.

It is all a very unfortunate situation. Of course, it's understandable that Adobe may hold firm to saying "we stopped supporting cf2016 in 2021,and cf11 in 2019, and cf10 in 2017". I don't see them creating a new out of band update for those. But perhaps they MAY come out with more clarity of what can be done for such folks, given the severity of this. Though even that is risky (for various reasons) and unlikely. I mean, look how little info they offer even for those on the supported versions.

As you know, of course, Brian--and others should keep in mind--it's dicey for a vendor to offer public info on vulnerabilities. They have to be VERY careful, and it's a fine line between too little and too much info. So it's a blessing we have folks like you and Pete (and others) stepping in to fill that void.

Let's see what the next steps may be, whether announced here or elsewhere. BTW, folks may be interested to hear that Brian has a blog where he writes (sometimes uniquely and extensively) on cf security matters at https://hoyahaxa.blo..., though as of this writing there are no posts since early July, before the 3 recent cf updates. And of course Pete Freitag has long been the cf community security maven, though he too has not blogged on this (last post in May, at https://www.petefrei...), though Pete HAS of course kept users of his https://hackmycf.com... service informed about the cf updates.
Thanks Charlie. At this point, I don't plan to blog about the WDDX or cfclient vulnerabilities unless I discover anything materially new or significant. (Fortunately, only CFCs with remote methods appear to be the primary ingress point.) I probably will blog about some of my recent vulnerabilities that were fixed by Adobe, but it may be awhile until I get around to that. :)

As far as advice or what can be done for older (unsupported) versions of CF, I'll share here what I tweeted two weeks ago in the midst of this -- https://twitter.com/...

"You should *not* expose any unsupported versions (i.e., no security patches) of CF to the public Internet. Migrate to a supported CFML platform *now*. Many of these vulns affect older, unsupported versions of CF too. CF2018 was EOL on 7/13, but it received APSB23-47 patches.

ColdFusion can be secured, but you actively need to take the steps to secure it. Patches are step zero. ColdFusion (like many other platforms) is not alone in being nearly impossible to secure if you do not have access to the latest security patches. Run the lockdown tool, remove unnecessary components, restrict access to sensitive components.

History has shown that some critical CF vulns have had very long tails. Internal exploitation and lateral movement is always possible too, but the high volume of activity makes Internet compromise against vulnerable systems an almost-certainty."
# Posted By Brian | 8/1/23 9:03 AM
Amen on that second (elaborated) point, and ok on the first. Thanks for sharing both here.

And lest any reader think I don't share his admonition to "get off old, unsupported cf versions", I do indeed.

But I'm like a doctor in an er. I can say "if we don't amputate that infected foot you'll die of sepsis", but some people will refuse. We all know what's inevitable.

Some may liken it instead to the doc telling people not to smoke, but a person points out their 99-year old granny who smoked to her last days. OK, they may want to take their chances--like someone who feels their server's not been hacked in all these years, so why go through the hassle/expense of getting off that unsupported cf version now (even if to Lucee, so "zero cost")?

Of course, if we did exploratory surgery we may find the "cancer" (result of an exploit) IS there, and has gone undetected for days, months, nor years. But some people won't want to pay for/go in for that exploratory surgery. And just like in medicine, even such exploratory efforts may not readily spot all cancers/results of exploits. Some take far more effort to find, diagnose, and cure--even for someone willing to undertake the needed efforts.

Again, most of us can see what's almost certainly inevitable in this case, but all we can do is counsel folks--and even then only those who seek or find our counsel--or help those who seek additional assistance.

Put another way, we can only "lead the horse to water,but can't make it drink". But you've certainly fired the warning shot that may get the attention of some of the more stubborn bucks in the herd, if folks will forgive the stretched analogy. :-)
Does anyone know if the QoQ custom hotfix hf201800-4212383.jar still needs to be reinstalled after Update 19? I suspect this is the case, but one can always hope. :-)
# Posted By Shaun Smalldridge | 8/2/23 9:51 AM
I believe so, yes. Unfortunately I'm traveling at the moment so cannot confirm. Maybe someone else can, or I will later.
Copyright ©2024 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting