[Looking for Charlie's main web site?]

Updates released for ColdFusion 2023/2021, Mar 12 2024: possible breaking change, solutions

This is a very important heads-up for my readers: there was an important security update released today by Adobe for ColdFusion 2023 (update 7) and 2021 (update 13). While as always there's much to say about what's changed in this update, I want to make this important clarification:

It's very important that people read the technote before "just applying this update". There is a very important (and fundamental) change in how CFML processes variables, with regard to searching for scopes when no scope is indicated on a variable name. It's NOT that you "must scope all your variables", as some are asserting. But it's still almost certainly a BREAKING change in many CF apps, if they use unscoped variables under certain conditions (that I discuss below). The change is for the sake of security, but it's just one aspect of the security fixes in this update.

Anyway, there are 3 things you can consider doing to rectify/work-around this breaking change, as I discuss below (or see the update technote, for this and more). And you may reasonably wonder what the implications would be of using the workarounds. You may also wonder if this scope matter relates to the CVE listed in the APSB (linked to below). That's currently unclear. It does not. As well, note that the Adobe security bulletin (link below) shows the security fix to be only a P3 (priority 3, the lowest severity), not a P1 (priority 1, the highest), though it IS regarded as "critical".1

But then there are still other aspects of the update beyond this scope matter, and you should be aware of those also.

For more, read on.

[....Continue Reading....]

Comments
Hello Charlie, I hope setting the aforementioned JVM argument to true won't negate any part of this security update. Thanks for your very helpful writeup.
# Posted By Sergei L. | 3/13/24 8:32 AM
Yes, it WILL negate the specific aspect of the update regarding implicit scope searching. So also will changing it in application.cfc/cfm. It's a workaround, to be used responsibly. I have tweaked the post soon to make that point more clear. Thanks.
Not to the point of re-opening CVE-2024-20767 Arbitrary file system read critical vulnerability, hopefully.
# Posted By Sergei L. | 3/13/24 8:54 AM
I can't say, nor can anyone other than Adobe or the folks named in the apsb. There's no detail there, nor any as we search now for that cve. All places I find it mentioned indicate that there is no detail released currently.

But to be clear, in my pointing to these workarounds, I'm not saying anything the technote doesn't also.

Those with the concern you raise should ask Adobe. They don't necessarily (and don't see to) read my blog, so we should not necessarily expect them to see this and offer an answer here.

If anyone finds an answer or more detail elsewhere, please do share.
Thanks for the write-up, as always. It's been a rough year for CFML security updates - they seem to be happening every other month these days. But, I suppose it's a good thing that they are moving things forward.

I use CommandBox to do my local development, and it looks like they haven't pushed a new image yet for their `adobe2021` tag. I want to be able to test my sites locally first before I have Hostek apply the update. I suppose I could YOLO it, and just have them update (since I have `searchImplicitScopes` set to `false` everywhere already. But, I see there are Tomcat updates and what not that I would feel better about verifying first.
Ben, thanks for the kind regards. And first some good news: it's really not been "every other month". :-) The last one was in November, so 4 months ago. But sure, there was a major update in Oct, then before that 3 updates over the summer. That is what felt like a bombardment. But as you allude to, security trumps convenience these days.

As always, I'm just a messenger, one person trying help others, as indeed you so ably (and frequently) do. And we appreciate it!

As for Commandbox I can only say that they're always on the ball and a new one may already be in the oven. I don't see an update yet on Docker hub for them or Adobe (Adobe's forum thread said new images were in process).

Until then, yes, folks can just update their images like other folks using cf would (even if it feels icky on a container). Or yep, as I'd noted above, anyone can get a head start BEFORE doing the update by setting searchimplicitscopes=false in their application.cfc/cfm, to force the change in behavior that will become the default with the update. That will help them see if/when unscoped vars would be a problem (again, not ALWAYS).

While they could start updating their apps to use searchimplicitscopes=true as a "workaround", that should be carefully considered. First, the new jvm arg could be used instead (after the update). But EITHER workaround should be weighed against unlocking the very door Adobe was locking for the sake of security.

Here again, convenience (compatibility) vs security needs to be weighed carefully. This is the new reality (secure by default) being imposed on us from many directions: in CF, in IT generally, on our devices, and even in our daily lives.
For sure, I didn't mean to sound alarmist. Only, it seemed that there was a relatively quiet period for years; and then, a much increased cadence of security releases.

For now, I'm going to just wait for CommandBox to release an update. I tried to update via the CF Admin (within my Docker container) and it said that this particular update can't be done via the Admin since it changes the underlying engine (I forget the exact wording).
Yarrr, I think I accidentally unsubscribed via link in email. Just leaving a comment to be re-subscribed.
No problem, sir. :-) And thanks for the additional context.

Yep, Commandbox is itself its own environment, and just as doing cf updates might be frowned on in containers, it seems there are times when they're precluded by Commandbox.

Perhaps soon we may hear from Brad or Jon, on either or both matters (issues updating cf ourselves within Commandbox, or news on a new image).
Thanks as always for the write-up Charlie. I updated this morning locally and the first app I loaded had a variable error. I used the jvm switch to get around it but we have a pretty large codebase so I am not looking forward to the updates we will likely have to make.

I wasn't sure from the technote which "workaround" would go away so if I read your assessment correctly I will probably try setting "searchimplicitscopes=true" in our main Application.cfc since it seems that that will be support even when the JVM switch is no longer supported.

Thanks again! Always grateful that you have such a great place for CF information since it sometimes feels like a lonely CF world.
# Posted By Steve C | 3/14/24 9:39 AM
Thanks Charlie much appreciated. I am looking for a tool that could search my code base and identify unscoped variables. Some quick googling and found some references to a tool "varscoper4" and a few others but they are all very dated it looks like. I also saw Pete Freitag looks to be possibly considering adding this "unscoped variable search" to the code security scanner Fixinator.

Does anyone know of such a tool like that, that could be used now?
# Posted By Walt T | 3/14/24 10:24 AM
SteveC, thanks for the kind regards. To your question, I addressed that when I said "you will NOT be able to rely on that 3rd option beyond CF2023". The 3rd option was the jvm arg. The 2nd option was the application-level setting. I will add yet another sentence, if that may help some readers. But your question and this answer should also help those who may more likely see this for some reason. (I know it's all a lot to take in.)

Finally, as for your feeling like you're in a "lonely cf world", PLEASE see that last link I offered at the end to several available community help resources: you will find issues like this and many more discussed every day, whether you prefer email lists, web forums, slack channels, facebook discussions, linkedin discussions, and more.

You are definitely NOT alone. You are rarely the only experiencing some problem. And you need NOT rely solely on google searches or AI engines to find answers. And of course, some simply prefer talking to real people (flawed though they may be, in their knowledge and sometimes even in their online personalities!)
Walt, I am NOT aware of any such tool.

So first, yep, for those who pay for Pete's Fixinator tool, that will be a great addition (and certainly yet another reason for folks to consider it).

And FWIW, I can confirm that Adobe's Security Code Analyzer does not currently flag code with unscoped variables. (I just proved it with a test.) We can hope they will consider adding that, if they are not perhaps already working on it. And I've just filed a feature request on it: https://tracker.adob... Interested users of that tool should add a vote there. I'll also update my post to reflect this.

Finally, sure, someone could take the old varscoper (which was looking for a very specific problem) and try to modify it.

But let me note (as I do in that tracker ticket, and above in my post): it's not that ALL unscoped variables will fail with this change in behavior (or by setting searchimplicitscopes=false yourself). Any tool that flags ALL unscoped variables will produce MANY false positives (just like many are overreacting to this change, thinking they must change ALL unscoped variables).

Instead, it's that unscoped variables are still ALLOWED: they will only search those scopes in that first list I gave above (like variables, local, arguments, etc.). The change in behavior (searchimplicitscopes=false) will NO LONGER ALSO SEARCH scopes that can be manipulated from outside the request (like url, form, cookie, etc.)

And a tool deciding to flag that distinction is going to have to be pretty smart. But sure, some will accept just being told of ANY unscoped variables. That's just overkill. Again, lots to juggle here.

That's my current take on things, at least.
The JVM Argument approach works for me, but for some reason the application.cfc method does not seem to work for me. Thoughts?
# Posted By Vicki Pruskowski | 3/14/24 3:10 PM
Hi, Vicki. Long-time, no talk. So in what way do you mean it does not work?

And I assume you mean you're setting it to true on the jvm arg, then you're trying to set it to false in an application.cfc, perhaps to tighten things down in apps that don't need to be loosemed?

If so, are you finding that despite being set to false, it acts as if it's true?

As for the setting of this searchimplicitscopes, are you doing that inside or outside of any method in application.cfc? It should be outside.

I plan to do a post tomorrow with more on that. It's not new or unique to this setting, that it does not take affect except when set OUTSIDE of any methods in the application.cfc (the"pseudo-constructor" of the cfc).

But let's have you clarify more before I go further with guesses and solutions. :-)
Currently, as of 9:13 GMT the URL 'cfmodules.adobe.com/bundlesdependency.json' is missing the update 13 packages. The update of my live system to 13 this morning via CFAdmin applied the core update but none of the packages, it removed the existing packages which have been flagged for update by Adobe. After re-installing the Administrator package via CFPM I noticed it installed version 11 and not 13, this is when I checked the packagesurl and noticed its missing any reference to update 13. This worked fine in my Dev environment on Wednesday (13th). For whatever reason it looks like the online version of bundlesdependency.json is incorrect.

I also noticed when viewing the Update 13 URL from any other region than US (uk for example), the links to the hotfix installer and zip repository go to hotfix version 12.

UK page - https://helpx.adobe....

When viewed from the US, the links go to version 13 as they should.

US page - https://helpx.adobe....
# Posted By MaxUK | 3/15/24 4:51 AM
MaxUK, thanks for the details. This sort of thing has happened before, about the entire update--as I mention in the post, and have found to be a caching issue. (It's not clear where the caching issue lies, whether on Adobe's end, the user's end, or somewhere in the middle).

And sure, it could happen with the package updates specifically since they are a separate file (and url). Something interesting is that they only change in some but not all cf updates; which means the issue of an "old"/stale/cached package update jar happens LESS FREQUENTLY and isn't noticed by as many as the same problem with the primary update jar.

Not sure what any of us can do, as users. Perhaps Adobe could implement some sort of additional checking for/protection against the package jar being found to be out of sync with the version of the update jar when the latter is run.

Until then at least they DO report in the technote what version number to check was implemented, both for the update and the packages (when an update introduces package changes). But most people don't pay attention. So again, good for you to spot it, and good of you to bring it to attention here.

Would you consider opening a ticket at tracker.adobe.com, to report the problem and suggest the possible solution? If you do, please share the link here so folks can add a vote.
Readers here may want to note that I just found Max had raised his issue both here and on that Adobe forum thread about the update (that I linked to in the opening here). He raised it there about the same time as here, a few hours ago (which is OK by me).

Anyway, Adobe has responded there (looking into things). Here's the direct link to his comment there, and the threaded replies from Adobe follow it:
https://tinyurl.com/...

BTW, some may notice he's listed there as Adam36099131ak4s, but the message is the same, so I'll assume he just had that existing Adobe account name but uses a different pseudonym in other places, like this.
Hi, yes thats an old Adobe account I have :-) The caching issue seems to have settled a little so I'll cross my fingers for tomorrow morning when I attempt to re-apply the update to my live environment. I've downloaded the version 13 .jar and the packages .zip from the US site so will run a manual install if the same problem crops up again. I also logged the issue with CFSUP this morning so they're actively looking into it.

By the way, thank you for producing this fantastic CF resource, its helped me greatly over the years!
# Posted By MaxUK | 3/15/24 9:47 AM
Max/Adam, thanks for the update, and the kind regards. :-)
Had exactly the same issue as MaxUK today. Testing the update locally and it worked fine, did the update on the live server and it broke. CF Admininistrator disappeared, and a couple of other packages uninstalled themselves.

If I compare the 'bundlesdependency.json' file downloaded on my local server and my live server, the live one has no reference at all to any update 13 packages (searching for "2012.0.13") whereas the local one does.

When viewing the following URL about update 13 (uk version of the page):
https://helpx.adobe....

The manual installation instructions seem to be instructions for installing update 12.

If I view the US version of this page:
https://helpx.adobe....

The links in the manual installation instructions correctly link to files for update 13.
# Posted By Andrew L | 3/18/24 4:00 AM
Andrew, thanks for the details. I can confirm seeing the same problem of that UK version of the technote showing info for 13 but links for 12, for doing the manual update.

But about the first problem, help us out: did you apply the update via the cf admin, on both servers? Or this manual update?

And in the update install log (within hf-updates), does it show having applied update 12 or 13? With 0 "fatalerrors" reported in that log?

And then in the coldfusion-out.log and coldfusion-error.log, what errors are shown during the first startup of cf after having applied the update? That's when updates to the packages actually happen.
Hi Charlie - I was trying to set searchimplicitscopes to false so I could address any issues, but it wasn't working because I didn't realize it could not be within any methods. :-) THANK YOU!
# Posted By Vicki | 3/18/24 7:49 AM
Vicki, glad to have helped. I'd not gotten that new post out yet. I hope to at least tweak this post to add the clarification, to save folks having to read to here. :-)
Hi Charlie,

I applied the update on both my local and live copy via the CF admin. Locally it worked fine, on live the CF admin refreshed and then displayed a message about the adminapi module being missing.

After enabling this manually my site still wasn't working, the coldfusion error log showing errors about the 'document' and 'pdf' modules not being enabled.

I apologise for the lack of supporting information in my previous post. If I look at the install log for the update, it shows the following (I think these are the relevant bits of information):

java.class.path:
/opt/ColdFusion/bundles/updateinstallers/hotfix-013-330286.jar

Installed Feature(s) Standard, J2EE of Adobe ColdFusion 2021 Update 13

Install Begin: MARCH 18, 2024 6:15:46 AM GMT
Install End: MARCH 18, 2024 6:16:28 AM GMT

Summary
-------

Installation: Successful.

1625 Successes
0 Warnings
0 NonFatalErrors
0 FatalErrors



If I look at the 'bundlesdependency.json' that was downloaded and used for this update, it contains no mention of "2021.0.13".

The errors shown in my coldfusion-error.log for the first startup of cf afterwards are:

SEVERE: Servlet.service() for servlet [CfmServlet] in context with path [] threw exception [Servlet execution threw an exception] with root cause
coldfusion.server.ServiceFactory$ModuleNotAvailableException: The document package is not installed.


In coldfusion.out I only see this:

Mar 18, 2024 06:16:42 AM Information [main] - ColdFusion started
Mar 18, 2024 06:16:42 AM Information [main] - ColdFusion: application services are now available
Mar 18, 2024 06:16:44 AM Error [ajp-nio-0:0:0:0:0:0:0:1-8120-exec-10] - '' The specific sequence of files included or processed is: /var/www/*redacted*/htdocs/index.cfm, line: 615
# Posted By Andrew L | 3/18/24 8:05 AM
Andrew, you're not the only one to report this, but you can solve things.

Frist, let me note that I have NOT seen it on all the servers I've applied updates to--except one, yesterday. And for them, we did not take time to dig into WHY it happened (they just needed the update applied and their site working ASAP).

Like you, we dropped to the command line (as admin) and went to the cfusion/bin folder and ran cfpm install administrator. With that, despite an message suggesting we'd need to restart, we were able to get into the CF Admin.

Then we went in, and we saw that in the "Package Manager" page (second to last on the bottom left), in its "available packages", there was several which were not installed--including document and pdf, like you. To be clear, all packages should have been installed, since this was a typical full install which includes all of them.

We clicked "install all" and that solved it for them. Their app now worked. (Knowing this now, you could also just do a "cfpm install all" from the command line, instead of just "cfpm install administrator".)

Let us know how it goes. If you struggle and don't want to await back and forth here, I can certainly meet with you via a remote screenshare consulting session (as I mention in the post: more at carehart.org/consulting), and we should be able to get the update in place and working quickly.

Then if you want, we could do more digging into what it is about your setup (and those of some others) that makes things not "just work" as they should. There must be some explanation--especially of how it happens to some but not all of us.
Hi Charlie,

Attempted the update again this morning on our live server - as before just doing it via CF Admin, this time everything worked as expected and we're now happily running Update 13.

This may be an over simplistic view, but it seems to me like the entire problem is that when my server tried to get a copy of 'bundlesdependency.json' during the update process, sometimes it was presented with the correct file and sometimes it was served something completely different.

The day that my server broke, doing a request from my server directly to 'https://www.adobe.co...' resulted in a json file with no mention of update 13 packages. Doing the same request today resulted in the file I was expecting.

To be honest, this is a sufficient enough explanation for me as to why things didn't work first time around. Some kind of caching issue/load balancing issue at the adobe end meant that my server was given a dependencies file that instructed it to uninstall certain packages. This caused my website to stop working. Once the relevant packages were put back in, everything seems fine.

As one of the affected parties, if there is anything you would like from me to help debug/understand the issue then do let me know. Otherwise I think we're happy to press on!

Andrew
# Posted By Andrew L | 3/20/24 1:48 AM
Andrew, thanks for the update and of course glad to hear you're working again. As for the musing you offer (which may be confirmed at some point), can you confirm where your server is (country/region)?
I wonder if anyone has had problems with installation from the command line instead of CF Admin? It seems like that continues to be the safer option, but perhaps I am wrong.
# Posted By ssykes | 3/20/24 10:10 AM
Ssykes, I assume you mean, "are those reporting update install problems only having them when run from the admin?" It's possible, yes.

I'll note that I've recommended the manual install as an option in both the post and other comments (like to Andrew, most recently), so again yep, I'm with you. :-)
Hi Charlie,

Our live server which had the trouble is located in a Redstation datacentre in Gosport, UK. Our test server which had no issues is located in our offices in Exeter, UK.

Regards,
# Posted By Andrew L | 3/20/24 1:58 PM
Just wanted to mention to those who have been waiting for it... Fixinator has been updated to scan for unscoped variables that need to be fixed released to this update. The details / demo here: https://www.petefrei...

Charlie, not sure if you'd consider updating your blog post to mention this, but it might be useful for those that don't make it all the way down here to the depths of the comments :-)
Thanks for the news, Pete--and of course for the effort! And yep, I've now added a mention of it and link within the text above...but of course, even there people may not "make it to the depths" of all the content in the post itself. :-)
Was looking to see if you had any guidance on what defines "the next major release of ColdFusion"? Does that mean the next non-security HF update, one of their installer refreshes, or ColdFusion 202X?

Also, we've seen some confusion on if setting searchimplicitscopes key to TRUE in Application.cfc or Application.cfm will continue to be supported once the JVM flag -Dcoldfusion.searchimplicitscopes is removed?

We're trying to understand the timing implications to budget development time to work on this and knowing some deadlines would be good for making that happen.
# Posted By Leith T | 3/27/24 7:49 AM
Leith, yes, I address both those questions above. Briefly, my understanding is that the jvm arg will not exist beyond cf2023, and that the app-level setting will always override it and is not due to go away.

Your questions are understandable, and I do already address/answer them above after mentioning those two options. See the section above, "Three ways to address the change in implicit scope searching", and the few paragraphs after the 3 bullet points there. The direct link to that section is:
https://www.carehart...

I realize my post here is large, and many will only scan rather than read it, and won't see every word. I've done what I could with sections, bolding, and a summary, to help such folks. But some points simply must be read. :-) I'm always open to feedback.
We've discovered that capital Ds in date formats like "MM/DD/YYYY" are now interpreted as day-of-year where previously they were treated as day-of-month.
Hey Keith we had that issue a while back, couple years ago. It is discussed in one of Charlies post from around that time. Linked below.

https://www.carehart...
# Posted By Steve C | 3/28/24 10:09 AM
Steve, thanks for stepping in and sharing that. And Keith, I suspect you've just moved to CF2023 or 2021 after having been running on CF2018 or earlier until recently. If so, welcome.

1) So as Steve notes, I blogged on this problem (which started with CF2021 and remains) when I found it back in Nov 2020 when CF2021 first came out. That first post Steve offered does discuss it in detail. And as that post indicates, I did another when when Adobe offered a solution a couple of weeks later. In that post (https://www.carehart...), I explain how they created a JVM arg that we must add to make CF treat a capital D like in a dateformat mast the way it always did. Again, those using CF2023 and beyond also will need to use that, to get that backward compatibility.

Or see how I mention (in both posts) that some kind souls offered regex searches that one could use to find and correct such uses of D, in many possible variants of dateformat and related functions.

2) Of course, you're bringing this up here on this post about yet ANOTHER breaking change, both of which are surely frustrating. And as in that case, Adobe opted to go with the breaking change and a jvm arg to REVERSE it, rather than the other way around.

And though these happened 40 months apart, you're not alone in only experiencing the first one only now...which is like how I concluded THIS post with how THIS new problem will ALSO plague us as future folks move to CF2021 or 2023 and apply this update, or move to CF2024 and beyond, which will include this change to the default for searchimplicitscopes as a built-in change. A difference is that while Adobe is choosing (for security reasons) to NOT allow the jvm arg reverting that to work in CF2024 and beyond, again this JVM arg for the dateformat issue IS still supported in CF2023 and beyond.

3) Finally, back to the issue in THIS post and since I suspect my comment here may lead someone to ask: no, I can't see any possible regex that could reliably "find all unscoped variables". There are just WAY too many ways that code could name/use variables: with or without pound signs, with or without spaces around it, with or without parens around it, within quotes (as in isdefined), and on and on. More than that, not ALL unscoped vars MUST be changed, as I elaborate on in the post.

But as I noted in a previous comment and an update to the post, Pete Freitag has done the community a solid in having added to his Fixinator tool such scanning (as effectively as possible) for potentially risky unscoped variables.

And maybe someone will offer here or elsewhere a regex they work on which they think helps do the job. If it works well, I will certainly highlight it! :-)
For those who may have missed the news, Adobe released a special hotfix related to this issue of searchimplicitscopes. See their forum thread with more info, and see also comments from myself and others there, elaborating on things. It's here

https://community.ad...
1) Thank you as always for the helpful posts. Of course using your application tip I had manually checked my sites before Adobe came out with that log option.

2) I am DELIGHTED that one of the sites I inherited in 2007 was very much affected by this breaking change! Budgets are tight so I finally received approval to take it down. Who-hooo!

3)FYI: I found that I can always update via the Admin IF I manually control the server restart. Get the admin page up, get service panel up, stop any extra CF services, kick off the update, stop CF Application service, Start CF Application service, start other CF services, success.
# Posted By Susan | 4/8/24 3:48 PM
Hi, Susan. Glad to have helped. And yep, some people may well use this scope-search matter as justification to remove some long-untouched app. Good news for the rest is that the workarounds do buy you time to get the actual changes in place--and that new fix for logging them where needed should be a big help. I still plan a post on that. Just busy. (Finally had to push out today posts on more time-sensitive upcoming presentations.)

Lastly, on your point of how you can't run the CF updates from the CF Admin, your approach is fine--but there's another solution, which again I've been meaning to blog about. And now I feel compelled to ASAP, lest discussion of the matter here takes us too far off-topic. :-)
Copyright ©2024 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting