[Looking for Charlie's main web site?]

Solving failure in applying Oct 2023 CF updates, or avoiding that failure

If you try to apply ColdFusion updates (including the latest released Oct 6) via the CF admin or command line and find that the update fails, the problem may be due to the JVM you're using (within CF or at the command line). There's a simple solution, which I discuss in this post.

TLDR; If you've configured either CF2021's java home to use Java 11.0.20 or later, or CF2023's java home to use 17.0.8 or later, you may find that applying CF updates ia the Admin will fail. You can apply the update via the command line, adding a needed new jvm arg:
-Djdk.util.zip.disableZip64ExtraFieldValidation=true
(to be placed BEFORE the -jar arg) in the java -jar ... command, as I discuss more in the 5th bullet point below. (If I've lost you with that simple suggested, read the rest here. And all may benefit reading what precedes that suggestion, for context. I also offer other suggestions and info.)

Here are the topics in this post:

(I should add that this is not a post covering all the things that can go wrong in applying CF updates and how to resolve them. It's focused on this one particular recent problem. And this is a bit of a revision of a post I did in July when I first discovered the issue. In fact the Adobe CF update technotes from the August updates pointed to that post for folks to find more info on the issue. The technotes for the latest Oct update merely mention the need of the new jvm arg discussed below, but without much more detail.)

Background

As some will know (and I shared in social media on Friday), Adobe has released updates for CF2023 (update 5) and 2021 (update 11), which is great. See the Adobe discussion forum post about the updates with more on the news, and I plan a post on it (after I sift through some matters related to it).

But as people attempt to apply that update (or even prior ones), they may hit a snag--and it's covered in the update technotes, but many fail to read those.

The problem, introduced with Java updates starting in July 2023

The problem is in fact not an issue with CF, per se, but with Java. And it relates to news I had shared (in more detail) back in July, when that new JVM came out.

Specifically, if you have updated your CF2021 to use at least Java 11.0.20, or CF2023 to use at least Java 17.0.8, you will find you will NOT be able successfully apply CF updates (the latest or previous ones).. Similarly, if you traditionally apply the updates from the command line using such an updated Java version, you will hit the same problem--unless you add a needed extra JVM argument, as I will discuss below.

How the problem presents itself

Whether you apply the update via the CF Admin or from the command line, in the scenario above you may experience the following problems:

  • You will find when applying the update in the CF admin UI that it never changes from showing the update process "checking" on the state of the update, as if it's stuck doing that
  • you may try to refresh your browser, only to find that you can no longer access the CF Admin (and indeed some parts of your app may fail), because the update has fails
  • in trying to access the CF Admin, you may see the following error on screen or in the logs: "Message java.lang.NullPointerException: Cannot invoke "coldfusion.server.LicenseService.isStandard()" because the return value of "coldfusion.server.ServiceFactory.getLicenseService()" is null".
  • Similarly, if you view the CF startup logs you may see that same error, or perhaps "Unable to install Logging package: java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger" (in CF2021) or "Unable to install Logging package: java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Layout" (in CF2023)

But regardless of whether you try to access the Admin, you will find clear evidence of this problem in the hotfix update log, which will report an error near the bottom about "Error installing uninstaller: java.util.zip.ZipException: Invalid CEN header (invalid zip64 extra data field size)". That's indeed the crux of the problem here.

(And FWIW, I have a blog post that covers finding and using the CF update log.)

There's no need to panic. Again, the problem can be solved (or avoided).

Solving (or preventing) the problem, if you typically use the CF Admin to install CF updates

If you typically use the CF Admin to install updates, the way to solve (or prevent) the problem is to switch to running the CF update from the command line. But even then there are some gotchas to beware.

The CF update technotes have always shown how to do that, but those who use the CF Admin have likely ignored that. Time to look at that update technote. (There is always a link to the technote offered in the CF Admin ui where you would choose to apply an update. And if your admin is busted and you can't see that link now, here are links to the CF2023 update 5 technote and the CF2021 update 11 technote.)

Since you would have downloaded the update in the admin before trying to install it, you will find the update has been downloaded into the [coldfusion]/bundles/updateinstallers folder, where the updates you have downloaded will be shown as jar files placed there.

You would now want to run that jar using Java--whether with the Java bundled with CF itself or the Java you have configured CF to use. (Some contend that resources on the internet and from Adobe say you should only do the former, but you can do the latter. What you want to avoid is using a version of Java OTHER than that which your given CF version supports, as I've written about before, including a table of CF vs Java versions):

  1. open a command prompt (as admin, if on Windows, or run as sudo/su on *nix)
  2. cd to that bundles/updateinstallers directory within your CF install directory (such as
    cd C:\ColdFusion2023\bundles\updateinstallers
    (or the equivalent for CF2021 or on *nix)
  3. run java pointing to that jar (again, whether using the jvm included with CF or your more updated one, discussed as follows)

And yes, you can run this command-line update when the first install has failed to leave you in the state of getting the error discussed above. The new install will run the install over top of the failed one.

As for running that java -jar command, you have two choices:

a) If you use the JVM included with CF (2023 or 2021, respectively) to run the java command, you don't need to worry about the new jvm arg. You are solving the problem by using an "older" jvm (from before July 2023). I offer here an example for CF2023 on Windows, and I leave you to change it to suit CF2021 or *nix (these are each to be entered entirely on a single line). Again, these presume you ran the 3 previous steps just above.

For cf2023:

\coldfusion2023\jre\bin\java -jar hotfix-005-330608.jar
or for cf2021:
\coldfusion2021\jre\bin\java -jar hotfix-011-330247.jar

This will work by running the java command within the original CF jre/bin folder, which (unless you've changed it) comes with an "older" jvm (not affected by the more recently updated JVM discussed in this post). You don't HAVE to use this older jvm, though. See point b below.

(If this is your first time running the commandline update installer, see some thoughts I share below after point "b".)

b) If instead you want to use the JVM that you've changed CF to use (the newer one, wherever you may have installed it) to run the java command, then it's that if the JVM is from July 2023 or later (again, 11.0.20 or greater for CF2021, or 17.0.8 or greater for CF2023), then you need to add the new JVM arg introduced with that JVM, otherwise you will get the same failure as if you used the CF Admin to apply the update. Again, I offer here an example for Windows--and I presume you have your JVM installed in the traditional location that the JVM installer places newer versions of java since January 2023 (as I wrote about back then).

Again, these are each to be entered entirely on a single line, and they presume you've done the previous 3 steps above point "a" immediately above.:

for cf2023:

"\Program Files\Java\jdk-17\bin\java" -Djdk.util.zip.disableZip64ExtraFieldValidation=true -jar hotfix-005-330608.jar

or for cf2021:

"\Program Files\Java\jdk-11\bin\java" -Djdk.util.zip.disableZip64ExtraFieldValidation=true -jar hotfix-011-330247.jar

Thoughts if this may be your first time running the command line update installer

If this may be your first time applying the CF updates this way, note that you should see a new ui appear, to walk you through applying the update. Assuming you've opened this command prompt as admin (or used sudo/su on *nix), this installer will stop and start CF if running as a service and perform the update, just as would have happened from within the CF Admin. (If you get a prompt in Windows about it wanting to allow network access, this is a common prompt the first time you ever run java process from the command line.)

And when the update finishes, it will report success or failure (and as with applying the update within the CF admin, you can view the update log as I discussed above. It's a good practice to always check that log and make sure it reports all successes and no fatalerrors or nonfatalerrors, as I discuss in that other post. If on Windows you find you get one warning, about "_win64 does not exist", I have found that can be ignored.

Solving (or preventing) the problem, if you typically use the command line to install CF updates

Finally, for you who already install CF updates from the command line, you can follow the same steps above. Again, if you use the Jthe only thing you need to do is add that new JVM argument--assuming that the java command you run is in fact the updated version that introduced this problem starting in July.

Or as I note above, you could also switch to using an older java, even if only to apply this update.

Also, let me note here that those who apply the CF updates at the command line as a matter of course will know that there are other matters that have to do with obtaining and running the update if you don't have internet access--which extends to additional steps (covered in the update technotes) about extracting the bundle zip and/or running the cfpm script. I am not detailing these here, as they are not NEW steps related to this problem. Again, if you already do them you already know the drill. If it's new to you, please see the update technotes for more. (And I hope to do a post at some point with more on these other manual update steps.)

This will also likely still happen with the next Java update, due out Oct 17

You may notice I said above that this problem would happen, "if you have updated your CF2021 to use at least Java 11.0.20, or CF2023 to use at least Java 17.0.8" (emphasis added). That's because Oracle is due to release another update for Java on Oct 17, which should be 11.0.21 and 17.0.9, and there's no reason to think that they will have changed their stance on this issue. Assuming they do not, then we need to continue dealing with this with that JVM update as well.

Adobe plans to resolve this in the next CF update

But here's some VERY good news: Adobe has said (in various places online and) even in the update technotes that their next CF update should fix this problem. It's not clear how they will fix it (whether they will change their update jars to not require the arg, or embed the arg into the process that kicks off the update within the CF Admin). But it will be nice to be rid of this challenge.

Conclusion

Again, the good news is that while this is an annoying problem, a) it won't hit everyone (only those who update CF to use the JVM from July-forward), b) it's easily solved (once your comfortable with command-line CF update installation), and c) Adobe plans to resolve it for us with the next CF update.

Until then, I hope that this post helps folks. I do wish I could somehow make it simpler. (I have removed a lot of extraneous matters I shared in the July post, where I first shared the news. At the time this JVM change also affected even the ability to download CF updates. Adobe did fix that within weeks. Still, I feared some may not notice or pay attention to such an "old" post :-), so I thought it best to offer this newer one, as more and more people apply the latest updates to CF and Java.

(Standard Commandbox disclaimer: Yes, yes, if you use Commandbox, you may not need to worry about any of this. Same if using Adobe's CF containers, as they offer new images with each CF update. A lot of what I write is for people who DO still run CF "the old fashioned way", for whatever reason, and whether in dev or in prod--and yes, I realize Commandbox can be used to run CF Enterprise in prod! :-)

Finally, if you need help considering or applying the updates, or dealing with this or other problems applying them (or indeed any CF problem), I can help quickly via remote screenshare consulting. More on my rates, approach, satisfaction guarantee, online calendar, and more at https://carehart.org/consulting.

For more content like this from Charlie Arehart: Need more help with problems?
  • If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
  • See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
Comments
In the article you say CF 2021 Update 5, but i think you mean 11, and CF 2023 update 11, when you mean 5

thanks
# Posted By Kevin Fox | 10/11/23 10:56 AM
Yep, thx so much. It was late. :-) Will correct.
And now I have corrected that mistaken reference. Thankfully, the few other references to the update numbers were right.

But I do always want things to be correct, and I am happy to fix them immediately when I can. :-) So again, thanks for the heads-up.
After applying update to 2021 and 2023 CF servers and "upgrading" wsconfig connector for all sites - I am unable to access CF admin anymore execpt from localhost 127.0.0.1- befor that i had set up CFIDE folder as Virtual Drive in IIS along with changes to /config/wsconfig/1/uriworkermap by uncommenting !/CFIDE/* , I made those changes again but no success
# Posted By Sheikh | 10/11/23 10:25 PM
While it's potentially a grave mistake to opt to expose the CFIDE folder via iis (adobe blocked it in that uriworkermap.properties file for a reason), you may be being blocked from a purported change in the update. The release notes said that they'd changed the default for admin "allowed ip addresses", in some situations. Perhaps yours is one. Try adding whatever ip address you want to permit accessing the admin from.

Or consider accepting accessing it from localhost only.
Thanks Charlie - our servers are not www accessible and are behind duo and only accessible behind VPN , so that's kind of not a concern . i have added my ipadderssess to "allowed Ip addresses" without luck - If i recall , i have seen people complaining about "allowed Ipaddress" not working -

Im seeing following errors -

404 - File or directory not found.
The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable.
The requested URL was not found on this server!
If you entered the URL manually please check your spelling and try again.
Tomcat/ISAPI/isapi_redirector/1.2.46

Isapi_redirect.log showing following errors:

jk_check_path::jk_util.c (2476): Blocking this uri: [/CFIDE/administrator/index.cfm] since its starting with cfide


I am curious - on linux servers without GUI how to access CF Admin on 127.0.0.1 limitation ?
# Posted By Sheikh | 10/11/23 11:10 PM
As for the allowed ip addresses page, did you fill in the SECOND box on that page, rather than the first?

To your last question, one way is via ssh port forwarding. Another is via a web server acting as a proxy to the built-in web server--which circumvents ajp (and that uriworkermap.properties file, etc), for better or worse.

Finally, if you may prefer we resolve this directly (via remote consulting) rather than a lot of back and forth, we might solve things in as little as 15 mins. See carehart.org/consulting for more, including rates, approach, satisfaction guarantee, and online calendar.
second Box
Allowed IP Addresses for accessing ColdFusion Administrator and ColdFusion Internal Directories
# Posted By Sheikh | 10/11/23 11:24 PM
Ok on your last comment. Just had to check.

And as that came in I was writing this to expand on my last comment:

Yet another option, of course, is to open an ip-specific firewall hole for that Cf built-in web server (8500, by default), in conjunction with that allowed ip limitation in the admin. Even in a server not connected to the internet, you should be as worried about hacks from within your network as from without.

And of course a real web server (such as nginx port-forwarding to cf's) can also be configured with additional protections like ip limitations, private certificates, additional security protections etc. While the cf built-in web server (which is tomcat's web server) can do SOME of that, other web servers are far more popular and well-documented.
Appreciate your blog Charlie. To @Sheikh, what you are observing is exactly what I am observing in my environment. The update deployed with no failures. And ColdFusion starts and runs just fine. I even got into the CF Administrator. But then I went and updated the isapi_redirect.dll file and I get the exact errors that you are getting within the isapi_redirect.log, "Blocking this uri: [/CFIDE/administrator/index.cfm] since its starting with cfide"
# Posted By neowire | 10/13/23 10:02 AM
Guys, bottom line is that this is indeed new behavior the connector. It's not documented, sadly, but it's intended as a security protection for cf...to prevent access to the CFIDE folder for requests via the connector, meaning from iis or apache. And it's more effective than the previous block in uriworkermap.properties, as that was case-sensitive. (Try for yourself, those who may find that shocking to discover.)

I've found no docs offering anyway to disable or modify this new behavior, though it may be possible.

Those "needing" to enable can circumvent it in iis or apache by using those to port forward to the cf built-in web server. (With iis, you must add its free ARR feature to do that.) Or you can use ssh tunneling (long discussed in lockdown guide, including section 6.4 of the 2023 guide.)

Finally, I know some argue "the DOD stig makes me disable that built-in web server". I addressed that in another thread 3 days ago (which included you, neowire, but no reply yet from you):
https://community.ad...

And someone happened to reply just an hour ago as I wrote. I have added a reply to them there.

And feedback is welcome here, but if on this specific matter it could be hard for some to keep up with both, consider checking out that thread before commenting here, and decide if it may make more sense there (and reach people there who don't know about this here).
@Charlie, Sheikh, and neowire - I've observed the same the new connector behavior you all mention. The release notes don't say much about the new connectors, but they do state "Wsconfig update now creates a backup of the modified files." I haven't gone through a ColdFusion upgrade to either new release, but I can say that running a "wsocnifg -update ..." on a new CF2021U11 server will back up the connector config files and library. So reverting to these backed-up files might be the best way to revert if needed.

However - for those that don't need remote access through a connector to CFAdmin, this is a big security improvement. I've shared some more thoughts on that here - https://hoyahaxa.blo...
# Posted By Brian | 10/18/23 3:50 PM
Note that if you are on Linux you can use a ssh tunnel to connect to the built in web server on port 8500 (or whatever port you might be running on).

There is an example of the ssh command you would use in the lockdown guide.

Basically how it works is you open a tunnel on your desktop that maps a local port number of your choosing to the remote server port (eg 8500) and the traffic runs through the ssh protocol when you browse to that mapped port on localhost.
# Posted By Pete Freitag | 10/19/23 8:42 AM
Brian, thanks. I have some feedback I'll share there on the post.

Pete, thanks also, though fwiw I covered that in a comment above (https://www.carehart...), but perhaps I should add the info to the post, as surely others will read that without considering all the "commentary". :-)
Ah, of course you did already mention that - I missed that, thanks
# Posted By Pete Freitag | 10/19/23 1:37 PM
@Pete - thank you for you CFBreak emails! I've been building out a new environment for CF2021 when I ran into this issue this week and after reading (and re-reading) all the release notes and not seeing anything I assumed I had mis-configured something and spent a day trying to debug why I could no longer access CFAdmin. Then today I'm reading CFBreak and hmmm - an article about the wsconfig connector and ... not being able to access CFAdmin!!!!
# Posted By Jim Priest | 10/20/23 2:06 PM
To give you a bit more background on the DoD side, different organizations might enforce the STIG differently. Our problem comes in not with the Coldfusion STIG but with the Windows server STIGs. The only way to log on to a server is as an administrator but when you do you are not allowed to access any resources outside of the DoD network. They way our organization has decided to enforce this is to not allow execution of a web browser when logged on as administrator. So for us, there is no way around it.
# Posted By Jim | 2/16/24 3:40 PM
Jim, you're clarifying that the only way you are ALLOWED (by the Windows stig) to access the CF Admin is from OFF the server. Fair enough. (I realize others make the contention that they "can't use the CF built-in web server" because of the dated CF stig from 2014: more in the CF forum thread I pointed to in a comment here back in October. But that's not your issue here.)

So you lament (with others) that this change to the CF connector (not allowing access to the CF Admin) now effectively precludes you accessing the CF Admin at all, right? And that there's "no way around it"?

But that's not true. There are in fact SEVERAL ways you could "get around such a restriction", and none are hacks or tricks. They're just different from what most folks using the CF Admin are used to. (And most have been mentioned above in comments, but I realize some either may not read them or might miss something--or ignore it, because their focus is on "what will it take to change Adobe's change in allowed behavior".)

1) First, of course you could open a firewall hole to access the port of the CF built-in web server (since it seems you're not held to that CF stig's restriction against it). But that seems so simple an option that I guess you have a reason that that, too, is "not allowed". Is that perhaps because you're not allowed to open ports? Everything must go through port 80 or 443?

2) Either way, a second thing one can do is setup your external web server (IIS or Apache) to have a site that is NOT connected to CF via the connector (via the CF "web server configuration tool"), but rather is configured to FORWARD requests to that CF built-in web server port. (Then you're not opening that CF built-in webserver port for access OFF the server. It's being accessed only via a forward from the web server on the same machine. And you can use the web server's capabilities to restrict that other ways, like via IP or requiring additional auth, etc.) I had mentioned this in comments on Oct 11 and 17 above, where I noted also that with IIS you'd need to enable its ARR feature to get that port forwarding. Otherwise, Apache, nginx, and others offer it as an inherent feature.

3) Yet another option is to setup SSH port tunneling, where you get the security (and optionally key-based auth) of SSH but have it forward (even from off the server) to that CF built-in web server port ON the server. I'd mentioned that also in that Oct 17 comment, where I also pointed out how this is actually discussed (for Linux) in the CF lockdown guide.

But note that such port forwarding is NOT limited to Linux. One can do it just as well with Windows. (And Windows has had an ssh client built-in for some years now, along with an optionally enabled ssh server.)

Hope that helps someone. Perhaps I need to do a post or talk on these alternatives. I'm open to hearing of others.

(I should add that one could use just forego the CF Admin and use the new cfsetup CLI tool added in CF2021, which lets one add/change/delete/view any admin settings, or export/import them via JSON. But I realize that may not suit everyone as an alternative to accessing the CF Admin "at all". Then too, the CF2023 Central Config Server or CCS feature could also reduce the need for people to access the CF Admin at all. But the focus above is getting to the Admin without using an external web server's AJP connection, as made via the CF web server config tool.)
Copyright ©2024 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting