Solving failure in applying Oct 2023 CF updates, or avoiding that failure
TLDR; If you've configured either CF2021's java home to use Java 11.0.20 or later, or CF2023's java home to use 17.0.8 or later, you may find that applying CF updates ia the Admin will fail. You can apply the update via the command line, adding a needed new jvm arg:
-Djdk.util.zip.disableZip64ExtraFieldValidation=true
(to be placed BEFORE the -jar arg) in the java -jar ... command, as I discuss more in the 5th bullet point below. (If I've lost you with that simple suggested, read the rest here. And all may benefit reading what precedes that suggestion, for context. I also offer other suggestions and info.)
Here are the topics in this post:
- Background
- The problem, introduced with Java updates starting in July 2023
- How the problem presents itself
- Solving (or preventing) the problem, if you typically use the CF Admin to install CF updates
- Solving (or preventing) the problem, if you typically use the command line to install CF updates
- This will also likely still happen with the next Java update, due out Oct 17
- Adobe plans to resolve this in the next CF update
(I should add that this is not a post covering all the things that can go wrong in applying CF updates and how to resolve them. It's focused on this one particular recent problem. And this is a bit of a revision of a post I did in July when I first discovered the issue. In fact the Adobe CF update technotes from the August updates pointed to that post for folks to find more info on the issue. The technotes for the latest Oct update merely mention the need of the new jvm arg discussed below, but without much more detail.)
Background
As some will know (and I shared in social media on Friday), Adobe has released updates for CF2023 (update 5) and 2021 (update 11), which is great. See the Adobe discussion forum post about the updates with more on the news, and I plan a post on it (after I sift through some matters related to it).
But as people attempt to apply that update (or even prior ones), they may hit a snag--and it's covered in the update technotes, but many fail to read those.
The problem, introduced with Java updates starting in July 2023
The problem is in fact not an issue with CF, per se, but with Java. And it relates to news I had shared (in more detail) back in July, when that new JVM came out.
Specifically, if you have updated your CF2021 to use at least Java 11.0.20, or CF2023 to use at least Java 17.0.8, you will find you will NOT be able successfully apply CF updates (the latest or previous ones).. Similarly, if you traditionally apply the updates from the command line using such an updated Java version, you will hit the same problem--unless you add a needed extra JVM argument, as I will discuss below.
How the problem presents itself
Whether you apply the update via the CF Admin or from the command line, in the scenario above you may experience the following problems:
- You will find when applying the update in the CF admin UI that it never changes from showing the update process "checking" on the state of the update, as if it's stuck doing that
- you may try to refresh your browser, only to find that you can no longer access the CF Admin (and indeed some parts of your app may fail), because the update has fails
- in trying to access the CF Admin, you may see the following error on screen or in the logs: "Message java.lang.NullPointerException: Cannot invoke "coldfusion.server.LicenseService.isStandard()" because the return value of "coldfusion.server.ServiceFactory.getLicenseService()" is null".
- Similarly, if you view the CF startup logs you may see that same error, or perhaps "Unable to install Logging package: java.lang.NoClassDefFoundError: org/apache/logging/log4j/Logger" (in CF2021) or "Unable to install Logging package: java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Layout" (in CF2023)
But regardless of whether you try to access the Admin, you will find clear evidence of this problem in the hotfix update log, which will report an error near the bottom about "Error installing uninstaller: java.util.zip.ZipException: Invalid CEN header (invalid zip64 extra data field size)". That's indeed the crux of the problem here.
(And FWIW, I have a blog post that covers finding and using the CF update log.)
There's no need to panic. Again, the problem can be solved (or avoided).
Solving (or preventing) the problem, if you typically use the CF Admin to install CF updates
If you typically use the CF Admin to install updates, the way to solve (or prevent) the problem is to switch to running the CF update from the command line. But even then there are some gotchas to beware.
The CF update technotes have always shown how to do that, but those who use the CF Admin have likely ignored that. Time to look at that update technote. (There is always a link to the technote offered in the CF Admin ui where you would choose to apply an update. And if your admin is busted and you can't see that link now, here are links to the CF2023 update 5 technote and the CF2021 update 11 technote.)
Since you would have downloaded the update in the admin before trying to install it, you will find the update has been downloaded into the [coldfusion]/bundles/updateinstallers folder, where the updates you have downloaded will be shown as jar files placed there.
You would now want to run that jar using Java--whether with the Java bundled with CF itself or the Java you have configured CF to use. (Some contend that resources on the internet and from Adobe say you should only do the former, but you can do the latter. What you want to avoid is using a version of Java OTHER than that which your given CF version supports, as I've written about before, including a table of CF vs Java versions):
- open a command prompt (as admin, if on Windows, or run as sudo/su on *nix)
- cd to that bundles/updateinstallers directory within your CF install directory (such as
cd C:\ColdFusion2023\bundles\updateinstallers
(or the equivalent for CF2021 or on *nix) - run java pointing to that jar (again, whether using the jvm included with CF or your more updated one, discussed as follows)
And yes, you can run this command-line update when the first install has failed to leave you in the state of getting the error discussed above. The new install will run the install over top of the failed one.
As for running that java -jar command, you have two choices:
a) If you use the JVM included with CF (2023 or 2021, respectively) to run the java command, you don't need to worry about the new jvm arg. You are solving the problem by using an "older" jvm (from before July 2023). I offer here an example for CF2023 on Windows, and I leave you to change it to suit CF2021 or *nix (these are each to be entered entirely on a single line). Again, these presume you ran the 3 previous steps just above.
For cf2023:
This will work by running the java command within the original CF jre/bin folder, which (unless you've changed it) comes with an "older" jvm (not affected by the more recently updated JVM discussed in this post). You don't HAVE to use this older jvm, though. See point b below.
(If this is your first time running the commandline update installer, see some thoughts I share below after point "b".)
b) If instead you want to use the JVM that you've changed CF to use (the newer one, wherever you may have installed it) to run the java command, then it's that if the JVM is from July 2023 or later (again, 11.0.20 or greater for CF2021, or 17.0.8 or greater for CF2023), then you need to add the new JVM arg introduced with that JVM, otherwise you will get the same failure as if you used the CF Admin to apply the update. Again, I offer here an example for Windows--and I presume you have your JVM installed in the traditional location that the JVM installer places newer versions of java since January 2023 (as I wrote about back then).
Again, these are each to be entered entirely on a single line, and they presume you've done the previous 3 steps above point "a" immediately above.:
for cf2023:
or for cf2021:
Thoughts if this may be your first time running the command line update installer
If this may be your first time applying the CF updates this way, note that you should see a new ui appear, to walk you through applying the update. Assuming you've opened this command prompt as admin (or used sudo/su on *nix), this installer will stop and start CF if running as a service and perform the update, just as would have happened from within the CF Admin. (If you get a prompt in Windows about it wanting to allow network access, this is a common prompt the first time you ever run java process from the command line.)
And when the update finishes, it will report success or failure (and as with applying the update within the CF admin, you can view the update log as I discussed above. It's a good practice to always check that log and make sure it reports all successes and no fatalerrors or nonfatalerrors, as I discuss in that other post. If on Windows you find you get one warning, about "_win64 does not exist", I have found that can be ignored.
Solving (or preventing) the problem, if you typically use the command line to install CF updates
Finally, for you who already install CF updates from the command line, you can follow the same steps above. Again, if you use the Jthe only thing you need to do is add that new JVM argument--assuming that the java command you run is in fact the updated version that introduced this problem starting in July.
Or as I note above, you could also switch to using an older java, even if only to apply this update.
Also, let me note here that those who apply the CF updates at the command line as a matter of course will know that there are other matters that have to do with obtaining and running the update if you don't have internet access--which extends to additional steps (covered in the update technotes) about extracting the bundle zip and/or running the cfpm script. I am not detailing these here, as they are not NEW steps related to this problem. Again, if you already do them you already know the drill. If it's new to you, please see the update technotes for more. (And I hope to do a post at some point with more on these other manual update steps.)
This will also likely still happen with the next Java update, due out Oct 17
You may notice I said above that this problem would happen, "if you have updated your CF2021 to use at least Java 11.0.20, or CF2023 to use at least Java 17.0.8" (emphasis added). That's because Oracle is due to release another update for Java on Oct 17, which should be 11.0.21 and 17.0.9, and there's no reason to think that they will have changed their stance on this issue. Assuming they do not, then we need to continue dealing with this with that JVM update as well.
Adobe plans to resolve this in the next CF update
But here's some VERY good news: Adobe has said (in various places online and) even in the update technotes that their next CF update should fix this problem. It's not clear how they will fix it (whether they will change their update jars to not require the arg, or embed the arg into the process that kicks off the update within the CF Admin). But it will be nice to be rid of this challenge.
Conclusion
Again, the good news is that while this is an annoying problem, a) it won't hit everyone (only those who update CF to use the JVM from July-forward), b) it's easily solved (once your comfortable with command-line CF update installation), and c) Adobe plans to resolve it for us with the next CF update.
Until then, I hope that this post helps folks. I do wish I could somehow make it simpler. (I have removed a lot of extraneous matters I shared in the July post, where I first shared the news. At the time this JVM change also affected even the ability to download CF updates. Adobe did fix that within weeks. Still, I feared some may not notice or pay attention to such an "old" post :-), so I thought it best to offer this newer one, as more and more people apply the latest updates to CF and Java.
(Standard Commandbox disclaimer: Yes, yes, if you use Commandbox, you may not need to worry about any of this. Same if using Adobe's CF containers, as they offer new images with each CF update. A lot of what I write is for people who DO still run CF "the old fashioned way", for whatever reason, and whether in dev or in prod--and yes, I realize Commandbox can be used to run CF Enterprise in prod! :-)
Finally, if you need help considering or applying the updates, or dealing with this or other problems applying them (or indeed any CF problem), I can help quickly via remote screenshare consulting. More on my rates, approach, satisfaction guarantee, online calendar, and more at https://carehart.org/consulting.
For more content like this from Charlie Arehart:Need more help with problems?
- Signup to get his blog posts by email:
- Follow his blog RSS feed
- View the rest of his blog posts
- View his blog posts on the Adobe CF portal
- If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
- See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
thanks
But I do always want things to be correct, and I am happy to fix them immediately when I can. :-) So again, thanks for the heads-up.
Or consider accepting accessing it from localhost only.
Im seeing following errors -
404 - File or directory not found.
The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable.
The requested URL was not found on this server!
If you entered the URL manually please check your spelling and try again.
Tomcat/ISAPI/isapi_redirector/1.2.46
Isapi_redirect.log showing following errors:
jk_check_path::jk_util.c (2476): Blocking this uri: [/CFIDE/administrator/index.cfm] since its starting with cfide
I am curious - on linux servers without GUI how to access CF Admin on 127.0.0.1 limitation ?
To your last question, one way is via ssh port forwarding. Another is via a web server acting as a proxy to the built-in web server--which circumvents ajp (and that uriworkermap.properties file, etc), for better or worse.
Finally, if you may prefer we resolve this directly (via remote consulting) rather than a lot of back and forth, we might solve things in as little as 15 mins. See carehart.org/consulting for more, including rates, approach, satisfaction guarantee, and online calendar.
Allowed IP Addresses for accessing ColdFusion Administrator and ColdFusion Internal Directories
And as that came in I was writing this to expand on my last comment:
Yet another option, of course, is to open an ip-specific firewall hole for that Cf built-in web server (8500, by default), in conjunction with that allowed ip limitation in the admin. Even in a server not connected to the internet, you should be as worried about hacks from within your network as from without.
And of course a real web server (such as nginx port-forwarding to cf's) can also be configured with additional protections like ip limitations, private certificates, additional security protections etc. While the cf built-in web server (which is tomcat's web server) can do SOME of that, other web servers are far more popular and well-documented.
I've found no docs offering anyway to disable or modify this new behavior, though it may be possible.
Those "needing" to enable can circumvent it in iis or apache by using those to port forward to the cf built-in web server. (With iis, you must add its free ARR feature to do that.) Or you can use ssh tunneling (long discussed in lockdown guide, including section 6.4 of the 2023 guide.)
Finally, I know some argue "the DOD stig makes me disable that built-in web server". I addressed that in another thread 3 days ago (which included you, neowire, but no reply yet from you):
https://community.ad...
And someone happened to reply just an hour ago as I wrote. I have added a reply to them there.
And feedback is welcome here, but if on this specific matter it could be hard for some to keep up with both, consider checking out that thread before commenting here, and decide if it may make more sense there (and reach people there who don't know about this here).
However - for those that don't need remote access through a connector to CFAdmin, this is a big security improvement. I've shared some more thoughts on that here - https://hoyahaxa.blo...
There is an example of the ssh command you would use in the lockdown guide.
Basically how it works is you open a tunnel on your desktop that maps a local port number of your choosing to the remote server port (eg 8500) and the traffic runs through the ssh protocol when you browse to that mapped port on localhost.
Pete, thanks also, though fwiw I covered that in a comment above (https://www.carehart...), but perhaps I should add the info to the post, as surely others will read that without considering all the "commentary". :-)
So you lament (with others) that this change to the CF connector (not allowing access to the CF Admin) now effectively precludes you accessing the CF Admin at all, right? And that there's "no way around it"?
But that's not true. There are in fact SEVERAL ways you could "get around such a restriction", and none are hacks or tricks. They're just different from what most folks using the CF Admin are used to. (And most have been mentioned above in comments, but I realize some either may not read them or might miss something--or ignore it, because their focus is on "what will it take to change Adobe's change in allowed behavior".)
1) First, of course you could open a firewall hole to access the port of the CF built-in web server (since it seems you're not held to that CF stig's restriction against it). But that seems so simple an option that I guess you have a reason that that, too, is "not allowed". Is that perhaps because you're not allowed to open ports? Everything must go through port 80 or 443?
2) Either way, a second thing one can do is setup your external web server (IIS or Apache) to have a site that is NOT connected to CF via the connector (via the CF "web server configuration tool"), but rather is configured to FORWARD requests to that CF built-in web server port. (Then you're not opening that CF built-in webserver port for access OFF the server. It's being accessed only via a forward from the web server on the same machine. And you can use the web server's capabilities to restrict that other ways, like via IP or requiring additional auth, etc.) I had mentioned this in comments on Oct 11 and 17 above, where I noted also that with IIS you'd need to enable its ARR feature to get that port forwarding. Otherwise, Apache, nginx, and others offer it as an inherent feature.
3) Yet another option is to setup SSH port tunneling, where you get the security (and optionally key-based auth) of SSH but have it forward (even from off the server) to that CF built-in web server port ON the server. I'd mentioned that also in that Oct 17 comment, where I also pointed out how this is actually discussed (for Linux) in the CF lockdown guide.
But note that such port forwarding is NOT limited to Linux. One can do it just as well with Windows. (And Windows has had an ssh client built-in for some years now, along with an optionally enabled ssh server.)
Hope that helps someone. Perhaps I need to do a post or talk on these alternatives. I'm open to hearing of others.
(I should add that one could use just forego the CF Admin and use the new cfsetup CLI tool added in CF2021, which lets one add/change/delete/view any admin settings, or export/import them via JSON. But I realize that may not suit everyone as an alternative to accessing the CF Admin "at all". Then too, the CF2023 Central Config Server or CCS feature could also reduce the need for people to access the CF Admin at all. But the focus above is getting to the Admin without using an external web server's AJP connection, as made via the CF web server config tool.)