Many find after applying a ColdFusion update that either CF won't start at all, or they can't access the ColdFusion Admin, or some part of CF or their app doesn't work. The problem may be simply that there was an error in the update process CF did, and it may be rather easily confirmed and resolved.
In this post, I share several tips and observations to help resolve this, based on my years of providing remote CF troubleshooting support.
The TLDR version: check the ColdFusion update log (not logs in the normal CF "logs" folder. More detail below.) And if there are errors, try stopping CF (and its related services) yourself and then either try the update again, or if it still fails, try to manually apply the update from the command line. If that's enough to get you going, great--especially if you ARE in panic mode! (If the "problem" you need to solve, instead, is that you can't get CF to show you updates because you're behind a firewall preventing outbound internet access, I help with that also, toward the end.)
For most people, though, even those "simple things to do" can prove challenging (and understandably so). And you may find different resources on the web offering perhaps truncated discussions of the topics, which is why I elaborate on things in this post.
And even if you're in a panic, it may take only about 10 minutes to read this whole post. (You can also hire me to help instead, of course. See the link above.) Hope the info to follow is helpful for you.
For the more visual folks, I also addressed much of this in an hour-long presentation I gave at various events in recent years, the last being recorded on the CFMeetup (as linked to there).
Check the update log...yes, there is one, but it's oddly formatted
So the first thing to do after you update ColdFusion (10 or greater) is to check the update install. And I do NOT mean here the logs in the CF "logs" folder. There are lines among those which will tell you THAT the updater applied but they will not tell you IF the update was applied successfully.
So yes, there is a log devoted to tracking the success of the update, and it's in a different location (and oddly formatted). And you really should check it even if the update seems to or reported itself to be "successful". It's easy to do once you know how:
- See the [cf]\cfusion\hf-updates\ folder. (I use [cf] to generically refer to the cf10, 11, or 2016 folder.) And in there, find a subfolder named for the number of the update you applied (like hf-10-00021, or in 2016, like hf-2016-00002-299200).
- There is a log in there whose name has the date/time of the install (or uninstall) of the update.
- And in that update install (or uninstall) log, about 100 lines down from the top (do NOT go to the bottom to look for it) is a table with a count of successes, warnings, and errors.
The table may look like this:
If the table reports all successes, great. Your update took. You can move along with your day. :-)
If it reports even one error or nonfatal error (and even it only reports warnings), then something was amiss with the update, and you can try to read through the remainder of the log to see if among the several hundred remaining lines it may explain what happened. That can be challenging because in some updates, many of lines in the log may have the word "error" or "warning" in referring to files that were changed in the update.
But do try to find what in the log may have been the problem.
The most common cause of ColdFusion update failures: CF did not stop
In nearly all cases when there is an error in the update, the answer is surprisingly simple (and it may even report it plainly if you look), but knowing to look is challenging enough. Addressing the problem can be even more challenging for most, but it too is fairly simple once you see how.
So the most common reason I find is that the update failed to stop the instance (of CF or other related processes). The log may even report it specifically on lines like the following:
- Failed to copy hotfix files:C:\Users\cfuser\288630.tmp\dist\cfusion: Failed to copy the hotfix files to the target location. Retry installation after ensuring that the server is not running or files are not locked by the server.
- Failed to copy the hotfix files to the target location:C:\ColdFusion11\cfusion FATAL ERROR - C:\ColdFusion11\cfusion\jetty\webapps\PDFgServlet\Resources\bin\HTML2PDFConverter.exe (The process cannot access the file because it is being used by another process)
Another error you may see is:
And there are others you may see. (I welcome comment pointing out others, so that I can add them here and hope people find this post while searching for such errors.)
So here's the issue: the update mechanism (in the CF Admin) is supposed to stop the instance, then backup the files to be changed, update them, and then bring it back up.
But if the instance does NOT come down (or not fast enough), then when the update tries to proceed with replacing or removing files to be changed, it cannot (because the files are locked by the OS), and that's why the update fails--not always, but nearly always, in my experience. We can debate whether Adobe should improve the process of the update so that it GUARANTEES that CF is down before proceeding, but until that's resolved, this info applied.
(And the update mechanism has improved. In CF10, a common problem is that once you click the "ok" button to let the updater run, you will find that within about 15-30 seconds, your CF Admin will refresh on its own, and if CF is not yet back up, you will get an error reporting "" or "Service Temporary Unavailable" and that "The server is temporarily unable to service your request". In that case, it may just be that it tried to refresh before it was back up. Just try refreshing the CF admin, or check if CF is indeed back up. In CF11, they fixed that so that the installer reports that it's checking the status and should let you know when it's all done. I say "should", because sometimes it does not, even though the update is indeed finished and worked. Again, if CF is back up, try refreshing the admin and see if you get a prompt to login then check if the update applied.)
Related to all this, one trick I recommend with people, to anticipate both problems above, is that they should always arrange to watch the CF process(es) during the update (at the OS level) to confirm that the CF instance (at a minimum) does indeed stop and start. In the case of Windows, you can watch in the Services panel, which you can refresh with F5 or the Action>Refresh menu command or the Refresh icon on under the menu.) Or in any OS watch the list of running processes to see ColdFusion disappear and reappear. If you do see it go down and come up, there's a greater chance that the update will take. Still, you may miss whether it happens, or there could still be errors, so again do look at the update log to confirm the success or failures.)
The most common solution, step 1: manually stop CF
So if there were errors, and either the log tells you that CF did not stop or you're just not sure, the solution nearly always is first just that you'll want to stop CF yourself, and then run the update manually.
For those who may have jumped to this "solution" step, I discussed in the previous section how the built-in CF update process will only wait a set amount of time before proceeding to try to do the update.
But by stopping CF yourself, you can make sure it's really down (whether via Services or command-line alternatives, or if you choose just to kill the process). The point is, once it's down there's then less reason the files should be locked when the update runs. (And you should make sure you really no longer see coldfusion running as a process, and may want to also stop all related CF services, like the jetty/add-on service, the odbc services if you use those, etc.)
The most common solution, step 1a: start CF and try the update again
Indeed, I'll just note that sometimes, this one step alone of stopping CF yourself and then restarting it may be enough to let the update run if you try it again!
It's just that sometimes when you stop CF at one point in time, it may go down faster than when the updater tried to stop it at a previous point in time. The matter of why CF sometimes is slow to go down or come back up is really a topic for another blog post. My point, though, is that you may find that if you do stop and start it, you may well find that you can just go ahead and apply the update via the Admin as you did before and it may "suddenly" just work!
But it may not, and if so, read on.
The most common solution, step 1b: manually install the update from the command line
For many people, for whatever reason, they may find that they just can't get the update to run from within the UI of the CF admin, as they find it keeps failing. For them, the next solution is that will need to stop CF themselves and keep it stopped for the update to work.
Of course, there's one REALLY big problem with stopping CF: you cannot then go into the CF Admin to run the updates as you did before. Doh! And this is where most get stuck.
The good news is that you can run the updates yourself, from the command line. That's where still more get stuck, but it's really quite simple, and once you are guided through it, you may even opt to always apply updates this way to avoid the above hassle!
As for running the CF update from the command line, basically, you just want to get to the location where the updates are placed when downloaded by the CF Admin ([cf]\cfusion\hf-updates), and from there run the Java command with the "-jar" argument to point to the update jar to be run. This will open a java-based installer (looks quite a lot like the CF installer itself), and it will go through a few screens to perform the update.
While I could elaborate on that process, others have already done so (and it's covered some in the CF docs). For instance, see this Adobe blog post, which includes info on how to run the java command from within the CF folder, for those who otherwise don't find "java" runs from their command line. While this post was written for a specific update of CF10, the info applies just as well to any update for CF10, 11, or 2016.
Even so, I will add a couple of tips for you to keep in mind:
- While the update process says it will try to stop CF, I would still stop CF myself before running the manual update. (And no, there's no harm that it's already stopped when the update says it will try to stop it.)
- You might still want to use the suggestion I made above to watch the services/processes to confirm that CF does go down and come back up, especially if you did not think to stop CF yourself.
- Indeed, do make sure that CF does come back up. While the installer SHOULD start CF, I have seen times when it did not, even when there were no errors in the update. In that case, start it as a service like you normally would.
- And yes you SHOULD still check the install log to confirm the count of successes and errors. It will be created just as described above, and put in the same place, regardless of whether the update is run from the CF admin or via this command-line java installer.
A gotcha to beware: make sure to use the same bit-level Java that CF is using
(Update: I added this section after the original post, to add this potentially important gotcha.)
If you DO use the manual command-line installer, there are a couple of potential gotchas. The first is easy to not even notice--until CF won't start. You apply the update at the command line, and confirm (per above) that the update had no errors, but then the CF service won't start. And if you try to start it from the command-line (by going to the /cfusion/bin folder and running cfstart), you find it reports "error loading" the jvm.dll. What the heck happened?
This tripped me up the first time it happened. I ultimately figured it out: the client I was doing it with had for some reason implemented a 32-bit Java JRE on their machine, though CF was implemented as 64-bit (and was pointing to a 64-bit JVM). There's nothing WRONG with having two different bit-level JVM's, as in this case, if you have something that needs the 32-bit jvm. But to be clear, CF does not use it.
But in this case, the 32-bit JVM was the "public jre", whereby when the java command above for the update was run (from a folder that did not have its own java executable), it was this 32-bit version of Java which ran. And the problem with that is that then the CF updater ran in 32-bit mode, and it implemented files into CF that were now 32-bit versions, and that's why CF then would not start (since its jvm was 64-bit but these 32-bit files within CF were being loaded).
So first, how would you know if the java you see running at the command line IS a 32-bit one? Well, if you ran the command with the "-version" argument, to reports its version info:
if in the info on the last line shown starts with "Java HotSpot(TM) 64-Bit Server VM", that's a 64-bit one (obviously), but if it just says "Java HotSpot(TM) Server VM", that is a 32-bit one (not as obvious). If it IS a 32-bit one, and your CF (and the JVM it uses) is 64-bit, then you MUST NOT use THAT 32-bit java to run the updater.
(Before I tell you the RIGHT one to run, you want to first uninstall the update you just did, to UNDO those "wrong files" that got implemented. I'll show you how to do that in a moment. But first, let's continue what you SHOULD have done, in case folks reading this have not yet made the mistake, but caught it before they ran the wrong bit-level java command.)
Instead, you want to point to a 64-bit JVM. You could point at the one in CF's JRE/bin folder, or if you had changed CF to use another JVM, you could point to that one. If it was installed in a folder like C:\Program Files\Java\jdk1.8.0_144, then you could run the Java within THAT folder with this command (note the use of quotes, because "program files" has a space in its name):
"C:\Program Files\Java\jdk1.8.0_192\bin\java" -version
and you'd also use it to then run the update, using the -jar command and pointing to the hotfix jar, as discussed in the previous section.
And what if you have run the installer (with the 32-bit java) before seeing this post, and CF is not starting? Well, you would want to uninstall the update you just applied (and get those 32-bit java files, among other updated files, removed). To do that, go to the "uninstall" directory under the folder for the update you just applied (within cfusion\hf-updates), and use that same 32-bit java to uninstall the update:
After that uninstall runs (and you check the log, as discussed above, to make sure there were no errors), CF will revert back to whatever update level you were before applying that last update. And you will find that CF DOES now start. But now, start things over. Run the update you were wanting to apply in the first place (using the java -jar pointing to the update jar, as discussed in the previous section above), but this time be SURE to run the 64-bit Java command as just discussed. Then check the update log, and you should find that CF starts. Yay!
Another gotcha to beware, when CF is started by the manual installer
Whether you have the last problem or not, there's one other potential gotcha when you use the manual command-line installer. The good news is that it's easily resolved. I've mentioned that after running the installer, the updater will try to start CF. That's a good thing. But watch out.
You want to watch for what user CF is "running as" after the command-line installer starts it. (You can see the user name which is running any process in Task Manager in Windows or via top in Linux.)
The problem is that I have found sometimes while the manual update process will properly start CF as service (in Windows or Linux), at other times I have seen it start CF WITHOUT running it as a service and worse, running as the user who ran the update.
On the surface, this may not seem a problem. CF is "running" at least, but this could be a real problem if the CF service (which it should have run as) is configured to run as some other user: whether a specific user (best for security), or the default of "local system" (Windows) or root (Linux). The issue is that the user YOU'RE logged in with (to do the update) may be none of these, and if CF runs under THAT user, there are some CFML operations which may not work because of that (you may not have proper permissions).
So if you don't see the service running, or you see the CF task running but not as the user who it would have been running as if started as the service, then kill the process (in Windows, go to Task Manager, find the coldfusion.exe, and kill it), and then start the service yourself (in Services, in Windows, for instance), just like you normally would. All should then be well.
Is CF running now after the update? Great!
So with all these possible issues resolved, if CF now starts (as the right user, see above), and you can access your admin, and it shows that the update has indeed applied, then you're good to go, and glad to have helped.
(As for how to check that an update has applied, other than using the log discussed above, you can also check it in the "updates" page of the admin, of course, on its "installed updates" tab. Perhaps better would be to look at the "system info" page, via the "i" in the top right corner of the admin, or the "settings summary" page listed on the left menu in the admin, which both report the CF and jvm version that CF is running.)
Again, on the surface, all of the above is quite simple once you understand it (finding if there are errors in the logs, stopping CF and running the update manually). It might be literally a 5-minute job when you know what you're doing.
But for folks who don't know, they can be down for hours or even days, which is tragic. I've even seen people go so far as to try (or be told to) uninstall/reinstall CF in the hopes of fixing such problems...and all it may have taken is stopping CF before applying the update!
See below for more on getting help if you do still struggle with some problem.
One other "problem" and its solution: how to download the CF updates manually
Before concluding this post, it may be that the "problem" someone has in applying these updates is very different: they may find that they can't even USE the CF Admin feature to download the update jars, because their server doesn't have internet access (for security reasons).
The good news is that there is indeed a solution to that, but again it seems many don't know about it (I don't see it generally offered by the community when people complain of this problem online).
Adobe has pages for each release listing the available jars for CF10, 11, and 2016, such as https://helpx.adobe.com/coldfusion/kb/coldfusion-10-updates.html. (You can change that to 11 or 2016 to get the jars for those.)
You would just download the desired jar file from there, and proceed with the manual command-line update process as described above. (Do beware that sometimes some versions of Windows or browsers will mistakenly convert a jar file into a zip when you downloaded it. If that happens, just rename the extension back to "jar" before running the "java -jar" command above.)
Still having problems? Reach out for remote help
If you're still having problems getting an update applied, you can certainly add a comment here and I'll try to get back to it, but perhaps better would be just to reach out for paid assistance. Sorry if this seems like a sales pitch, but I share the above based on my experience helping people. This sort of CF server troubleshooting is all I do all day every day, for hundreds of clients a year and over a thousand the past 10 years.
And some people would really just rather have (or would fix problems query by having) someone help ensure all this is considered and done right in 15-30 minutes rather than struggle on their own to consider all the issues above.
I can nearly always work remotely (via shared desktop solutions you may favor like gotomeeting, webex, or my preference join.me, among others), so you don't need to "give me remote access" (like RDP or ssh, which require you give me or create new credentials, or may require opening a firewall hole, or require VPN access). With a shared desktop solution, none of those are necessary, and we'll look into the problem together (I'll guide you to the problem and solution).
And we may not need very long at all: I can solve many problems in less than an hour. You can find more on my rates, approach, satisfaction guarantee, client references, and more at that consulting/support page on my site.
For some problems, there is just only so much one can reasonably anticipate in advance--even considering the ins and outs as I do in blog posts, presentations, videos, and more. Often there are just too many variables to consider and fully resolve a given problem generically (though I do try, as seen above).
In cases where the info I or others share doesn't suffice, or if you have some other knotty problem, don't suffer it for long. I'd even go so far as to say don't spend more than 15 minutes trying to google for a solution (where you may be led down all sorts of rabbit holes from well-meaning people, or perhaps dated info), if instead you can lean on someone who knows what they're doing and can get you to your solution--targeted to your specific problem, your environment, and your skill level--perhaps without much more time. Especially when your satisfaction us guaranteed!
There's just no substitute for the calm, experienced hand of someone who's "been there, done that", when it comes to problems like applying updates as above, or if you update your JVM and find CF won't start, or challenges like migrating to new CF releases, or updating/tuning the web server connector in CF10 and above, and so on.
If these problems were simple to solve, you'd find the answer in a tweet. Because they are not (lots of moving parts, common misconceptions), many struggle, perhaps finding only info that someone felt was "the least necessary to say", and leaving you to ferret out the details, or worse the solution to problems when things go wrong and they "left out that detail".
Whether in public resources or direct consulting, I not only want to help you solve your problem but also help you understand it: what happened, why it happened, and how to perhaps prevent it in the future!
Let me know what you think about the problems and solutions above.