[Looking for Charlie's main web site?]

CF911: Are you finding CF (or CF Admin) busted after applying a hotfix? A few possible reasons

Many people have reported that they find after applying certain ColdFusion cumulative hotfixes (CHFs), security hotfixes (SHFs) and hotfixes (HFs) that either CF Admin or perhaps even some feature of CF is busted.

The "good" news is that there are at least a few really common explanations of what may have happened, which I'll explain them here. Once you consider them, you may find either that you can confirm this is what happened (and fix it), or if nothing else you can keep it in mind if you're ever applying them yourself in the future.

For some of the hotfixes in particular, among the errors people may get related to this are several related to the ESAPIUtils class: "Element ESAPIUTILS is undefined in a Java object", "Could not initialize class coldfusion.security.ESAPIUtils", "Class not found: coldfusion.security.ESAPIUtils", and so on. Others find the CF Admin fails to work (entirely or in part). Others find that previously working CF web pages now fail.

But even if you're not getting these particular errors, the info here should help. Honestly, I feel it's required reading for anyone applying any CF hotfixes, since the mistakes are so easy to make.

I help people solve this very problem often, whether on CF mailing lists or in my independent consulting as a CF troubleshooter. And I've been meaning to post these thoughts for months but as you'll see below, there's a fair bit to consider, if you look beyond what one may try to communicate in a tweet or simple mailing list answer. After doing just that today on yet another forum, I decided to offer the expanded explanation, and hope that it may help someone.

First, some background.

What's different about recent CHFs, SHFs, and HFs (past couple of years)

Before addressing the problem, it would help to understand what's different about hotfixes (for 8 and 9) the past couple of years which can make this an issue in the first place.

It used to be that most hotfixes (CHF, SHF, HF) were just a single jar file, that you would (per the technote for the hotfix) just either upload using the CF admin "system information" page or drop into the appropriate \lib\updates folder (\coldfusion9\lib\updates or \jrun4\servers\[instancename]\[cfusionear]\[cfusionwar]\lib\updates).

But in recent years, the hotfixes have gotten more complicated (solving important security problems, often), including now often needing to update entire directories, whether in the WEB-INF or the CFIDE (where both the CF Admin and also various UI features for CFML are stored).

Well, if you make a mistake in doing that (updating those directories, or doing it in the wrong place), then you may find that some part of CF is busted, whether the Admin or perhaps even part of your app (which relies on finding those properly updated files within the CFIDE directory).

Let me make one important note before some jump in here to (reasonably) bemoan the situation.

Note that Adobe is addressing this with the next version of CF

Yes, it is unfortunate that these problems can happen. And the good news is that Adobe is addressing it in the next release of CF. They have pointed out in a number of recent public presentations and blog entries that they will be adding an automated hotfix mechanism in the next release, for now code-named Zeus. But until then, the following may prove helpful.

Also, for those reading this blog entry who may not know where CF hotfixes, cumulative hotfixes, and security hotfixes are located, see the following:

Now, back to the problem at hand: how you may make mistakes in applying hotfixes that may leave things busted. My observation (in helping lots of people with this problem) is that the root cause is primarily down to either of a few mistakes one can easily make.

Mistake 1: Applying hotfix for a version other than what you really have

This one is remarkably easy to do. When you're looking at the list of hotfixes on either of the pages above, and especially on pages that list hotfixes for multiple versions, such as the security hotfixes do, it would be very easy to inadvertently follow the steps for 9.0 when you are on 9.0.1, or vice-versa. Same with 9.0.2. (It's almost impossible to do with CF10 and its auto-hotfix mechanism--though perhaps as we move to any subsequent versions it could be an issue if people may change the URL for the hotfix feed.)

And the tragic thing is that there's nothing about the hotfix process that can stop you if you are using the wrong files. Since they all involve just dropping files into directories, there's nothing that will stop you or warn you if you are applying the wrong files for the version you have. It's simply incumbent on you to make sure you double check that you are getting the right files.

Mistake 1a: Doing the steps from technote "section 1" when you should do "section 2", or vice-versa

This is related to the mistake above. Again, especially on the security hotfixes, they may have an indication that even within a single release (such as 9.0.1), there may be a "section 1" set of steps (and files) that are to be used if you HAVE applied the security hotfix previous to that, or "section 2" if you have not. An example is the technote for this security hotfix.

If you're not careful, you may either use the wrong section, or if you are flipping back and forth between your file explorer and your browser, you may inadvertently grab the right version of one file/zip and then the wrong version of another.

Mistake 1b: Running the 9.0.1 installer on a 9.0.2 installation

This is a sneaky and devastating mistake that's just too easy to do. If you have installed CF 9.0.2, note that you MUST NOT run the 9.0.1 updater against that. Sadly, because the 9.0.1 update installer came out a couple years before 9.0.2, there's nothing that will stop it from running. And if you do, your CF installation will be busted.

What is 9.0.2? It is not just an update to 9.0.1. Instead, it is a complete installer for CF9, which was made available on the Adobe site in May 2012 when CF10 came out, because Verity was removed from CF10 and Adobe also needed to pull the 9.0 installer which included Verity. So while those who got 9.0 did need to add 9.0.1 to update it further, those who got 9.0.2 must NOT run the 9.0.1 installer.

(To check your version, use the "System Information" or "i" icon in the top right of the CF Admin, or the "Settings Summary" link on the left of the Admin, or if you have no access to the Admin--perhaps because it's now busted--run a CFML page that does a cfdump of the server scope. The version is reported there.)

Mistake 2: Updating the wrong directory CFIDE

The next problem could be that when you have finally downloaded the correct and necessary files, and are following the steps provided in the hotfix technote, you might install updates related to CFIDE into the wrong place. It's surprisingly easy to do. (Fortunately, not all hotfixes involve updating the CFIDE folder, but for those which do, this is VERY important to get right.)

I noted earlier that often the hotfixes may now have you update code in the CFIDE directory. Most know that to be where the CF Admin is stored. But it's also where various files are stored (by default, unless you move them) to be used by various UI elements of CF, in CFIDE\scripts. For instance, all the ajaxy things are stored there, as are the older cfform elements. There's also CSS, XSL, and other stuff there. That's why those things can break (silently, since it's the client's browser that now can't find what it wants, or finds the wrong version of a file, based on CF generating code that expects to use the "updated" versions.)

Well, here's the problem: when the technotes tell you to extract the zip into the CFIDE directory, they often leave you to figure out where that should go.

And the problem is that you may have many. For one thing, when you install CF, you're asked if you want to use the built-in web server, or an external one. If you do the former, then the CFIDE is put in the [cf]\wwwroot. if you do the latter, it's stored wherever you tell it is your web server root, whether inetpub\wwwroot, or htdocs, or wherever.

And if you use the Multiserver form of deployment, you may find that even if you install CF initially to integrate with your web server (in which case it would put the CFIDE in perhaps the inetpub\wwwroot, if/when you create new instances, they by default each have their own CFIDE deep within the instance, such as [JRun4]\servers\[instancename]\cfusion.ear\cfusion.war\CFIDE.

So when you run the update, you need to be POSITIVE that you are updating the one that your CF instance is really using, which means you may need to update many.

And if you're not careful, you may actually find you currently have a CFIDE directory in both kinds of places (and more), either because you have had multiple versions of CF installed, or someone has copied the files from one place to another to "fix" a problem. Then also, some admins (and hosting providers) even repeatedly copy the CFIDE (or perhaps just CFIDE/scripts) to the docroot for different web sites, trying to help with some other issues.

The upshot then is that you could easily have more than one CFIDE.

But it gets better: you can also set things up so that you can access the CF Admin either through the built-in web server, or through the external web server, or both. So besides the question of where the CFIDE lives, there is also the matter of where your web server (whichever you're using) is expecting to find the CFIDE.

And this goes for the CFIDE/scripts. Whether an admin has made multiple copies of it, or just manually added a virtual directory definition to their web server, this means that what matters most (for "finding" the CFIDE to update) is where the web server says it's looking.

Now, before someone gets too burdened with these prospects, note that I'm just explaining some of the many things that CAN happen. You may have a simple environment, and only one (or maybe two) places where the CFIDE lives.

So where to look to know where to update the CFIDE?

Given all the above, what are you to do?

Well here's one tip (not mentioned in the technotes): if you look in the CF Admin, you will find in the Mappings page that there is in fact a mapping for where *CF* at least thinks that the CFIDE is located. So first thing before applying the hotfix, check out that mapping, and at least be sure to update THAT location. That may be all you need.

But second (and even despite doing the above), do look at your web server (IIS or Apache), and see where THEY think that the CFIDE is located. Remember, that's what your browser will end up using (whether for the Admin, if you access it via those web servers, or for your end users running apps on your server). The point again is that they may show different CFIDE mappings, either for different sites, or within different sites.

Yes, it's lamentable. But the problem is caused by people doing things on their own that they need to remember they've done. And no, I doubt Adobe really will be able to fully resolve this, since they'd have to go digging in your web server to find the info. (They could, but should they? since they didn't cause the problem?)

And consider too that it may be that the different CFIDEs are leftovers from different CF versions you may have installed, and each may have been installed differently (one using the built-in web server for the CF Admin, one using the external web server).

So as the captain in the old show, "Hill Street Blues", used to say to his patrolmen after their morning briefing, "be careful out there".

Mistake 2a: Updating the wrong directory lib\updates

Related to the previous mistake of updating the wrong CFIDE location, there is also the potential problem of updating the wrong lib\updates directory. Repeating a point I made above, some people do the update of the hotfix jars by dropping them directly into the appropriate \lib\updates folder, rather than using the CF Admin "system information" page's upload feature.

Well, it's then important that you are careful about just where you drop those files. In a deployment of CF9 for instance, if you're running ColdFusion Standard or the Enterprise in the Server form of deployment, that would (on Windows) be \coldfusion9\lib\updates. In an Enterprise Multiserver form of deployment, it would be \jrun4\servers\[instancename]\[cfusionear]\[cfusionwar]\lib\updates. And note as well that in Enterprise Multiserver (multiple instances), you must remember to update ALL the instances.

And similarly, if the hotfix involves updating the WEB-INF directory, that too is within the CF9 (or CF8) directory, in the webroot for the built-in CF web server (even if you do not enable it and/or connect CF to an external web server). That location is either (for CF9) in \coldfusion9\wwwroot\ or in \jrun4\servers\[instancename]\[cfusionear]\[cfusionwar], and you need to update all the instances.

Mistake 3: Extracting the zips at the wrong level

This next mistake is different from the above, and yet applies to that step (extracting the zip for the CFIDE). But it also applies to extracting the zip for the WEB-INF, or any other directory where a hotfix says to extract a zip.

Here's the problem: it's possible that even if you point your zip extracting tool (winzip, izarc, winrar, copy/paste, whatever) to the "right" directory, you may mistakenly (or the tool may) extract the zip in such a way that what you extract is not at the level you point to, but one level below it.

I've seen it happen, to myself and others. (And some zip tools are more prone to it than others, especially winrar, in my experience.)

Here's the thing, if the technote says, for instance, to extract from the zip the web-inf directory, if you're not careful you may end up putting a NEW \web-inf under your current one! So you end up with \web-inf\web-inf. That will NOT, of course, update the needed files within the "real" \web-inf directory.

And of course the same could happen when you extract the CFIDE. You could end up with CFIDE\CFIDE.

Indeed, I've seen people have it happen when they extract the jar to the lib\updates.

So again be VERY careful, and double check things as you proceed.

Mistake 4: Be sure to do ALL the steps in the technote

Finally, another mistake I've seen people make is that they miss one or more steps in the technote. Again, these recent hotfixes have gotten more and more complicated, and have many steps.

Again, the automated hotfix mechanism in Zeus should address that.

But until then, it's incumbent upon you to make sure you follow each and every step, carefully and completely. Hope the above helps.

(BTW, there's a slightly related issue: you may also think you can skip over intermediate CHFs, like from having none to going to CHF 2, or going from CHF 1 to CHF4, etc. You can, but beware that there may be some "manual" hotfixes that were mentioned in an earlier CHF that are not repeated or included in the later one. I addressed that separate problem in a past entry, "CFMyths: "If/when I apply Cumulative Hotfixes, I need apply only the latest CHF, right?"">.)

What you can do, if it's broken

So the steps above are what to be aware of as you're going through the hotfix update process. But what if you did it and it's busted, and you found this note after the fact?

Well, I hope you can use the info above to simply double-check what was done. For instance, check if the directories have mistakenly got things in the wrong place (like if you see a web-inf\web-inf).

Look also at the dates of the files in the zips, and make sure that the dates you show (where files SHOULD BE) are in fact the same dates.

As for the CFIDE issues, besides the above, you may also benefit from just doing a search of your hard drive to find any and all CFIDE directories. You may find many. (I have a blog entry on some great tools to help with that, on Windows.)

Again you can check for which are being used by your web server and make sure they are correct for whatever version of CF you mean to be running (for that URL on that web server).

Hope that helps. Or did I leave anything out? Any questions? Comments? Let me know, either way.

Finally, if all this seems just too overwhelming to contemplate or resolve, or you just are too busy to want to bother, you can also hire me to help, and generally we can do a LOT to resolve things in less than an hour.

Comments
Thanks Charlie - I usually have to help people/companies with this exact issue. The current patch process has become overly complex and I'm glad Adobe is addressing this. I have found one useful method to apply these complex patches (esp multi-instance setups) is to script it out (bash or batch files). First I will build a 'staging' location for the updated CFIDE and WEB-INF folders and extract everything into those. If I have a server that is un-patched, I will do all the steps for all the patches and build a master CFIDE and WEB-INF folder that has all updates applied. Then either copy or script the copy to the proper locations for each instance. This helps me ensure I have all the updates applied and I can recheck or reapply if I needed to. It takes a little bit to do it and if you've never patched a ColdFusion server you may want to practice a bit doing each patch one at a time and then doing them the I described. Practice on your DEV setup - its much more forgiving that a clients product server :)
# Posted By Wil Genovese | 10/21/11 3:34 PM
Thanks for sharing your thoughts, Wil. Yep, that's a worthwhile step, but I'd warn folks that unless their servers really are identical, that they'd still want to be very careful to confirm that this "updated" CFIDE (and certainly web-inf) is correct for that destination server.

In fact, in that case it would be important not to miss the step to save off a copy of the folder before overwriting it. :-) But sure, your idea can help someone reduce the risk when doing it to multiple machines.
# Posted By Charlie Arehart | 10/23/11 11:14 AM
Charlie, let me be clear - I do this PER server because yes, each server can be and many times are different.
# Posted By Wil Genovese | 10/23/11 12:51 PM
I believe that I may be an unwitting victim of an improperly updated server over at HostMySite, As I have a CF8 site running on shared hosting for nearly three years now that is now erring out on every page that required CF8 Ajax tags ... Apparently the security sandbox is not properly propagating permissions ...

The one constant in the error stack is ...

"coldfusion.security.ESAPIUtils.getValidateInput(ESAPIUtils.java:387) at coldfusion.tagext.html.ajax.HtmlAssembler.setContainerId(HtmlAssembler.java:543) at coldfusion.tagext.html.ajax. "

plus what ever ajax tag is being called ...

Two weeks now and HMS has yet to find a solution ...
# Posted By Edward Beckett | 10/23/11 9:07 PM
Hi Edward, yes, that is indeed another example of what can break because of a busted hotfix implementation. I've seen other errors related to ESAPI, such as "Object Instantiation Exception. Class not found: coldfusion.security.ESAPIUtils" and "Error: org/owasp/esapi/errors/EncodingException" (the latter which might appear to the user as a blank CF Admin, with this error in the application log).

Have you tried my suggestion of reviewing carefully the steps for whatever HF or CHF you see was applied (based on what' in the lib/updated dir)? Do let us kno whow you get on.
# Posted By Charlie Arehart | 10/24/11 4:58 AM
When running multiple instances, we've found it very helpful to be in a *nix environment because softlinks can be made for just the portions of the CFIDE that are actually needed for the cfform tags.

I'm actually a little worried about the Zeus automated hotfix on multiple-instance machines, it's hard enough to get right with a human running things.
# Posted By Chris Champion | 10/25/11 6:12 PM
Charlie,

I pretty much got fed up with manually applying hotfixes and security updates, so created Unofficial Updater 2 back in April. It is nothing more than a packaged Ant script that will correctly apply published patches to CF 8.0.1 or 9.0.1. It is published at uu2.riaforge.org and all the source is available on github.
# Posted By David Epler | 11/3/11 6:15 PM
Cool stuff, David. Thanks for sharing it.

I've gone ahead and created a new section for such hotfix management tools (to go along with tools like MerlinManager and cfUpdater):

http://www.cf411.com...

If anyone knows of still others, do let me know.
# Posted By Charlie Arehart | 11/3/11 9:07 PM
Awesome stuff! Thanks guys
# Posted By Nate | 2/17/12 11:52 PM
Still another tip is to beware deleting the wrong jar files when applying a hotfix, as discussed here: http://sebduggan.com...

The converse is also to fail to delete any that you're told to delete. Be careful in both cases to follow whatever the technote says to do.
# Posted By Charlie Arehart | 3/15/12 11:03 PM
Thank you for the article. I was intimidated by installing the latest Coldfusion updates after reading some horror stories, but I followed your advice exactly (especially the CFIDE location part) and it helped me achieve a flawless upgrade. Thanks!
# Posted By Dan Krueger | 5/23/12 1:45 AM
Very good to hear, Dan. Thanks for sharing.
# Posted By Charlie Arehart | 5/25/12 4:28 AM
I just wanted to post a follow-up comment here on this blog entry from a couple of years ago.

Dave Epler had kindly shared back then that he had created the "unofficial updater" tool (UU2) to help with the very challenges of correctly applying updates in CF 9 and earlier, and I thanked him for that then, as many surely do still! :-)

But I would want to add a caution, related to the whole point of my blog entry (about the risk of not quite properly doing updates).

People using UU2 should be very careful to notice that it asks you were it should find and update your CFIDE folders, and just as I discuss above, be very careful about that.

Indeed, since it updates just the one CFIDE folder you point it to, beware that if you have your web server/sites configured to point at still other CFIDE folders, and those remain NOT updated, then again things can seem busted.

Like I said above, it's incumbent on you to search your server for any CFIDEs and be sure to update all that are supposed to be used by web sites running against the same version of CF, with the hotfixes applied to it.

While the UU2 tool doesn't provide for you to point to more than one, you could just run a tool after the fact like beyondcompare or other comparison tools. I list some at http://www.cf411.com...

And Dave has mentioned to me in the past that you could also just rerun the uu2 tool a second time (or more), naming the different CFIDE locations you want it to update. (I do believe it will still update everything else it is supposed to, not just these files. But his point is that that is a way to address this potential issue.)

If anyone (Dave or others) has more to share on that, feel free to comment.
# Posted By Charlie Arehart | 11/7/13 1:24 PM
BlogCFC was created by Raymond Camden. This blog is running version 5.005.

Managed Hosting Services provided by
Managed Dedicated Hosting