How and why your sites may break, and what to do, after applying March 2020 update to CF2018 or 2016
Instead, it's that after applying it, your CF web sites served via IIS or Apache WILL likely break initially, until you take one at least and perhaps two extra steps. The good news is that these steps are both easy and documented by Adobe in the update technotes, but they do require that someone do them, if needed. Let me explain.
[Update: I did an abbreviated version of this post on the Adobe CF portal: Three reasons your sites may break, and how to fix them, after applying March 2020 update to CF2018 or 2016. Note I also titled it differently. Just trying many ways to get people's attention. That post may interest some, either to read first (but my TLDR below also tries to abbreviate things also), or especially if you may prefer to give others a link to a post on this matter that is not as "dense" as this one. :-) I do point to this post from there, of course, for the many additional details that some may appreciate.]
Sadly, because many people don't bother to read the CF update technotes (linked to below), and they just apply the CF updates, they are not noticing this issue until they or their users start screaming because their sites are down. There's also a fair bit of "screaming" in the CF community, and folks responding may not know the info that I (or Adobe) have shared, to get things "working again", so I hope this helps bring some calm, and most important the clear solution/s needed.
For those who favor brevity, here's my attempt at a "brief" explanation. More details follow it.
The issue(s) stem from important security fixes that Adobe has implemented (related to the "Ghostcat" Tomcat vulnerability), which apply to the "AJP connector" that is enabled by default for connections between web servers like IIS or Apache and CF.
After applying these latest Mar 2020 CF updates, web sites served from IIS or Apache will likely fail. Here, briefly, is what to do and why (update since original posting: I added the 3rd point):
- First, you MUST "upgrade" any CF web server connector for IIS or Apache after applying the update. The issue is that the update puts a new "secret" into the CF configuration (the AJP line of the server.xml file), and the wsconfig upgrade puts that "secret" into the web connector (workers.properties) configuration file. (See below for where to find those, etc.)
The issue is that while such a secret was optional before, it's now mandated (by the Tomcat AJP connector update), but the CF update only does "one side" of the required update. This needed connector "upgrade" does the other.
Until you run that, requests will fail (with a 403). I discuss how to do it, below. (And even if you may have specified such a requiredSecret previously in the server.xml, the new Tomcat version which this update implements has changed the attribute name for the secret, so your prior settings won't work.)
- Second, there are a few scenarios where you will likely get 503 errors and need to implement a required IP "address" attribute, which would also be set for the AJP connector in CF's server.xml file.
The issue is that PRIOR to this update, CF (the Tomcat AJP connector embedded within it) was willing to accept requests from ANY ip address (which wasn't tragic, in that it's non-standard port that was blocked by most firewalls). So you didn't HAVE to set such an "address" attribute, though perhaps you did. But AFTER this update, the default behavior (if no "address" is set) is that the AJP connector is willing to accept requests ONLY made to CF as 127.0.0.1, presumed to be from a web server on the same machine.
Certainly that is an issue if the web server is on some OTHER machine (a rather unusual setup for most CF folks), or if you have modified your hosts file so that localhost binds to some other IP address on your machine.
But even if on the web server is on the same machine, the requests to CF will fail if it somehow sends requests to CF as some OTHER ip address (even ::1, the IPV6 equivalent), then you MUST set this "address" field on the AJP connector naming that IP address, or requests will fail (with a 503). I discuss below a bit more about this address attribute.
I discuss below those other scenarios, which includes even simply using Apache, which I and others are finding that for some reason it sends requests to ::1, even though nothing obvious would suggest that it should. More below.
- Third, you MAY need to make still one more change, if you STILL get errors (403 errors) after making the changes above, to add yet another new Tomcat AJP connector attribute, allowedRequestAttributesPattern, on the AJP connector line (the same one discussed above) found in the server.xml file. The shortcut to get things working (if this is required) is to set allowedRequestAttributesPattern=".*" (that's a dot and an asterisk).
This issue is that after this update, the Tomcat connector has been changed (again, by the Tomcat team) to have a certain limited number of request attributes that the connector will allow to be passed in. If for any reason your web server passes in other headers, the request will fail. You could (and should) find what headers those are, and only enable those, but it's harder than it may seem, for a number of reasons.
But I will note that I've found that if you simply use Apache's and rewrite CF page requests using mod_rewrite or .htaccess, that will indeed add a header and trigger this problem. So again, the simple solution above is the quickest way to get things working again. More below.
To be clear, if you only access your web sites through CF's internal web server (which by default will be a port like 8500), or if you use a reverse proxy or other means to connect your web server to CF, then these two steps do not apply to you.
Finally, setting asides the details of this update for a moment, if you may have problems after applying the update (or you feel that none of the 3 points above apply for you), see yet another blog post I've done, "Having problems after applying a CF update? What to check, and how to recover!"
For some readers, the above is all they need to know and they can take it from there.
For most though, I suspect they will need or want more detail which I share or link to below. I also address some reasonable questions one may have.
I've organized the post into two sections. First I cover a few "tactical response" points (how to deal with the problem), and then a few "strategic response" points (some thoughts for you and for Adobe to consider):
Tactical response (dealing with the problem immediately):
- How you can easily upgrade your CF web server connector
- "I never ran this wsconfig tool, so does this not apply to my sites?"
- Finding more about the update from Adobe
- How your sites may fail: what you may see
- More on the IP address attribute: how your web server calls CF
- On secrets vs requiredSecrets
- More on the allowedRequestAttributesPattern attribute
- What if I am in cf11 or earlier (or choose not to apply the Mar 2020 update), but want the security fix?
Strategic response (thinking about the problem)
- How would you have known of the need to upgrade the connector?
- How such a connector upgrade was often recommended before, but is required with this update
- How Adobe could make it MORE CLEAR that we might need to take these two extra steps
- How these two "extra steps" will apply to those who skip this updates but do the next one
- How a new CF installer for CF2018 and 2016 would really help us now
- These changes are indeed ALL that this update implements
Let's talk first about your "tactical response" to this update and the troubles it may cause for you and others. First and foremost, how do you "fix" the problem. Let me tackle the web connector upgrade first.
How you can easily upgrade your web server connector
Fortunately, it is QUITE simple. You just need to run the CF "web server configuration" or wsconfig tool, and if you use the UI for it then it's now especially easy since CF2016, since you just need to select a connector and click the "upgrade" button. I discuss this in more detail (including command-line options) in a Nov 2019 post discussing when and how to upgrade the web server connector.
As I will discuss later, the need to do it is not new, only the mandatory nature after this update.
Finally, note that you MUST update ALL connectors that you have, which point to the CF instance that you have updated. Again, until you do, those will fail.
"I never ran this wsconfig tool, so does this not apply to my sites?"
I have said above that this "connector upgrade" is needed for sites connected to CF from a web server like IIS or Apache, as have been connected using the Adobe CF "web server configuration" tool.
You may say, "I haven't used that tool". Well, let me clarify that in CF2016 and earlier, the CF installer would also ask (if it detected such a web server) whether you wanted to connect CF to that web server. If you said yes, then THAT process created this "connector", and you DO need therefore to "upgrade" it. (If you may not have noticed, in CF2018 that part of the installer no longer exists, so anyone wanting to connect CF to their web sites on such a server this way would have to run the wsconfig tool.)
So as I noted in the opening, the only people who can ignore this "connector upgrade" are those who really DO NOT use this connector, which is
- those who only use the CF built-in web server (as may be typical of those doing merely development or testing), and
- those who may use a reverse proxy or other means to connect their web server to CF (where the web server "forwards" requests to that internal web server, without using the "connector"). That topic is beyond the scope of this post, so I will not elaborate on it.
Finding more about the update from Adobe
You might wonder how you could have known of this "need to update the web server connector" (or the other IP address issue, which may affect some).
I will note that Adobe has indeed offered considerable information about the most recent update, including mentioning that you needed to upgrade the web connector. But it could have been lost/missed. Also, some people don't bother reading any of these things. They just click the button to update the connector, and so were blithely unaware of the consequences.
I just don't think they make it as clear as they could that the connector upgrade was REQUIRED or your sites would break. Nor does the text box for the update, in the CF Admin.. I will discuss how I have raised these to Adobe, later in the "strategic response" section below.
How your sites may fail: what you may see
Since some people may only find this post because they google for the error they may see, let me point those out.
The 403 error they may see
First, if you miss the needed step to upgrade the web server connector after these updates, and therefore the needed "secret" is not being passed in, then requests made to your site(s) through your web server that uses that AJP connector will fail.
Same if your web server is passing in unexpected headers and you need to add the allowedRequestAttributesPattern attribute (see the third point in my TLDR at the opening).
In either case, the error may look like this (on CF2018):
HTTP Status 403 Forbidden
<hr class="line" />
<b>Type</b> Status Report<br /><b>Description</b> The server understood the request but refuses to authorize it.
<hr class="line" />
While on CF2016 (upon applying update 14) will report the Tomcat version as 8.5.42. (These are the Tomcat versions that the Mar 2020 CF updates implement, which address the Ghostcat issue.)
No, the error doesn't more say clearly that "your web server connection needs to pass in the required secret", or "set that allowedrequestattributespattern", and that's for wise security reasons, but it can through folks off.
Then again, it's also possible that you/your users may see a different error, or perhaps even an error page for your site, or even just a blank page, depending on the configuration of error handling in the web server or CF.
(I have been meaning to find and document if there is a corresponding log entry that may confirm these problems are happening. I didn't want to hold up posting everything else before finding and documenting that. I will add that here, if I find it.)
The 503 error they may see
If instead the problem is about the second point in my TLDR at the opening, about the IP address being used to connect from your web server to CF, then you would get an entirely different error after the update (perhaps showing a 503 status code), such as this:
Service Temporarily Unavailable!
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
There's a bit more about the ip address issue in the CF update technotes, including a reference to what to look for in the CF logs to help know what the IP address should be, if needed--though I will note that where they say that the "starting protocolhandler" message appears in the coldfusion-out.log, I found it instead in the coldfusion-error.log.
Let's dig into this ip address matter a bit more.
More on the IP address attribute: how your web server calls CF
As I have reiterated, if for any of a few reasons CF is being called by your web server using some IP address other than 127.0.0.1, then your requests to CF from that web server will fail. You MUST implement an IP "address" field, in CF's server.xml file, for the AJP connector. This was NOT needed prior to this update, because CF (or technically, the Tomcat embedded within it) was willing to accept requests to that AJP connector from ANY IP address.
Personal opinion/rant: such "open access" that the connector "allowed" may seem tragic (and it's really what has been behind the uproar about this "ghostcat" vulberability), but note that the AJP port is a non-standard one that would be blocked by any decent firewall. (In the case of a native Tomcat installation, the default AJP port was 8009. In the case of CF, it's been different default values for different CF versions, such as 8016 in CF2016, or 8018 in CF2018).
I do think a lot of the hype over ghostcat (and the pressure for Adobe to "fix" this) was overblown, but Tomcat felt the pressure to change it and did, and now Adobe has implemented these Tomcat changes, so we must go along. And of course, it's never a bad thing to have multiple levels of security, from the firewall, to this ip address limitation, to the new "secrets" required.
Anyway, the issue is that AFTER this Tomcat (CF) update, the default behavior is that the AJP connector will be willing by default to accept ONLY requests sent to CF as 127.0.0.1. [Update: when I first wrote this post, I said "or ::1", the ipv6 equivaent to that. I said that based on my reading of the Tomcat docs. But time and the heartbreak of many has proven that it is indeed ONLY 127.0.0.1 that it accepts, by default.]
But first, some readers may be confused, saying "I don't tell my users to use that or any ip address". I get it. That's not the issue. I've said each time above that this is about the ip address that the web server uses to talk to CF. It's NOT about the ip address you or users use to reach your web server. Those are totally different things.
And I mentioned at the outset (in the TLDR) 3 common ways you may find that you get this failure, because your web server is NOT talking to CF via 127.0.0.1. Let's elaborate on them.
Handling if your web server is on a different machine than CF
First, of course, if CF and the web server are on DIFFERENT machines (a rather uncommon mode called "distributed cf" in the CF docs), then of course an address MUST be specified, to allow access into CF/Tomcat from that web server's IP address.
Note that it is possible that someone had already added that "address" attribute previously even before it was "required" for this reason (its name has not changed with this update).
Handling if you have changed the IP address that localhost resolves to
Another way this IP address issue can catch you out even if you have CF and the web server on the same machine is if you have for some reason modified your hosts file, so that "localhost" resolves to an IP address other than the traditional loopback address, 127.0.0.1.
Related to this, note that the default indication in your CF web connector workers.properties file has it connect to "localhost" as the host. It may look like this:
Again, to solve this, add the named IP address as an "address" on the ajp line of server.xml. (You might think you could/should change your hosts file to NOT point it to such a specific address, but that could impact other things you may have going on your machine which are relying on that.)
On Apache using ::1, when it may not seem it should
I have mentioned above how the main problem causing this need to set the "address" attribute is if something clearly would cause your web server to point to CF via some different IP address.
But there's at least one occasion I've found where people are getting this 403 error and the simple solution is to set the "address" to "::1", even though it doesn't seem obvious that they should.
For now, the most common is folks using Apache (again, even on the same machine as CF). This is proving to be so even though a ping of localhost on their machine did not return that as what localhost resolves to. And also even though localhost is the value of the host name listed in the workers.properties file of their CF connector.
If you're in this boat, just try it (address="::1"), if things are failing despite all other attempts.
As I learn more, or if indeed I find that it can happen with IIS in some situations, I will update this.
Beware changing the workers.properties host value
Speaking of the workers.properties file, do beware the temptation of trying to solve this (when CF and the web server are on teh same machine) by changing the worker.cfusion.host=localhost to point to an ip address like ::1. That may well "work" instead, for now.
But understand that the next time add a new connector, that will default to the localhost phrase (unless you change it), and you'd have the same problem again.
If you change the address attribute in the server.xml AJP line, then you will resolve this for your setup once and for all.
Handling when multiple web servers connect to the same CF
What if you may have multiple web servers set to connect into one CF instance?
I'll note that the CF update technotes and the Tomcat docs for the AJP connector refer to this address field as setting "the ip address" (singular) of the web server, at least at this writing.
But I did try entering a comma-separated list of multiple addresses, and not only did CF start and show no errors in any logs, but I had listed the valid IP address as second in the list, and a request DID work--whereas if I listed that bad IP address alone (and restarted CF), I got the 503 shown above.
On secrets vs requiredSecrets
Perhaps you heard about this "ghostcat" problem before Adobe implemented these updates, or you read about locking down the AJP connector for whatever reason. Perhaps you had taken steps on your own to secure the AJP connector in CF, by defining a "requiredSecret" in the server.xml's AJP line.
That was indeed the name of the attribute as Tomcat defined it, before this update by the Tomcat team to the AJP connector which the CF update has implemented.
After this update, though, the attribute is instead named "secret", and the CF update will put such a newly-named "secret" attribute (with that value Adobe chooses) onto the AJP line of your connector in the server.xml file.
This CF update WILL leave in place any requiredSecret you may have defined. There will be no harm, in that Tomcat ignores the requiredSecret attribute from the older versions of Tomcat, though it could be confusing to some observers of that line in that file.
Note also that the needed connector update (above) WILL CHANGE any current secret you may already have had there (which worked previously with that "requiredSecret" value you had created in the server.xml). It will set the workers.properties secret value to the value for the newly created "secret" attribute in the server.xml.
(And all this is what the update SHOULD do, for things to work. I'm just confirming that it does.)
More on the allowedRequestAttributesPattern attribute
I mentioned at the opening (in the TLDR) how requests will also fail after the update if the request sent from the web server to CF (and thus the updated AJP connector in CF) has any headers other than the dozen or so allowed, as listed in the Tomcat docs on the AJP connector.
And I'd mentioned that you could just override that new default by setting allowedRequestAttributesPattern="." on the AJP connector in the server.xml file.
I also mentioned that one situation where this was proving to be needed was if one uses Apache and does rewrites, whether with mod_rewrite or .htaccess files. Such a rewrite is adding a new header.
How did we find that? And what could we do with that info?
How would one see what header is being sent in, which would cause the 404?
One way that may work is if you enable debug-level logging for the AJP connector. You can do that (for IIS) in the isapi_redirect.properties file with:
Or for Apache in mod_jk.conf (or jk.conf or http.conf, or wherever the AJP configuration is), with:
I have observed that that logging shows what headers come in on a request, though I will admit I have not done it while someone was experiencing the 404 due to this "allowed headers" problem. I've seen them when requests DID work, for sure.
Also, from a CF perspective, there are a couple of ways to see what headers are sent into a CF request, at least when the request IS working, such as when you may use that .* pattern (to get past the problem temporarily), or if you may roll back the update.
First, there's the CFML function gethttprequestdata (or more specifically, coldfusion gethttprequestdata().headers). Second is with FusionReactor, and its ability to see the "details" of a request, one of which is a "headers" tab.
That's how we found that it was a header called "site-name".
What to do once you have that header
With that knowledge, one COULD try adding that as THE pattern for that new allowedRequestAttributesPattern attribute (rather than use the "/*" value to "allow any).
If you do that, note that it's a regex, so with a dash in the name like that, you would have to escape it, so the pattern would be "Xsite\-name". And we did that, it "worked" (in that now the "rewritten" site would run. And FWIW, sites that were NOT being "rewritten" were working regardless.)
What's not clear from the Tomcat docs is whether in doing that, we are ADDING to or OVER-RIDING/overwriting the default values listed on the Tomcat doc page. Again, I plan to do some more research and testing on things. (I find virtually none by anyone else on the web.) If I learn more, I will share it here.
What if I am in cf11 or earlier (or choose not to apply the Mar 2020 update) but want the security fix?
While it's risky for so many reasons to be running CF11 or earlier (since such are not supported/updated by Adobe), note that Adobe did offer a special technote for those in this situation who want to address (manually) the vulnerability closed in this Mar 2020 CF update.
Now, moving on to more "strategic" questions and answers, for you and for Adobe to consider...
How would you have known of the need to upgrade the connector?
First, I will note that the CF update technotes have (since CF2016) offered a table at the bottom of the page, indicating whether an upgrade to the connector is required. (They use the term "recreation", and I'll discuss that more later.)
Second, the technotes for this update (for each of CF2018 update 8 and CF2016 update 14) do also specifically indicate the need to do the connector upgrade, adding additional discussions similar to some of what I have shared here. (They actually share quite a bit more, including about the IP address matter also, which some readers will appreciate. )
How such a connector upgrade was often recommended before, but is required with this update
FWIW, while the technotes have indicated the occasional need (after some updates) to update the web server connector after applying the update, often people would miss this point. Again, many people don't read the technotes. And the small text box shown on the CF Admin page with the updates sometimes DID indicate a need to upgrade the connector, but sometimes it would not (even when it was required, as in the case with this update, at least as I write.)
Or people would skip some updates (since they are indeed cumulative), and so they may not have bothered to look at the technote (or the update text box) for the updates they were skipping. But at least that table at the bottom of the technote did help people to know which updates DID call for the connector to be updated.
And all this is among the reasons I created my other blog post on upgrading the connector, to help folks know when and how to do it, especially since it's so easy since CF2016.
But my point here is to say that if one did skip (or miss) the need to update the connector in prior updates, the problem was merely that there COULD be problems. Sometimes people who missed or skipped the upgrade didn't notice any.
With this latest CF update, however, it is MANDATORY that you update any web server connector(s) after the update, so that it puts in that needed secret that CF generates (and it also updates the connector dll or so. See my o ther blog post for more on that).
How Adobe could make it MORE CLEAR that we might need to take these two extra steps
So as I have related above, one concern I have is that (at least at this writing), the need to upgrade the connector it isn't made as clearly at the top of the technote, nor is it indicated in the text box that appears for these two updates. I hope Adobe will fix that soon. I have asked them directly, in addition to asking here.
How these two "extra steps" will apply to those who skip this updates but do the next one
If you've read the past few points here, I have made the observation that people do sometimes skip CF updates. Well, let's say a new CF update comes out in the next month or months. Then the people who have skipped THIS CF update will of course face the SAME problems, if they created a connector while at an update earlier than this Mar 2020 one.
And we will go through another round of people experiencing broken sites, and complaining loudly in the community.
How a new CF installer for CF2018 and 2016 would really help us now
I should say that the previous point only applies until Adobe may create a new installer for CF2018 or 2016, which bundles this newer connector, while I hope they may consider.
Doing that would also greatly help all those who use the CF Admin update mechanism and currently trip over two problems:
- the UI issues in the CF admin updates page, which makes it hard to select updates and click on the update buttons (fixed in CF2018 update 7 and CF2016 update 13)
- the issue that those on updates earlier than those from June 2019 (CF2018 update 4 and CF2016 update 11) must FIRST implement THOSE CF updates before they can apply any later ones
These changes are indeed ALL that this update implements
Finally, some people have wondered if this update may have address any other issues (like the long-awaited and now-pressing samesite cookie issue that I wrote about this week). Sadly, it did not.
All it addressed was these issues (and updating Tomcat related to them).
FWIW, I did a compare of all the files in CF before and after the change, and this set of changes (the addition of the secret in the server.xml, and the updating of Tomcat-specific jar files related to this security change) were indeed the only changes that the update performed.
Pro tip: Someone might want to say, "well, duh, Charlie. You could look at the hotfix_filelist.log in the update's folder, under cfusion/hf-updates" folder, to see what files were changed. But that's actually not quite so.
While that file DOES list ALL the files changed during the update, we have to keep in mind that since again CF updates are CUMULATIVE, each update lays down ALL the latest versions of files changed in ALL the previous updates. So you can't just look at that file and see, "yep, only these 20 or so files were changed due specifically to THIS latest update". Again, I did a compare of the complete CF folder, before and after the update, to confirm things.
Note that Adobe did NOT update the Tomcat version, only the AJP CONNECTOR itself
This section is an addition since the original posting. As I was digging into things (and helping someone with the update), I found out that whereas I thought this CF update updated Tomcat for us, I can say now that it did NOT. CF is running the same Tomcat version that would have been implemented if you'd applied the CF updates from Sep 2019, which are update 5 for CF2018, and update 12 for CF2016. You can confirm this in the CF Admin "settings summary" page, which will report (after those updates, and including this one) that CF2018 is running on Tomcat 9.0.21, and CF2016 is running on 8.5.42.
The thing is, the "official" Tomcat version which implemented the changes for Ghostcat (discussed in this blog post here) were introduced in Tomcat 9.0.31 and 8.5.51, respectively. But CF is NOT running those Tomcat versions (as of this update).
So what gives? We can only conclude that Adobe took a shortcut, and ONLY updated the AJP connector. That's not necessarily "wrong". Maybe it was expedient, allowing them to get this update out ASAP, without dealing with a "full" Tomcat update.
But it is confusing, for those who dig into the whole Ghostcat issue, and who understand Tomcat versions. I just wanted to point that out here, for those who may care to know (or who may dig further, in which case I welcome comments, of course).
Conclusion, and a plea
So phew, that's another mega post. I realize that some will have stopped at the top. Some may have dropped off on the way. My goal with these it to provide enough info for anyone who takes the time to read them, to understand the issue more completely than can be covered in twitter, slack, facebook, linkedin, etc.
If you may feel overwhelmed in considering it all, let me note that I can certainly help you implement these updates, typically in less than 15 minutes. See my consulting page for more on my rates, approach, satisfaction guarantee, and more.
And if you're looking at how to use the current downtime with the pandemic, I can help also with any CF updates, as well as connector upgrades, and JVM updates, and lots more related to install/config/admin of CF, troubleshooting/tuning, upgrading/migrating, deployment of CF on Docker, and lots more.
Finally, if you read all the way to the bottom here, gold star for you!
Could you return the favor, and take a moment to share your thoughts? I put a LOT of time into posts like these, hoping to help people (and indeed, they may well help folks so that they don't need to leverage my consulting!) But often I find I get very little feedback, pro or con. Your encouragement is appreciated, and even brickbats are tolerated in helping make things better.
For more content like this:
Need more help with problems?
- Signup to get my blog posts by email:
- Follow my blog RSS feed
- View the rest of my blog posts
- View my blog posts on the Adobe CF portal
- If you may prefer direct help, rather than digging around here/elsewhere or via comments, I can help via my consulting services
- See that for more on how I can help a) over the web, safely and securely, b) usually very quickly, c) teaching you as we go, and d) with satisfaction guaranteed