[Looking for Charlie's main web site?]

How and why your sites may break, and what to do, after applying March 2020 update to CF2018 or 2016

This is a critical warning to anyone who may apply the recent CF2018 Update 8 or CF2016 Update 14, released Tuesday of this week. To be clear, I do not mean with this warning to suggest that you should NOT apply the update! It implements an important security fix.

Instead, it's that after applying it, your CF web sites served via IIS or Apache WILL likely break initially, until you take one at least and perhaps two extra steps. The good news is that these steps are both easy and documented by Adobe in the update technotes, but they do require that someone do them, if needed. Let me explain.

[Update: I did an abbreviated version of this post on the Adobe CF portal: Three reasons your sites may break, and how to fix them, after applying March 2020 update to CF2018 or 2016. Note I also titled it differently. Just trying many ways to get people's attention. That post may interest some, either to read first (but my TLDR below also tries to abbreviate things also), or especially if you may prefer to give others a link to a post on this matter that is not as "dense" as this one. :-) I do point to this post from there, of course, for the many additional details that some may appreciate.]

Sadly, because many people don't bother to read the CF update technotes (linked to below), and they just apply the CF updates, they are not noticing this issue until they or their users start screaming because their sites are down. There's also a fair bit of "screaming" in the CF community, and folks responding may not know the info that I (or Adobe) have shared, to get things "working again", so I hope this helps bring some calm, and most important the clear solution/s needed.

TLDR:

For those who favor brevity, here's my attempt at a "brief" explanation. More details follow it.

The issue(s) stem from important security fixes that Adobe has implemented (related to the "Ghostcat" Tomcat vulnerability), which apply to the "AJP connector" that is enabled by default for connections between web servers like IIS or Apache and CF.

After applying these latest Mar 2020 CF updates, web sites served from IIS or Apache will likely fail. Here, briefly, is what to do and why (update since original posting: I added the 3rd point):

  • First, you MUST "upgrade" any CF web server connector for IIS or Apache after applying the update. The issue is that the update puts a new "secret" into the CF configuration (the AJP line of the server.xml file), and the wsconfig upgrade puts that "secret" into the web connector (workers.properties) configuration file. (See below for where to find those, etc.)

    The issue is that while such a secret was optional before, it's now mandated (by the Tomcat AJP connector update), but the CF update only does "one side" of the required update. This needed connector "upgrade" does the other.

    Until you run that, requests will fail (with a 403). I discuss how to do it, below. (And even if you may have specified such a requiredSecret previously in the server.xml, the new Tomcat version which this update implements has changed the attribute name for the secret, so your prior settings won't work.)

  • Second, there are a few scenarios where you will likely get 503 errors and need to implement a required IP "address" attribute, which would also be set for the AJP connector in CF's server.xml file.

    The issue is that PRIOR to this update, CF (the Tomcat AJP connector embedded within it) was willing to accept requests from ANY ip address (which wasn't tragic, in that it's non-standard port that was blocked by most firewalls). So you didn't HAVE to set such an "address" attribute, though perhaps you did. But AFTER this update, the default behavior (if no "address" is set) is that the AJP connector is willing to accept requests ONLY made to CF as 127.0.0.1, presumed to be from a web server on the same machine.

    Certainly that is an issue if the web server is on some OTHER machine (a rather unusual setup for most CF folks), or if you have modified your hosts file so that localhost binds to some other IP address on your machine.

    But even if on the web server is on the same machine, the requests to CF will fail if it somehow sends requests to CF as some OTHER ip address (even ::1, the IPV6 equivalent), then you MUST set this "address" field on the AJP connector naming that IP address, or requests will fail (with a 503). I discuss below a bit more about this address attribute.

    I discuss below those other scenarios, which includes even simply using Apache, which I and others are finding that for some reason it sends requests to ::1, even though nothing obvious would suggest that it should. More below.

  • Third, you MAY need to make still one more change, if you STILL get errors (403 errors) after making the changes above, to add yet another new Tomcat AJP connector attribute, allowedRequestAttributesPattern, on the AJP connector line (the same one discussed above) found in the server.xml file. The shortcut to get things working (if this is required) is to set allowedRequestAttributesPattern=".*" (that's a dot and an asterisk).

    This issue is that after this update, the Tomcat connector has been changed (again, by the Tomcat team) to have a certain limited number of request attributes that the connector will allow to be passed in. If for any reason your web server passes in other headers, the request will fail. You could (and should) find what headers those are, and only enable those, but it's harder than it may seem, for a number of reasons.

    But I will note that I've found that if you simply use Apache's and rewrite CF page requests using mod_rewrite or .htaccess, that will indeed add a header and trigger this problem. So again, the simple solution above is the quickest way to get things working again. More below.

To be clear, if you only access your web sites through CF's internal web server (which by default will be a port like 8500), or if you use a reverse proxy or other means to connect your web server to CF, then these two steps do not apply to you.

Finally, setting asides the details of this update for a moment, if you may have problems after applying the update (or you feel that none of the 3 points above apply for you), see yet another blog post I've done, "Having problems after applying a CF update? What to check, and how to recover!"

For some readers, the above is all they need to know and they can take it from there.

For most though, I suspect they will need or want more detail which I share or link to below. I also address some reasonable questions one may have.

I've organized the post into two sections. First I cover a few "tactical response" points (how to deal with the problem), and then a few "strategic response" points (some thoughts for you and for Adobe to consider):

Tactical response (dealing with the problem immediately):

Strategic response (thinking about the problem)

Let's talk first about your "tactical response" to this update and the troubles it may cause for you and others. First and foremost, how do you "fix" the problem. Let me tackle the web connector upgrade first.

How you can easily upgrade your web server connector

Fortunately, it is QUITE simple. You just need to run the CF "web server configuration" or wsconfig tool, and if you use the UI for it then it's now especially easy since CF2016, since you just need to select a connector and click the "upgrade" button. I discuss this in more detail (including command-line options) in a Nov 2019 post discussing when and how to upgrade the web server connector.

As I will discuss later, the need to do it is not new, only the mandatory nature after this update.

Finally, note that you MUST update ALL connectors that you have, which point to the CF instance that you have updated. Again, until you do, those will fail.

"I never ran this wsconfig tool, so does this not apply to my sites?"

I have said above that this "connector upgrade" is needed for sites connected to CF from a web server like IIS or Apache, as have been connected using the Adobe CF "web server configuration" tool.

You may say, "I haven't used that tool". Well, let me clarify that in CF2016 and earlier, the CF installer would also ask (if it detected such a web server) whether you wanted to connect CF to that web server. If you said yes, then THAT process created this "connector", and you DO need therefore to "upgrade" it. (If you may not have noticed, in CF2018 that part of the installer no longer exists, so anyone wanting to connect CF to their web sites on such a server this way would have to run the wsconfig tool.)

So as I noted in the opening, the only people who can ignore this "connector upgrade" are those who really DO NOT use this connector, which is

  • those who only use the CF built-in web server (as may be typical of those doing merely development or testing), and
  • those who may use a reverse proxy or other means to connect their web server to CF (where the web server "forwards" requests to that internal web server, without using the "connector"). That topic is beyond the scope of this post, so I will not elaborate on it.

Finding more about the update from Adobe

You might wonder how you could have known of this "need to update the web server connector" (or the other IP address issue, which may affect some).

I will note that Adobe has indeed offered considerable information about the most recent update, including mentioning that you needed to upgrade the web connector. But it could have been lost/missed. Also, some people don't bother reading any of these things. They just click the button to update the connector, and so were blithely unaware of the consequences.

For more, see a blog post announcing the update(s) earlier this week, as well as the CF 2018 update 8 technote and the CF2016 update 14 technote.

I just don't think they make it as clear as they could that the connector upgrade was REQUIRED or your sites would break. Nor does the text box for the update, in the CF Admin.. I will discuss how I have raised these to Adobe, later in the "strategic response" section below.

How your sites may fail: what you may see

Since some people may only find this post because they google for the error they may see, let me point those out.

The 403 error they may see

First, if you miss the needed step to upgrade the web server connector after these updates, and therefore the needed "secret" is not being passed in, then requests made to your site(s) through your web server that uses that AJP connector will fail.

Same if your web server is passing in unexpected headers and you need to add the allowedRequestAttributesPattern attribute (see the third point in my TLDR at the opening).

In either case, the error may look like this (on CF2018):

HTTP Status 403 Forbidden
<hr class="line" />
<b>Type</b> Status Report<br /><b>Description</b> The server understood the request but refuses to authorize it.
<hr class="line" />
Apache Tomcat/9.0.21

While on CF2016 (upon applying update 14) will report the Tomcat version as 8.5.42. (These are the Tomcat versions that the Mar 2020 CF updates implement, which address the Ghostcat issue.)

No, the error doesn't more say clearly that "your web server connection needs to pass in the required secret", or "set that allowedrequestattributespattern", and that's for wise security reasons, but it can through folks off.

Then again, it's also possible that you/your users may see a different error, or perhaps even an error page for your site, or even just a blank page, depending on the configuration of error handling in the web server or CF.

(I have been meaning to find and document if there is a corresponding log entry that may confirm these problems are happening. I didn't want to hold up posting everything else before finding and documenting that. I will add that here, if I find it.)

The 503 error they may see

If instead the problem is about the second point in my TLDR at the opening, about the IP address being used to connect from your web server to CF, then you would get an entirely different error after the update (perhaps showing a 503 status code), such as this:

Service Temporarily Unavailable!

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Tomcat/ISAPI/isapi_redirector/1.2.46

There's a bit more about the ip address issue in the CF update technotes, including a reference to what to look for in the CF logs to help know what the IP address should be, if needed--though I will note that where they say that the "starting protocolhandler" message appears in the coldfusion-out.log, I found it instead in the coldfusion-error.log.

Let's dig into this ip address matter a bit more.

More on the IP address attribute: how your web server calls CF

As I have reiterated, if for any of a few reasons CF is being called by your web server using some IP address other than 127.0.0.1, then your requests to CF from that web server will fail. You MUST implement an IP "address" field, in CF's server.xml file, for the AJP connector. This was NOT needed prior to this update, because CF (or technically, the Tomcat embedded within it) was willing to accept requests to that AJP connector from ANY IP address.

Personal opinion/rant: such "open access" that the connector "allowed" may seem tragic (and it's really what has been behind the uproar about this "ghostcat" vulberability), but note that the AJP port is a non-standard one that would be blocked by any decent firewall. (In the case of a native Tomcat installation, the default AJP port was 8009. In the case of CF, it's been different default values for different CF versions, such as 8016 in CF2016, or 8018 in CF2018).

I do think a lot of the hype over ghostcat (and the pressure for Adobe to "fix" this) was overblown, but Tomcat felt the pressure to change it and did, and now Adobe has implemented these Tomcat changes, so we must go along. And of course, it's never a bad thing to have multiple levels of security, from the firewall, to this ip address limitation, to the new "secrets" required.

Anyway, the issue is that AFTER this Tomcat (CF) update, the default behavior is that the AJP connector will be willing by default to accept ONLY requests sent to CF as 127.0.0.1. [Update: when I first wrote this post, I said "or ::1", the ipv6 equivaent to that. I said that based on my reading of the Tomcat docs. But time and the heartbreak of many has proven that it is indeed ONLY 127.0.0.1 that it accepts, by default.]

But first, some readers may be confused, saying "I don't tell my users to use that or any ip address". I get it. That's not the issue. I've said each time above that this is about the ip address that the web server uses to talk to CF. It's NOT about the ip address you or users use to reach your web server. Those are totally different things.

And I mentioned at the outset (in the TLDR) 3 common ways you may find that you get this failure, because your web server is NOT talking to CF via 127.0.0.1. Let's elaborate on them.

Handling if your web server is on a different machine than CF

First, of course, if CF and the web server are on DIFFERENT machines (a rather uncommon mode called "distributed cf" in the CF docs), then of course an address MUST be specified, to allow access into CF/Tomcat from that web server's IP address.

Note that it is possible that someone had already added that "address" attribute previously even before it was "required" for this reason (its name has not changed with this update).

Handling if you have changed the IP address that localhost resolves to

Another way this IP address issue can catch you out even if you have CF and the web server on the same machine is if you have for some reason modified your hosts file, so that "localhost" resolves to an IP address other than the traditional loopback address, 127.0.0.1.

Related to this, note that the default indication in your CF web connector workers.properties file has it connect to "localhost" as the host. It may look like this:

worker.cfusion.host=localhost
. As such, that localhost name resolve now to this "other" IP address you have set in your hosts file, and you'd get the same error, as if CF and the web server were on different machines.

Again, to solve this, add the named IP address as an "address" on the ajp line of server.xml. (You might think you could/should change your hosts file to NOT point it to such a specific address, but that could impact other things you may have going on your machine which are relying on that.)

On Apache using ::1, when it may not seem it should

I have mentioned above how the main problem causing this need to set the "address" attribute is if something clearly would cause your web server to point to CF via some different IP address.

But there's at least one occasion I've found where people are getting this 403 error and the simple solution is to set the "address" to "::1", even though it doesn't seem obvious that they should.

For now, the most common is folks using Apache (again, even on the same machine as CF). This is proving to be so even though a ping of localhost on their machine did not return that as what localhost resolves to. And also even though localhost is the value of the host name listed in the workers.properties file of their CF connector.

If you're in this boat, just try it (address="::1"), if things are failing despite all other attempts.

As I learn more, or if indeed I find that it can happen with IIS in some situations, I will update this.

Beware changing the workers.properties host value

Speaking of the workers.properties file, do beware the temptation of trying to solve this (when CF and the web server are on teh same machine) by changing the worker.cfusion.host=localhost to point to an ip address like ::1. That may well "work" instead, for now.

But understand that the next time add a new connector, that will default to the localhost phrase (unless you change it), and you'd have the same problem again.

If you change the address attribute in the server.xml AJP line, then you will resolve this for your setup once and for all.

Handling when multiple web servers connect to the same CF

What if you may have multiple web servers set to connect into one CF instance?

I'll note that the CF update technotes and the Tomcat docs for the AJP connector refer to this address field as setting "the ip address" (singular) of the web server, at least at this writing.

But I did try entering a comma-separated list of multiple addresses, and not only did CF start and show no errors in any logs, but I had listed the valid IP address as second in the list, and a request DID work--whereas if I listed that bad IP address alone (and restarted CF), I got the 503 shown above.

On secrets vs requiredSecrets

Perhaps you heard about this "ghostcat" problem before Adobe implemented these updates, or you read about locking down the AJP connector for whatever reason. Perhaps you had taken steps on your own to secure the AJP connector in CF, by defining a "requiredSecret" in the server.xml's AJP line.

That was indeed the name of the attribute as Tomcat defined it, before this update by the Tomcat team to the AJP connector which the CF update has implemented.

After this update, though, the attribute is instead named "secret", and the CF update will put such a newly-named "secret" attribute (with that value Adobe chooses) onto the AJP line of your connector in the server.xml file.

This CF update WILL leave in place any requiredSecret you may have defined. There will be no harm, in that Tomcat ignores the requiredSecret attribute from the older versions of Tomcat, though it could be confusing to some observers of that line in that file.

Note also that the needed connector update (above) WILL CHANGE any current secret you may already have had there (which worked previously with that "requiredSecret" value you had created in the server.xml). It will set the workers.properties secret value to the value for the newly created "secret" attribute in the server.xml.

(And all this is what the update SHOULD do, for things to work. I'm just confirming that it does.)

More on the allowedRequestAttributesPattern attribute

I mentioned at the opening (in the TLDR) how requests will also fail after the update if the request sent from the web server to CF (and thus the updated AJP connector in CF) has any headers other than the dozen or so allowed, as listed in the Tomcat docs on the AJP connector.

I mentioned that one situation where this was proving to be needed was if one uses Apache and does rewrites, whether with mod_rewrite or .htaccess files. Such a rewrite is adding a new header.

How would one see what header is being sent in, and what to do with it?

There are a couple of ways to see what headers are sent into a CF request. Unfortunately, you can only see them when the request IS working, such as when you use that .* pattern, or if you roll back the update.

One way is with the CFML function gethttprequestdata (or more specifically, coldfusion gethttprequestdata().headers). Another way is with FusionReactor, and its ability to see the "details" of a request, one of which is a "headers" tab.

As I was working with one client using such Apache rewrites, we found the header added to be "XSite-name".

With that knowledge, one COULD try adding that as THE pattern for that new allowedRequestAttributesPattern attribute. If you do that, note that it's a regex, so with a dash in the name like that, you would have to escape it, so the pattern would be "Xsite\-name". And we did that, it "worked" (in that now the "rewritten" site would run. And FWIW, sites that were NOT being "rewritten" were working regardless.)

What's not clear from the Tomcat docs is whether in doing that, we are ADDING to or OVER-RIDING/overwriting the default values listed on the Tomcat doc page. Again, I plan to do some more research and testing on things. (I find virtually none by anyone else on the web.) If I learn more, I will share it here.

What if I am in cf11 or earlier (or choose not to apply the Mar 2020 update) but want the security fix?

While it's risky for so many reasons to be running CF11 or earlier (since such are not supported/updated by Adobe), note that Adobe did offer a special technote for those in this situation who want to address (manually) the vulnerability closed in this Mar 2020 CF update.

For more, see the technote . It's rather spartan, but in conjunction with the info I share here, some will be able to make sense of it. For others, I'm avaliable to help implement this remotely.


Now, moving on to more "strategic" questions and answers, for you and for Adobe to consider...

How would you have known of the need to upgrade the connector?

First, I will note that the CF update technotes have (since CF2016) offered a table at the bottom of the page, indicating whether an upgrade to the connector is required. (They use the term "recreation", and I'll discuss that more later.)

Second, the technotes for this update (for each of CF2018 update 8 and CF2016 update 14) do also specifically indicate the need to do the connector upgrade, adding additional discussions similar to some of what I have shared here. (They actually share quite a bit more, including about the IP address matter also, which some readers will appreciate. )

How such a connector upgrade was often recommended before, but is required with this update

FWIW, while the technotes have indicated the occasional need (after some updates) to update the web server connector after applying the update, often people would miss this point. Again, many people don't read the technotes. And the small text box shown on the CF Admin page with the updates sometimes DID indicate a need to upgrade the connector, but sometimes it would not (even when it was required, as in the case with this update, at least as I write.)

Or people would skip some updates (since they are indeed cumulative), and so they may not have bothered to look at the technote (or the update text box) for the updates they were skipping. But at least that table at the bottom of the technote did help people to know which updates DID call for the connector to be updated.

And all this is among the reasons I created my other blog post on upgrading the connector, to help folks know when and how to do it, especially since it's so easy since CF2016.

But my point here is to say that if one did skip (or miss) the need to update the connector in prior updates, the problem was merely that there COULD be problems. Sometimes people who missed or skipped the upgrade didn't notice any.

With this latest CF update, however, it is MANDATORY that you update any web server connector(s) after the update, so that it puts in that needed secret that CF generates (and it also updates the connector dll or so. See my o ther blog post for more on that).

How Adobe could make it MORE CLEAR that we might need to take these two extra steps

So as I have related above, one concern I have is that (at least at this writing), the need to upgrade the connector it isn't made as clearly at the top of the technote, nor is it indicated in the text box that appears for these two updates. I hope Adobe will fix that soon. I have asked them directly, in addition to asking here.

How these two "extra steps" will apply to those who skip this updates but do the next one

If you've read the past few points here, I have made the observation that people do sometimes skip CF updates. Well, let's say a new CF update comes out in the next month or months. Then the people who have skipped THIS CF update will of course face the SAME problems, if they created a connector while at an update earlier than this Mar 2020 one.

And we will go through another round of people experiencing broken sites, and complaining loudly in the community.

How a new CF installer for CF2018 and 2016 would really help us now

I should say that the previous point only applies until Adobe may create a new installer for CF2018 or 2016, which bundles this newer connector, while I hope they may consider.

Doing that would also greatly help all those who use the CF Admin update mechanism and currently trip over two problems:

  • the UI issues in the CF admin updates page, which makes it hard to select updates and click on the update buttons (fixed in CF2018 update 7 and CF2016 update 13)
  • the issue that those on updates earlier than those from June 2019 (CF2018 update 4 and CF2016 update 11) must FIRST implement THOSE CF updates before they can apply any later ones
And tragically, all these problems also affect anyone currently installing either CF 2018 or 2016, who then try to apply "the latest" updates.

These changes are indeed ALL that this update implements

Finally, some people have wondered if this update may have address any other issues (like the long-awaited and now-pressing samesite cookie issue that I wrote about this week). Sadly, it did not.

All it addressed was these issues (and updating Tomcat related to them).

FWIW, I did a compare of all the files in CF before and after the change, and this set of changes (the addition of the secret in the server.xml, and the updating of Tomcat-specific jar files related to this security change) were indeed the only changes that the update performed.

Pro tip: Someone might want to say, "well, duh, Charlie. You could look at the hotfix_filelist.log in the update's folder, under cfusion/hf-updates" folder, to see what files were changed. But that's actually not quite so.

While that file DOES list ALL the files changed during the update, we have to keep in mind that since again CF updates are CUMULATIVE, each update lays down ALL the latest versions of files changed in ALL the previous updates. So you can't just look at that file and see, "yep, only these 20 or so files were changed due specifically to THIS latest update". Again, I did a compare of the complete CF folder, before and after the update, to confirm things.

Note that Adobe did NOT update the Tomcat version, only the AJP CONNECTOR itself

This section is an addition since the original posting. As I was digging into things (and helping someone with the update), I found out that whereas I thought this CF update updated Tomcat for us, I can say now that it did NOT. CF is running the same Tomcat version that would have been implemented if you'd applied the CF updates from Sep 2019, which are update 5 for CF2018, and update 12 for CF2016. You can confirm this in the CF Admin "settings summary" page, which will report (after those updates, and including this one) that CF2018 is running on Tomcat 9.0.21, and CF2016 is running on 8.5.42.

The thing is, the "official" Tomcat version which implemented the changes for Ghostcat (discussed in this blog post here) were introduced in Tomcat 9.0.31 and 8.5.51, respectively. But CF is NOT running those Tomcat versions (as of this update).

So what gives? We can only conclude that Adobe took a shortcut, and ONLY updated the AJP connector. That's not necessarily "wrong". Maybe it was expedient, allowing them to get this update out ASAP, without dealing with a "full" Tomcat update.

But it is confusing, for those who dig into the whole Ghostcat issue, and who understand Tomcat versions. I just wanted to point that out here, for those who may care to know (or who may dig further, in which case I welcome comments, of course).

Conclusion, and a plea

So phew, that's another mega post. I realize that some will have stopped at the top. Some may have dropped off on the way. My goal with these it to provide enough info for anyone who takes the time to read them, to understand the issue more completely than can be covered in twitter, slack, facebook, linkedin, etc.

If you may feel overwhelmed in considering it all, let me note that I can certainly help you implement these updates, typically in less than 15 minutes. See my consulting page for more on my rates, approach, satisfaction guarantee, and more.

And if you're looking at how to use the current downtime with the pandemic, I can help also with any CF updates, as well as connector upgrades, and JVM updates, and lots more related to install/config/admin of CF, troubleshooting/tuning, upgrading/migrating, deployment of CF on Docker, and lots more.

Finally, if you read all the way to the bottom here, gold star for you!

Could you return the favor, and take a moment to share your thoughts? I put a LOT of time into posts like these, hoping to help people (and indeed, they may well help folks so that they don't need to leverage my consulting!) But often I find I get very little feedback, pro or con. Your encouragement is appreciated, and even brickbats are tolerated in helping make things better.

For more content like this: Need more help with problems?
  • If you may prefer direct help, rather than digging around here/elsewhere or via comments, I can help via my consulting services
  • See that for more on how I can help a) over the web, safely and securely, b) usually very quickly, c) teaching you as we go, and d) with satisfaction guaranteed
Comments
Thanks again Charlie for covering our rears! Stay safe and great work as always.
Greg - BMChosting team lead
# Posted By Greg Eslick | 3/21/20 10:48 AM
And thank you, Greg. Glad to help, and all the more to hear that in helping a host, I'm helping many times more people still! :-)
# Posted By charlie arehart | 3/21/20 12:57 PM
Readers: if you have already read my post here, as originally written last night, please note that I have added a new 3rd point (a possible step you may need to do after the update) in the TLDR section at the top.

Also, I have added a new section at the bottom, "Note that Adobe did NOT update the Tomcat version, only the AJP CONNECTOR itself", which may interest some of you.

For now, I didn't think it worth creating an entirely new/different post for these points, as they don't really affect most people--and would only be "new" info on the post for those who read it since I posted this last night.
# Posted By Charlie Arehart | 3/21/20 6:55 PM
Thanks Charlie, really appreciate this detailed info!
# Posted By Chris | 3/22/20 8:00 PM
Thanks, Chris.
# Posted By Charlie Arehart | 3/22/20 8:01 PM
First of all, many thanks to you Charlie for your helpful posts, which I have been reading for many, many years and which have often saved my day here in Germany.

After updating ColdFusion 2016 from Update 12 to Upadate 14 on my development server, I encountered a nasty little bug.

For years I have been using "localhost" in the "workers.properties" configuration file for all server instances as host (for example: worker.cfusion.host=localhost) without any problems. After the update to ColdFusion 2016 Update 14 it was no longer possible to establish connections to the ColdFusion Server instances, although I added all new secret values ??to the server instances in the "workers.properties" file (as described by Adobe) manually.

It took me a while to try to use the IP address 127.0.0.1 instead of "localhost" (for example: worker.cfusion.host=127.0.0.1), although this is the same (under the hood). But after this (not really logical) change, everything runs without errors and all connections to the ColdFusion Server instances can be made correctly. It seems that ColdFusion Update 14 does not like "worker.cfusion.host=localhost" anymore. :-(

My server environment:
- Debian Linux 9.12
- Apache 2.4.25
- Nginx 1.16.1 (runs as proxy)
- ColdFusion 2016 Update 14

Maybe this comment will also help other developers in the ColdFusion community. If so, I would be happy.

Greetings from Germany
Knut
# Posted By Knut | 3/24/20 7:01 AM
Thanks for the kind regards, Knut, and hapoy to have helpers (indeed for so mahy years).

As for your issue, it's not a bug. I suspect it's that you have a hosts file that sets localhost on your machine to your own ip address.

If you do a ping localhost or other command at the command line on the machine, what up address does it report? I suspect it's not the traditional localhost/loop back address.

And that's fine normally, but again one of the Tomcat changes (which the CF update incorporates) is default support for ONLY requests from the traditional localhost/loopback addresses, 127.0.0.1 or its ipv6 equivalents, as I indicated above.

And your change of the workers.properies is certainly one way to force the cf connector to "use" that address (so things work with the change)

But as I also noted, you could have also configured the address atrribute on the ajp element of the server.xml file to name the ip address also.

In your case, though, it may be easiest/better to just reconsider the hosts file setting that may be redirecting the localhost domain name to your specific ip address.

There are various reasons people have done that over the years. If you can't recall yours, maybe try commenting it out (the localhost line in the hosts file) and see how it goes.

If it works, you could then set the workers.properties back to localhost. Note that any future connector upgrade or creation will do that, so this is why I think it's the better solution than you changing the file.

Let us know how it goes. If you confirm it's what I suspect, I will tweak the post to add a mention of this point. Thanks for bringing it up. Hope I've solved it for you.
# Posted By Charlie Arehart | 3/24/20 7:21 AM
Knut, if you got emailed my last reply, I hope you may see this before replying to it. Please visit the site to see an update I made to the 3rd and next to last paragraphs of my previous comment, as I realized those were necessary for your situation.
# Posted By Charlie Arehart | 3/24/20 7:32 AM
I have also now added a new section on this point, which Knut raised and about which I have guessed at an answer. I will look forward to his reply/confirmation, but since I suspect it really is his issue, I added the new section <a href="#localhostchanged">Handling if you have changed the IP address that localhost resolves to</a>, and I also added a mention of it on the second point in the TLDR at the top. I could see it hitting people unexpectedly, and there are the 3 possible solutions one could consider, as I elaborate in that section.
# Posted By charlie arehart | 3/24/20 10:44 AM
Charly, thank you for your answer.

I checked the entries in the "/etc/hosts" file. It all looks completely normal and unchanged. The IP address 127.0.0.1 and :: 1 are also bound to "locahost".

As you suggested, I'll try the "address" attribute in the server.xml file to name the ip address in the connector. I will report whether this alternative way also works.

Thank you for your kind support.
# Posted By Knut | 3/25/20 6:00 AM
Thanks, but most simply, what ip addtesss do you get if you ping localhost? And what address otherwise were you planning to add as the "address" in server.xml? Very curious to hear the conclusion for you.
# Posted By Charlie Arehart | 3/25/20 6:04 AM
When I do a "ping localhost" at the console I get:

PING localhost(server2.xyz.de (::1)) 56 data bytes
64 bytes from server2.xyz.de (::1): icmp_seq=1 ttl=64 time=0.021 ms

I've masked the real domain name with "xyz".
# Posted By Knut | 3/25/20 6:51 AM
I did some trial/error investigation.

Example 1
- Adding the additional attribute "address=localhost" to the connector in server.xml
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: No connectivity

Example 2
- Adding the additional attribute "address=127.0.0.1" to the connector in server.xml
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: No connectivity

Example 3
- Adding the additional attribute "address=127.0.0.1" to the connector in server.xml
- Commenting OUT the "worker.cfusion.host=localhost" entry in the "workers.properties" file
RESULT: No connectivity

Working examples (in my case):
- NO additional attribute "address=127.0.0.1" to the connector in server.xml
- Only using "worker.cfusion.host=127.0.0.1" in the "workers.properties" file
RESULT: Connectivity

- Additional attribute "address=127.0.0.1" to the connector in server.xml
- Using "worker.cfusion.host=127.0.0.1" in the "workers.properties" file
RESULT: Connectivity

So in my view it does not make any difference to add the additional attribute "address=127.0.0.1" (or not) to the connector in server.xml. It seems to be mandantory (at least in my case) to use "worker.cfusion.host=127.0.0.1" in the "workers.properties" file.

My original overall configuration is basically based on the descriptions from Adobe at: https://helpx.adobe.... Under point 4, "worker.cfusion.host=localhost" is also specified, but at least in my case this no longer works with "ColdFusion Update 14". I have to use IP 127.0.0.1 instead of localhost in the "workers.properties" file.
# Posted By Knut | 3/25/20 7:26 AM
Thanks for all that.

So first, I agree something seems amiss. I don't think it's a Linux (or Unix thing), as some with such curious challenges have wondered. Time will tell.

You show the ping reporting ::1. That SHOULD be accepted by the updated ajp connector, by default.

But since you're scrambling to find anything that works, how about adding address="::1" on the ajp connector line in server.xml? You don't show trying that among your many attempts.

As I explain in my post, I recommend against changing the workers.properties file for such workarounds if possible, because the next connector install would set it to localhost by default. And while connector upgrades don't typically change that, one could.

Let us know what you find. Also, did you confirm 0 errors in the update log?
# Posted By Charlie Arehart | 3/25/20 8:06 AM
Short note:

>> ...but since I suspect it really is his issue, I added the new section ... Handling if you have changed the IP address that localhost resolves to...

I'm not a linux erxpert, but I think in my case "localhost" resolves to "127.0.0.1" and "::1". This is the server config I'm using since years. This has never been changed.
# Posted By Knut | 3/25/20 8:10 AM
>> But since you're scrambling to find anything that works, how about adding address="::1" on the ajp connector line in server.xml? You don't show trying that among your many attempts.

Okay, I will also try "::1" and give a report to you.
# Posted By Knut | 3/25/20 8:13 AM
Did you write your previous comment ("short note") before or after reading my comment above it? I can't tell, and I realize our messages could have passed each other in the ether.

Either way, I don't quite get what you were getting at. But the real key is to here how things go, per your (and my) last comment. :)
# Posted By Charlie Arehart | 3/25/20 8:19 AM
Oh, and about my mention earlier of "I still don't think it's a unix thing", I was confusing a separate discussion I'd been having with someone privately. I realize you said you were on Linux, instead. I will correct my comment, to avoid confusing later readers.

Even so, I will say I don't think your challenge is a Linux thing, either, if anyone reading may wonder (as some have been questioning, about possible OS differences).
# Posted By Charlie Arehart | 3/25/20 8:24 AM
>> Did you write your previous comment ("short note") before or after reading my comment above it? I can't tell, and I realize our messages could have passed each other in the ether.

Our posts unfortunately overlap.

I've tried to use "::1" for the additional address attribute in the connector and it works. Great.

Working examples (in my case):

- Additional attribute address="::1" in the connector in the "server.xml" file
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: Connectivity
CONCLUSION: I will use this in this form for all server instances in the future.


I think that normally the connector should resolve "localhost" to "127.0.0.1" and/or "::1" as standard. This is what the connector did bevor the U14 update. I can't see the reason why it is mandantory (in my case) to add the additional "address" attribut in the connector after updating to CF-2016 U14. This is a wired behavior (in my view).

Thank you for your tips and your support.
# Posted By Knut | 3/25/20 8:43 AM
Short additional note:

I just realized, that my two machines ("development" an "production" server) deliver different results for "ping localhost".

Output for development server "ping localhost" at the console:

PING localhost(server2.xyz.de (::1)) 56 data bytes
64 bytes from server2.xyz.de (::1): icmp_seq=1 ttl=64 time=0.021 ms

Output for production server "ping localhost" at the console:

PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.023 ms


I've contacted my hoster to check, why the outputs are different.
# Posted By Knut | 3/25/20 9:19 AM
Yep, I agree that it should not be "needed", but I proposed it in case it might help--and I'm really glad to hear that it did. I had mentioned how the Tomcat docs said that localhost addresses would be supported out of the box, with the update to AJP (which CF incorporates).

The specific wording (on the Tomcat doc page I link to above) says, "By default, the connector will listen on the loopback address." It's looking like that may be only 127.0.0.1, which I find shocking. I almost think something ELSE must explain things.

But if you want to try it, you can use multiple addresses (as I do also indicate above), so address="127.0.0.1,::1". That would cover you either way.

Again, I agree: it should NOT be needed. But until we (all) figure out what's amiss here, it may be the best recommendation, and I could update my post to reflect it. That said, I already added a new section above (as mentioned in a previous comment) before I heard back from you, so I would like to do a little more testing of confirmation before updating the post.

Still, I'll let this comment stand for now, for those with the gumption to read down to here among these now 7000+ words. :-)
# Posted By charlie arehart | 3/25/20 12:44 PM
Thanks for the very detailed information Charlie. As usual you make valid points. You're articles are always a good reference so please keep up the good work. Stay safe. Cheers!
# Posted By Miguel-F | 3/25/20 1:04 PM
>> But if you want to try it, you can use multiple addresses (as I do also indicate above), so address="127.0.0.1,::1". That would cover you either way.

Good suggestion, I can try it.
Do you think that address="127.0.0.1,::1,localhost" should also work?
# Posted By Knut | 3/25/20 1:21 PM
Did some testing.
Neither address="127.0.0.1,::1" works, nor address="127.0.0.1,::1,localhost" works.

In my case it's only working with: address="::1"

Best regards
Knut
# Posted By Knut | 3/25/20 1:47 PM
Hi Charlie,

Thank you very much for this post and the much needed extra explanation about what was happening. One slight edit if I may suggest, the Adobe kb article says that that allowedRequestAttributesPattern is suppose to be ".*" and not "*". It took me a while before I realized I had gotten it wrong.
# Posted By Samuel Smith | 3/26/20 4:30 PM
Miguel: thanks very much.

Knut: that's odd. I had done it (adding two), and it did work (as I said in the post, I even reversed the order and put one that would fail first, and it still worked). I wonder if there can be some other explanation.

But glad to see that at least just ::1 worked. I have been doing some work that might suggest that Tomcat is ONLY by default accepting from 127.0.0.1, and not ALSO ::1 as one would expect (and I'd said). The odd thing again is that I find that to be an issue on some machines but not others. Will share more as I learn more.
# Posted By charlie arehart | 3/26/20 4:56 PM
Samuel, thank you very much (and sorry, I had meant to also comment on your point in my last comment). That was indeed a mistake, in both this post and the shorter one I did at the Adobe CF portal (now listed at the top here). I apologize to everyone for that mistake, and of course I have corrected it here and there.
# Posted By charlie arehart | 3/26/20 5:02 PM
I was unable to get ACF update 8 to work at all after following the suggestions in this post.

Prior to seeing this post, my production server failed after I installed update 8 (Win Server2018, Apache 2.4). I tried updating the web connector, but no joy. I panicked and was able to uninstall the update and got it running again at update 7.

Tried the update again on my development box (Win10, Apache 2.4) with worse results. I tried the suggested fixes in this post, but couldn't get the CF service to restart. Gave up and reinstalled the whole enchilada. Ran update 4 and then update 8 before even setting up the web connector. Same problem: service wouldn't start and wouldn't start after trying to change server.xml AJP settings. I'm wondering if there is something specific to the combination of Windows and Apache that is causing me grief. Any suggestions appreciated.
# Posted By Adam Read | 3/26/20 5:29 PM
No, Adam. Update 8 would not have anything that would itslef keep cf from starting. That problem could be caused with ANY CF update, and I pointed above to my blog post on sovimg such problems.

PLEASE read that. You will get cf running again.

Or I can help you directly, likely in as little as 15 minutes. You won't pay for time you don't find valuable.
# Posted By Charlie Arehart | 3/26/20 7:31 PM
Also, Adam, since you're using Apache, let me add that once you get CF running again (see my last comment), you should then see if the original problems you had (with the connector) is fixed by using allowedRequestAttributesPattern=".*", as discussed in prior comments and a correction I made yesterday.

I will be doing more research and documentation on the matter (why it's needed, when, and better alternatives to opening things up that way).
# Posted By charlie arehart | 3/27/20 1:58 PM
Adding to my last comment, I just posted some more detail in a comment on a CF forum message from someone else, and I shared there some more detail--my initial findings. It may interest some readers, until I can get more documented:

https://community.ad...
# Posted By charlie arehart | 3/27/20 2:04 PM
Thanks as always, Charlie, for these posts. I feel like every CF update over the past 20 years has given me some degree of anxiety, but it's great to know you're always here to help us through.
# Posted By Rocky Rosen | 3/29/20 6:32 PM
Thanks, so much, Rocky. Happy to be able to help.
# Posted By Charlie Arehart | 3/29/20 6:34 PM
Copyright ©2020 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting