[Looking for Charlie's main web site?]

How and why your sites may break, and what to do, after applying March 2020 update to CF2018 or 2016

Note: This blog post is from 2020. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.
This is a critical warning to anyone who may apply the recent CF2018 Update 8 or CF2016 Update 14, released Tuesday of this week (on Mar 20, 2020). And readers in the future should note it will apply if and as you may update CF from any update BEFORE this one to any update AFTER this one.

To be clear, I do not mean with this warning to suggest that you should NOT apply the update! It implements an important security fix.

Instead, it's that after applying it, your CF web sites served via IIS or Apache WILL likely break initially, until you take one at least and perhaps two extra steps. The good news is that these steps are both easy and documented by Adobe in the update technotes, but they do require that someone do them, if needed. Let me explain.

[Update: I did an abbreviated version of this post on the Adobe CF portal: Three reasons your sites may break, and how to fix them, after applying March 2020 update to CF2018 or 2016. Note I also titled it differently. Just trying many ways to get people's attention. That post may interest some, either to read first (but my TLDR below also tries to abbreviate things also), or especially if you may prefer to give others a link to a post on this matter that is not as "dense" as this one. :-) I do point to this post from there, of course, for the many additional details that some may appreciate.]

Sadly, because many people don't bother to read the CF update technotes (linked to below), and they just apply the CF updates, they are not noticing this issue until they or their users start screaming because their sites are down. There's also a fair bit of "screaming" in the CF community, and folks responding may not know the info that I (or Adobe) have shared, to get things "working again", so I hope this helps bring some calm, and most important the clear solution/s needed.

TLDR:

For those who favor brevity, here's my attempt at a "brief" explanation. More details follow it.

The issue(s) stem from important security fixes that Adobe has implemented (related to the "Ghostcat" Tomcat vulnerability), which apply to the "AJP connector" that is enabled by default for connections between web servers like IIS or Apache and CF.

After applying these latest Mar 2020 CF updates, web sites served from IIS or Apache will likely fail. Here, briefly, is what to do and why (update since original posting: I added the 3rd point):

  • First, you MUST "upgrade" any CF web server connector for IIS or Apache after applying the update. The issue is that the update puts a new "secret" into the CF configuration (the AJP line of the server.xml file), and the wsconfig upgrade puts that "secret" into the web connector (workers.properties) configuration file. (See below for where to find those, etc.)

    The issue is that while such a secret was optional before, it's now mandated (by the Tomcat AJP connector update), but the CF update only does "one side" of the required update. This needed connector "upgrade" does the other.

    Until you run that, requests will fail (with a 403). I discuss how to do it, below. (And even if you may have specified such a requiredSecret previously in the server.xml, the new Tomcat version which this update implements has changed the attribute name for the secret, so your prior settings won't work.)

  • Second, there are a few scenarios where you will likely get 503 errors and need to implement a required IP "address" attribute, which would also be set for the AJP connector in CF's server.xml file.

    The issue is that PRIOR to this update, CF (the Tomcat AJP connector embedded within it) was willing to accept requests from ANY ip address (which wasn't tragic, in that it's non-standard port that was blocked by most firewalls). So you didn't HAVE to set such an "address" attribute, though perhaps you did. But AFTER this update, the default behavior (if no "address" is set) is that the AJP connector is willing to accept requests ONLY made to CF as 127.0.0.1, presumed to be from a web server on the same machine.

    Certainly that is an issue if the web server is on some OTHER machine (a rather unusual setup for most CF folks), or if you have modified your hosts file so that localhost binds to some other IP address on your machine.

    But even if on the web server is on the same machine, the requests to CF will fail if it somehow sends requests to CF as some OTHER ip address (even ::1, the IPV6 equivalent), then you MUST set this "address" field on the AJP connector naming that IP address, or requests will fail (with a 503). I discuss below a bit more about this address attribute.

    I discuss below those other scenarios, which includes even simply using Apache, which I and others are finding that for some reason it sends requests to ::1, even though nothing obvious would suggest that it should. More below.

  • Third, you MAY need to make still one more change, if you STILL get errors (403 errors) after making the changes above, to add yet another new Tomcat AJP connector attribute, allowedRequestAttributesPattern, on the AJP connector line (the same one discussed above) found in the server.xml file. The shortcut to get things working (if this is required) is to set allowedRequestAttributesPattern=".*" (that's a dot and an asterisk).

    This issue is that after this update, the Tomcat connector has been changed (again, by the Tomcat team) to have a certain limited number of request attributes that the connector will allow to be passed in. If for any reason your web server passes in other headers, the request will fail. You could (and should) find what headers those are, and only enable those, but it's harder than it may seem, for a number of reasons.

    But I will note that I've found that if you simply use Apache's and rewrite CF page requests using mod_rewrite or .htaccess, that will indeed add a header and trigger this problem. So again, the simple solution above is the quickest way to get things working again. More below.

To be clear, if you only access your web sites through CF's internal web server (which by default will be a port like 8500), or if you use a reverse proxy or other means to connect your web server to CF, then these three steps do not apply to you.

Finally, setting asides the details of this update for a moment, if you may have problems after applying the update (or you feel that none of the 3 points above apply for you), see yet another blog post I've done, "Having problems after applying a CF update? What to check, and how to recover!"

For some readers, the above is all they need to know and they can take it from there.

For most though, I suspect they will need or want more detail which I share or link to below. I also address some reasonable questions one may have.

I've organized the post into two sections. First I cover a few "tactical response" points (how to deal with the problem), and then a few "strategic response" points (some thoughts for you and for Adobe to consider):

Tactical response (dealing with the problem immediately):

Strategic response (thinking about the problem)

Let's talk first about your "tactical response" to this update and the troubles it may cause for you and others. First and foremost, how do you "fix" the problem. Let me tackle the web connector upgrade first.

How you can easily upgrade your web server connector

Fortunately, it is QUITE simple. You just need to run the CF "web server configuration" or wsconfig tool, and if you use the UI for it then it's now especially easy since CF2016, since you just need to select a connector and click the "upgrade" button. I discuss this in more detail (including command-line options) in a Nov 2019 post discussing when and how to upgrade the web server connector.

As I will discuss later, the need to do it is not new, only the mandatory nature after this update.

Finally, note that you MUST update ALL connectors that you have, which point to the CF instance that you have updated. Again, until you do, those will fail.

"I never ran this wsconfig tool, so does this not apply to my sites?"

I have said above that this "connector upgrade" is needed for sites connected to CF from a web server like IIS or Apache, as have been connected using the Adobe CF "web server configuration" tool.

You may say, "I haven't used that tool". Well, let me clarify that in CF2016 and earlier, the CF installer would also ask (if it detected such a web server) whether you wanted to connect CF to that web server. If you said yes, then THAT process created this "connector", and you DO need therefore to "upgrade" it. (If you may not have noticed, in CF2018 that part of the installer no longer exists, so anyone wanting to connect CF to their web sites on such a server this way would have to run the wsconfig tool.)

So as I noted in the opening, the only people who can ignore this "connector upgrade" are those who really DO NOT use this connector, which is

  • those who only use the CF built-in web server (as may be typical of those doing merely development or testing), and
  • those who may use a reverse proxy or other means to connect their web server to CF (where the web server "forwards" requests to that internal web server, without using the "connector"). That topic is beyond the scope of this post, so I will not elaborate on it.

Finding more about the update from Adobe

You might wonder how you could have known of this "need to update the web server connector" (or the other IP address issue, which may affect some).

I will note that Adobe has indeed offered considerable information about the most recent update, including mentioning that you needed to upgrade the web connector. But it could have been lost/missed. Also, some people don't bother reading any of these things. They just click the button to update the connector, and so were blithely unaware of the consequences.

For more, see a blog post announcing the update(s) earlier this week, as well as the CF 2018 update 8 technote and the CF2016 update 14 technote.

I just don't think they make it as clear as they could that the connector upgrade was REQUIRED or your sites would break. Nor does the text box for the update, in the CF Admin.. I will discuss how I have raised these to Adobe, later in the "strategic response" section below.

How your sites may fail: what you may see

Since some people may only find this post because they google for the error they may see, let me point those out.

The 403 error they may see

First, if you miss the needed step to upgrade the web server connector after these updates, and therefore the needed "secret" is not being passed in, then requests made to your site(s) through your web server that uses that AJP connector will fail.

Same if your web server is passing in unexpected headers and you need to add the allowedRequestAttributesPattern attribute (see the third point in my TLDR at the opening).

In either case, the error may look like this (on CF2018):

HTTP Status 403 Forbidden
<hr class="line" />
<b>Type</b> Status Report<br /><b>Description</b> The server understood the request but refuses to authorize it.
<hr class="line" />
Apache Tomcat/9.0.21

While on CF2016 (upon applying update 14) will report the Tomcat version as 8.5.42. (These are the Tomcat versions that the Mar 2020 CF updates implement, which address the Ghostcat issue.)

No, the error doesn't more say clearly that "your web server connection needs to pass in the required secret", or "set that allowedrequestattributespattern", and that's for wise security reasons, but it can through folks off.

Then again, it's also possible that you/your users may see a different error, or perhaps even an error page for your site, or even just a blank page, depending on the configuration of error handling in the web server or CF.

(I have been meaning to find and document if there is a corresponding log entry that may confirm these problems are happening. I didn't want to hold up posting everything else before finding and documenting that. I will add that here, if I find it.)

The 503 error they may see

If instead the problem is about the second point in my TLDR at the opening, about the IP address being used to connect from your web server to CF, then you would get an entirely different error after the update (perhaps showing a 503 status code), such as this:

Service Temporarily Unavailable!

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Tomcat/ISAPI/isapi_redirector/1.2.46

There's a bit more about the ip address issue in the CF update technotes, including a reference to what to look for in the CF logs to help know what the IP address should be, if needed--though I will note that where they say that the "starting protocolhandler" message appears in the coldfusion-out.log, I found it instead in the coldfusion-error.log.

Let's dig into this ip address matter a bit more.

More on the IP address attribute: how your web server calls CF

As I have reiterated, if for any of a few reasons CF is being called by your web server using some IP address other than 127.0.0.1, then your requests to CF from that web server will fail. You MUST implement an IP "address" field, in CF's server.xml file, for the AJP connector. This was NOT needed prior to this update, because CF (or technically, the Tomcat embedded within it) was willing to accept requests to that AJP connector from ANY IP address.

Personal opinion/rant: such "open access" that the connector "allowed" may seem tragic (and it's really what has been behind the uproar about this "ghostcat" vulberability), but note that the AJP port is a non-standard one that would be blocked by any decent firewall. (In the case of a native Tomcat installation, the default AJP port was 8009. In the case of CF, it's been different default values for different CF versions, such as 8016 in CF2016, or 8018 in CF2018).

I do think a lot of the hype over ghostcat (and the pressure for Adobe to "fix" this) was overblown, but Tomcat felt the pressure to change it and did, and now Adobe has implemented these Tomcat changes, so we must go along. And of course, it's never a bad thing to have multiple levels of security, from the firewall, to this ip address limitation, to the new "secrets" required.

Anyway, the issue is that AFTER this Tomcat (CF) update, the default behavior is that the AJP connector will be willing by default to accept ONLY requests sent to CF as 127.0.0.1. [Update: when I first wrote this post, I said "or ::1", the ipv6 equivaent to that. I said that based on my reading of the Tomcat docs. But time and the heartbreak of many has proven that it is indeed ONLY 127.0.0.1 that it accepts, by default.]

But first, some readers may be confused, saying "I don't tell my users to use that or any ip address". I get it. That's not the issue. I've said each time above that this is about the ip address that the web server uses to talk to CF. It's NOT about the ip address you or users use to reach your web server. Those are totally different things.

And I mentioned at the outset (in the TLDR) 3 common ways you may find that you get this failure, because your web server is NOT talking to CF via 127.0.0.1. Let's elaborate on them.

Handling if your web server is on a different machine than CF

First, of course, if CF and the web server are on DIFFERENT machines (a rather uncommon mode called "distributed cf" in the CF docs), then of course an address MUST be specified, to allow access into CF/Tomcat from that web server's IP address.

Note that it is possible that someone had already added that "address" attribute previously even before it was "required" for this reason (its name has not changed with this update).

Handling if you have changed the IP address that localhost resolves to

Another way this IP address issue can catch you out even if you have CF and the web server on the same machine is if you have for some reason modified your hosts file, so that "localhost" resolves to an IP address other than the traditional loopback address, 127.0.0.1.

Related to this, note that the default indication in your CF web connector workers.properties file has it connect to "localhost" as the host. It may look like this:

worker.cfusion.host=localhost
. As such, that localhost name resolve now to this "other" IP address you have set in your hosts file, and you'd get the same error, as if CF and the web server were on different machines.

Again, to solve this, add the named IP address as an "address" on the ajp line of server.xml. (You might think you could/should change your hosts file to NOT point it to such a specific address, but that could impact other things you may have going on your machine which are relying on that.)

On Apache using ::1, when it may not seem it should

I have mentioned above how the main problem causing this need to set the "address" attribute is if something clearly would cause your web server to point to CF via some different IP address.

But there's at least one occasion I've found where people are getting this 403 error and the simple solution is to set the "address" to "::1", even though it doesn't seem obvious that they should.

For now, the most common is folks using Apache (again, even on the same machine as CF). This is proving to be so even though a ping of localhost on their machine did not return that as what localhost resolves to. And also even though localhost is the value of the host name listed in the workers.properties file of their CF connector.

If you're in this boat, just try it (address="::1"), if things are failing despite all other attempts.

As I learn more, or if indeed I find that it can happen with IIS in some situations, I will update this.

Beware changing the workers.properties host value

Speaking of the workers.properties file, do beware the temptation of trying to solve this (when CF and the web server are on teh same machine) by changing the worker.cfusion.host=localhost to point to an ip address like ::1. That may well "work" instead, for now.

But understand that the next time add a new connector, that will default to the localhost phrase (unless you change it), and you'd have the same problem again.

If you change the address attribute in the server.xml AJP line, then you will resolve this for your setup once and for all.

Handling when multiple web servers connect to the same CF

What if you may have multiple web servers set to connect into one CF instance?

I'll note that the CF update technotes and the Tomcat docs for the AJP connector refer to this address field as setting "the ip address" (singular) of the web server, at least at this writing.

But I did try entering a comma-separated list of multiple addresses, and not only did CF start and show no errors in any logs, but I had listed the valid IP address as second in the list, and a request DID work--whereas if I listed that bad IP address alone (and restarted CF), I got the 503 shown above.

On secrets vs requiredSecrets

Perhaps you heard about this "ghostcat" problem before Adobe implemented these updates, or you read about locking down the AJP connector for whatever reason. Perhaps you had taken steps on your own to secure the AJP connector in CF, by defining a "requiredSecret" in the server.xml's AJP line.

That was indeed the name of the attribute as Tomcat defined it, before this update by the Tomcat team to the AJP connector which the CF update has implemented.

After this update, though, the attribute is instead named "secret", and the CF update will put such a newly-named "secret" attribute (with that value Adobe chooses) onto the AJP line of your connector in the server.xml file.

This CF update WILL leave in place any requiredSecret you may have defined. There will be no harm, in that Tomcat ignores the requiredSecret attribute from the older versions of Tomcat, though it could be confusing to some observers of that line in that file.

Note also that the needed connector update (above) WILL CHANGE any current secret you may already have had there (which worked previously with that "requiredSecret" value you had created in the server.xml). It will set the workers.properties secret value to the value for the newly created "secret" attribute in the server.xml.

(And all this is what the update SHOULD do, for things to work. I'm just confirming that it does.)

More on the allowedRequestAttributesPattern attribute

I mentioned at the opening (in the TLDR) how requests will also fail after the update if the request sent from the web server to CF (and thus the updated AJP connector in CF) has any headers other than the dozen or so allowed, as listed in the Tomcat docs on the AJP connector.

And I'd mentioned that you could just override that new default by setting allowedRequestAttributesPattern="." on the AJP connector in the server.xml file.

I also mentioned that one situation where this was proving to be needed was if one uses Apache and does rewrites, whether with mod_rewrite or .htaccess files. Such a rewrite is adding a new header.

How did we find that? And what could we do with that info?

How would one see what header is being sent in, which would cause the 404?

One way that may work is if you enable debug-level logging for the AJP connector. You can do that (for IIS) in the isapi_redirect.properties file with:

log_level=debug

Or for Apache in mod_jk.conf (or jk.conf or http.conf, or wherever the AJP configuration is), with:

JkLogLevel info

I have observed that that logging shows what headers come in on a request, though I will admit I have not done it while someone was experiencing the 404 due to this "allowed headers" problem. I've seen them when requests DID work, for sure.

Also, from a CF perspective, there are a couple of ways to see what headers are sent into a CF request, at least when the request IS working, such as when you may use that .* pattern (to get past the problem temporarily), or if you may roll back the update.

First, there's the CFML function gethttprequestdata (or more specifically, coldfusion gethttprequestdata().headers). Second is with FusionReactor, and its ability to see the "details" of a request, one of which is a "headers" tab.

That's how we found that it was a header called "site-name".

What to do once you have that header

With that knowledge, one COULD try adding that as THE pattern for that new allowedRequestAttributesPattern attribute (rather than use the "/*" value to "allow any).

If you do that, note that it's a regex, so with a dash in the name like that, you would have to escape it, so the pattern would be "Xsite\-name". And we did that, it "worked" (in that now the "rewritten" site would run. And FWIW, sites that were NOT being "rewritten" were working regardless.)

What's not clear from the Tomcat docs is whether in doing that, we are ADDING to or OVER-RIDING/overwriting the default values listed on the Tomcat doc page. Again, I plan to do some more research and testing on things. (I find virtually none by anyone else on the web.) If I learn more, I will share it here.

What if I am in cf11 or earlier (or choose not to apply the Mar 2020 update) but want the security fix?

While it's risky for so many reasons to be running CF11 or earlier (since such are not supported/updated by Adobe), note that Adobe did offer a special technote for those in this situation who want to address (manually) the vulnerability closed in this Mar 2020 CF update.

For more, see the technote . It's rather spartan, but in conjunction with the info I share here, some will be able to make sense of it. For others, I'm avaliable to help implement this remotely.


Now, moving on to more "strategic" questions and answers, for you and for Adobe to consider...

How would you have known of the need to upgrade the connector?

First, I will note that the CF update technotes have (since CF2016) offered a table at the bottom of the page, indicating whether an upgrade to the connector is required. (They use the term "recreation", and I'll discuss that more later.)

Second, the technotes for this update (for each of CF2018 update 8 and CF2016 update 14) do also specifically indicate the need to do the connector upgrade, adding additional discussions similar to some of what I have shared here. (They actually share quite a bit more, including about the IP address matter also, which some readers will appreciate. )

How such a connector upgrade was often recommended before, but is required with this update

FWIW, while the technotes have indicated the occasional need (after some updates) to update the web server connector after applying the update, often people would miss this point. Again, many people don't read the technotes. And the small text box shown on the CF Admin page with the updates sometimes DID indicate a need to upgrade the connector, but sometimes it would not (even when it was required, as in the case with this update, at least as I write.)

Or people would skip some updates (since they are indeed cumulative), and so they may not have bothered to look at the technote (or the update text box) for the updates they were skipping. But at least that table at the bottom of the technote did help people to know which updates DID call for the connector to be updated.

And all this is among the reasons I created my other blog post on upgrading the connector, to help folks know when and how to do it, especially since it's so easy since CF2016.

But my point here is to say that if one did skip (or miss) the need to update the connector in prior updates, the problem was merely that there COULD be problems. Sometimes people who missed or skipped the upgrade didn't notice any.

With this latest CF update, however, it is MANDATORY that you update any web server connector(s) after the update, so that it puts in that needed secret that CF generates (and it also updates the connector dll or so. See my o ther blog post for more on that).

How Adobe could make it MORE CLEAR that we might need to take these two extra steps

So as I have related above, one concern I have is that (at least at this writing), the need to upgrade the connector it isn't made as clearly at the top of the technote, nor is it indicated in the text box that appears for these two updates. I hope Adobe will fix that soon. I have asked them directly, in addition to asking here.

How these two "extra steps" will apply to those who skip this updates but do the next one

If you've read the past few points here, I have made the observation that people do sometimes skip CF updates. Well, let's say a new CF update comes out in the next month or months. Then the people who have skipped THIS CF update will of course face the SAME problems, if they created a connector while at an update earlier than this Mar 2020 one.

And we will go through another round of people experiencing broken sites, and complaining loudly in the community.

How a new CF installer for CF2018 and 2016 would really help us now

I should say that the previous point only applies until Adobe may create a new installer for CF2018 or 2016, which bundles this newer connector, while I hope they may consider.

Doing that would also greatly help all those who use the CF Admin update mechanism and currently trip over two problems:

  • the UI issues in the CF admin updates page, which makes it hard to select updates and click on the update buttons (fixed in CF2018 update 7 and CF2016 update 13)
  • the issue that those on updates earlier than those from June 2019 (CF2018 update 4 and CF2016 update 11) must FIRST implement THOSE CF updates before they can apply any later ones
And tragically, all these problems also affect anyone currently installing either CF 2018 or 2016, who then try to apply "the latest" updates.

These changes are indeed ALL that this update implements

Finally, some people have wondered if this update may have address any other issues (like the long-awaited and now-pressing samesite cookie issue that I wrote about this week). Sadly, it did not.

All it addressed was these issues (and updating Tomcat related to them).

FWIW, I did a compare of all the files in CF before and after the change, and this set of changes (the addition of the secret in the server.xml, and the updating of Tomcat-specific jar files related to this security change) were indeed the only changes that the update performed.

Pro tip: Someone might want to say, "well, duh, Charlie. You could look at the hotfix_filelist.log in the update's folder, under cfusion/hf-updates" folder, to see what files were changed. But that's actually not quite so.

While that file DOES list ALL the files changed during the update, we have to keep in mind that since again CF updates are CUMULATIVE, each update lays down ALL the latest versions of files changed in ALL the previous updates. So you can't just look at that file and see, "yep, only these 20 or so files were changed due specifically to THIS latest update". Again, I did a compare of the complete CF folder, before and after the update, to confirm things.

Note that Adobe did NOT update the Tomcat version, only the AJP CONNECTOR itself

This section is an addition since the original posting. As I was digging into things (and helping someone with the update), I found out that whereas I thought this CF update updated Tomcat for us, I can say now that it did NOT. CF is running the same Tomcat version that would have been implemented if you'd applied the CF updates from Sep 2019, which are update 5 for CF2018, and update 12 for CF2016. You can confirm this in the CF Admin "settings summary" page, which will report (after those updates, and including this one) that CF2018 is running on Tomcat 9.0.21, and CF2016 is running on 8.5.42.

The thing is, the "official" Tomcat version which implemented the changes for Ghostcat (discussed in this blog post here) were introduced in Tomcat 9.0.31 and 8.5.51, respectively. But CF is NOT running those Tomcat versions (as of this update).

So what gives? We can only conclude that Adobe took a shortcut, and ONLY updated the AJP connector. That's not necessarily "wrong". Maybe it was expedient, allowing them to get this update out ASAP, without dealing with a "full" Tomcat update.

But it is confusing, for those who dig into the whole Ghostcat issue, and who understand Tomcat versions. I just wanted to point that out here, for those who may care to know (or who may dig further, in which case I welcome comments, of course).

Conclusion, and a plea

So phew, that's another mega post. I realize that some will have stopped at the top. Some may have dropped off on the way. My goal with these it to provide enough info for anyone who takes the time to read them, to understand the issue more completely than can be covered in twitter, slack, facebook, linkedin, etc.

If you may feel overwhelmed in considering it all, let me note that I can certainly help you implement these updates, typically in less than 15 minutes. See my consulting page for more on my rates, approach, satisfaction guarantee, and more.

And if you're looking at how to use the current downtime with the pandemic, I can help also with any CF updates, as well as connector upgrades, and JVM updates, and lots more related to install/config/admin of CF, troubleshooting/tuning, upgrading/migrating, deployment of CF on Docker, and lots more.

Finally, if you read all the way to the bottom here, gold star for you!

Could you return the favor, and take a moment to share your thoughts? I put a LOT of time into posts like these, hoping to help people (and indeed, they may well help folks so that they don't need to leverage my consulting!) But often I find I get very little feedback, pro or con. Your encouragement is appreciated, and even brickbats are tolerated in helping make things better.

For more content like this from Charlie Arehart: Need more help with problems?
  • If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
  • See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
Comments
Thanks again Charlie for covering our rears! Stay safe and great work as always.
Greg - BMChosting team lead
And thank you, Greg. Glad to help, and all the more to hear that in helping a host, I'm helping many times more people still! :-)
Readers: if you have already read my post here, as originally written last night, please note that I have added a new 3rd point (a possible step you may need to do after the update) in the TLDR section at the top.

Also, I have added a new section at the bottom, "Note that Adobe did NOT update the Tomcat version, only the AJP CONNECTOR itself", which may interest some of you.

For now, I didn't think it worth creating an entirely new/different post for these points, as they don't really affect most people--and would only be "new" info on the post for those who read it since I posted this last night.
Thanks Charlie, really appreciate this detailed info!
# Posted By Chris | 3/22/20 8:00 PM
Thanks, Chris.
First of all, many thanks to you Charlie for your helpful posts, which I have been reading for many, many years and which have often saved my day here in Germany.

After updating ColdFusion 2016 from Update 12 to Upadate 14 on my development server, I encountered a nasty little bug.

For years I have been using "localhost" in the "workers.properties" configuration file for all server instances as host (for example: worker.cfusion.host=localhost) without any problems. After the update to ColdFusion 2016 Update 14 it was no longer possible to establish connections to the ColdFusion Server instances, although I added all new secret values ??to the server instances in the "workers.properties" file (as described by Adobe) manually.

It took me a while to try to use the IP address 127.0.0.1 instead of "localhost" (for example: worker.cfusion.host=127.0.0.1), although this is the same (under the hood). But after this (not really logical) change, everything runs without errors and all connections to the ColdFusion Server instances can be made correctly. It seems that ColdFusion Update 14 does not like "worker.cfusion.host=localhost" anymore. :-(

My server environment:
- Debian Linux 9.12
- Apache 2.4.25
- Nginx 1.16.1 (runs as proxy)
- ColdFusion 2016 Update 14

Maybe this comment will also help other developers in the ColdFusion community. If so, I would be happy.

Greetings from Germany
Knut
# Posted By Knut | 3/24/20 7:01 AM
Thanks for the kind regards, Knut, and hapoy to have helpers (indeed for so mahy years).

As for your issue, it's not a bug. I suspect it's that you have a hosts file that sets localhost on your machine to your own ip address.

If you do a ping localhost or other command at the command line on the machine, what up address does it report? I suspect it's not the traditional localhost/loop back address.

And that's fine normally, but again one of the Tomcat changes (which the CF update incorporates) is default support for ONLY requests from the traditional localhost/loopback addresses, 127.0.0.1 or its ipv6 equivalents, as I indicated above.

And your change of the workers.properies is certainly one way to force the cf connector to "use" that address (so things work with the change)

But as I also noted, you could have also configured the address atrribute on the ajp element of the server.xml file to name the ip address also.

In your case, though, it may be easiest/better to just reconsider the hosts file setting that may be redirecting the localhost domain name to your specific ip address.

There are various reasons people have done that over the years. If you can't recall yours, maybe try commenting it out (the localhost line in the hosts file) and see how it goes.

If it works, you could then set the workers.properties back to localhost. Note that any future connector upgrade or creation will do that, so this is why I think it's the better solution than you changing the file.

Let us know how it goes. If you confirm it's what I suspect, I will tweak the post to add a mention of this point. Thanks for bringing it up. Hope I've solved it for you.
Knut, if you got emailed my last reply, I hope you may see this before replying to it. Please visit the site to see an update I made to the 3rd and next to last paragraphs of my previous comment, as I realized those were necessary for your situation.
I have also now added a new section on this point, which Knut raised and about which I have guessed at an answer. I will look forward to his reply/confirmation, but since I suspect it really is his issue, I added the new section <a href="#localhostchanged">Handling if you have changed the IP address that localhost resolves to</a>, and I also added a mention of it on the second point in the TLDR at the top. I could see it hitting people unexpectedly, and there are the 3 possible solutions one could consider, as I elaborate in that section.
Charly, thank you for your answer.

I checked the entries in the "/etc/hosts" file. It all looks completely normal and unchanged. The IP address 127.0.0.1 and :: 1 are also bound to "locahost".

As you suggested, I'll try the "address" attribute in the server.xml file to name the ip address in the connector. I will report whether this alternative way also works.

Thank you for your kind support.
# Posted By Knut | 3/25/20 6:00 AM
Thanks, but most simply, what ip addtesss do you get if you ping localhost? And what address otherwise were you planning to add as the "address" in server.xml? Very curious to hear the conclusion for you.
When I do a "ping localhost" at the console I get:

PING localhost(server2.xyz.de (::1)) 56 data bytes
64 bytes from server2.xyz.de (::1): icmp_seq=1 ttl=64 time=0.021 ms

I've masked the real domain name with "xyz".
# Posted By Knut | 3/25/20 6:51 AM
I did some trial/error investigation.

Example 1
- Adding the additional attribute "address=localhost" to the connector in server.xml
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: No connectivity

Example 2
- Adding the additional attribute "address=127.0.0.1" to the connector in server.xml
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: No connectivity

Example 3
- Adding the additional attribute "address=127.0.0.1" to the connector in server.xml
- Commenting OUT the "worker.cfusion.host=localhost" entry in the "workers.properties" file
RESULT: No connectivity

Working examples (in my case):
- NO additional attribute "address=127.0.0.1" to the connector in server.xml
- Only using "worker.cfusion.host=127.0.0.1" in the "workers.properties" file
RESULT: Connectivity

- Additional attribute "address=127.0.0.1" to the connector in server.xml
- Using "worker.cfusion.host=127.0.0.1" in the "workers.properties" file
RESULT: Connectivity

So in my view it does not make any difference to add the additional attribute "address=127.0.0.1" (or not) to the connector in server.xml. It seems to be mandantory (at least in my case) to use "worker.cfusion.host=127.0.0.1" in the "workers.properties" file.

My original overall configuration is basically based on the descriptions from Adobe at: https://helpx.adobe.... Under point 4, "worker.cfusion.host=localhost" is also specified, but at least in my case this no longer works with "ColdFusion Update 14". I have to use IP 127.0.0.1 instead of localhost in the "workers.properties" file.
# Posted By Knut | 3/25/20 7:26 AM
Thanks for all that.

So first, I agree something seems amiss. I don't think it's a Linux (or Unix thing), as some with such curious challenges have wondered. Time will tell.

You show the ping reporting ::1. That SHOULD be accepted by the updated ajp connector, by default.

But since you're scrambling to find anything that works, how about adding address="::1" on the ajp connector line in server.xml? You don't show trying that among your many attempts.

As I explain in my post, I recommend against changing the workers.properties file for such workarounds if possible, because the next connector install would set it to localhost by default. And while connector upgrades don't typically change that, one could.

Let us know what you find. Also, did you confirm 0 errors in the update log?
Short note:

>> ...but since I suspect it really is his issue, I added the new section ... Handling if you have changed the IP address that localhost resolves to...

I'm not a linux erxpert, but I think in my case "localhost" resolves to "127.0.0.1" and "::1". This is the server config I'm using since years. This has never been changed.
# Posted By Knut | 3/25/20 8:10 AM
>> But since you're scrambling to find anything that works, how about adding address="::1" on the ajp connector line in server.xml? You don't show trying that among your many attempts.

Okay, I will also try "::1" and give a report to you.
# Posted By Knut | 3/25/20 8:13 AM
Did you write your previous comment ("short note") before or after reading my comment above it? I can't tell, and I realize our messages could have passed each other in the ether.

Either way, I don't quite get what you were getting at. But the real key is to here how things go, per your (and my) last comment. :)
Oh, and about my mention earlier of "I still don't think it's a unix thing", I was confusing a separate discussion I'd been having with someone privately. I realize you said you were on Linux, instead. I will correct my comment, to avoid confusing later readers.

Even so, I will say I don't think your challenge is a Linux thing, either, if anyone reading may wonder (as some have been questioning, about possible OS differences).
>> Did you write your previous comment ("short note") before or after reading my comment above it? I can't tell, and I realize our messages could have passed each other in the ether.

Our posts unfortunately overlap.

I've tried to use "::1" for the additional address attribute in the connector and it works. Great.

Working examples (in my case):

- Additional attribute address="::1" in the connector in the "server.xml" file
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: Connectivity
CONCLUSION: I will use this in this form for all server instances in the future.


I think that normally the connector should resolve "localhost" to "127.0.0.1" and/or "::1" as standard. This is what the connector did bevor the U14 update. I can't see the reason why it is mandantory (in my case) to add the additional "address" attribut in the connector after updating to CF-2016 U14. This is a wired behavior (in my view).

Thank you for your tips and your support.
# Posted By Knut | 3/25/20 8:43 AM
Short additional note:

I just realized, that my two machines ("development" an "production" server) deliver different results for "ping localhost".

Output for development server "ping localhost" at the console:

PING localhost(server2.xyz.de (::1)) 56 data bytes
64 bytes from server2.xyz.de (::1): icmp_seq=1 ttl=64 time=0.021 ms

Output for production server "ping localhost" at the console:

PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.023 ms


I've contacted my hoster to check, why the outputs are different.
# Posted By Knut | 3/25/20 9:19 AM
Yep, I agree that it should not be "needed", but I proposed it in case it might help--and I'm really glad to hear that it did. I had mentioned how the Tomcat docs said that localhost addresses would be supported out of the box, with the update to AJP (which CF incorporates).

The specific wording (on the Tomcat doc page I link to above) says, "By default, the connector will listen on the loopback address." It's looking like that may be only 127.0.0.1, which I find shocking. I almost think something ELSE must explain things.

But if you want to try it, you can use multiple addresses (as I do also indicate above), so address="127.0.0.1,::1". That would cover you either way.

Again, I agree: it should NOT be needed. But until we (all) figure out what's amiss here, it may be the best recommendation, and I could update my post to reflect it. That said, I already added a new section above (as mentioned in a previous comment) before I heard back from you, so I would like to do a little more testing of confirmation before updating the post.

Still, I'll let this comment stand for now, for those with the gumption to read down to here among these now 7000+ words. :-)
Thanks for the very detailed information Charlie. As usual you make valid points. You're articles are always a good reference so please keep up the good work. Stay safe. Cheers!
# Posted By Miguel-F | 3/25/20 1:04 PM
>> But if you want to try it, you can use multiple addresses (as I do also indicate above), so address="127.0.0.1,::1". That would cover you either way.

Good suggestion, I can try it.
Do you think that address="127.0.0.1,::1,localhost" should also work?
# Posted By Knut | 3/25/20 1:21 PM
Did some testing.
Neither address="127.0.0.1,::1" works, nor address="127.0.0.1,::1,localhost" works.

In my case it's only working with: address="::1"

Best regards
Knut
# Posted By Knut | 3/25/20 1:47 PM
Hi Charlie,

Thank you very much for this post and the much needed extra explanation about what was happening. One slight edit if I may suggest, the Adobe kb article says that that allowedRequestAttributesPattern is suppose to be ".*" and not "*". It took me a while before I realized I had gotten it wrong.
# Posted By Samuel Smith | 3/26/20 4:30 PM
Miguel: thanks very much.

Knut: that's odd. I had done it (adding two), and it did work (as I said in the post, I even reversed the order and put one that would fail first, and it still worked). I wonder if there can be some other explanation.

But glad to see that at least just ::1 worked. I have been doing some work that might suggest that Tomcat is ONLY by default accepting from 127.0.0.1, and not ALSO ::1 as one would expect (and I'd said). The odd thing again is that I find that to be an issue on some machines but not others. Will share more as I learn more.
Samuel, thank you very much (and sorry, I had meant to also comment on your point in my last comment). That was indeed a mistake, in both this post and the shorter one I did at the Adobe CF portal (now listed at the top here). I apologize to everyone for that mistake, and of course I have corrected it here and there.
I was unable to get ACF update 8 to work at all after following the suggestions in this post.

Prior to seeing this post, my production server failed after I installed update 8 (Win Server2018, Apache 2.4). I tried updating the web connector, but no joy. I panicked and was able to uninstall the update and got it running again at update 7.

Tried the update again on my development box (Win10, Apache 2.4) with worse results. I tried the suggested fixes in this post, but couldn't get the CF service to restart. Gave up and reinstalled the whole enchilada. Ran update 4 and then update 8 before even setting up the web connector. Same problem: service wouldn't start and wouldn't start after trying to change server.xml AJP settings. I'm wondering if there is something specific to the combination of Windows and Apache that is causing me grief. Any suggestions appreciated.
No, Adam. Update 8 would not have anything that would itslef keep cf from starting. That problem could be caused with ANY CF update, and I pointed above to my blog post on sovimg such problems.

PLEASE read that. You will get cf running again.

Or I can help you directly, likely in as little as 15 minutes. You won't pay for time you don't find valuable.
Also, Adam, since you're using Apache, let me add that once you get CF running again (see my last comment), you should then see if the original problems you had (with the connector) is fixed by using allowedRequestAttributesPattern=".*", as discussed in prior comments and a correction I made yesterday.

I will be doing more research and documentation on the matter (why it's needed, when, and better alternatives to opening things up that way).
Adding to my last comment, I just posted some more detail in a comment on a CF forum message from someone else, and I shared there some more detail--my initial findings. It may interest some readers, until I can get more documented:

https://community.ad...
Thanks as always, Charlie, for these posts. I feel like every CF update over the past 20 years has given me some degree of anxiety, but it's great to know you're always here to help us through.
# Posted By Rocky Rosen | 3/29/20 6:32 PM
Thanks, so much, Rocky. Happy to be able to help.
I have also seen where the Tomcat server resolution would delay processing by up one second. But adding `address="::1"` to the server.xml AJP line the problem goes away. If you noticed a slow down but not an error this would something to investigate.
# Posted By Bilal | 4/14/20 1:13 PM
Thanks, Bilal. I'd see the discussions of that on the Lucee list, but I'd hoped to get more clarity on when it does or doesn't happen (the delay), including whether anything would influence it happening (or not) on cf or tomcat without lucee. Then I lost track of the concern.

So I'm glad you mentioned it here, in case it may help someone. Thanks, and as always thanks for your efforts on the alternative boncode connector.
Once again you da man! Thank you for this Charlie!
Thanks for the encouragement, Brian. :-)
Thank you Charlie!

Unfortunately I just found your post/blog after finding the solution myself after working on it the entire day.

I worked on my development desktop running Windows 10, Apache 2.4.41 (Win64) and Coldfusion 2016 which had hotfix 13 before I updated it to hotfix 15 (I had skipped 14).

My finding is that the connector definition (without address=) in server.xml resolves to 127.0.0.1 while the localhost in worker.cfusion.host=localhost in workers.properties resolves to ::1.

Putting "address=::1" in the connector definition in server.xml alone solved the problem. Mine was 503.

You could also use any of these combinations:

connector with address=::1 and workers with localhost (as mentioned above)
connector with no address and workers with 127.0.0.1
connector with address=::1 and workers with ::1
connector with address=127.0.0.1 and workers with 127.0.0.1

Thank you anyway. I was a pleasure and reassured me of my assessment reading your entire post.
# Posted By Claude | 4/16/20 8:53 PM
Bummer to hear that you were troubled all day before finding this.. And yep, on how the address="::1" seems a solution needed especially for apache, as I did indicate in the post.

There's a lot to the post, of course. That's also why I made the more brief post on the Adobe site, as I noted at the top. I have plans for another resource, especially to help folks installing cf and then doing updates.
Just adding a +1 to the note about BonCode connector. We also had to add address="::1", otherwise it was incredibly slow. Just in case anyone sees the same and banging head against the wall. Charlie, legend as always, these articles are invaluable.
# Posted By Dave Quested | 4/19/20 5:09 PM
Thanks so much, Dave, for both the kind regards and the additional info.
Thanks Charlie for CF update8 Traoubleshooting,
I red this page and doing as following... but doesn't work website that shown 500 error. not 503, 403 error.
Could you advice me please.
---
Step1: Amazon ColdFusion (2018 release - Ubuntu) : Made instance : 64bitAMI.
Installed CF success : ColdFusion:Y , API Manager:N , PMT:N, Servlet(RDS,JS Debug etc) None.
Created new file index.cfm on {CF_HOME}/cfusion/wwwroot/. then brows it and shown it's content.

Step2: ColdFusion 2018 Update 4 : success on CF Administrator on chrome browser.

Step3: ColdFusion 2018 Update 5 : success on CF Administrator on chrome browser.

Step4: ColdFusion 2018 Update 6 : success on CF Administrator on chrome browser.

Step5: ColdFusion 2018 Update 8 : messaged in popup "Install Status: Server taking too long to restart. You will have to manually restart server and check update status".
Reload web page CFIDE/administrator/ was 500 error.

Step6: Login on SSH and check two files.
server.xml : additional `secret="D978D4AB-AE67-4F2D-A129-6E95075FFF81"` in AJP connector line.
workers.properties : nothing `worker.cfusion.secret`.

Step7: doing conname `./wsconfig -upgrade -v`

Step8: check workers.properties file.
additional 1 line `worker.cfusion.secret=D978D4AB-AE67-4F2D-A129-6E95075FFF81`.

Step9: restart cfusion and apache2
stop cfusion , stop apache2, start apache2, start cfusion

Step10: brows CFIDE/administrator
shown 500 error.
And index.cfm was500 error, too.

Step11: Changed workres.properties file
Changed from `worker.cfusion.host=localhost` to
`worker.cfusion.host=127.0.0.1`.

Step12: restart cfusion and apache2 then brows CFIDE/administrator
shown 500 error.
And index.cfm was500 error, too.

----------
I have try and try many type setting for a several days but does not work cfusion...

best regards
# Posted By Taka Iga | 4/30/20 2:40 AM
Taka, see the first link I offered, at the end of the tldr section, where I explain what to do if you have issues after applying ANY cf update.

In your case, you said in step 10 that you applied update 8 and then could not access the CF admin. I assume you are using a port for that, like 8500 (the default). That then is NOT using the ajp web server connector (and the changes I regarding it which I discuss in this post).

As the other post explains, your next step is to a) confirm if there were errors in the update log, then if so b) re-apply the update, whether in the admin or at the command line. Details in that post.

Let us know how it goes. Or I can help you via a screen share, in a consulting session of likely less than an hour. You don't have to pay if I don't solve it.
Thank you for quickly response, Charlie san.
Is the first link "Having problems after applying a CF update? What to check, and how to recover!", this link URL same this post(.../2020/3/20/...). It missed link page !? the page URL is .../2019/03/.., right !?
I respond just info, I'm going to read the page you offered and check my environment.
Best Regards
# Posted By Taka Iga | 4/30/20 8:07 AM
Wow, thanks for that. The URL was indeed wrong (it was a copy/paste error, linking instead to the same url as the very first link in the post, for the "brief" version of this post). It's sad to think no one else noticed or reported it all this time. I have corrected it. Again, thanks.

And yes, let's hear how it goes for you.
Hi, Charlie san,

I checked and tried, but not work yet. same status before.
How can I check other points, could you advise me please.

I done as below...
1. I check "install log" file of Update1, 4, 5, 6 and Update8. All Update was Success, ZERO error/fatal error.

2. Re-Update Update8 by command line, shown "This update is already installed."

3. Stop CF, stop Apaceh2, start CF, start apache and brows page as
"http://{IP address}/index.cfm" => shown 500 error.
"http://{IP address}/CFIDE/administrator/" => 404 Not Found.
for example,
"http://{IP address}/index.html" => shown "Apache2 Ubuntu Default Page".
"http://{IP address}:8500/CFIDE/administrator/" => 500 error.

hummm...

---
BTW, you fixed LINK ^^, but the link page was WHITE page and your other blogs too...(I kept the page by capture ^^).
# Posted By Taka Iga | 5/1/20 3:22 AM
Taka, if there are indeed 0 errors in the install log for update 8, then there's no need to reinstall, and that's also why you cannot.

So now the key is that you are confirming that you get an error in the cf admin and you are CONFIRMING that you were accessing it via port 8500. That tells us again that whatever your problem is (with that), it's not about the ajp connector.

But can you confirm as well that you WERE using thst port 8500 to access the admin before the update?

If not, were you accessing the admin without a port? That would not be allowed by default in cf 2018 (or 2016). You'd have to tweak the connector properties to allow that. Did you?

If so, can you confirm that cf is even listening on port 8500, for its built in web server/biws? It may be some other port.

Bottom line: if cf is running and the biws is working (and the admin accessible), then your other issues COULD still be about the ajp connector. Did you do all the things listed at the top? Upgrade the connector, add the address attribute, if needed? Add the allowedRequestAtrributesPatten, if needed? See above for more.

If you may write back to ask continued questions, please answer every question I ask in this comment. You can also email me at Charlie (at) carehart (dot) org, which may be easier/better than extended discussion here.

Finally, I will say again that if you just want to get the problem solved without all this back and forth we could solve it in a quick consulting session.
Oh, and thank you Taka for pointing out how the link I updated goes to a blank page. That is indeed a very unfortunate thing to discover. To be clear, it's not on my site but Adobe's. I have bought it to their attention.

What's especially sad is that it seems to be limited to pages of mine. I can think of no explanation for that. They have said someone will work on it next week. I am trying to remain calm about it, but it is indeed disconcerting.
Charlie san,
I email you. Could you see it and advice me please...
Yes, I access CF Administrator with post :8500. not use Buildin server.
I tyied connector option "address", "allowedRequestAtrributesPatten".
thank you so much !
# Posted By Taka Iga | 5/5/20 3:10 AM
Thank you for your write up! It's nuts that the adobe kb has so little info on error 403. I got it working by setting the allowedRequestAttributePattern to .* but figuring out the actual request attributes and how to set them as a RegEx has been a challenge.
I ran gethttprequestdata().headers like you mentioned and got 16 different values back. In addition to Xsite-name I got stuff like "accept" "connection" "host"... Could you clue me in how to specify them in the allowedRequestAttributesPattern attribute? "Xsite\-name|accept|connection|host..."
Again, thank you for your support!
# Posted By Rob | 5/5/20 10:27 PM
We are also suffering from the 403 error, and would appreciate any help you can offer. We're running
macOS mojave v. 10.14.6
apache httpd v. 2.4.34 (configured for https)
mod_jk.so v. 1.2.46
apache tomcat v. 9.0.34

$ ping localhost
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.036 ms
...

In server.xml (with 'address="127.0.0.1"' or 'address="127.0.0.1,::1"', httpd couldn't connect to tomcat at all:
<Connector port="8009"
protocol="AJP/1.3"
redirectPort="8443"
address="::1"
secret="somePassword"
allowedRequestAttributesPattern=".*" />

previously, in tomcat v. 8.0.21:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />q

In localhost_access_log.2020-05-18.txt:
xx.xxx.xxx.143 - - [18/May/2020:13:00:43 -0700] "GET /mpstruc HTTP/1.1" 403 618

workers.properties:
workers.tomcat_home=/Library/Tomcat
ps=/
worker.list=ajp13
worker.ajp13.port=8009
worker.ajp13.host=localhost
worker.ajp13.type=ajp13
worker.ajp13.secret="somePassword"

/etc/hosts:
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost

A distillation from httpd mod_jk_log:

init_ws_service::mod_jk.c (1179): Service protocol=HTTP/1.1 method=GET ssl=true host=(null) addr=76.21.65.212 name=ourwebsite.edu p
ort=443 auth=(null) user=(null) laddr=128.200.165.4 raddr=76.21.65.212 uaddr=76.21.65.212 uri=/mpstruc/newsfeed
ajp_get_endpoint::jk_ajp_common.c (3357): (ajp13) acquired connection pool slot=0 after 0 retries
ajp_marshal_into_msgb::jk_ajp_common.c (681): (ajp13) ajp marshaling done
ajp_service::jk_ajp_common.c (2588): processing ajp13 with 2 retries
ajp_send_request::jk_ajp_common.c (1719): (ajp13) no usable connection found, will create a new one.
jk_open_socket::jk_connect.c (674): socket TCP_NODELAY set to On
jk_open_socket::jk_connect.c (798): trying to connect socket 31 to ::1:8009
jk_open_socket::jk_connect.c (824): socket 31 [::1c1e:1f49:0:0:56240 -> ::f00:c43:b281:f1fd:8009] connected
ajp_connection_tcp_send_message::jk_ajp_common.c (1264): sending to ajp13 pos=4 len=380 max=8192
ajp_connection_tcp_send_message::jk_ajp_common.c (1264): 0000 12 34 01 78 02 02 00 08 48 54 54 50 2F 31 2E 31 - .4.x....HTTP/1.1
...
ajp_send_request::jk_ajp_common.c (1779): (ajp13) request body to send 0 - request body to resend 0
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): received from ajp13 pos=0 len=54 max=8192
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0000 04 01 93 00 03 34 30 33 00 00 03 A0 01 00 17 74 - .....403.......t
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0010 65 78 74 2F 68 74 6D 6C 3B 63 68 61 72 73 65 74 - ext/html;charset
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0020 3D 75 74 66 2D 38 00 A0 02 00 02 65 6E 00 A0 03 - =utf-8.....en...
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0030 00 03 36 31 38 00 00 00 00 00 00 00 00 00 00 00 - ..618...........
ajp_unmarshal_response::jk_ajp_common.c (736): (ajp13) status = 403
ajp_unmarshal_response::jk_ajp_common.c (743): Number of headers is = 3
ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13) Header[0] [Content-Type] = [text/html;charset=utf-8]
ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13) Header[1] [Content-Language] = [en]
ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13) Header[2] [Content-Length] = [618]
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): received from ajp13 pos=0 len=622 max=8192
# Posted By Craig S. | 5/18/20 4:07 PM
Craig, would you confirm that you're simply asking as a user of tomcat, rather than coldfusion (running on tomcat)? I ask because your ajp port of 8009 is not what cf would set.

One may ask, why does it matter? But if you have installed your own tomcat, and configure by hand your ajp files and how apache points to them, you have to be sure you properly updated the right files.

For instance, you state the version of your mod_jk.so, but you don't indicate where it exists. Are you sure apache is loading it? Have you checked its conf files, to confirm? And even if you think all's well with that, what if you rbamed the so file, or the conf file? Does httpd still start? It should not. That's a quick sanity check. I have seen people chasing the wrong problem by mistaken assumptions.

And this is key, because if the wrong so is running, or that secret for the workers.properties is wrong, you'd get a 403.

Same if somehow your server.xml changes were not to the right one. I'd tweak that in a way that should stop tomcat running. Btw, it could come up but not process pages. So be sure to check a page running via its web server (the http connector in server.xml), to make sure tomcat IS running, separate from requests via ajp/apache.

I realize this may sound like crazy ideas, but you're presenting that "all is well" with your config files as presented, and I concur. But clearly something's not right.

One other thing you might try: changing the localhost value in workers.properties to ::1, to match your server.xml. Of course, restart apache after that, or tomcat after changing server.xml. (Some folks don't realize that chbages to the former are loaded by apache rather than tomcat).)

Let us know how it goes, even if that still nothing works. In that case, I'd think it time to enable more logging or even network monitoring, to find what's really going on.
Hi Charlie,

Yes, we are not running cold fusion, just tomcat behind apache httpd.

Apple did not provide the mod_jk.so module in the standard module location (/usr/libexec/apache2/) in macOS 10.14, and wouldn't allow me to add it there, so I put it in /etc/apache2/libexec (etc/apache2 is the apache home directory). httpd.conf includes it with (showing some context):

LoadModule hfs_apple_module libexec/apache2/mod_hfs_apple.so
LoadModule jk_module /etc/apache2/libexec/mod_jk.so

and the log /var/log/apache2/error_log.1589414400 indicates that httpd is finding it:

[Tue May 19 12:02:50.526249 2020] [mpm_prefork:notice] [pid 102] AH00163: Apache/2.4.34 (Unix) LibreSSL/2.5.5 mod_jk/1.2.46 configured -- resuming normal operations

Apache seems to be running fine, and is serving our static pages. Tomcat also seems to be running fine, and hearing requests (e.g., see the excerpt from the localhost_access_log in my previous post), but denying access to all requests.

I've tried changing workers.properties worker.ajp13.host=localhost to worker.ajp13.host=::1, no difference (and I always restart after such a config change).
I've set all the logging levels in tomcat/conf/logging.properties to ALL, and the logging for mod_jk to "JKLogLevel debug". While I'm getting a lot of information from the httpd side of things, the only thing tomcat wants to tell me is, again, that 403 response in the localhost_access_log. So I'm not sure how to enable more logging at this point. What kind of network monitoring would you suggest?

Nothing is crazy at this point - we've been pulling our hair out for a couple of weeks! Thanks for taking a look!
# Posted By Craig S. | 5/19/20 4:07 PM
Interesting challenge, Craig. But again, it has to be some configuration issue.

It does seem you've covered all your bases for the typical things that trip folks up. And yep, the mod_jk debugging logging does confirm that the request is getting from apache TO Tomcat.

Still, I have a couple other ideas.

First, does your password have special characters? Have you tried setting it to a simple string, just for the sake of testing? If that works, perhaps there was an issue in the passing of that.

Second, is it possible that the request you're making is not going through mod_jk but mod_proxy_ajp? In that case, the latter would not honor the secret set in the workers.properties, but would instead be set on the proxypass line, from what I have read (I have not tested this).

Third, you haven't indicated what Tomcat version you're using. And the mod_jk.so is not reporting to be the latest. While I realize you may have reason not to want to be "on the latest", it seems worth trying just to find out if updating both might resolve this, as there may be some bug.

In fact, there was one about incoming headers having a new line character that was tripping up Tomcat, since fixed. (Your debug output doesn't show that, but I have not investigated to find out how that looks in the log if that's the issue, and if it's even tracked there.)

Indeed, one could argue that a real challenge is that even the ajp debug logging doesn't tell us WHY the request gets a 403. That seems reasonable to expect, but the folks behind Tomcat are VERY careful and thorough and always have well-reasoned explanations for why they have done (or not done) something.

And to that very point, if none of the above works, I'd recommend you take the problem to the tomcat-users mailing list. That is THE best place to get resolution to challenges like this, as most questions are answered by the very people who wrote tomcat and the ajp connector.

And of course, they have been dealing with ramifications of these ghostcat changes for months now. So they may think of something readily that we have missed here. You can learn more here: http://tomcat.apache...

But before anyone would write their first note to that list, they should note the indications on that page of HOW questions should be asked. Even better, read a few threads in the list before making a first post to take the pulse of things (there's a link to the online archives--even already from today as I write).

If you look at threads where there have been a few or several replies, you'll get a good sense of how things go--and when they go well, or not. :-) Sometimes it can be a bit like watching a cross between the scenes when Dorothy's crew approached the Wizard of Oz, and the courtroom scenes in My Cousin Vinny. I'm not trying to scare anyone off. Again, they are whip smart as a group and great at solving knotty challenges. Just be prepared in how you present things.

If you do ask there, and especially if you get a resolution, I hope you will return here to let folks know (either the answer or a link to the online discussion).
And Rob, sorry I missed your note from a couple weeks ago. I'd offer to you the same suggestion I made to Craig at the end of my last comment. Yours is a good question, not answered as clearly in the docs as some might hope. If you agree, take it to the tomcat-users list (with the same caution I shared with Craig).

And again, if you get an answer, please post it here for others to know. Thanks.
As for Taka's issue (before Craig and Rob), let me update those following along. He and I did start working together offline on his issue--but it seems NOT about the AJP connector.

His problem is that NO CF requests run, even using CF's "built-in web server" (which is the Tomcat web server, rather than Apache talking to CF/Tomcat over AJP), after applying the latest (April) CF update. And for those interested, there are no errors in the CF update log.

I have now heard of one other person with the same problem, but only one other like it, so there's some combination of characteristics they share. I'm trying to figure that out with each.

And Adobe has gotten involved also (with the other client I'm working with). If we learn something, I will share it here for everyone to know. But to be clear, it seems unrelated to the topic of this post, which is about the changes regarding AJP processing per the March updates to CF 2018 and 2016.
Hey, Craig and Rob, any news?
Charlie, thanks for your help with this problem, which has finally been resolved. We're running macOS 10.14.6, upgraded from Apple-installed httpd v.2.4.34 to Homebrew-installed v. 2.4.43, and tomcat v.8.0.21 (when this started) to v.8.5.4, to 9.0.34, and finally v.9.0.35. Moving from mod_jk.so 1.2.46 to 1.2.48 was problematic - no binary available, and compiling from source wasn't working, so I stuck with 1.2.46.
Ultimately, once those upgrades were made, the final, unintuitive change that finally got things working was, in workers.properties, going from

worker.ajp13.secret="somePassword"

to

worker.ajp13.secret=somePassword

On the tomcat side, in server.xml, the double quotes are needed, as in

secret="somePassword"
# Posted By Craig S. | 6/4/20 5:49 PM
Wow, ok. Thanks for sharing. Hope it may help others.
Charlie san,
I read a message from Craig (Thanks Craig san). I tried that mod_jk.so compiling from source code v.1.2.48.
but AWS CF Administrator (:8500) do not work yet.
I'm continuing read this web site.
Thanks all
# Posted By Taka | 6/4/20 10:27 PM
Charlie - Thank you so much for posting this detailed response. I have been struggling with a unstable Apache/CF setup since applying update 9 and following the tech notes. We are using RHEL7, Apache 2.4 and CF2018, and have invested (wasted) so much time and effort trying to get this resolved. We even stripped back the services reinstalled both CF and Apache and modified a great number of Selinux settings. Finally after adding address="::1" to the AJP connector in the server.xml file we are back in business. If we had not found your post, we will have dropped Coldfusion - despite having worked with it for 10 years +
# Posted By Peter | 6/16/20 9:35 AM
Thanks, Peter, and glad I could help: especially to save you from the brink of leaving CF.

I had in fact expressed my concern about that in my first comment in the Adobe blog post on the the day it was announced (https://coldfusion.a...), where my opening sentence was, "Wow, I suspect this update is going to start a lot of trouble for a lot of people."

In the weeks since I and others (including Tomcat folks) have tried to share more and more info to help folks, but there's no doubting that anyone updating from that point forward (for the first time, such as in your going to CF2018 update 9) will hit this, so we will see people struggle with this for years...or until Adobe simply creates a new installer that embeds the latest updates to CF and the connector, and perhaps reconsiders tweaking the wsconfig tool to help people get the RIGHT value for that address attribute, or whether to enable that allowedrequestattributespattern override. Different strokes for different folks.
And Taka, to your issue, that's not really with this matter related to the web server connector. Instead, that's about an issue that it seems would hit anyone who is running on the AWS AMI from Orbiterra. I have been in touch with Adobe about this with another client of mine, and have mentioned how it hit you also. For now, their only answer is to propose an alternative AMI. I am pressing to get them to figure out what it is about THIS AMI that keeps one from being able to update CF. Obviously, it's not a good situation to be unable to update.

But to be clear, it's a different problem than this blog post's focus. So keep in touch with me directly rather than here--and anyone interested in the topic can reach out to me also (see the contact info offered here on my site). Perhaps I will create a new blog post about it, especially if a specific solution is identified.
when i upgraded my coldfusion 16 to update 14, the coldfusion instance would not start. got an abort error when i tried to start the service. when i did cfstart.bat got the same error an abort. i was able to look at the even viewer log was getting the following error:

- System

- Provider

[ Name] Application Error

- EventID 1000

[ Qualifiers] 0

Level 2

Task 100

Keywords 0x80000000000000

- TimeCreated

[ SystemTime] 2020-09-22T01:30:20.000000000Z

EventRecordID 235026

Channel Application

Computer WIN-

Security


- EventData

coldfusion.exe
2016.0.0.0
5b92a2b1
coldfusion.exe
2016.0.0.0
5b92a2b1
80000003
00000000000341c2
ea0
01d6907fecf895a4
C:\ColdFusion2016\cfusion\bin\coldfusion.exe
C:\ColdFusion2016\cfusion\bin\coldfusion.exe
2d04d3f3-fc73-11ea-9db5-d4ae52b9744e

I went round and round trying to get the coldfusion.exe to start, so i uninstalled and fortunately the uninstall worked and was able to start coldfusion and get the website up and running again.

nowhere does this issue get addressed.
# Posted By darrell | 9/21/20 9:11 PM
Darrell, it's not unusual to have errors after applying any cf update. There are various reasons and easy solutions.

I discuss these in another post:
https://coldfusion.a...

Neither the cause nor the solution is indicated in that Windows Event Log. The fact that you could uninstall the update is an indication how you can simply try again, after considering what I offer in that post.

If somehow you still struggle, I can offer direct remote assistance. I'm sure we could resolve it, and likely have cf updated for you in less than 15 mins. If interested, check out carehart.org/consulting.
Hi,

Just to say thanks for these posts! I have a fresh install of CF2021 in VirtualBox Ubuntu Server 20.04 LTS with Apache mod_rewrite. The rewrites to .cfm files did not work at all giving Tomcat's 403.

After banging my head and going all over the net for hours searching for fix it turned out to be as simple as adding that allowedRequestAttributesPattern=". *" to the server.xml file.

You saved my day!

BR,

Aarni
# Posted By Aarni | 5/12/21 6:38 AM
Thanks again Charlie for all the excellent info.

On the requiredSecret attribute, after installing the CF2018 update 12 I had two servers returning 403 errors for cfm files, I tried adding allowedRequestAttributesPattern=".*" and an IP address to the AJP connector line as well as wsconfig upgrades. It wasn't until I removed the requiredSecret attribute that they started working.
# Posted By Chris | 9/21/21 12:35 AM
Chris, that is indeed curious. If the cf update was successful, along with the cf web server connector update, then that "old" requireddecret should be ignored, as I discuss above.

The fact that you had to remove it would have me worried that either of those was not successfully updated. I discuss above how to check if a cf update is successful (it's not enough to confirm merely that it has been installed). For more, see my post that I point to in that above section:
https://coldfusion.a...

As for checking that the wsconfig is updated, I discuss that in the blog post I point to on updating the connector itself:
https://www.carehart... You'd want to see the isapi_redirect.dll (for iis) or mod_jk.so (for Apache) had a date of just before the date of the cf update you'd applied (assuming that update did call for an update to the connector, as most do). In your case, having updated to cf2018 u12 (from this year), the connector dll or so file should be from 2021. Is it?

If you want any help double-checking all that, as I note above we could do that on a consulting basis in perhaps as little as 15 mins. More at carehart.org/consulting.

Or let us know if you solve things yourself.
Thanks Charlie, I thought I'd mention it in case it was of interest to you or anyone else with the same issue.

I've installed the update on four Linux/Apache servers, all update logs are without warnings or errors, and all now have the same August 2021 date on mod_jk.so.

The two other servers didn't have the requiredSecret attribute and had no 403 errors, I've now tested adding requiredSecret="test" to those and both went to 403 with it added.

I would definitely call on your services if there was still a problem, but removing the redundant requiredSecret seems to have been the fix for the servers that had it.
# Posted By Chris | 9/21/21 9:36 PM
Ok, but when you say "redundant", do you mean what you deem to be "unneeded"? I would argue one SHOULD be needed unless something's amiss.

But I realize you feel you're all set, so I share this as much for future readers. If you, I, or anyone else ever learns something more about this, perhaps it will be added here.

Glad you're back in business. :-)
Thanks Charlie, my read of your 'On secrets vs requiredSecrets' section above was that the secret attribute replaced requiredSecret, and newer Tomcat versions would only be using the secret attribute?
# Posted By Chris | 9/21/21 9:55 PM
My bad. It's late (for me), and I'm seeing this on my phone. Yes, you're right: requiredSecret is the OLD attribute name. It should be ignored in a properly updated CF/Tomcat deployment. The fact that it CAUSED an impact is disconcerting.

But you're set. If things change or we learn more, we can add it here.
Many thanks for confirming Charlie, and again for providing so much helpful info on CF!
# Posted By Chris | 9/21/21 11:49 PM
Chris, I think you may have just saved me another round of headaches. I tried the update over the weekend and was struggling to find out why I was getting 403 errors before I reverted back to the previous update. I now see I have this requiredSecret attribute in there. I will purge that and give this another try during our next update window.
# Posted By Chris Braster | 9/28/21 8:30 AM
Hi Chris, it would be good to hear if it solves the problem for you too, good luck and please keep us posted!
# Posted By Chris | 9/29/21 12:07 AM
I will definitely report back. Looking like early Sunday morning is our next go time. Fingers crossed dropping the requiredSecrets does the trick.
# Posted By Chris Braster | 9/29/21 6:06 AM
Reporting back that my second go at the update was a complete success. Just needed to remove the requiredSecret attribute from the connector in server.xml, apply the new update, and done. Thanks again Chris, and thanks Charlie for doing everything that you do for the community.
# Posted By Chris Braster | 10/3/21 2:45 AM
Thanks to you both. This has indeed hit others, it seems. I've not been able to recreate it. The questions are first what caused that old requiredSectets to be there...or worse, be suddenly put there. Then second, why is it being paid attention to, if the newer "secret" attribute is there--in which case itnseems the old requiredSectets attribute should be ignored.

Also, we need to k ow what exactly is causing any recent change in behavior regarding all this? The tomcat team (and something cf simply includes)? Or the Adobe cf team, who do modify the tomcat ajp web server connector (and implement it with the cf wsconfig tool)?

And finally, does it only happen in some situations? Some previous cf update vs another? Some previous cf version vs another? Some previous wsconfig version vs another?

Again, let's see what we learn over time.
Very glad to hear it worked for you too Chris, thanks for the update.

Charlie, I believe the requiredSecrets were there prior to update 12 on the servers I had the problem with, which had updates 4 and 11 installed when they were set up earlier this year.
# Posted By Chris | 10/3/21 11:49 PM
Here's an update on the "new" 403 errors found to be occurring recently. These are the ones discussed in the past few comments here over the past couple of weeks. These seem to be caused when the "older" requiredSecret attribute exists (in the server.xml ajp connector line) IN ADDITION TO the "newer" secret attribute on the same connector line.

Until recently, it seemed the "older" one was was ignored, but now its presence is causing 403 errors for some. Why?

First, some folks are saying they did not PUT that the requiredSecret there, and that as far as they know it's been there a while (not causing trouble, even after the ghostcat updates to the connector and CF discussed in this post starting in March 2020).

It seems THAT perhaps it's presence is a leftover from their having run the Adobe CF Lockdown tool, which had been introduced in CF2018. It seems THAT put this "old" requiredSecret in place (then), for the sake of connector security.

Again, the ghostcat changes led Tomcat to create a NEW "secret" attribute instead (and a related secretRequired boolean, whose name was too close to the old attribute, in my opinion).

Anyway, it seems the presence of BOTH the "secret" and "requiredSecret" was being ignored until recently. Perhaps there was a change in Tomcat in the Adobe CF connector.

So for now, those getting a 403 that can't be solved by the other means above (such as the address attribute) should check to see simply if they have both a secret AND requiredSecret on the ajp connector line in server.xml. If so, remove the latter (maybe copy it off to a comment line), and restart CF, and see if your 403 goes away.

Feel free to share here whether it does or does not work for you, even months from now as I write.
I can confirm that I ran the Lockdown Tool on the initial installation of CF 2018 and that was also where I was thinking the requiredSecret originally came from.
# Posted By Chris Braster | 10/22/21 4:36 PM
Hi Charlie,

I updated from 2018,0,06,316308 to 2018,0,13,329786, and ColdFusion restarted without errors.

According to the Update 13 instructions, I need to follow the post-installation steps for Update 8.

This is a locked-down installation that already had the requiredSecret attribute, and the web server and ColdFusion are on the same machine. In this scenario, the instructions say: "If you have already locked down ColdFusion, then you need not take any action, since ColdFusion instances are already configured with the requiredSecret attribute and your web server also has the secret information with it to communicate."

Should I upgrade the connectors anyway?

Thanks for any info!

JG
# Posted By JG | 12/22/21 11:37 AM
You do need to update the connector, as it's changed since then. And that Cf 2018 update 8 will also implement a new (different) "secret" in the server.xml that will require the connector update to change the worker.properties to have that NEW secret value.

Then you need to remove the old "requiredSecret" in the server.xml, due to the Tomcat change in a recent cf update.
Thanks for the instructions, Charlie!

Upgrading the connectors caused 403 errors, but removing the requiredSecret in server.xml fixed that.

Happy holidays!
# Posted By JG | 12/22/21 12:34 PM
Sweet, thanks, and same to you and all.
Hi Charlie,

Is there a way of adding a secret value to a manually created connector that is connecting to IIS.

I have installed CF2018, completed all the updates, and then added a manual configured connector for iis following the adobe documentation, as per my software vendor advice. I am just getting 500 errors and I feel it must be the connector that is playing up.

Any advice would be greatly appreciated

Laurie
# Posted By Laurie | 1/19/22 10:10 AM
To interested readers, I offered Laurie an answer elsewhere that he asked about this the same day, and which I'd seen and replied to first:
https://coldfusion.a...
Copyright ©2024 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting