[Looking for Charlie's main web site?]

How and why your sites may break, and what to do, after applying March 2020 update to CF2018 or 2016

Note: This blog post is from 2020. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.
This is a critical warning to anyone who may apply the recent CF2018 Update 8 or CF2016 Update 14, released Tuesday of this week (on Mar 20, 2020). And readers in the future should note it will apply if and as you may update CF from any update BEFORE this one to any update AFTER this one.

To be clear, I do not mean with this warning to suggest that you should NOT apply the update! It implements an important security fix.

Instead, it's that after applying it, your CF web sites served via IIS or Apache WILL likely break initially, until you take one at least and perhaps two extra steps. The good news is that these steps are both easy and documented by Adobe in the update technotes, but they do require that someone do them, if needed. Let me explain.

[Update: I did an abbreviated version of this post on the Adobe CF portal: Three reasons your sites may break, and how to fix them, after applying March 2020 update to CF2018 or 2016. Note I also titled it differently. Just trying many ways to get people's attention. That post may interest some, either to read first (but my TLDR below also tries to abbreviate things also), or especially if you may prefer to give others a link to a post on this matter that is not as "dense" as this one. :-) I do point to this post from there, of course, for the many additional details that some may appreciate.]

Sadly, because many people don't bother to read the CF update technotes (linked to below), and they just apply the CF updates, they are not noticing this issue until they or their users start screaming because their sites are down. There's also a fair bit of "screaming" in the CF community, and folks responding may not know the info that I (or Adobe) have shared, to get things "working again", so I hope this helps bring some calm, and most important the clear solution/s needed.

[....Continue Reading....]

Comments
Thanks again Charlie for covering our rears! Stay safe and great work as always.
Greg - BMChosting team lead
And thank you, Greg. Glad to help, and all the more to hear that in helping a host, I'm helping many times more people still! :-)
Readers: if you have already read my post here, as originally written last night, please note that I have added a new 3rd point (a possible step you may need to do after the update) in the TLDR section at the top.

Also, I have added a new section at the bottom, "Note that Adobe did NOT update the Tomcat version, only the AJP CONNECTOR itself", which may interest some of you.

For now, I didn't think it worth creating an entirely new/different post for these points, as they don't really affect most people--and would only be "new" info on the post for those who read it since I posted this last night.
Thanks Charlie, really appreciate this detailed info!
# Posted By Chris | 3/22/20 8:00 PM
Thanks, Chris.
First of all, many thanks to you Charlie for your helpful posts, which I have been reading for many, many years and which have often saved my day here in Germany.

After updating ColdFusion 2016 from Update 12 to Upadate 14 on my development server, I encountered a nasty little bug.

For years I have been using "localhost" in the "workers.properties" configuration file for all server instances as host (for example: worker.cfusion.host=localhost) without any problems. After the update to ColdFusion 2016 Update 14 it was no longer possible to establish connections to the ColdFusion Server instances, although I added all new secret values ??to the server instances in the "workers.properties" file (as described by Adobe) manually.

It took me a while to try to use the IP address 127.0.0.1 instead of "localhost" (for example: worker.cfusion.host=127.0.0.1), although this is the same (under the hood). But after this (not really logical) change, everything runs without errors and all connections to the ColdFusion Server instances can be made correctly. It seems that ColdFusion Update 14 does not like "worker.cfusion.host=localhost" anymore. :-(

My server environment:
- Debian Linux 9.12
- Apache 2.4.25
- Nginx 1.16.1 (runs as proxy)
- ColdFusion 2016 Update 14

Maybe this comment will also help other developers in the ColdFusion community. If so, I would be happy.

Greetings from Germany
Knut
# Posted By Knut | 3/24/20 7:01 AM
Thanks for the kind regards, Knut, and hapoy to have helpers (indeed for so mahy years).

As for your issue, it's not a bug. I suspect it's that you have a hosts file that sets localhost on your machine to your own ip address.

If you do a ping localhost or other command at the command line on the machine, what up address does it report? I suspect it's not the traditional localhost/loop back address.

And that's fine normally, but again one of the Tomcat changes (which the CF update incorporates) is default support for ONLY requests from the traditional localhost/loopback addresses, 127.0.0.1 or its ipv6 equivalents, as I indicated above.

And your change of the workers.properies is certainly one way to force the cf connector to "use" that address (so things work with the change)

But as I also noted, you could have also configured the address atrribute on the ajp element of the server.xml file to name the ip address also.

In your case, though, it may be easiest/better to just reconsider the hosts file setting that may be redirecting the localhost domain name to your specific ip address.

There are various reasons people have done that over the years. If you can't recall yours, maybe try commenting it out (the localhost line in the hosts file) and see how it goes.

If it works, you could then set the workers.properties back to localhost. Note that any future connector upgrade or creation will do that, so this is why I think it's the better solution than you changing the file.

Let us know how it goes. If you confirm it's what I suspect, I will tweak the post to add a mention of this point. Thanks for bringing it up. Hope I've solved it for you.
Knut, if you got emailed my last reply, I hope you may see this before replying to it. Please visit the site to see an update I made to the 3rd and next to last paragraphs of my previous comment, as I realized those were necessary for your situation.
I have also now added a new section on this point, which Knut raised and about which I have guessed at an answer. I will look forward to his reply/confirmation, but since I suspect it really is his issue, I added the new section <a href="#localhostchanged">Handling if you have changed the IP address that localhost resolves to</a>, and I also added a mention of it on the second point in the TLDR at the top. I could see it hitting people unexpectedly, and there are the 3 possible solutions one could consider, as I elaborate in that section.
Charly, thank you for your answer.

I checked the entries in the "/etc/hosts" file. It all looks completely normal and unchanged. The IP address 127.0.0.1 and :: 1 are also bound to "locahost".

As you suggested, I'll try the "address" attribute in the server.xml file to name the ip address in the connector. I will report whether this alternative way also works.

Thank you for your kind support.
# Posted By Knut | 3/25/20 6:00 AM
Thanks, but most simply, what ip addtesss do you get if you ping localhost? And what address otherwise were you planning to add as the "address" in server.xml? Very curious to hear the conclusion for you.
When I do a "ping localhost" at the console I get:

PING localhost(server2.xyz.de (::1)) 56 data bytes
64 bytes from server2.xyz.de (::1): icmp_seq=1 ttl=64 time=0.021 ms

I've masked the real domain name with "xyz".
# Posted By Knut | 3/25/20 6:51 AM
I did some trial/error investigation.

Example 1
- Adding the additional attribute "address=localhost" to the connector in server.xml
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: No connectivity

Example 2
- Adding the additional attribute "address=127.0.0.1" to the connector in server.xml
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: No connectivity

Example 3
- Adding the additional attribute "address=127.0.0.1" to the connector in server.xml
- Commenting OUT the "worker.cfusion.host=localhost" entry in the "workers.properties" file
RESULT: No connectivity

Working examples (in my case):
- NO additional attribute "address=127.0.0.1" to the connector in server.xml
- Only using "worker.cfusion.host=127.0.0.1" in the "workers.properties" file
RESULT: Connectivity

- Additional attribute "address=127.0.0.1" to the connector in server.xml
- Using "worker.cfusion.host=127.0.0.1" in the "workers.properties" file
RESULT: Connectivity

So in my view it does not make any difference to add the additional attribute "address=127.0.0.1" (or not) to the connector in server.xml. It seems to be mandantory (at least in my case) to use "worker.cfusion.host=127.0.0.1" in the "workers.properties" file.

My original overall configuration is basically based on the descriptions from Adobe at: https://helpx.adobe.... Under point 4, "worker.cfusion.host=localhost" is also specified, but at least in my case this no longer works with "ColdFusion Update 14". I have to use IP 127.0.0.1 instead of localhost in the "workers.properties" file.
# Posted By Knut | 3/25/20 7:26 AM
Thanks for all that.

So first, I agree something seems amiss. I don't think it's a Linux (or Unix thing), as some with such curious challenges have wondered. Time will tell.

You show the ping reporting ::1. That SHOULD be accepted by the updated ajp connector, by default.

But since you're scrambling to find anything that works, how about adding address="::1" on the ajp connector line in server.xml? You don't show trying that among your many attempts.

As I explain in my post, I recommend against changing the workers.properties file for such workarounds if possible, because the next connector install would set it to localhost by default. And while connector upgrades don't typically change that, one could.

Let us know what you find. Also, did you confirm 0 errors in the update log?
Short note:

>> ...but since I suspect it really is his issue, I added the new section ... Handling if you have changed the IP address that localhost resolves to...

I'm not a linux erxpert, but I think in my case "localhost" resolves to "127.0.0.1" and "::1". This is the server config I'm using since years. This has never been changed.
# Posted By Knut | 3/25/20 8:10 AM
>> But since you're scrambling to find anything that works, how about adding address="::1" on the ajp connector line in server.xml? You don't show trying that among your many attempts.

Okay, I will also try "::1" and give a report to you.
# Posted By Knut | 3/25/20 8:13 AM
Did you write your previous comment ("short note") before or after reading my comment above it? I can't tell, and I realize our messages could have passed each other in the ether.

Either way, I don't quite get what you were getting at. But the real key is to here how things go, per your (and my) last comment. :)
Oh, and about my mention earlier of "I still don't think it's a unix thing", I was confusing a separate discussion I'd been having with someone privately. I realize you said you were on Linux, instead. I will correct my comment, to avoid confusing later readers.

Even so, I will say I don't think your challenge is a Linux thing, either, if anyone reading may wonder (as some have been questioning, about possible OS differences).
>> Did you write your previous comment ("short note") before or after reading my comment above it? I can't tell, and I realize our messages could have passed each other in the ether.

Our posts unfortunately overlap.

I've tried to use "::1" for the additional address attribute in the connector and it works. Great.

Working examples (in my case):

- Additional attribute address="::1" in the connector in the "server.xml" file
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: Connectivity
CONCLUSION: I will use this in this form for all server instances in the future.


I think that normally the connector should resolve "localhost" to "127.0.0.1" and/or "::1" as standard. This is what the connector did bevor the U14 update. I can't see the reason why it is mandantory (in my case) to add the additional "address" attribut in the connector after updating to CF-2016 U14. This is a wired behavior (in my view).

Thank you for your tips and your support.
# Posted By Knut | 3/25/20 8:43 AM
Short additional note:

I just realized, that my two machines ("development" an "production" server) deliver different results for "ping localhost".

Output for development server "ping localhost" at the console:

PING localhost(server2.xyz.de (::1)) 56 data bytes
64 bytes from server2.xyz.de (::1): icmp_seq=1 ttl=64 time=0.021 ms

Output for production server "ping localhost" at the console:

PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.023 ms


I've contacted my hoster to check, why the outputs are different.
# Posted By Knut | 3/25/20 9:19 AM
Yep, I agree that it should not be "needed", but I proposed it in case it might help--and I'm really glad to hear that it did. I had mentioned how the Tomcat docs said that localhost addresses would be supported out of the box, with the update to AJP (which CF incorporates).

The specific wording (on the Tomcat doc page I link to above) says, "By default, the connector will listen on the loopback address." It's looking like that may be only 127.0.0.1, which I find shocking. I almost think something ELSE must explain things.

But if you want to try it, you can use multiple addresses (as I do also indicate above), so address="127.0.0.1,::1". That would cover you either way.

Again, I agree: it should NOT be needed. But until we (all) figure out what's amiss here, it may be the best recommendation, and I could update my post to reflect it. That said, I already added a new section above (as mentioned in a previous comment) before I heard back from you, so I would like to do a little more testing of confirmation before updating the post.

Still, I'll let this comment stand for now, for those with the gumption to read down to here among these now 7000+ words. :-)
Thanks for the very detailed information Charlie. As usual you make valid points. You're articles are always a good reference so please keep up the good work. Stay safe. Cheers!
# Posted By Miguel-F | 3/25/20 1:04 PM
>> But if you want to try it, you can use multiple addresses (as I do also indicate above), so address="127.0.0.1,::1". That would cover you either way.

Good suggestion, I can try it.
Do you think that address="127.0.0.1,::1,localhost" should also work?
# Posted By Knut | 3/25/20 1:21 PM
Did some testing.
Neither address="127.0.0.1,::1" works, nor address="127.0.0.1,::1,localhost" works.

In my case it's only working with: address="::1"

Best regards
Knut
# Posted By Knut | 3/25/20 1:47 PM
Hi Charlie,

Thank you very much for this post and the much needed extra explanation about what was happening. One slight edit if I may suggest, the Adobe kb article says that that allowedRequestAttributesPattern is suppose to be ".*" and not "*". It took me a while before I realized I had gotten it wrong.
# Posted By Samuel Smith | 3/26/20 4:30 PM
Miguel: thanks very much.

Knut: that's odd. I had done it (adding two), and it did work (as I said in the post, I even reversed the order and put one that would fail first, and it still worked). I wonder if there can be some other explanation.

But glad to see that at least just ::1 worked. I have been doing some work that might suggest that Tomcat is ONLY by default accepting from 127.0.0.1, and not ALSO ::1 as one would expect (and I'd said). The odd thing again is that I find that to be an issue on some machines but not others. Will share more as I learn more.
Samuel, thank you very much (and sorry, I had meant to also comment on your point in my last comment). That was indeed a mistake, in both this post and the shorter one I did at the Adobe CF portal (now listed at the top here). I apologize to everyone for that mistake, and of course I have corrected it here and there.
I was unable to get ACF update 8 to work at all after following the suggestions in this post.

Prior to seeing this post, my production server failed after I installed update 8 (Win Server2018, Apache 2.4). I tried updating the web connector, but no joy. I panicked and was able to uninstall the update and got it running again at update 7.

Tried the update again on my development box (Win10, Apache 2.4) with worse results. I tried the suggested fixes in this post, but couldn't get the CF service to restart. Gave up and reinstalled the whole enchilada. Ran update 4 and then update 8 before even setting up the web connector. Same problem: service wouldn't start and wouldn't start after trying to change server.xml AJP settings. I'm wondering if there is something specific to the combination of Windows and Apache that is causing me grief. Any suggestions appreciated.
No, Adam. Update 8 would not have anything that would itslef keep cf from starting. That problem could be caused with ANY CF update, and I pointed above to my blog post on sovimg such problems.

PLEASE read that. You will get cf running again.

Or I can help you directly, likely in as little as 15 minutes. You won't pay for time you don't find valuable.
Also, Adam, since you're using Apache, let me add that once you get CF running again (see my last comment), you should then see if the original problems you had (with the connector) is fixed by using allowedRequestAttributesPattern=".*", as discussed in prior comments and a correction I made yesterday.

I will be doing more research and documentation on the matter (why it's needed, when, and better alternatives to opening things up that way).
Adding to my last comment, I just posted some more detail in a comment on a CF forum message from someone else, and I shared there some more detail--my initial findings. It may interest some readers, until I can get more documented:

https://community.ad...
Thanks as always, Charlie, for these posts. I feel like every CF update over the past 20 years has given me some degree of anxiety, but it's great to know you're always here to help us through.
# Posted By Rocky Rosen | 3/29/20 6:32 PM
Thanks, so much, Rocky. Happy to be able to help.
I have also seen where the Tomcat server resolution would delay processing by up one second. But adding `address="::1"` to the server.xml AJP line the problem goes away. If you noticed a slow down but not an error this would something to investigate.
# Posted By Bilal | 4/14/20 1:13 PM
Thanks, Bilal. I'd see the discussions of that on the Lucee list, but I'd hoped to get more clarity on when it does or doesn't happen (the delay), including whether anything would influence it happening (or not) on cf or tomcat without lucee. Then I lost track of the concern.

So I'm glad you mentioned it here, in case it may help someone. Thanks, and as always thanks for your efforts on the alternative boncode connector.
Once again you da man! Thank you for this Charlie!
Thanks for the encouragement, Brian. :-)
Thank you Charlie!

Unfortunately I just found your post/blog after finding the solution myself after working on it the entire day.

I worked on my development desktop running Windows 10, Apache 2.4.41 (Win64) and Coldfusion 2016 which had hotfix 13 before I updated it to hotfix 15 (I had skipped 14).

My finding is that the connector definition (without address=) in server.xml resolves to 127.0.0.1 while the localhost in worker.cfusion.host=localhost in workers.properties resolves to ::1.

Putting "address=::1" in the connector definition in server.xml alone solved the problem. Mine was 503.

You could also use any of these combinations:

connector with address=::1 and workers with localhost (as mentioned above)
connector with no address and workers with 127.0.0.1
connector with address=::1 and workers with ::1
connector with address=127.0.0.1 and workers with 127.0.0.1

Thank you anyway. I was a pleasure and reassured me of my assessment reading your entire post.
# Posted By Claude | 4/16/20 8:53 PM
Bummer to hear that you were troubled all day before finding this.. And yep, on how the address="::1" seems a solution needed especially for apache, as I did indicate in the post.

There's a lot to the post, of course. That's also why I made the more brief post on the Adobe site, as I noted at the top. I have plans for another resource, especially to help folks installing cf and then doing updates.
Just adding a +1 to the note about BonCode connector. We also had to add address="::1", otherwise it was incredibly slow. Just in case anyone sees the same and banging head against the wall. Charlie, legend as always, these articles are invaluable.
# Posted By Dave Quested | 4/19/20 5:09 PM
Thanks so much, Dave, for both the kind regards and the additional info.
Thanks Charlie for CF update8 Traoubleshooting,
I red this page and doing as following... but doesn't work website that shown 500 error. not 503, 403 error.
Could you advice me please.
---
Step1: Amazon ColdFusion (2018 release - Ubuntu) : Made instance : 64bitAMI.
Installed CF success : ColdFusion:Y , API Manager:N , PMT:N, Servlet(RDS,JS Debug etc) None.
Created new file index.cfm on {CF_HOME}/cfusion/wwwroot/. then brows it and shown it's content.

Step2: ColdFusion 2018 Update 4 : success on CF Administrator on chrome browser.

Step3: ColdFusion 2018 Update 5 : success on CF Administrator on chrome browser.

Step4: ColdFusion 2018 Update 6 : success on CF Administrator on chrome browser.

Step5: ColdFusion 2018 Update 8 : messaged in popup "Install Status: Server taking too long to restart. You will have to manually restart server and check update status".
Reload web page CFIDE/administrator/ was 500 error.

Step6: Login on SSH and check two files.
server.xml : additional `secret="D978D4AB-AE67-4F2D-A129-6E95075FFF81"` in AJP connector line.
workers.properties : nothing `worker.cfusion.secret`.

Step7: doing conname `./wsconfig -upgrade -v`

Step8: check workers.properties file.
additional 1 line `worker.cfusion.secret=D978D4AB-AE67-4F2D-A129-6E95075FFF81`.

Step9: restart cfusion and apache2
stop cfusion , stop apache2, start apache2, start cfusion

Step10: brows CFIDE/administrator
shown 500 error.
And index.cfm was500 error, too.

Step11: Changed workres.properties file
Changed from `worker.cfusion.host=localhost` to
`worker.cfusion.host=127.0.0.1`.

Step12: restart cfusion and apache2 then brows CFIDE/administrator
shown 500 error.
And index.cfm was500 error, too.

----------
I have try and try many type setting for a several days but does not work cfusion...

best regards
# Posted By Taka Iga | 4/30/20 2:40 AM
Taka, see the first link I offered, at the end of the tldr section, where I explain what to do if you have issues after applying ANY cf update.

In your case, you said in step 10 that you applied update 8 and then could not access the CF admin. I assume you are using a port for that, like 8500 (the default). That then is NOT using the ajp web server connector (and the changes I regarding it which I discuss in this post).

As the other post explains, your next step is to a) confirm if there were errors in the update log, then if so b) re-apply the update, whether in the admin or at the command line. Details in that post.

Let us know how it goes. Or I can help you via a screen share, in a consulting session of likely less than an hour. You don't have to pay if I don't solve it.
Thank you for quickly response, Charlie san.
Is the first link "Having problems after applying a CF update? What to check, and how to recover!", this link URL same this post(.../2020/3/20/...). It missed link page !? the page URL is .../2019/03/.., right !?
I respond just info, I'm going to read the page you offered and check my environment.
Best Regards
# Posted By Taka Iga | 4/30/20 8:07 AM
Wow, thanks for that. The URL was indeed wrong (it was a copy/paste error, linking instead to the same url as the very first link in the post, for the "brief" version of this post). It's sad to think no one else noticed or reported it all this time. I have corrected it. Again, thanks.

And yes, let's hear how it goes for you.
Hi, Charlie san,

I checked and tried, but not work yet. same status before.
How can I check other points, could you advise me please.

I done as below...
1. I check "install log" file of Update1, 4, 5, 6 and Update8. All Update was Success, ZERO error/fatal error.

2. Re-Update Update8 by command line, shown "This update is already installed."

3. Stop CF, stop Apaceh2, start CF, start apache and brows page as
"http://{IP address}/index.cfm" => shown 500 error.
"http://{IP address}/CFIDE/administrator/" => 404 Not Found.
for example,
"http://{IP address}/index.html" => shown "Apache2 Ubuntu Default Page".
"http://{IP address}:8500/CFIDE/administrator/" => 500 error.

hummm...

---
BTW, you fixed LINK ^^, but the link page was WHITE page and your other blogs too...(I kept the page by capture ^^).
# Posted By Taka Iga | 5/1/20 3:22 AM
Taka, if there are indeed 0 errors in the install log for update 8, then there's no need to reinstall, and that's also why you cannot.

So now the key is that you are confirming that you get an error in the cf admin and you are CONFIRMING that you were accessing it via port 8500. That tells us again that whatever your problem is (with that), it's not about the ajp connector.

But can you confirm as well that you WERE using thst port 8500 to access the admin before the update?

If not, were you accessing the admin without a port? That would not be allowed by default in cf 2018 (or 2016). You'd have to tweak the connector properties to allow that. Did you?

If so, can you confirm that cf is even listening on port 8500, for its built in web server/biws? It may be some other port.

Bottom line: if cf is running and the biws is working (and the admin accessible), then your other issues COULD still be about the ajp connector. Did you do all the things listed at the top? Upgrade the connector, add the address attribute, if needed? Add the allowedRequestAtrributesPatten, if needed? See above for more.

If you may write back to ask continued questions, please answer every question I ask in this comment. You can also email me at Charlie (at) carehart (dot) org, which may be easier/better than extended discussion here.

Finally, I will say again that if you just want to get the problem solved without all this back and forth we could solve it in a quick consulting session.
Oh, and thank you Taka for pointing out how the link I updated goes to a blank page. That is indeed a very unfortunate thing to discover. To be clear, it's not on my site but Adobe's. I have bought it to their attention.

What's especially sad is that it seems to be limited to pages of mine. I can think of no explanation for that. They have said someone will work on it next week. I am trying to remain calm about it, but it is indeed disconcerting.
Charlie san,
I email you. Could you see it and advice me please...
Yes, I access CF Administrator with post :8500. not use Buildin server.
I tyied connector option "address", "allowedRequestAtrributesPatten".
thank you so much !
# Posted By Taka Iga | 5/5/20 3:10 AM
Thank you for your write up! It's nuts that the adobe kb has so little info on error 403. I got it working by setting the allowedRequestAttributePattern to .* but figuring out the actual request attributes and how to set them as a RegEx has been a challenge.
I ran gethttprequestdata().headers like you mentioned and got 16 different values back. In addition to Xsite-name I got stuff like "accept" "connection" "host"... Could you clue me in how to specify them in the allowedRequestAttributesPattern attribute? "Xsite\-name|accept|connection|host..."
Again, thank you for your support!
# Posted By Rob | 5/5/20 10:27 PM
We are also suffering from the 403 error, and would appreciate any help you can offer. We're running
macOS mojave v. 10.14.6
apache httpd v. 2.4.34 (configured for https)
mod_jk.so v. 1.2.46
apache tomcat v. 9.0.34

$ ping localhost
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.036 ms
...

In server.xml (with 'address="127.0.0.1"' or 'address="127.0.0.1,::1"', httpd couldn't connect to tomcat at all:
<Connector port="8009"
protocol="AJP/1.3"
redirectPort="8443"
address="::1"
secret="somePassword"
allowedRequestAttributesPattern=".*" />

previously, in tomcat v. 8.0.21:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />q

In localhost_access_log.2020-05-18.txt:
xx.xxx.xxx.143 - - [18/May/2020:13:00:43 -0700] "GET /mpstruc HTTP/1.1" 403 618

workers.properties:
workers.tomcat_home=/Library/Tomcat
ps=/
worker.list=ajp13
worker.ajp13.port=8009
worker.ajp13.host=localhost
worker.ajp13.type=ajp13
worker.ajp13.secret="somePassword"

/etc/hosts:
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost

A distillation from httpd mod_jk_log:

init_ws_service::mod_jk.c (1179): Service protocol=HTTP/1.1 method=GET ssl=true host=(null) addr=76.21.65.212 name=ourwebsite.edu p
ort=443 auth=(null) user=(null) laddr=128.200.165.4 raddr=76.21.65.212 uaddr=76.21.65.212 uri=/mpstruc/newsfeed
ajp_get_endpoint::jk_ajp_common.c (3357): (ajp13) acquired connection pool slot=0 after 0 retries
ajp_marshal_into_msgb::jk_ajp_common.c (681): (ajp13) ajp marshaling done
ajp_service::jk_ajp_common.c (2588): processing ajp13 with 2 retries
ajp_send_request::jk_ajp_common.c (1719): (ajp13) no usable connection found, will create a new one.
jk_open_socket::jk_connect.c (674): socket TCP_NODELAY set to On
jk_open_socket::jk_connect.c (798): trying to connect socket 31 to ::1:8009
jk_open_socket::jk_connect.c (824): socket 31 [::1c1e:1f49:0:0:56240 -> ::f00:c43:b281:f1fd:8009] connected
ajp_connection_tcp_send_message::jk_ajp_common.c (1264): sending to ajp13 pos=4 len=380 max=8192
ajp_connection_tcp_send_message::jk_ajp_common.c (1264): 0000 12 34 01 78 02 02 00 08 48 54 54 50 2F 31 2E 31 - .4.x....HTTP/1.1
...
ajp_send_request::jk_ajp_common.c (1779): (ajp13) request body to send 0 - request body to resend 0
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): received from ajp13 pos=0 len=54 max=8192
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0000 04 01 93 00 03 34 30 33 00 00 03 A0 01 00 17 74 - .....403.......t
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0010 65 78 74 2F 68 74 6D 6C 3B 63 68 61 72 73 65 74 - ext/html;charset
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0020 3D 75 74 66 2D 38 00 A0 02 00 02 65 6E 00 A0 03 - =utf-8.....en...
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0030 00 03 36 31 38 00 00 00 00 00 00 00 00 00 00 00 - ..618...........
ajp_unmarshal_response::jk_ajp_common.c (736): (ajp13) status = 403
ajp_unmarshal_response::jk_ajp_common.c (743): Number of headers is = 3
ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13) Header[0] [Content-Type] = [text/html;charset=utf-8]
ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13) Header[1] [Content-Language] = [en]
ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13) Header[2] [Content-Length] = [618]
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): received from ajp13 pos=0 len=622 max=8192
# Posted By Craig S. | 5/18/20 4:07 PM
Craig, would you confirm that you're simply asking as a user of tomcat, rather than coldfusion (running on tomcat)? I ask because your ajp port of 8009 is not what cf would set.

One may ask, why does it matter? But if you have installed your own tomcat, and configure by hand your ajp files and how apache points to them, you have to be sure you properly updated the right files.

For instance, you state the version of your mod_jk.so, but you don't indicate where it exists. Are you sure apache is loading it? Have you checked its conf files, to confirm? And even if you think all's well with that, what if you rbamed the so file, or the conf file? Does httpd still start? It should not. That's a quick sanity check. I have seen people chasing the wrong problem by mistaken assumptions.

And this is key, because if the wrong so is running, or that secret for the workers.properties is wrong, you'd get a 403.

Same if somehow your server.xml changes were not to the right one. I'd tweak that in a way that should stop tomcat running. Btw, it could come up but not process pages. So be sure to check a page running via its web server (the http connector in server.xml), to make sure tomcat IS running, separate from requests via ajp/apache.

I realize this may sound like crazy ideas, but you're presenting that "all is well" with your config files as presented, and I concur. But clearly something's not right.

One other thing you might try: changing the localhost value in workers.properties to ::1, to match your server.xml. Of course, restart apache after that, or tomcat after changing server.xml. (Some folks don't realize that chbages to the former are loaded by apache rather than tomcat).)

Let us know how it goes, even if that still nothing works. In that case, I'd think it time to enable more logging or even network monitoring, to find what's really going on.
Hi Charlie,

Yes, we are not running cold fusion, just tomcat behind apache httpd.

Apple did not provide the mod_jk.so module in the standard module location (/usr/libexec/apache2/) in macOS 10.14, and wouldn't allow me to add it there, so I put it in /etc/apache2/libexec (etc/apache2 is the apache home directory). httpd.conf includes it with (showing some context):

LoadModule hfs_apple_module libexec/apache2/mod_hfs_apple.so
LoadModule jk_module /etc/apache2/libexec/mod_jk.so

and the log /var/log/apache2/error_log.1589414400 indicates that httpd is finding it:

[Tue May 19 12:02:50.526249 2020] [mpm_prefork:notice] [pid 102] AH00163: Apache/2.4.34 (Unix) LibreSSL/2.5.5 mod_jk/1.2.46 configured -- resuming normal operations

Apache seems to be running fine, and is serving our static pages. Tomcat also seems to be running fine, and hearing requests (e.g., see the excerpt from the localhost_access_log in my previous post), but denying access to all requests.

I've tried changing workers.properties worker.ajp13.host=localhost to worker.ajp13.host=::1, no difference (and I always restart after such a config change).
I've set all the logging levels in tomcat/conf/logging.properties to ALL, and the logging for mod_jk to "JKLogLevel debug". While I'm getting a lot of information from the httpd side of things, the only thing tomcat wants to tell me is, again, that 403 response in the localhost_access_log. So I'm not sure how to enable more logging at this point. What kind of network monitoring would you suggest?

Nothing is crazy at this point - we've been pulling our hair out for a couple of weeks! Thanks for taking a look!
# Posted By Craig S. | 5/19/20 4:07 PM
Interesting challenge, Craig. But again, it has to be some configuration issue.

It does seem you've covered all your bases for the typical things that trip folks up. And yep, the mod_jk debugging logging does confirm that the request is getting from apache TO Tomcat.

Still, I have a couple other ideas.

First, does your password have special characters? Have you tried setting it to a simple string, just for the sake of testing? If that works, perhaps there was an issue in the passing of that.

Second, is it possible that the request you're making is not going through mod_jk but mod_proxy_ajp? In that case, the latter would not honor the secret set in the workers.properties, but would instead be set on the proxypass line, from what I have read (I have not tested this).

Third, you haven't indicated what Tomcat version you're using. And the mod_jk.so is not reporting to be the latest. While I realize you may have reason not to want to be "on the latest", it seems worth trying just to find out if updating both might resolve this, as there may be some bug.

In fact, there was one about incoming headers having a new line character that was tripping up Tomcat, since fixed. (Your debug output doesn't show that, but I have not investigated to find out how that looks in the log if that's the issue, and if it's even tracked there.)

Indeed, one could argue that a real challenge is that even the ajp debug logging doesn't tell us WHY the request gets a 403. That seems reasonable to expect, but the folks behind Tomcat are VERY careful and thorough and always have well-reasoned explanations for why they have done (or not done) something.

And to that very point, if none of the above works, I'd recommend you take the problem to the tomcat-users mailing list. That is THE best place to get resolution to challenges like this, as most questions are answered by the very people who wrote tomcat and the ajp connector.

And of course, they have been dealing with ramifications of these ghostcat changes for months now. So they may think of something readily that we have missed here. You can learn more here: http://tomcat.apache...

But before anyone would write their first note to that list, they should note the indications on that page of HOW questions should be asked. Even better, read a few threads in the list before making a first post to take the pulse of things (there's a link to the online archives--even already from today as I write).

If you look at threads where there have been a few or several replies, you'll get a good sense of how things go--and when they go well, or not. :-) Sometimes it can be a bit like watching a cross between the scenes when Dorothy's crew approached the Wizard of Oz, and the courtroom scenes in My Cousin Vinny. I'm not trying to scare anyone off. Again, they are whip smart as a group and great at solving knotty challenges. Just be prepared in how you present things.

If you do ask there, and especially if you get a resolution, I hope you will return here to let folks know (either the answer or a link to the online discussion).
And Rob, sorry I missed your note from a couple weeks ago. I'd offer to you the same suggestion I made to Craig at the end of my last comment. Yours is a good question, not answered as clearly in the docs as some might hope. If you agree, take it to the tomcat-users list (with the same caution I shared with Craig).

And again, if you get an answer, please post it here for others to know. Thanks.
As for Taka's issue (before Craig and Rob), let me update those following along. He and I did start working together offline on his issue--but it seems NOT about the AJP connector.

His problem is that NO CF requests run, even using CF's "built-in web server" (which is the Tomcat web server, rather than Apache talking to CF/Tomcat over AJP), after applying the latest (April) CF update. And for those interested, there are no errors in the CF update log.

I have now heard of one other person with the same problem, but only one other like it, so there's some combination of characteristics they share. I'm trying to figure that out with each.

And Adobe has gotten involved also (with the other client I'm working with). If we learn something, I will share it here for everyone to know. But to be clear, it seems unrelated to the topic of this post, which is about the changes regarding AJP processing per the March updates to CF 2018 and 2016.
Hey, Craig and Rob, any news?
Charlie, thanks for your help with this problem, which has finally been resolved. We're running macOS 10.14.6, upgraded from Apple-installed httpd v.2.4.34 to Homebrew-installed v. 2.4.43, and tomcat v.8.0.21 (when this started) to v.8.5.4, to 9.0.34, and finally v.9.0.35. Moving from mod_jk.so 1.2.46 to 1.2.48 was problematic - no binary available, and compiling from source wasn't working, so I stuck with 1.2.46.
Ultimately, once those upgrades were made, the final, unintuitive change that finally got things working was, in workers.properties, going from

worker.ajp13.secret="somePassword"

to

worker.ajp13.secret=somePassword

On the tomcat side, in server.xml, the double quotes are needed, as in

secret="somePassword"
# Posted By Craig S. | 6/4/20 5:49 PM
Wow, ok. Thanks for sharing. Hope it may help others.
Charlie san,
I read a message from Craig (Thanks Craig san). I tried that mod_jk.so compiling from source code v.1.2.48.
but AWS CF Administrator (:8500) do not work yet.
I'm continuing read this web site.
Thanks all
# Posted By Taka | 6/4/20 10:27 PM
Charlie - Thank you so much for posting this detailed response. I have been struggling with a unstable Apache/CF setup since applying update 9 and following the tech notes. We are using RHEL7, Apache 2.4 and CF2018, and have invested (wasted) so much time and effort trying to get this resolved. We even stripped back the services reinstalled both CF and Apache and modified a great number of Selinux settings. Finally after adding address="::1" to the AJP connector in the server.xml file we are back in business. If we had not found your post, we will have dropped Coldfusion - despite having worked with it for 10 years +
# Posted By Peter | 6/16/20 9:35 AM
Thanks, Peter, and glad I could help: especially to save you from the brink of leaving CF.

I had in fact expressed my concern about that in my first comment in the Adobe blog post on the the day it was announced (https://coldfusion.a...), where my opening sentence was, "Wow, I suspect this update is going to start a lot of trouble for a lot of people."

In the weeks since I and others (including Tomcat folks) have tried to share more and more info to help folks, but there's no doubting that anyone updating from that point forward (for the first time, such as in your going to CF2018 update 9) will hit this, so we will see people struggle with this for years...or until Adobe simply creates a new installer that embeds the latest updates to CF and the connector, and perhaps reconsiders tweaking the wsconfig tool to help people get the RIGHT value for that address attribute, or whether to enable that allowedrequestattributespattern override. Different strokes for different folks.
And Taka, to your issue, that's not really with this matter related to the web server connector. Instead, that's about an issue that it seems would hit anyone who is running on the AWS AMI from Orbiterra. I have been in touch with Adobe about this with another client of mine, and have mentioned how it hit you also. For now, their only answer is to propose an alternative AMI. I am pressing to get them to figure out what it is about THIS AMI that keeps one from being able to update CF. Obviously, it's not a good situation to be unable to update.

But to be clear, it's a different problem than this blog post's focus. So keep in touch with me directly rather than here--and anyone interested in the topic can reach out to me also (see the contact info offered here on my site). Perhaps I will create a new blog post about it, especially if a specific solution is identified.
when i upgraded my coldfusion 16 to update 14, the coldfusion instance would not start. got an abort error when i tried to start the service. when i did cfstart.bat got the same error an abort. i was able to look at the even viewer log was getting the following error:

- System

- Provider

[ Name] Application Error

- EventID 1000

[ Qualifiers] 0

Level 2

Task 100

Keywords 0x80000000000000

- TimeCreated

[ SystemTime] 2020-09-22T01:30:20.000000000Z

EventRecordID 235026

Channel Application

Computer WIN-

Security


- EventData

coldfusion.exe
2016.0.0.0
5b92a2b1
coldfusion.exe
2016.0.0.0
5b92a2b1
80000003
00000000000341c2
ea0
01d6907fecf895a4
C:\ColdFusion2016\cfusion\bin\coldfusion.exe
C:\ColdFusion2016\cfusion\bin\coldfusion.exe
2d04d3f3-fc73-11ea-9db5-d4ae52b9744e

I went round and round trying to get the coldfusion.exe to start, so i uninstalled and fortunately the uninstall worked and was able to start coldfusion and get the website up and running again.

nowhere does this issue get addressed.
# Posted By darrell | 9/21/20 9:11 PM
Darrell, it's not unusual to have errors after applying any cf update. There are various reasons and easy solutions.

I discuss these in another post:
https://coldfusion.a...

Neither the cause nor the solution is indicated in that Windows Event Log. The fact that you could uninstall the update is an indication how you can simply try again, after considering what I offer in that post.

If somehow you still struggle, I can offer direct remote assistance. I'm sure we could resolve it, and likely have cf updated for you in less than 15 mins. If interested, check out carehart.org/consulting.
Hi,

Just to say thanks for these posts! I have a fresh install of CF2021 in VirtualBox Ubuntu Server 20.04 LTS with Apache mod_rewrite. The rewrites to .cfm files did not work at all giving Tomcat's 403.

After banging my head and going all over the net for hours searching for fix it turned out to be as simple as adding that allowedRequestAttributesPattern=". *" to the server.xml file.

You saved my day!

BR,

Aarni
# Posted By Aarni | 5/12/21 6:38 AM
Thanks again Charlie for all the excellent info.

On the requiredSecret attribute, after installing the CF2018 update 12 I had two servers returning 403 errors for cfm files, I tried adding allowedRequestAttributesPattern=".*" and an IP address to the AJP connector line as well as wsconfig upgrades. It wasn't until I removed the requiredSecret attribute that they started working.
# Posted By Chris | 9/21/21 12:35 AM
Chris, that is indeed curious. If the cf update was successful, along with the cf web server connector update, then that "old" requireddecret should be ignored, as I discuss above.

The fact that you had to remove it would have me worried that either of those was not successfully updated. I discuss above how to check if a cf update is successful (it's not enough to confirm merely that it has been installed). For more, see my post that I point to in that above section:
https://coldfusion.a...

As for checking that the wsconfig is updated, I discuss that in the blog post I point to on updating the connector itself:
https://www.carehart... You'd want to see the isapi_redirect.dll (for iis) or mod_jk.so (for Apache) had a date of just before the date of the cf update you'd applied (assuming that update did call for an update to the connector, as most do). In your case, having updated to cf2018 u12 (from this year), the connector dll or so file should be from 2021. Is it?

If you want any help double-checking all that, as I note above we could do that on a consulting basis in perhaps as little as 15 mins. More at carehart.org/consulting.

Or let us know if you solve things yourself.
Thanks Charlie, I thought I'd mention it in case it was of interest to you or anyone else with the same issue.

I've installed the update on four Linux/Apache servers, all update logs are without warnings or errors, and all now have the same August 2021 date on mod_jk.so.

The two other servers didn't have the requiredSecret attribute and had no 403 errors, I've now tested adding requiredSecret="test" to those and both went to 403 with it added.

I would definitely call on your services if there was still a problem, but removing the redundant requiredSecret seems to have been the fix for the servers that had it.
# Posted By Chris | 9/21/21 9:36 PM
Ok, but when you say "redundant", do you mean what you deem to be "unneeded"? I would argue one SHOULD be needed unless something's amiss.

But I realize you feel you're all set, so I share this as much for future readers. If you, I, or anyone else ever learns something more about this, perhaps it will be added here.

Glad you're back in business. :-)
Thanks Charlie, my read of your 'On secrets vs requiredSecrets' section above was that the secret attribute replaced requiredSecret, and newer Tomcat versions would only be using the secret attribute?
# Posted By Chris | 9/21/21 9:55 PM
My bad. It's late (for me), and I'm seeing this on my phone. Yes, you're right: requiredSecret is the OLD attribute name. It should be ignored in a properly updated CF/Tomcat deployment. The fact that it CAUSED an impact is disconcerting.

But you're set. If things change or we learn more, we can add it here.
Many thanks for confirming Charlie, and again for providing so much helpful info on CF!
# Posted By Chris | 9/21/21 11:49 PM
Chris, I think you may have just saved me another round of headaches. I tried the update over the weekend and was struggling to find out why I was getting 403 errors before I reverted back to the previous update. I now see I have this requiredSecret attribute in there. I will purge that and give this another try during our next update window.
# Posted By Chris Braster | 9/28/21 8:30 AM
Hi Chris, it would be good to hear if it solves the problem for you too, good luck and please keep us posted!
# Posted By Chris | 9/29/21 12:07 AM
I will definitely report back. Looking like early Sunday morning is our next go time. Fingers crossed dropping the requiredSecrets does the trick.
# Posted By Chris Braster | 9/29/21 6:06 AM
Reporting back that my second go at the update was a complete success. Just needed to remove the requiredSecret attribute from the connector in server.xml, apply the new update, and done. Thanks again Chris, and thanks Charlie for doing everything that you do for the community.
# Posted By Chris Braster | 10/3/21 2:45 AM
Thanks to you both. This has indeed hit others, it seems. I've not been able to recreate it. The questions are first what caused that old requiredSectets to be there...or worse, be suddenly put there. Then second, why is it being paid attention to, if the newer "secret" attribute is there--in which case itnseems the old requiredSectets attribute should be ignored.

Also, we need to k ow what exactly is causing any recent change in behavior regarding all this? The tomcat team (and something cf simply includes)? Or the Adobe cf team, who do modify the tomcat ajp web server connector (and implement it with the cf wsconfig tool)?

And finally, does it only happen in some situations? Some previous cf update vs another? Some previous cf version vs another? Some previous wsconfig version vs another?

Again, let's see what we learn over time.
Very glad to hear it worked for you too Chris, thanks for the update.

Charlie, I believe the requiredSecrets were there prior to update 12 on the servers I had the problem with, which had updates 4 and 11 installed when they were set up earlier this year.
# Posted By Chris | 10/3/21 11:49 PM
Here's an update on the "new" 403 errors found to be occurring recently. These are the ones discussed in the past few comments here over the past couple of weeks. These seem to be caused when the "older" requiredSecret attribute exists (in the server.xml ajp connector line) IN ADDITION TO the "newer" secret attribute on the same connector line.

Until recently, it seemed the "older" one was was ignored, but now its presence is causing 403 errors for some. Why?

First, some folks are saying they did not PUT that the requiredSecret there, and that as far as they know it's been there a while (not causing trouble, even after the ghostcat updates to the connector and CF discussed in this post starting in March 2020).

It seems THAT perhaps it's presence is a leftover from their having run the Adobe CF Lockdown tool, which had been introduced in CF2018. It seems THAT put this "old" requiredSecret in place (then), for the sake of connector security.

Again, the ghostcat changes led Tomcat to create a NEW "secret" attribute instead (and a related secretRequired boolean, whose name was too close to the old attribute, in my opinion).

Anyway, it seems the presence of BOTH the "secret" and "requiredSecret" was being ignored until recently. Perhaps there was a change in Tomcat in the Adobe CF connector.

So for now, those getting a 403 that can't be solved by the other means above (such as the address attribute) should check to see simply if they have both a secret AND requiredSecret on the ajp connector line in server.xml. If so, remove the latter (maybe copy it off to a comment line), and restart CF, and see if your 403 goes away.

Feel free to share here whether it does or does not work for you, even months from now as I write.
I can confirm that I ran the Lockdown Tool on the initial installation of CF 2018 and that was also where I was thinking the requiredSecret originally came from.
# Posted By Chris Braster | 10/22/21 4:36 PM
Hi Charlie,

I updated from 2018,0,06,316308 to 2018,0,13,329786, and ColdFusion restarted without errors.

According to the Update 13 instructions, I need to follow the post-installation steps for Update 8.

This is a locked-down installation that already had the requiredSecret attribute, and the web server and ColdFusion are on the same machine. In this scenario, the instructions say: "If you have already locked down ColdFusion, then you need not take any action, since ColdFusion instances are already configured with the requiredSecret attribute and your web server also has the secret information with it to communicate."

Should I upgrade the connectors anyway?

Thanks for any info!

JG
# Posted By JG | 12/22/21 11:37 AM
You do need to update the connector, as it's changed since then. And that Cf 2018 update 8 will also implement a new (different) "secret" in the server.xml that will require the connector update to change the worker.properties to have that NEW secret value.

Then you need to remove the old "requiredSecret" in the server.xml, due to the Tomcat change in a recent cf update.
Thanks for the instructions, Charlie!

Upgrading the connectors caused 403 errors, but removing the requiredSecret in server.xml fixed that.

Happy holidays!
# Posted By JG | 12/22/21 12:34 PM
Sweet, thanks, and same to you and all.
Hi Charlie,

Is there a way of adding a secret value to a manually created connector that is connecting to IIS.

I have installed CF2018, completed all the updates, and then added a manual configured connector for iis following the adobe documentation, as per my software vendor advice. I am just getting 500 errors and I feel it must be the connector that is playing up.

Any advice would be greatly appreciated

Laurie
# Posted By Laurie | 1/19/22 10:10 AM
To interested readers, I offered Laurie an answer elsewhere that he asked about this the same day, and which I'd seen and replied to first:
https://coldfusion.a...
Copyright ©2024 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting