How and why your sites may break, and what to do, after applying March 2020 update to CF2018 or 2016
Note: This blog post is from 2020. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.This is a critical warning to anyone who may apply the recent CF2018 Update 8 or CF2016 Update 14, released Tuesday of this week (on Mar 20, 2020). And readers in the future should note it will apply if and as you may update CF from any update BEFORE this one to any update AFTER this one.
To be clear, I do not mean with this warning to suggest that you should NOT apply the update! It implements an important security fix.
Instead, it's that after applying it, your CF web sites served via IIS or Apache WILL likely break initially, until you take one at least and perhaps two extra steps. The good news is that these steps are both easy and documented by Adobe in the update technotes, but they do require that someone do them, if needed. Let me explain.
[Update: I did an abbreviated version of this post on the Adobe CF portal: Three reasons your sites may break, and how to fix them, after applying March 2020 update to CF2018 or 2016. Note I also titled it differently. Just trying many ways to get people's attention. That post may interest some, either to read first (but my TLDR below also tries to abbreviate things also), or especially if you may prefer to give others a link to a post on this matter that is not as "dense" as this one. :-) I do point to this post from there, of course, for the many additional details that some may appreciate.]
Sadly, because many people don't bother to read the CF update technotes (linked to below), and they just apply the CF updates, they are not noticing this issue until they or their users start screaming because their sites are down. There's also a fair bit of "screaming" in the CF community, and folks responding may not know the info that I (or Adobe) have shared, to get things "working again", so I hope this helps bring some calm, and most important the clear solution/s needed.
Greg - BMChosting team lead
Also, I have added a new section at the bottom, "Note that Adobe did NOT update the Tomcat version, only the AJP CONNECTOR itself", which may interest some of you.
For now, I didn't think it worth creating an entirely new/different post for these points, as they don't really affect most people--and would only be "new" info on the post for those who read it since I posted this last night.
After updating ColdFusion 2016 from Update 12 to Upadate 14 on my development server, I encountered a nasty little bug.
For years I have been using "localhost" in the "workers.properties" configuration file for all server instances as host (for example: worker.cfusion.host=localhost) without any problems. After the update to ColdFusion 2016 Update 14 it was no longer possible to establish connections to the ColdFusion Server instances, although I added all new secret values ??to the server instances in the "workers.properties" file (as described by Adobe) manually.
It took me a while to try to use the IP address 127.0.0.1 instead of "localhost" (for example: worker.cfusion.host=127.0.0.1), although this is the same (under the hood). But after this (not really logical) change, everything runs without errors and all connections to the ColdFusion Server instances can be made correctly. It seems that ColdFusion Update 14 does not like "worker.cfusion.host=localhost" anymore. :-(
My server environment:
- Debian Linux 9.12
- Apache 2.4.25
- Nginx 1.16.1 (runs as proxy)
- ColdFusion 2016 Update 14
Maybe this comment will also help other developers in the ColdFusion community. If so, I would be happy.
Greetings from Germany
Knut
As for your issue, it's not a bug. I suspect it's that you have a hosts file that sets localhost on your machine to your own ip address.
If you do a ping localhost or other command at the command line on the machine, what up address does it report? I suspect it's not the traditional localhost/loop back address.
And that's fine normally, but again one of the Tomcat changes (which the CF update incorporates) is default support for ONLY requests from the traditional localhost/loopback addresses, 127.0.0.1 or its ipv6 equivalents, as I indicated above.
And your change of the workers.properies is certainly one way to force the cf connector to "use" that address (so things work with the change)
But as I also noted, you could have also configured the address atrribute on the ajp element of the server.xml file to name the ip address also.
In your case, though, it may be easiest/better to just reconsider the hosts file setting that may be redirecting the localhost domain name to your specific ip address.
There are various reasons people have done that over the years. If you can't recall yours, maybe try commenting it out (the localhost line in the hosts file) and see how it goes.
If it works, you could then set the workers.properties back to localhost. Note that any future connector upgrade or creation will do that, so this is why I think it's the better solution than you changing the file.
Let us know how it goes. If you confirm it's what I suspect, I will tweak the post to add a mention of this point. Thanks for bringing it up. Hope I've solved it for you.
I checked the entries in the "/etc/hosts" file. It all looks completely normal and unchanged. The IP address 127.0.0.1 and :: 1 are also bound to "locahost".
As you suggested, I'll try the "address" attribute in the server.xml file to name the ip address in the connector. I will report whether this alternative way also works.
Thank you for your kind support.
PING localhost(server2.xyz.de (::1)) 56 data bytes
64 bytes from server2.xyz.de (::1): icmp_seq=1 ttl=64 time=0.021 ms
I've masked the real domain name with "xyz".
Example 1
- Adding the additional attribute "address=localhost" to the connector in server.xml
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: No connectivity
Example 2
- Adding the additional attribute "address=127.0.0.1" to the connector in server.xml
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: No connectivity
Example 3
- Adding the additional attribute "address=127.0.0.1" to the connector in server.xml
- Commenting OUT the "worker.cfusion.host=localhost" entry in the "workers.properties" file
RESULT: No connectivity
Working examples (in my case):
- NO additional attribute "address=127.0.0.1" to the connector in server.xml
- Only using "worker.cfusion.host=127.0.0.1" in the "workers.properties" file
RESULT: Connectivity
- Additional attribute "address=127.0.0.1" to the connector in server.xml
- Using "worker.cfusion.host=127.0.0.1" in the "workers.properties" file
RESULT: Connectivity
So in my view it does not make any difference to add the additional attribute "address=127.0.0.1" (or not) to the connector in server.xml. It seems to be mandantory (at least in my case) to use "worker.cfusion.host=127.0.0.1" in the "workers.properties" file.
My original overall configuration is basically based on the descriptions from Adobe at: https://helpx.adobe.... Under point 4, "worker.cfusion.host=localhost" is also specified, but at least in my case this no longer works with "ColdFusion Update 14". I have to use IP 127.0.0.1 instead of localhost in the "workers.properties" file.
So first, I agree something seems amiss. I don't think it's a Linux (or Unix thing), as some with such curious challenges have wondered. Time will tell.
You show the ping reporting ::1. That SHOULD be accepted by the updated ajp connector, by default.
But since you're scrambling to find anything that works, how about adding address="::1" on the ajp connector line in server.xml? You don't show trying that among your many attempts.
As I explain in my post, I recommend against changing the workers.properties file for such workarounds if possible, because the next connector install would set it to localhost by default. And while connector upgrades don't typically change that, one could.
Let us know what you find. Also, did you confirm 0 errors in the update log?
>> ...but since I suspect it really is his issue, I added the new section ... Handling if you have changed the IP address that localhost resolves to...
I'm not a linux erxpert, but I think in my case "localhost" resolves to "127.0.0.1" and "::1". This is the server config I'm using since years. This has never been changed.
Okay, I will also try "::1" and give a report to you.
Either way, I don't quite get what you were getting at. But the real key is to here how things go, per your (and my) last comment. :)
Even so, I will say I don't think your challenge is a Linux thing, either, if anyone reading may wonder (as some have been questioning, about possible OS differences).
Our posts unfortunately overlap.
I've tried to use "::1" for the additional address attribute in the connector and it works. Great.
Working examples (in my case):
- Additional attribute address="::1" in the connector in the "server.xml" file
- Using "worker.cfusion.host=localhost" in the "workers.properties" file
RESULT: Connectivity
CONCLUSION: I will use this in this form for all server instances in the future.
I think that normally the connector should resolve "localhost" to "127.0.0.1" and/or "::1" as standard. This is what the connector did bevor the U14 update. I can't see the reason why it is mandantory (in my case) to add the additional "address" attribut in the connector after updating to CF-2016 U14. This is a wired behavior (in my view).
Thank you for your tips and your support.
I just realized, that my two machines ("development" an "production" server) deliver different results for "ping localhost".
Output for development server "ping localhost" at the console:
PING localhost(server2.xyz.de (::1)) 56 data bytes
64 bytes from server2.xyz.de (::1): icmp_seq=1 ttl=64 time=0.021 ms
Output for production server "ping localhost" at the console:
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.023 ms
I've contacted my hoster to check, why the outputs are different.
The specific wording (on the Tomcat doc page I link to above) says, "By default, the connector will listen on the loopback address." It's looking like that may be only 127.0.0.1, which I find shocking. I almost think something ELSE must explain things.
But if you want to try it, you can use multiple addresses (as I do also indicate above), so address="127.0.0.1,::1". That would cover you either way.
Again, I agree: it should NOT be needed. But until we (all) figure out what's amiss here, it may be the best recommendation, and I could update my post to reflect it. That said, I already added a new section above (as mentioned in a previous comment) before I heard back from you, so I would like to do a little more testing of confirmation before updating the post.
Still, I'll let this comment stand for now, for those with the gumption to read down to here among these now 7000+ words. :-)
Good suggestion, I can try it.
Do you think that address="127.0.0.1,::1,localhost" should also work?
Neither address="127.0.0.1,::1" works, nor address="127.0.0.1,::1,localhost" works.
In my case it's only working with: address="::1"
Best regards
Knut
Thank you very much for this post and the much needed extra explanation about what was happening. One slight edit if I may suggest, the Adobe kb article says that that allowedRequestAttributesPattern is suppose to be ".*" and not "*". It took me a while before I realized I had gotten it wrong.
Knut: that's odd. I had done it (adding two), and it did work (as I said in the post, I even reversed the order and put one that would fail first, and it still worked). I wonder if there can be some other explanation.
But glad to see that at least just ::1 worked. I have been doing some work that might suggest that Tomcat is ONLY by default accepting from 127.0.0.1, and not ALSO ::1 as one would expect (and I'd said). The odd thing again is that I find that to be an issue on some machines but not others. Will share more as I learn more.
Prior to seeing this post, my production server failed after I installed update 8 (Win Server2018, Apache 2.4). I tried updating the web connector, but no joy. I panicked and was able to uninstall the update and got it running again at update 7.
Tried the update again on my development box (Win10, Apache 2.4) with worse results. I tried the suggested fixes in this post, but couldn't get the CF service to restart. Gave up and reinstalled the whole enchilada. Ran update 4 and then update 8 before even setting up the web connector. Same problem: service wouldn't start and wouldn't start after trying to change server.xml AJP settings. I'm wondering if there is something specific to the combination of Windows and Apache that is causing me grief. Any suggestions appreciated.
PLEASE read that. You will get cf running again.
Or I can help you directly, likely in as little as 15 minutes. You won't pay for time you don't find valuable.
I will be doing more research and documentation on the matter (why it's needed, when, and better alternatives to opening things up that way).
https://community.ad...
So I'm glad you mentioned it here, in case it may help someone. Thanks, and as always thanks for your efforts on the alternative boncode connector.
Unfortunately I just found your post/blog after finding the solution myself after working on it the entire day.
I worked on my development desktop running Windows 10, Apache 2.4.41 (Win64) and Coldfusion 2016 which had hotfix 13 before I updated it to hotfix 15 (I had skipped 14).
My finding is that the connector definition (without address=) in server.xml resolves to 127.0.0.1 while the localhost in worker.cfusion.host=localhost in workers.properties resolves to ::1.
Putting "address=::1" in the connector definition in server.xml alone solved the problem. Mine was 503.
You could also use any of these combinations:
connector with address=::1 and workers with localhost (as mentioned above)
connector with no address and workers with 127.0.0.1
connector with address=::1 and workers with ::1
connector with address=127.0.0.1 and workers with 127.0.0.1
Thank you anyway. I was a pleasure and reassured me of my assessment reading your entire post.
There's a lot to the post, of course. That's also why I made the more brief post on the Adobe site, as I noted at the top. I have plans for another resource, especially to help folks installing cf and then doing updates.
I red this page and doing as following... but doesn't work website that shown 500 error. not 503, 403 error.
Could you advice me please.
---
Step1: Amazon ColdFusion (2018 release - Ubuntu) : Made instance : 64bitAMI.
Installed CF success : ColdFusion:Y , API Manager:N , PMT:N, Servlet(RDS,JS Debug etc) None.
Created new file index.cfm on {CF_HOME}/cfusion/wwwroot/. then brows it and shown it's content.
Step2: ColdFusion 2018 Update 4 : success on CF Administrator on chrome browser.
Step3: ColdFusion 2018 Update 5 : success on CF Administrator on chrome browser.
Step4: ColdFusion 2018 Update 6 : success on CF Administrator on chrome browser.
Step5: ColdFusion 2018 Update 8 : messaged in popup "Install Status: Server taking too long to restart. You will have to manually restart server and check update status".
Reload web page CFIDE/administrator/ was 500 error.
Step6: Login on SSH and check two files.
server.xml : additional `secret="D978D4AB-AE67-4F2D-A129-6E95075FFF81"` in AJP connector line.
workers.properties : nothing `worker.cfusion.secret`.
Step7: doing conname `./wsconfig -upgrade -v`
Step8: check workers.properties file.
additional 1 line `worker.cfusion.secret=D978D4AB-AE67-4F2D-A129-6E95075FFF81`.
Step9: restart cfusion and apache2
stop cfusion , stop apache2, start apache2, start cfusion
Step10: brows CFIDE/administrator
shown 500 error.
And index.cfm was500 error, too.
Step11: Changed workres.properties file
Changed from `worker.cfusion.host=localhost` to
`worker.cfusion.host=127.0.0.1`.
Step12: restart cfusion and apache2 then brows CFIDE/administrator
shown 500 error.
And index.cfm was500 error, too.
----------
I have try and try many type setting for a several days but does not work cfusion...
best regards
In your case, you said in step 10 that you applied update 8 and then could not access the CF admin. I assume you are using a port for that, like 8500 (the default). That then is NOT using the ajp web server connector (and the changes I regarding it which I discuss in this post).
As the other post explains, your next step is to a) confirm if there were errors in the update log, then if so b) re-apply the update, whether in the admin or at the command line. Details in that post.
Let us know how it goes. Or I can help you via a screen share, in a consulting session of likely less than an hour. You don't have to pay if I don't solve it.
Is the first link "Having problems after applying a CF update? What to check, and how to recover!", this link URL same this post(.../2020/3/20/...). It missed link page !? the page URL is .../2019/03/.., right !?
I respond just info, I'm going to read the page you offered and check my environment.
Best Regards
And yes, let's hear how it goes for you.
I checked and tried, but not work yet. same status before.
How can I check other points, could you advise me please.
I done as below...
1. I check "install log" file of Update1, 4, 5, 6 and Update8. All Update was Success, ZERO error/fatal error.
2. Re-Update Update8 by command line, shown "This update is already installed."
3. Stop CF, stop Apaceh2, start CF, start apache and brows page as
"http://{IP address}/index.cfm" => shown 500 error.
"http://{IP address}/CFIDE/administrator/" => 404 Not Found.
for example,
"http://{IP address}/index.html" => shown "Apache2 Ubuntu Default Page".
"http://{IP address}:8500/CFIDE/administrator/" => 500 error.
hummm...
---
BTW, you fixed LINK ^^, but the link page was WHITE page and your other blogs too...(I kept the page by capture ^^).
So now the key is that you are confirming that you get an error in the cf admin and you are CONFIRMING that you were accessing it via port 8500. That tells us again that whatever your problem is (with that), it's not about the ajp connector.
But can you confirm as well that you WERE using thst port 8500 to access the admin before the update?
If not, were you accessing the admin without a port? That would not be allowed by default in cf 2018 (or 2016). You'd have to tweak the connector properties to allow that. Did you?
If so, can you confirm that cf is even listening on port 8500, for its built in web server/biws? It may be some other port.
Bottom line: if cf is running and the biws is working (and the admin accessible), then your other issues COULD still be about the ajp connector. Did you do all the things listed at the top? Upgrade the connector, add the address attribute, if needed? Add the allowedRequestAtrributesPatten, if needed? See above for more.
If you may write back to ask continued questions, please answer every question I ask in this comment. You can also email me at Charlie (at) carehart (dot) org, which may be easier/better than extended discussion here.
Finally, I will say again that if you just want to get the problem solved without all this back and forth we could solve it in a quick consulting session.
What's especially sad is that it seems to be limited to pages of mine. I can think of no explanation for that. They have said someone will work on it next week. I am trying to remain calm about it, but it is indeed disconcerting.
I email you. Could you see it and advice me please...
Yes, I access CF Administrator with post :8500. not use Buildin server.
I tyied connector option "address", "allowedRequestAtrributesPatten".
thank you so much !
I ran gethttprequestdata().headers like you mentioned and got 16 different values back. In addition to Xsite-name I got stuff like "accept" "connection" "host"... Could you clue me in how to specify them in the allowedRequestAttributesPattern attribute? "Xsite\-name|accept|connection|host..."
Again, thank you for your support!
macOS mojave v. 10.14.6
apache httpd v. 2.4.34 (configured for https)
mod_jk.so v. 1.2.46
apache tomcat v. 9.0.34
$ ping localhost
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.036 ms
...
In server.xml (with 'address="127.0.0.1"' or 'address="127.0.0.1,::1"', httpd couldn't connect to tomcat at all:
<Connector port="8009"
protocol="AJP/1.3"
redirectPort="8443"
address="::1"
secret="somePassword"
allowedRequestAttributesPattern=".*" />
previously, in tomcat v. 8.0.21:
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />q
In localhost_access_log.2020-05-18.txt:
xx.xxx.xxx.143 - - [18/May/2020:13:00:43 -0700] "GET /mpstruc HTTP/1.1" 403 618
workers.properties:
workers.tomcat_home=/Library/Tomcat
ps=/
worker.list=ajp13
worker.ajp13.port=8009
worker.ajp13.host=localhost
worker.ajp13.type=ajp13
worker.ajp13.secret="somePassword"
/etc/hosts:
127.0.0.1 localhost
255.255.255.255 broadcasthost
::1 localhost
A distillation from httpd mod_jk_log:
init_ws_service::mod_jk.c (1179): Service protocol=HTTP/1.1 method=GET ssl=true host=(null) addr=76.21.65.212 name=ourwebsite.edu p
ort=443 auth=(null) user=(null) laddr=128.200.165.4 raddr=76.21.65.212 uaddr=76.21.65.212 uri=/mpstruc/newsfeed
ajp_get_endpoint::jk_ajp_common.c (3357): (ajp13) acquired connection pool slot=0 after 0 retries
ajp_marshal_into_msgb::jk_ajp_common.c (681): (ajp13) ajp marshaling done
ajp_service::jk_ajp_common.c (2588): processing ajp13 with 2 retries
ajp_send_request::jk_ajp_common.c (1719): (ajp13) no usable connection found, will create a new one.
jk_open_socket::jk_connect.c (674): socket TCP_NODELAY set to On
jk_open_socket::jk_connect.c (798): trying to connect socket 31 to ::1:8009
jk_open_socket::jk_connect.c (824): socket 31 [::1c1e:1f49:0:0:56240 -> ::f00:c43:b281:f1fd:8009] connected
ajp_connection_tcp_send_message::jk_ajp_common.c (1264): sending to ajp13 pos=4 len=380 max=8192
ajp_connection_tcp_send_message::jk_ajp_common.c (1264): 0000 12 34 01 78 02 02 00 08 48 54 54 50 2F 31 2E 31 - .4.x....HTTP/1.1
...
ajp_send_request::jk_ajp_common.c (1779): (ajp13) request body to send 0 - request body to resend 0
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): received from ajp13 pos=0 len=54 max=8192
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0000 04 01 93 00 03 34 30 33 00 00 03 A0 01 00 17 74 - .....403.......t
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0010 65 78 74 2F 68 74 6D 6C 3B 63 68 61 72 73 65 74 - ext/html;charset
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0020 3D 75 74 66 2D 38 00 A0 02 00 02 65 6E 00 A0 03 - =utf-8.....en...
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 0030 00 03 36 31 38 00 00 00 00 00 00 00 00 00 00 00 - ..618...........
ajp_unmarshal_response::jk_ajp_common.c (736): (ajp13) status = 403
ajp_unmarshal_response::jk_ajp_common.c (743): Number of headers is = 3
ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13) Header[0] [Content-Type] = [text/html;charset=utf-8]
ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13) Header[1] [Content-Language] = [en]
ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13) Header[2] [Content-Length] = [618]
ajp_connection_tcp_get_message::jk_ajp_common.c (1459): received from ajp13 pos=0 len=622 max=8192
One may ask, why does it matter? But if you have installed your own tomcat, and configure by hand your ajp files and how apache points to them, you have to be sure you properly updated the right files.
For instance, you state the version of your mod_jk.so, but you don't indicate where it exists. Are you sure apache is loading it? Have you checked its conf files, to confirm? And even if you think all's well with that, what if you rbamed the so file, or the conf file? Does httpd still start? It should not. That's a quick sanity check. I have seen people chasing the wrong problem by mistaken assumptions.
And this is key, because if the wrong so is running, or that secret for the workers.properties is wrong, you'd get a 403.
Same if somehow your server.xml changes were not to the right one. I'd tweak that in a way that should stop tomcat running. Btw, it could come up but not process pages. So be sure to check a page running via its web server (the http connector in server.xml), to make sure tomcat IS running, separate from requests via ajp/apache.
I realize this may sound like crazy ideas, but you're presenting that "all is well" with your config files as presented, and I concur. But clearly something's not right.
One other thing you might try: changing the localhost value in workers.properties to ::1, to match your server.xml. Of course, restart apache after that, or tomcat after changing server.xml. (Some folks don't realize that chbages to the former are loaded by apache rather than tomcat).)
Let us know how it goes, even if that still nothing works. In that case, I'd think it time to enable more logging or even network monitoring, to find what's really going on.
Yes, we are not running cold fusion, just tomcat behind apache httpd.
Apple did not provide the mod_jk.so module in the standard module location (/usr/libexec/apache2/) in macOS 10.14, and wouldn't allow me to add it there, so I put it in /etc/apache2/libexec (etc/apache2 is the apache home directory). httpd.conf includes it with (showing some context):
LoadModule hfs_apple_module libexec/apache2/mod_hfs_apple.so
LoadModule jk_module /etc/apache2/libexec/mod_jk.so
and the log /var/log/apache2/error_log.1589414400 indicates that httpd is finding it:
[Tue May 19 12:02:50.526249 2020] [mpm_prefork:notice] [pid 102] AH00163: Apache/2.4.34 (Unix) LibreSSL/2.5.5 mod_jk/1.2.46 configured -- resuming normal operations
Apache seems to be running fine, and is serving our static pages. Tomcat also seems to be running fine, and hearing requests (e.g., see the excerpt from the localhost_access_log in my previous post), but denying access to all requests.
I've tried changing workers.properties worker.ajp13.host=localhost to worker.ajp13.host=::1, no difference (and I always restart after such a config change).
I've set all the logging levels in tomcat/conf/logging.properties to ALL, and the logging for mod_jk to "JKLogLevel debug". While I'm getting a lot of information from the httpd side of things, the only thing tomcat wants to tell me is, again, that 403 response in the localhost_access_log. So I'm not sure how to enable more logging at this point. What kind of network monitoring would you suggest?
Nothing is crazy at this point - we've been pulling our hair out for a couple of weeks! Thanks for taking a look!
It does seem you've covered all your bases for the typical things that trip folks up. And yep, the mod_jk debugging logging does confirm that the request is getting from apache TO Tomcat.
Still, I have a couple other ideas.
First, does your password have special characters? Have you tried setting it to a simple string, just for the sake of testing? If that works, perhaps there was an issue in the passing of that.
Second, is it possible that the request you're making is not going through mod_jk but mod_proxy_ajp? In that case, the latter would not honor the secret set in the workers.properties, but would instead be set on the proxypass line, from what I have read (I have not tested this).
Third, you haven't indicated what Tomcat version you're using. And the mod_jk.so is not reporting to be the latest. While I realize you may have reason not to want to be "on the latest", it seems worth trying just to find out if updating both might resolve this, as there may be some bug.
In fact, there was one about incoming headers having a new line character that was tripping up Tomcat, since fixed. (Your debug output doesn't show that, but I have not investigated to find out how that looks in the log if that's the issue, and if it's even tracked there.)
Indeed, one could argue that a real challenge is that even the ajp debug logging doesn't tell us WHY the request gets a 403. That seems reasonable to expect, but the folks behind Tomcat are VERY careful and thorough and always have well-reasoned explanations for why they have done (or not done) something.
And to that very point, if none of the above works, I'd recommend you take the problem to the tomcat-users mailing list. That is THE best place to get resolution to challenges like this, as most questions are answered by the very people who wrote tomcat and the ajp connector.
And of course, they have been dealing with ramifications of these ghostcat changes for months now. So they may think of something readily that we have missed here. You can learn more here: http://tomcat.apache...
But before anyone would write their first note to that list, they should note the indications on that page of HOW questions should be asked. Even better, read a few threads in the list before making a first post to take the pulse of things (there's a link to the online archives--even already from today as I write).
If you look at threads where there have been a few or several replies, you'll get a good sense of how things go--and when they go well, or not. :-) Sometimes it can be a bit like watching a cross between the scenes when Dorothy's crew approached the Wizard of Oz, and the courtroom scenes in My Cousin Vinny. I'm not trying to scare anyone off. Again, they are whip smart as a group and great at solving knotty challenges. Just be prepared in how you present things.
If you do ask there, and especially if you get a resolution, I hope you will return here to let folks know (either the answer or a link to the online discussion).
And again, if you get an answer, please post it here for others to know. Thanks.
His problem is that NO CF requests run, even using CF's "built-in web server" (which is the Tomcat web server, rather than Apache talking to CF/Tomcat over AJP), after applying the latest (April) CF update. And for those interested, there are no errors in the CF update log.
I have now heard of one other person with the same problem, but only one other like it, so there's some combination of characteristics they share. I'm trying to figure that out with each.
And Adobe has gotten involved also (with the other client I'm working with). If we learn something, I will share it here for everyone to know. But to be clear, it seems unrelated to the topic of this post, which is about the changes regarding AJP processing per the March updates to CF 2018 and 2016.
Ultimately, once those upgrades were made, the final, unintuitive change that finally got things working was, in workers.properties, going from
worker.ajp13.secret="somePassword"
to
worker.ajp13.secret=somePassword
On the tomcat side, in server.xml, the double quotes are needed, as in
secret="somePassword"
I read a message from Craig (Thanks Craig san). I tried that mod_jk.so compiling from source code v.1.2.48.
but AWS CF Administrator (:8500) do not work yet.
I'm continuing read this web site.
Thanks all
I had in fact expressed my concern about that in my first comment in the Adobe blog post on the the day it was announced (https://coldfusion.a...), where my opening sentence was, "Wow, I suspect this update is going to start a lot of trouble for a lot of people."
In the weeks since I and others (including Tomcat folks) have tried to share more and more info to help folks, but there's no doubting that anyone updating from that point forward (for the first time, such as in your going to CF2018 update 9) will hit this, so we will see people struggle with this for years...or until Adobe simply creates a new installer that embeds the latest updates to CF and the connector, and perhaps reconsiders tweaking the wsconfig tool to help people get the RIGHT value for that address attribute, or whether to enable that allowedrequestattributespattern override. Different strokes for different folks.
But to be clear, it's a different problem than this blog post's focus. So keep in touch with me directly rather than here--and anyone interested in the topic can reach out to me also (see the contact info offered here on my site). Perhaps I will create a new blog post about it, especially if a specific solution is identified.
- System
- Provider
[ Name] Application Error
- EventID 1000
[ Qualifiers] 0
Level 2
Task 100
Keywords 0x80000000000000
- TimeCreated
[ SystemTime] 2020-09-22T01:30:20.000000000Z
EventRecordID 235026
Channel Application
Computer WIN-
Security
- EventData
coldfusion.exe
2016.0.0.0
5b92a2b1
coldfusion.exe
2016.0.0.0
5b92a2b1
80000003
00000000000341c2
ea0
01d6907fecf895a4
C:\ColdFusion2016\cfusion\bin\coldfusion.exe
C:\ColdFusion2016\cfusion\bin\coldfusion.exe
2d04d3f3-fc73-11ea-9db5-d4ae52b9744e
I went round and round trying to get the coldfusion.exe to start, so i uninstalled and fortunately the uninstall worked and was able to start coldfusion and get the website up and running again.
nowhere does this issue get addressed.
I discuss these in another post:
https://coldfusion.a...
Neither the cause nor the solution is indicated in that Windows Event Log. The fact that you could uninstall the update is an indication how you can simply try again, after considering what I offer in that post.
If somehow you still struggle, I can offer direct remote assistance. I'm sure we could resolve it, and likely have cf updated for you in less than 15 mins. If interested, check out carehart.org/consulting.
Just to say thanks for these posts! I have a fresh install of CF2021 in VirtualBox Ubuntu Server 20.04 LTS with Apache mod_rewrite. The rewrites to .cfm files did not work at all giving Tomcat's 403.
After banging my head and going all over the net for hours searching for fix it turned out to be as simple as adding that allowedRequestAttributesPattern=". *" to the server.xml file.
You saved my day!
BR,
Aarni
On the requiredSecret attribute, after installing the CF2018 update 12 I had two servers returning 403 errors for cfm files, I tried adding allowedRequestAttributesPattern=".*" and an IP address to the AJP connector line as well as wsconfig upgrades. It wasn't until I removed the requiredSecret attribute that they started working.
The fact that you had to remove it would have me worried that either of those was not successfully updated. I discuss above how to check if a cf update is successful (it's not enough to confirm merely that it has been installed). For more, see my post that I point to in that above section:
https://coldfusion.a...
As for checking that the wsconfig is updated, I discuss that in the blog post I point to on updating the connector itself:
https://www.carehart... You'd want to see the isapi_redirect.dll (for iis) or mod_jk.so (for Apache) had a date of just before the date of the cf update you'd applied (assuming that update did call for an update to the connector, as most do). In your case, having updated to cf2018 u12 (from this year), the connector dll or so file should be from 2021. Is it?
If you want any help double-checking all that, as I note above we could do that on a consulting basis in perhaps as little as 15 mins. More at carehart.org/consulting.
Or let us know if you solve things yourself.
I've installed the update on four Linux/Apache servers, all update logs are without warnings or errors, and all now have the same August 2021 date on mod_jk.so.
The two other servers didn't have the requiredSecret attribute and had no 403 errors, I've now tested adding requiredSecret="test" to those and both went to 403 with it added.
I would definitely call on your services if there was still a problem, but removing the redundant requiredSecret seems to have been the fix for the servers that had it.
But I realize you feel you're all set, so I share this as much for future readers. If you, I, or anyone else ever learns something more about this, perhaps it will be added here.
Glad you're back in business. :-)
But you're set. If things change or we learn more, we can add it here.
Also, we need to k ow what exactly is causing any recent change in behavior regarding all this? The tomcat team (and something cf simply includes)? Or the Adobe cf team, who do modify the tomcat ajp web server connector (and implement it with the cf wsconfig tool)?
And finally, does it only happen in some situations? Some previous cf update vs another? Some previous cf version vs another? Some previous wsconfig version vs another?
Again, let's see what we learn over time.
Charlie, I believe the requiredSecrets were there prior to update 12 on the servers I had the problem with, which had updates 4 and 11 installed when they were set up earlier this year.
Until recently, it seemed the "older" one was was ignored, but now its presence is causing 403 errors for some. Why?
First, some folks are saying they did not PUT that the requiredSecret there, and that as far as they know it's been there a while (not causing trouble, even after the ghostcat updates to the connector and CF discussed in this post starting in March 2020).
It seems THAT perhaps it's presence is a leftover from their having run the Adobe CF Lockdown tool, which had been introduced in CF2018. It seems THAT put this "old" requiredSecret in place (then), for the sake of connector security.
Again, the ghostcat changes led Tomcat to create a NEW "secret" attribute instead (and a related secretRequired boolean, whose name was too close to the old attribute, in my opinion).
Anyway, it seems the presence of BOTH the "secret" and "requiredSecret" was being ignored until recently. Perhaps there was a change in Tomcat in the Adobe CF connector.
So for now, those getting a 403 that can't be solved by the other means above (such as the address attribute) should check to see simply if they have both a secret AND requiredSecret on the ajp connector line in server.xml. If so, remove the latter (maybe copy it off to a comment line), and restart CF, and see if your 403 goes away.
Feel free to share here whether it does or does not work for you, even months from now as I write.
I updated from 2018,0,06,316308 to 2018,0,13,329786, and ColdFusion restarted without errors.
According to the Update 13 instructions, I need to follow the post-installation steps for Update 8.
This is a locked-down installation that already had the requiredSecret attribute, and the web server and ColdFusion are on the same machine. In this scenario, the instructions say: "If you have already locked down ColdFusion, then you need not take any action, since ColdFusion instances are already configured with the requiredSecret attribute and your web server also has the secret information with it to communicate."
Should I upgrade the connectors anyway?
Thanks for any info!
JG
Then you need to remove the old "requiredSecret" in the server.xml, due to the Tomcat change in a recent cf update.
Upgrading the connectors caused 403 errors, but removing the requiredSecret in server.xml fixed that.
Happy holidays!
Is there a way of adding a secret value to a manually created connector that is connecting to IIS.
I have installed CF2018, completed all the updates, and then added a manual configured connector for iis following the adobe documentation, as per my software vendor advice. I am just getting 500 errors and I feel it must be the connector that is playing up.
Any advice would be greatly appreciated
Laurie
https://coldfusion.a...