Part 3: Adobe hotfix released for "Serious security threat for ColdFusion servers"

@YASHAS Thanks so much for the update. I also saw the comment from Shilpi on the CF Team blog, and I have now updated the blog entry here to remove my warning about this issue for 9.0.2, as well as have sent out a tweet retracting my warning there, and of course I have left a note of thanks for her clarification on that CF team blog entry.

Was all in the spirit of making sure things were clear and correct, given how important this fix was. Glad to see it so. :-)

@Garry, sorry for the delay in responding. I can't always stay up on the blog comments every day, because it's just one of many ways that I help people throughout the day.

There can be any number of possible explanations for the problems you are seeing, both with respect to the prod box and the tabs not appearing in the CF10 updates page, and with the dev box and the command-line JAR not working.

I'll give it a shot, especially in case they may help others who may find this some day, even if they may not necessarily all apply to you (but I think the key points will).

1) First, for the failure to see the tabs on the prod box, I suspect it's that there's a problem with the browser not getting a response for the calls made within the page to show those tabs. The code calls into the /CFIDE/scripts directory.

So before going any further, have you done anything to block the entire /CFIDE directory? Many people over-reacted to this recent attack and did just that. I warned several times in my entries here not to do that, but of course many are not reading these, or missed it if they did read it.

You don't want to block /CFIDE/scripts because that's used by many things (CFML tags like CFFORM and all the Ajax tags/functions), and it's used by this CF10 updates page, because it uses tabs, as created by the CFLAYOUT type="tab" tag (at least, that's what I discern from looking at the web server request logs when this page is called).

So as a simple test, on your server where you "don't see the tabs on the updates page", what happens if you browse:

/CFIDE/scripts/ajax/resources/ext/images/default/tabs/xd-tab-strip-bg.gif

By this I mean, once you have your CF Admin on screen, modify the URL to remove the /CFIDE/administrator/index.cfm and replace it with that line above. Does it work, returning an image? Or do you get some error? It could be, again, that someone has inappropriately blocked all access to the /CFIDE/, thus blocking the /CFIDE/scripts folder.

Or maybe there's something that has caused this or other files deep in that folder structure to not be there.

Note also that someone on your CF Admin may have changed the scriptsrc location, so that CF is now building the HTML for that page to use a different path for these files. So if you do a view source on the frame (in that Admin page), find a line near the top of the page in a line that should look like this:

script type="text/javascript" src="/CFIDE/scripts/ajax/package/cfajax.js"

The starting folder for path listed in the SRC attribute may vary for you. Anyway, whatever value is in that src attribute, what happens when you browse that, as above? In this case, for this cfajax.js file, you should be shown or prompted to download a javascript file. If not, if you get some error, then again it says that's there's something wrong with either your web server or CF configuration.

One other related tip: it can be helpful (whenever any web page's interface if not working as expected) to use a tool to help you identify if there are any errors on the page (such as missing files for images, javascript, css, etc.) For identifying such problems, it's best to use one of many developer tools, either available within the browser or as an add-on. See a blog entry I did on that last year, http://www.carehart....

If that is not enough to help you, or if it's not that, you can write back but I can't promise to respond here today (and would not respond tomorrow, Sunday), but if you contact me for support (www.carehart.org/consulting) then I should be able to respond today. More on that in a moment.

2) Second, as for your problem when you tried to run the update jar from the command line, when you say "system error log event", I'll assume you mean an entry in the Windows Event Viewer, specifically in the system log. That often happens when people try to change their JVM (the java.home line in their jvm.config or the corresponding "java virtual machine path" field in the CF Admin "java & jvm" page. Did you by any chance make that change before restarting the instance after you'd applied the hotfix?

Or if that's not it, rather than focus on the Windows event log, look instead at the coldfusion-out.log which you'll find in the [cf10]\[instance]\logs directory (if you only have one instance, then it's [cf10]\cfusion\logs). That log holds the equivalent of the CF "console log" information, since there is no "console" because you are starting CF as a service (which we can assume because of the error you report happening in the Windows Event Viewer).

The coldfusion-out.log should have information reflecting more details on what's happening during the startup of CF, which should explain why it failed to start.

One other problem could be a permissions issue. For instance, perhaps you or someone there changed the account under which CF is being started in the Windows service (using the "Log on" tab in the Service's properties). Well, if you run the update (the jar) from the command line as a user that differs, or does not share the same privileges, that could cause a problem.

For folks who leave the service running as "System" (the local system account), they can get into trouble running the jar from the command line if they don't start that command prompt with administrating privileges (even if they are a user in the Windows "administrator" group).

So you may want to try it again, by launching the command-prompt using a right-click and choosing "run as administrator". Then run the jar as you did before.

Finally, if none of that helps, then with respect to the application of the update, note that there is a log created in the [cf10]\[instance]\hf-updates\ directory, within a folder named for each update applied.

3) I do hope some of that may help. Sorry it wasn't a quick answer, but as I said at the outset, you could have any number of problems. And some people reading this may have ones that you don't, but they may find this some day in searching for a solution.

<CFSoapBox>

Of course, some people reading this will sniff that I can't seem to say anything in less than a paragraph! They may even assert that I make troubleshooting seem "complicated".

I would counter (since this is all I do all day each day, helping people solve CF troubleshooting problems) that I've simply learned that few problems are simple.

As we've seen above, there can be many interrelationships of things, even in one seeming "problem and its solution". There are also different ways that you or someone else on your server may have changed things which could be causing your problem.

Rather than just say "try this", I share the details to try to help you solve the problem, diagnostically, which is what I do when I help people in a consulting session.

It's like the difference between giving someone turn-by-turn directions, and giving them a map. With the former, if you miss one turn, you might go a long time before realizing it and get horribly lost.

I see that happen a lot with people when they try to solve problems with twitter-length answers or one-paragraph blog entries, forum messages, or mailing list messages that they may find, which "seem related to their problem".

So whether I'm offering possible solutions to someone's problem in a blog comment (or forum thread, or list message) or in a consulting session, I tend to offer the additional detail because my experience tells me that most people would benefit. For those who don't, well, I guess the help is not for you.

</CFSoapBox>

Anyway, let me conclude by saying that if these challenges are important and time-sensitive for you, I'd recommend that it may be more effective for us to work together looking at your machine online, and talking on the phone. You can learn more about my troubleshooting services at www.carehart.org/consulting/. I may or may not get back to any reply you offer today (and won't see any tomorrow, Sunday.)

Again, hope some of this helps. I'll look forward to feedback from you or others on any of that (the technical discussion or the soapbox), as I may consider turning either or both into a blog post of their own. :-)

Charlie,

Thank you for your incredibly detailed input!

I have to confess that I fell into the "knee jerk" category, and had placed a request filtering rule on the /scripts directory, mainly because our site doesn't use any of the CF Ajax functions. Once I'd removed that everything was back to normal.

Thanks again.

Mike, I can think of at least one way, yes. And I discussed it in the Part 2 entry in this series (linked to above). See specifically the section there, "Caution: Don't be lulled into a false sense of security/confidence!".

Also, you say you "got hacked", but you don't give enough info for us to know that you were hacked by the same exploit described here (and addressed in that hotfix by Adobe).

Then too, you refer only to "the patest patch". It's not clear if you also undertook the steps to modify your web server configuration for the vital CFIDE/administrator, CFIDE/adminapi, and CFIDE/componentutils folders that were key in this hack (and could lead to still others). Even if you did, see my first point in the comment here about how one even doing that may not do all they need to.

Finally, beware also that in "applying the patch" you may well have simply made a mistake, and therefore not have been protected in the way you assume. I have seen that happen very often, and in fact I blogged substantially on it a couple of years ago. You should check that out as well (and don't let the title fool you, if you don't feel that your CF or admin is necessarily "busted"):

http://www.carehart....

And if you feel like there's just too much in all these entries for you to track, note then also the section above (in this Part 3 entry), "Can't I just get someone to help take care of this stuff for me?" :-)

Hope that helps.

PS To others who have commented in recent weeks, again I've been overwhelmed. I will get to them. I just had a moment to reply to Mike here, with what seemed especially important information to re-assert.

They also found a file called wtf.exe. I guess what I am curious about is if they potentially got in a different way and not by this way at all?

Well, I was compromised around Christmas, and discovered it in early Jan, and it's been a battle ever since. I locked everything down thanks to Charlie's blog posts in Feb, but still had files changed. Turns out there were multiple files uploaded all over the site. They were smart - they were dropping them in folders that hadn't been modified in years, and they were putting old modification dates on things so I couldn't just look at a folder and know something new was in it.

After reading Mike's posts I realized I probably had malware on there as well. Yup, a whole bunch of it. I also discovered Kaspersky Security Center 9 should definitely NOT be installed on a machine running ColdFusion (it totally borked CF and took hours to fix). Managed to get the malware removed (I hope), and locked the machine down further.

We now have CF running under its own account, and it's limited to the web directory. Also, we've limited the folders and files it has write access to, so even if they gain access in the future, they shouldn't be able to upload or modify anything.

Such a frustrating time.

Well, my tale of woe isn't over yet. Got hit again last night, even after the changes I made the other day. Turns out they created an account that had admin permissions, so they still had full run of the server.

I was also hit by at least two groups since Christmas, as I found some very different templates that did the same things. Some were encrypted templates, some weren't. What a mess.

A client we don't host, but do develop for and help out when she needs it ran into this just yesterday on her CF9 server that hadn't been updated. Luckily, it's a server with only one website, and it appears that the hackers really only wanted access in order to send emails out from the server (which is not great, but at least they didn't damage the server or the site!). Also luckily, they were stupid enough to use an existing run-once-a-minute scheduled task instead of creating a new one, and they also left it changed instead of changing it back once the script was on the server - the client noticed when that task wasn't working.

Thank you so much for the detailed information that helped me figure out exactly what happened and what they did and how to keep it from happening again!

Was there ever a part 4 to this?