[Looking for Charlie's main web site?]

I'll be speaking on the CFMeetup this week, doing a CF911 talk

Just a heads up: I'll be speaking on the CFMeetup this week, presenting a new talk, "CF911: ColdFusion Tools for When the Stuff Hits the Fan".

While you can find the details (and optional RSVP info) on the meetup event page, I'll repeat the description to save you having to go there:

If you run a production CF server, you've likely hit a situation where the server seems to hang up. Do you know why it's happened? How to troubleshoot it? It may not be CF's fault at all, though it tends to get the blame. In this presentation, I'll point out a few tools (some free, some commercial, some built-in and some third-party) which you ought to know about when the stuff hits the fan. With the right tools and just a little understanding, you can be a CF troubleshooting superhero.

I'll be presenting in the noon (US Eastern Time) slot, this coming Thursday, Feb 18.

Of course, some of the concepts will apply as well to those using Railo, BlueDragon, etc. This is an expansion of an article of the same name that I did in the latest issue of the FusionAuthority Quarterly Update. (BTW, the magazine has converted to an online-only (though still subscription-based) format.)

Happy to help

This is part of my ongoing focus in recent years on CF server troubleshooting, to go along with the various articles, presentations, blog entries, tweets, and many mailing lists I contribute to.

I'll add also that if you ever find yourself in need of assistance, and have exhausted your review of the many resources on my site and elsewhere (or don't have time), I am available for direct, remote, short-term assistance. More at carehart.org/consulting/. I love helping solve problems and can generally do it very quickly.

That said, the talk is definitely NOT a sales pitch for my services. Like most of my site, the information in the talk is provided freely to those who like to "do it themselves".

Spying on ORM database interactions: Hibernate, Transfer, etc. on any CFML engine

As people use CF9's ORM feature (or other ORMs like Transfer and Reactor, or indeed Hibernate, on any CFML engine), they may be left wondering what sort of SQL interactions happen "under the covers" between the ORM framework and the database engine. Well, there are several ways you can watch them, as this entry will discuss, and it may be interesting to discover what's going on.

Of course, ORM support is just a different way that the CFML engine (through the ORM framework) sends SQL to a database via a regular DSN, just like any other request, so there's nothing really "tricky" about this. It's just about realizing that while you don't write the SQL yourself, it's still generated by the CFML engine/ORM framework, and you may not realize/consider the available tools which can spy on it, just like any other DB processing from within CF. Indeed, some people may not even realize how many options exist to spy on JDBC interactions from their CFML engine to the database engine.

The good news is that there are several approaches, some included in CF (some depending on the edition), and some available separately which would work in any edition of CF or the other CFML engines (Open BlueDragon, Railo, etc.), and with any of the ORM frameworks.

(FWIW, besides the aforementioned Transfer and Reactor, there are still others, which I mention in my CF411 list as CFML ORM Frameworks. Indeed, note that you can run Hibernate on CF prior to CF9.)

Built-in ORM Logging Option

Update: After posting this entry, I learned that there's yet another option, so I'm adding it here as the first to consider. There is a built-in option in the CF ORM setup where one can enable logging, settable in the application.cfc: see the this.ormsettings option and its available key/value pair, logSQL="true".

There are several resources where you can learn more on that. Besides the CF9 docs page on the ORM settings, there is also a blog entry by Adobe engineer Rupesh Kumar. I've not yet used it, so don't know how it compares to the other options below. I will leave them for all to consider.

Using FusionReactor or SeeFusion

Users of any CF edition (6+) or CFML engine can use tools like SeeFusion and FusionReactor, which have always had the ability to monitor database interactions by "wrapping" the datasource to be monitored. FusionReactor engineer John Hawksley has posted a recent article specifically on monitoring CF9's ORM interaction, in the FR Devnet site, Using FusionReactor's JDBC Driver Wrapper With ColdFusion 9 ORM. Its concepts would apply to any ORM, of course.

Similarly, I've written generically about its database monitoring feature in What is the FusionReactor datasource monitoring feature? Why would I use it? Powerful stuff. As I point out in that article, the concepts discussed apply as well to SeeFusion's ability to monitor queries by wrapping datasources.

CF Enterprise, with a caveat

I also mention in that article that those running CF 8 (or now 9) Enterprise will find that it has built-in wrapping of datasources, such that its Enterprise Server Monitor reports database interactivity, so that can also monitor the ORM interactions.

A key difference for FR and SeeFusion over that, though, is that while they all have means to highlight queries that exceed certain limits, only FR and SeeFusion have the built-in ability to log every DB interaction (FR to text logs and SF to a database, and they both do so with low overhead). The CF Enterprise Server Monitor has no logging ability at all, and while you can set its interface to lower the threshold for what active queries to show, it has no means to show queries which have executed in the past.

It's just that it can be useful sometimes to see every single DB interaction for a request (or all requests), and it may be all the more interesting for discovering/observing what's happening with ORM interactivity.

Another alternative CF feature

Still another little-known feature for spying on JDBC interactions in CF is by way of the JDBC "spy" feature, which does in fact allow logging of all JDBC interactions mde from within CF. This feature was first enabled by way of the DataDirect 3.5 driver update which was made available (as an optional upgrade for 6 and 7) in the CF 7.02 timeframe. I wrote about the Spy feature back back in Aug 2006.

Since then, CF 8 (and now 9) offer it instead as a new "log activity" option in the "advanced settings" for a datasource definition in the CF Admin (which is disabled by default). I pointed this out in another entry from 2007 as one of many easily missed changes for the CF 8 Admin.

This "log activity" output is not as easy to interpret as FusionReactor's logs, and can indeed be voluminous (moreso than FR's), so be careful. Anyway, it's one of the several ways you can monitor JDBC interactions between CFML and your DB engine. Again, any of these may be useful for monitoring any of your CFML/database interactions.

Generic DB Monitoring tools

Indeed, it's worth noting finally that while the focus here has been watching the DB interaction from CF (and the ORM framework) to the database (by watching the JDBC traffic going out of CF and returning), you could just as well watch the DB interactivity from the DB's perspective instead (watching it coming and and being returned).

There are many tools that can monitor database processing, available for each of the major databases (free and commercial). I list several such tools in one of my CF411 section, Database/SQL Monitoring Tools.

Hope all that's helpful, whether you use ORM or not.

You may have mistakenly applied an 8.0 CHF on a 8.0.1 CF server, and not realize it!

I just helped a customer today solve a problem where he swore he had applied the latest Cumulative Hotfix (CHF) for CF 8.0.1, but I showed him that instead he had mistakenly applied the CHF for 8.0. I know how it happened, and showed him. I hope how you can avoid the same mistake.

Update: David Collie of Adobe has informed us (via comments below) that he has now fixed the two pages with the problems reported here. We thank him so much! Even so, since some may have suffered the problem of getting the wrong hitfixes while the problem was in place, I'll leave this entry as is to help them.

The problem: you may have applied CHFs for 8.0 by mistake

When checking to confirm if an 8.0.1 server has applied the CF 8.0.1 cumulative hotfixes, make sure that the file in the [cf_root]/lib/updates directory has a jar file name with "801" in its name, like chf8010003.jar. The problem is that it could easily, mistakenly be an 8.0 CHF instead, such as chf8000003.jar (note the "800" reference).

How could that happen? More in a moment. But the problem is that CF wouldn't complain about this (while applying the fix in the CF Admin or if you just dropped it into the lib/updates and restarted CF.) It would just be ignored, and the CHF would not do anything--and you would NOT get the updates you expected in the 8.0.1 CHF.

The root cause of the mistake

So how does this happen? I know at least one source of the confusion. I've complained over and over about this: there's an updates page at the Adobe site which starts out offering a link to the 8.0.1 update (itself). There are no CHFs for it. Rather, the problem is that right after its shown, the page lists the 8.0.0 CHFs! See the screenshot below.

The problem is simply that no one has gone back and updates this page to put the links to the 8.0.1 CHFs. They're just not there.

So many people may easily miss that those are in fact the CHFs for 8.0, not 8.0.1. Worse, since there 3 of them (just as some may know there are 3 for 8.0.1, currently), it's even easier for someone to make this mistake and think they're getting the 8.0.1 CHFs.

Again, since applying them gives no error, I would bet that many are in this situation and don't even realize it (and are not getting the benefits of the hotfixes.) So follow those instructions above and check to ensure that you really do have a jar with 801 in the name (in either [ColdFusion8]\lib\updates in Server mode, or in [JRun4]\servers\[instancename]\cfusion-ear\cfusion-war\WEB-INF\cfusion\lib\updates in Multiserver mode.)

Is the customer to blame for being careless?

Some may sneer that those who make this mistake "get what they deserve", and that it's about "thinning the herd", if people are "so careless". Sorry, I don't agree.

This is a very easy mistake to make, especially for folks who by and large don't spend their days doing this sort of stuff, who might easily assume that what followed the 8.0.1 update on that page was its hotfixes.

And since this "updates" page I pointed to is what's offered on the right of the CF product page, it's just all the more easy to be found--and unfortunate that this problem has been allowed to linger.

So where does one find the 8.0.1 CHFs? and CFH 3?

I wish I could point you to one page that lists links all the the 8.0.1 CHFs, but sadly, that's still another point of annoyance. Some will know that there is this page, ColdFusion Hot Fixes (ColdFusion 8 and later).

But guess what, even IT does not have the CHF 3 listed! :-(

Please, Adobe, fix these things.

You can find CHF 3 for CF 8.01 here.

Anyway, do bookmark that updates page. We can only hope that in the future it will be the place to keep an eye on the latest CHFs for each release of CF going forward.

CF911: Easier thread dumps and stack traces in CF: how and why

You may have heard the value of taking thread dumps or stack traces when trying to understand and resolve problems with CF. They can be valuable to see what's really running on your server at the time it may seem hung or slow to respond. The problem is that they can be challenging to obtain, so here's how to get them even more easily.

(If you're not familiar with the value of thread dumps or stack traces, read on. The resources I point to get help you to appreciate their usefulness.)

Well, I was tooling around looking for some help on a topic when I happened upon an old entry from the awesome and ever-valuable blog of Steven Erat, formerly of Adobe and now of Webapper. The entry is An easier way to take ColdFusion thread dumps on Windows , and he actually points to entry in another classic blog by Brandon Purcell (formerly of Adobe and now with Universal Mind). Steven's entry also points to a classic Adobe technote on interpreting dumps and stack traces.

But both entries are from 2005, and since then there are still easier solutions for getting either a thread dump or stack trace, and I wanted to point them out (I'll also add a link to this entry in Steven's.)

For those on CF 6, 7, 8

First, for those running CF 6, 7, 8 (and other CFML engines, like Railo and Open BlueDragon), you can use either FusionReactor or SeeFusion, commercial monitoring tools, which each offer a simple single button to produce a thread dump (a stack trace for all threads running inside the JVM that underlies CF).

Better still, they also offer an option to get a stack trace for a single running CFML page request.

For those on CF 8 Enterprise

If you're on CF 8 Enterprise, you can take advantage of its Server Monitor, which also offers the ability to get a thread dump, by way of its "snapshots" feature.

I discuss the Server Monitor in a 4-part series of articles for Adobe, starting here.

Part 3 of the series discusses Snapshots in particular.

(I also have an an article introducing FusionReactor.)

The CF 8 Server Monitor can also provide the stack trace of a given request, but you need both "start monitoring" and "start profiling" to be enabled. If they are, you can see currently running requests in the Active Requests screen, and if you click on one, the stack trace appears in the middle of the request detail page.

But maybe you don't need to do a thread dump at all

All that said, I should point out that one of the values of thread dumps was to see what requests were running, which is incredibly valuable.

But if you do get either FusionReactor or SeeFusion, or can use the CF8 Enterprise Server Monitor, note that they all have a feature to show what requests are running.

That, alone, can be what you really want to know, and their interface is a LOT easier to read than a thread dump. :-) And the stack trace feature let's you see exactly what line of CFML code a request is running at the time you request the stack trace.

Thread dumps are still valuable

Still, thread dumps can still be valuable. For one thing, you may not always be on the server when you want to see what's running. Both FusionReactor and the CF8 Monitor have an option to trigger sending you a thread dump by email when certain conditions are met (too many requests, requests taking too long, too little memory, etc.).

Also, sometimes a problem is not caused by a running CF request. It could be another thread that's having a problem, whether one of the scheduler threads (which are used for background tasks like the client variable purge process), or cf-thread threads (used by CFTHREAD in CF8), and so on.

A tool to help analyze thread dumps

Indeed, Steven would also want to point out that Webapper (makers of SeeFusion) also have a free online tool called SeeStack which can help analyze thread dumps to make them a little easier to read.

Still more on thread dumps and stack traces

Further, the Adobe technote that Steven points to, Debugging thread dumps and server problems in ColdFusion MX 6.1 and 7.0, also gives considerable insight into understanding stack traces and thread dumps. It's also quite old and doesn't talk about things with respect to the more modern tools I've mentioned, but the information is valuable whether you use them or not.

In fact, much of it is about understanding stack traces (remember: a thread dump is a list of stack traces for all threads in the JVM), and connecting the dots in what they report and specific possible problems in your code or configuration.

I plan to do some entries in the near future, walking through use of these tools to solve common problems of CF servers being unavailable. Until then, check out also the "related entries" listed at the bottom of Steven's entry.

Of course, there are many other fine classic blog entries (and bloggers) who talk about CF troubleshooting, too. I also plan to offer something to help with finding those.

Monitor CF using Nagios: a useful new way

Nagios is an open source (Linux based) monitoring tool that many organizations use to monitor their IT infrastructure. Here are two resources to help you monitor it with CF, one of which is a new way that may give additional benefit to those already doing Nagios monitoring of CF.

Basic CF Monitoring in Nagios

First let me share that if you've not explored monitoring CF with Nagios at all, here's at least one blog entry discussing that.

Of course, you can easily use Nagios primarily to report whether CF's up or down, but you can also monitor basic statistics such as are provided via CFSTAT (also available on Windows via Perfmon, which also exposes generic stats about all processes), as demonstrated in that blog entry.

But you may have noticed that, if you run the Multiserver (multiple instances) version of CF, those CFSTAT and Windows Perfmon Stats aren't available on that version of CF.

So what to do if you wanted to monitor more about what's going on inside of CF?

FusionReactor Nagios Plug-in

If you're running FusionReactor Enterprise, you can now get that additional information via the free FusionReactor Nagios Plugin. It's a perl app that when implemented will expose certain key statistics in a way that Nagio can process them (and some are things that CFSTAT and the CF Permon stats don't report):

  • Count of current running requests, and total count of all requests run
  • Count of request queued by FusionReactor
  • Average request runtime (since server start), and Recent request runtime (in past 60s, by default)
  • Count of recent slow pages
  • Memory free in bytes and percentage
  • Memory Total and Max
  • Instance and System CPU Use percentage
  • Count of recently completed JDBC requests

Note that this is technically community-contributed feature (from an Intergral employee, David Stockton), not a supported feature.

If you're running or are interested in FusionReactor Enterprise, check it out.

CF911: CF pages get "no web application configured", but CF's "running"

If you find you can't run CF pages (they get "There is no web application configured to service your request") but CF seems to be "running", here's one possible explanation and how to diagnose/resolve the problem.

The problem outlined here has to do with being careful when you're modifying CF to enable/disable RDS (or doing any edits in the [cf]/wwwroot/WEB-INF/web.xml file that controls it.) This applies to CF 6, 7, or 8.

Of course, you may have a different cause for this kind of error. The focus here is this one cause and solution, but I hope that the information shared here on diagnosing the problem may help you if you've got a different problem.

The problem and solution

As many know, enabling/disabling RDS involves removing or adding comments around some XML entries, as explained in this technote.

After making the edits, though, you may find that you get this error when trying to run any CFML pages:

"500

There is no web application configured to service your request"

What happened is you made a mistake I did. I had RDS enabled, so I wanted to disable it, which meant adding comments.

Don't do this

So I modified the XML entry from this:

<!-- begin RDS -->
<servlet-mapping id="coldfusion_mapping_9">
<servlet-name>RDSServlet</servlet-name>
<url-pattern>/CFIDE/main/ide.cfm</url-pattern>
</servlet-mapping>
<!-- end RDS -->

to this:

<!-- <!-- begin RDS -->
<servlet-mapping id="coldfusion_mapping_9">
<servlet-name>RDSServlet</servlet-name>
<url-pattern>/CFIDE/main/ide.cfm</url-pattern>
</servlet-mapping>
<!-- end RDS --> -->

That won't work. The problem is that my comment surrounds the section of XML which also includes an existing commented line.

Do this

So the solution is to do something like this instead:

<!-- begin RDS -->
<!--
<servlet-mapping id="coldfusion_mapping_9">
<servlet-name>RDSServlet</servlet-name>
<url-pattern>/CFIDE/main/ide.cfm</url-pattern>
</servlet-mapping>
-->
<!-- end RDS -->

I've only commented out the significant XML entries, not the comment that preceded it. There's a second entry that needs to be commented out as well, as discussed in the technote.

Also, note that you'd get the same error if you mistakenly use CFML comments (three dashes) rather than HTML comments (2 dashes). It's easy to do if you happen to edit the XML file in a CFML editor. I've confirmed it gets the same error indications above.

Diagnosing and resolving this and similar errors

That's the problem and solution. But how did I know that was the cause of the problem. It wasn't as obvious as it may seem.

It seems that CF *is* running--but no, JRun is

As the title of this entry (and the error message above) indicates, the problem was that CF was reporting that there was "no web application configured". Well, it's not really CF reporting that. It's JRun.

Indeed, in this situation one would notice that the jrun.exe process is running, but no CF pages would run.

There's no error reported in the CF/logs

So you may be tempted to look in the CF logs (such as C:\ColdFusion8\logs) but you'll find nothing. In fact, you may note that the logs are not even updated with the time that you had started CF.

The thing is, CF never really started. JRun did, yes, but the CF web application itself (within JRun) did not.

The real diagnostic info is in the Runtime logs

You can see this more clearly if you look instead in the "runtime" logs, which if you run the Server version of CF are in [cf]\runtime\logs\. (If you're running the Multiserver mode or what some call multiple instances, they're in jrun4\logs.)

In the coldfusion-out.log file (or the corresponding -out.log file for whatever instance is failing, if you're running multiple instances), you may see something like this error:

error Error on line 333 of document file:/C:/ColdFusion8/wwwroot/WEB-INF/web.xml: Next character must be ">" terminating comment .
[1]org.xml.sax.SAXParseException: Next character must be ">" terminating comment .

Sure enough, line 333 was where I had made my edit to the first of the XML entries discussed above.

This was followed by other messages (along with lots of stack tracing info to wade through):

info Unregistering enterprise application: file__C__ColdFusion8_#Adobe_ColdFusion_8

error Deployer Service failed to deploy file:/C:/ColdFusion8/
   * Could not deploy web application at file:/C:/ColdFusion8/wwwroot/.
   * Failed to parse XML: file:/C:/ColdFusion8/wwwroot/
   * null

finally showing (after still more stack tracing info):

Server coldfusion ready (startup time: 7 seconds)

Again, we can be mislead by that, thinking, "ok, CF is up". But no, it's just saying that the JRun server named "coldfusion" (or whatever instance you may be running) had started, but the web app for CF itself did not deploy, as the previous error message above showed.

So the moral of the story is to be careful when editing these (and any) XML files.

Side note on RDS

I'll add a note that one other reason people don't use RDS is because it relies on a single password to be shared by all who would use it. Yes, I realize there's still more to the security concerns. Anyway, if that's the most significant issue for you, there's a new feature in CF 8 Enterprise to address that, in the new option to enable multiple different user accounts for both the Administrator and RDS.

And in fact, I just did an Adobe DevCenter article introducing this feature at length (and I point to more about the other security concerns to consider before enabling RDS.) It's supposed to be coming out later today (at 4pm eastern), and I would assume it will be at the CF dev center.

Having problems with SQL Server/Oracle/DB2/Sybase? Check out Confio Ignite

Hey folks, if you're having problems with your CF apps and you determine that (or wonder if) the cause may be due to problems in the database, check out Confio Ignite, a commercial tool that may be well worth the price for you.

Sure, there are many DB monitoring tools out there, but Ignite focuses on tracking, analyzing, reporting, and explaining wait events within the database--and you'd be amazed how often waits caused by your code, that of others, or from other operations in the DB are the explanation for poor performance. It can help target exactly what SQL statement or other operation is a cause of significant waits.

The tool presents the data aggregated over time, so you can view it per hour, day, week, etc. Great for both drilling down to find hot spots, and for viewing how coding/config improvements (resulting from your responding to the analyses) have led to performance improvements over time.

The tool runs with low overhead: it reads data that the DB provides, storing it in a database and providing a web-based interface to view that data. The process to read the data and create the repository (and present the web-based interface) can (and should) be done on a server separate from the server being monitored.

Here's a nice 2-minute demo. There's also a free trial, of course, and it's pretty quick and easy to install and benefit from.

As I noted in the title, it works with SQL Server, Oracle, DB2, or Sybase (sorry, not MySQL. Don't know why). And while it's a commercial product, it's not a ridiculously high price (as for some tools). I just learned of it in the past few weeks, and one customer of mine who tried it has been just thrilled with the results. I hope to write more about it later, but wanted to at least get this info out for folks to consider.

CF911: CF 8 Server Monitor reports "ColdFusion Server is unavailable" (solution)

Here's another entry in my CF911 series. If you try to open the CF8 server monitor and get the error "ColdFusion Server is unavailable", the problem may be in your web server configuration. In this entry, I help you confirm if you're getting the problem I refer to here, and of course I show the solution (3 actually), with a caveat.

Here's a screenshot of what you may see:

Note that this is not an error related to logging in. You do need to fill in a username to log into the Server Monitor, even if CF is set to only ask for a password when logging into the Admin. Just use "admin". This and other facets about the CF8 Server Monitor are covered in a 4-part series of articles I did in the Adobe Dev Center, starting here.

Confirming this is the cause of your Monitor challenge

From my observation, this error is related to a problem with the Flex client being able to talk to the server using a URL it needs to use, and the problem is web server related.

You can confirm if what I'm about to describe is your issue by trying to access the URL that the server monitor tries to use to access the Flex Gateway for CF, such as:

http://localhost/flex2gateway/

Actually, you should use whatever domain name/port you're using to access your CF Admin, which is then used when you ask it to open the CF8 monitor, which may be a URL like this:

http://yourserver/CFIDE/administrator/monitor/launch-monitor.cfm

Anyway, if that test attempt to open the /flex2gateway/ url comes back with a "file not found" (or 404) error, as opposed to a blank page, then you likely have the problem I'm describing, whereby your web server is mistakenly looking to verify that a file exists for the path you're specifying. You have 2 solutions.

First, let me note that this flex2gateway URL is not a file, nor a directory. It's a value intercepted by a servlet filter defined within CF. You need to tell your web server not to check for any existing file (it's trying to use one of the "default documents" that are used when only a path to the web server is provided.) Before launching into how to fix your web server, you may want to consider one other possibly simpler alternative.

Changing to use the Internal Web Server

Some will note that I've used no port above in the URL. That's why I point out for you to try whatever URL is used to access your Admin. In the case above (and the people who have reported this problem so far that I've seen, they've been trying to access the CF admin using their external web server, IIS.

If instead you were to use the CF internal web server to access the CF Admin, you'd have a port in the URL, like this:

http://yourserver:8500/CFIDE/administrator/monitor/launch-monitor.cfm

(or it could be 8300, or 8301: whatever is the port for accessing the built-in web server for CF, if you chose to implement that when CF was installed, and you are accessing the Admin that way.)

Well, I'd propose that if you DO use the internal web server, you probably won't get this error at all. The problem seems related to using IIS to access the Admin (and the CF 8 Server Monitor).

That said, I'll suggest that one quick solution folks can try is to see if indeed they can access their CF Admin (and monitor) using the internal web server. (If you can't or won't use it, I have the solution for getting it to work with IIS, in a moment.)

You just need to know what port to use to access the internal web server, if it's enabled.

First, you may find that if (on Windows) you use Start>Programs>Adobe>ColdFusion 8>Administrator that it will open using the built-in web server. If it does, see if using that gets you around this whole problem.

If that opens it with external web server (doesn't use a port like those above), or if you aren't on Windows and have no Start menu, you can also get the web server port (and indeed enabled it, if disabled) by way of the jrun.xml file. Rather than detail it here, I'll point you to a couple of resources:

Configuring the Macromedia ColdFusion MX built-in web server is an old technote, but the info still applies. Where it talks about disabling the internal web server, you'd want to reverse that, of course. There can be more subtleties and challenges to running the CF admin on the internal web server, if you don't configure it that way at the start, such as where are the /CFIDE files? Are they in the [cf]/wwwroot? or in your web server doc root, like inetpub/wwwroot? The built-in web server will look for them in the [cf]/wwwroot, so you may need to copy the CFIDE into this directory, or add a mapping to the built-in web server to point to the path as being located externally.

Making the change in IIS

Or you could just fix IIS to let you access the server monitor via IIS. The problem may be due to a setting in IIS (verify that files exist) that you may have caused to be set. (I don't know if it's set by default when CF is configured to integrate with a site, but I wouldn't think it was, so maybe this affects those who add new sites or configure things manually.)

And since this problem may affect other Flash/Flex apps trying to talk to CF, it may be worth doing for all such users. But this does come with a caveat to be aware of, if you might be using NTLM security to control access to files requested via IIS. More in a moment.

I offer the solution for IIS 6 and 7. I don't know if the same problem can affect Apache. If so, and anyone can offer the solution for there, please do comment.

Making the change in IIS 6

For IIS 6, launch the IIS Manager and select the web site which has the CF Admin you're trying to use. (It may be that you've also configured IIS so that ALL web sites are configured for CF, in which case this setting would be not at the site-level but at the root server-level, so you'd select the server name instead in the left IIS pane.)

From there, right-click and choose properties, and then select the "home directory" tab, then in the "application settings" area click the "configuration" button, and in the "wildcard mappings" section you should see something like "C:\ColdFusion8\runtime\lib\wsconfig\1\jrun_iis6_wildcard.dll" (which will be different, of course, for the JRun4-based Multiserver deployment).

This value is implemented here during the install of CF if you tell it to integrate with IIS, or by your running the CF Web Server Configuration tool after the fact.

Select it, and choose Edit, and if the "Verify that file exists" option is checked, un-check it. This setting can be confusing: you may think it means "verify that the named executable exists", but it doesn't. It refers to whether files requested and passed through this handler should be checked to confirm if THEY exist. Here's a depiction of the setting and how to get there.

Now try the URL above, and it should no longer give a 404. Then try again to login into the Admin. (Actually, you may find that you can just click the "cancel" button and it will login, even if the values for username and password are blank. I find this helpful when the CF server is temporarily unresponsive too, and the Monitor login screen pops up.) Hopefully the server monitor now works for you.

Note that this was NOT about changing the handler mapping for .cfm files, which also offers an option to control the "verify that file exists".

A caveat about access via IIS to NTLM secured files, and another alternative

Thanks to Mike Gillespie for the following notice and clarification. If you use NTLM security (windows integrated authentication in IIS) to secure files accessed via IIS, then you DO NOT WANT TO make the above change for your IIS site. I share below what he offered to me.

But I'd point out again that even with that issue, you could still use the built-in web server is a solution. Or, sticking with IIS, you could also create a new IIS site just for accessing the CF admin and monitor, and make the change above for that site only.

Anyway, if you do use NTLM security to control access to sites requested via IIS, consider the following:

The check that file exists option is required if you want to use NTLM perms to secure .cfm files. http://www.adobe.com/go/tn_18516 (Steps 1-4)

If you have a secure folder on your webserver put a .htm file and a .cfm file in it. Do not give your ID access to that folder. In IIS turn on clear text and NTLM auth.

With the "check that File Exists" option unchecked, try this test.

Try to access the .htm file in the browser - access denied

Try to access the .cfm file in the folder, - access GRANTED - so much for NTLM perms

Now check the box and try again (you will need to recycle cf and IIS)

Try to access the .htm file in the browser - access denied

Try to access the .cfm file in the folder, - access denied as it should be - but flash forms and server monitor are dead.

So "fixing" the Server Monitor problem on an authenticated server just broke the security of the server for the sake of monitoring... [frown>]

In a nutshell.

If you implement this so that CF pages can be authenticated against Windows Security http://www.adobe.com/go/tn_18516, then Flash forms break (and the server monitor too). So to get flash forms (and the server monitor) working, you have to implement this, which fixes flash forms (though every user gets their own personal file on disk on the server that has to be cleaned up) but it does not fix the server monitor.

It is the "check that file exists" selection that breaks the Server Monitor (and flash forms).

On a cf webserver that grants anonymous access there is no reason to check the "check that file exists" box. However, on a server that does authenticate users for NTLM file access, that box should be checked.

This section above was added after the entry was first posted.

The change for IIS 7. None needed?

For IIS 7, it's a little different. I actually run IIS 7 (Vista) and am not sure how/where the wildcard mapping equivalent got created (I may have fudged it manually), but it's now listed as a "Handler Mapping" (in the properties for a web site). In my case, it's labeled "AboMapperCustom-32635", but just look at those listed as handling "*" meaning all requests. It's listed with a value of IsapiModule in the Handler column. (If you're looking at a specific site, and the "Entry type" column says "Inherited", then there is another mapping at the server level, so select your server name in the left IIS panel, and repeat.)

Even so, I see no option to control "verify that file exists", so maybe this problem can't happen in IIS 7. I will say, FWIW, that there is indeed a an equivalent to that "verify that file exists" option, at least for specific extension handler mappings. Look a the one for .cfm, for instance. Double-click it to see its properties, and note a new button called "request restrictions". It has an option, "Invoke handler only if request is mapped to", and an option of "file". Again, though, this does not affect requests to non-cfm requests like that for the /flex2gateway/ URL.

About other Flex/Flash apps

As I said, it may be that the info above will help other Flex apps having trouble talking to CF (the CF8 monitor is a Flex app), but I'll note that this problem doesn't affect all Flex apps: only those that connect to CF via IIS.

For instance, on this same server where this problem occurred, there was never any problem using FusionReactor (which is also a Flex app). It was working fine the whole time. But then its default behavior is also to use its own Built-in web server, so requests weren't going through IIS. If I did try to use IIS to access FusionReactor, then it too failed (with a file not found), and the fix above solved that.

CF911: CF doesn't respond for extended lengths of time

This is the first of a series of blog entries I'll do under the heading of CF911, to share info I offer to people when I help people solve CF problems either on mailing lists or by my support consulting.

One problem: CF not responding

Someone asked for help on one of the Adobe forums, with a problem about users reporting that the server was not responding for extended periods.

After checking the logs (CF, Windows event, and IIS logs) and not "seeing anything to indicate errors", he wondered if somehow "you can set site availability windows".

I replied to him that, no, there are no features to set "windows of availability" for CF. So either CF was up and responding, or it wasn't. Even if it was, it may also be that his problem was with IIS. There are a few things that he could check.

First thing: check the Event logs for CF or IIS up/down

The first thing I'd do is check the Event Log (Application) to see if CF was indeed up or down. Look for event ids 3 and 4 for the "source" being the name for your CF server as shown in the event viewer.

Then I'd check if the web server itself may have gone down, using the same approach (though it's less likely). I'd also check the Event Logs (application or system) for any indications that the IIS application pool(s) may have have gone down. It's not unusual for the pools to recycle, but being down for an extended period may point out a web server problem (by which CF often gets blamed unfairly).

Next: Check the IIS logs

If that's not the issue, I'd then check the IIS logs, to see if indeed requests were being processed during that time. When people say "the server is down", that just means their requests were not processed. It may well be up and processing other requests. You'd also want to check whether cfm or other extensions were being processed at the time, their status codes, and the time-taken for each request (if enabled for the log in the properties for the site in IIS).

Next: Check the CF Built-in Web Server, if enabled

And when people report that it is "down", I'd also run requests using CF's built-in web server if it is/was enabled. If it responds on that web server, but not IIS, then that confirms a problem either with IIS or the web server connector between CF and it. The default port for CF's built-in web server is 8300, 8500, or some variation like it, depending on the version of CF and other things.

You can confirm if it's enabled and what port it would be by looking in the jrun.xml file. For more info, see resources like this CF 7 doc on the topic.

Using tools like FusionReactor, with even better logs

I'll point out that besides the web server logs, this is a problem that would be well served by having a tool like FusionReactor (fusion-reactor.com) and its great logging features that track all kinds of details about each page request. Of course, it's a great monitor and notification tool, with optional request protection features as well. (The CF8 monitor and SeeFusion are great too, but don't offer the kind of logs that FR does.)

Web Services specifically

He had also indicated that these were Flex apps calling CF web services that weren't returning data. He didn't indicate if other CF page requests were running, but since I've heard of problems of CF seemingly "down", the info above applies just as well to web service requests. As far as CF and IIS are concerned, those are regular requests, so they go through the same web server process and CF request queue.

(He didn't mention what version of CF he has--6, 7, or 8--and whether it's Standard or Enterprise. In CF8 Enterprise, there's a new feature to give web services requests their own set of request threads. You can see that in the CF Admin "Request Tuning" page.)

But he said specifically "during these times, calls to the web services never return any data". So, that's potentially still another issue. It may help to see what the web services would respond with if requested directly in the browser. Many don't realize you can request a CFC as a web service on the browser, which can help to confirm if/how it responds.

If you had a test.cfc with a method called getdata, that took an argument of myarg and a value of 1, you could request that on the browser using http://yourserver/yourpath/test.cfc?wsdl&method=getdata&myarg=1. If you need to pass in other arguments with simple values (text, numbers), just keep adding them.

Finally, you could also use a debugging Flash player or proxy tools like Charles to watch what traffic is going from/back to your Flex browser client. I list several such proxies in the CF411 resource I keep.

More help

There are surely still other things I could have thought of but I fired that off to see what he may say, and I've offered it here in case it may help someone. Feel free to raise questions here. (For anyone interested, the Adobe Forum thread this took place in was on here. If you want to offer something that may help the gent in question, feel free to reply there instead.)

I'll say in closing that if anyone has problems like the above and would like some help (someone looking over your shoulder to help you solve such problems), I'm happy to provide that sort of support, as discussed at carehart.org/consulting. But as you can see above (and in the forums, as well as many mailing lists), I'm also happy to share the info for people who prefer to solve problems on their own.

CFMAIL being detected as spam? Some solutions for CF 6, 7, and 8

Someone asked on a list about how to resolve the problem where messages sent via CFMAIL have a message-id value that can cause mail to be flagged as spam. I mentioned in an earlier entry how to solve this in CF8. In this entry, I offer a couple of solutions for CF 6 and 7.

To backup and explain the problem, some have lamented that in CFMX 6 and 7, CFMAIL used the name of your server where CF is installed, which might be something like "server1" or "bingo", as in:

Message-ID: <23070863.1197039960343.JavaMail.SYSTEM@Server1>

This might cause a recipient mail server to flag the mail as spam, if the mail server was a different domain name (like "yourcompany.com"). The bummer was that many found no way to fix this. Sure, in CF5 you could set it in a CFMAILPARAM to set a mail header, but CF 6 and above ignores that.

The good news for those on CF 8 is that a change makes the problem go away, which I talked about in a blog entry back in December.

But for those still on 6 or 7, I'd mentioned that there's an available code or (preferably) configuration fix to resolve the problem. Apologies for not posting it at the time, so here it is. I also explain what the issue is all about, for those not aware of it.

Simple CF5 solution doesn't work in 6, 7, or 8

First, let me point out that some may propose that one can just use CFMAILPARAM to set that message-id header, as was mentioned in this FAQ:

http://www.developer.be/index.cfm/fuseaction/faqDetail/FaqId/201.htm

Unfortunately, that's a very old FAQ and it no longer applies after CF5.

CFMX 6 (and 7 and 8) ignore that header you set and set the message-id themselves to the name of the physical server from which the mail's sent. Here's an example, from my laptop (which is named charlied620):

Message-ID: <23070863.1197039960343.JavaMail.SYSTEM@CharlieD620>

On your server, it could be that the problem is that the server is a real domain name, but it's not the same name as the SMTP server through which you're sending email via CFMAIL, and some servers won't let mail through with FROM addresses having a different domain name.

Why doesn't CF just use the SERVER set in CFMAIL or the CF Admin?

You might wish/expect it would just use name of the mail server as specified in the CF Admin mail server setting or the CFMAIL SERVER attribute, but it does not (at least, not prior to CF8, as discussed in my earlier entry.)

But while researching this problem for the person, I found this, where a very compelling solution was proposed. I tested it, and it does indeed work.

It turns out that you can get CFMAIL to use a specific mail server, either of 2 ways, for CF 6, 7, or 8 (again, on CF8, the easier solution is that I discussed in the earlier entry). Ken Smith of Adobe offered it also in this Adobe forum thread. If you make this change, either server-wide or per page/app, you will now find that the CFMAIL sets the message-id to use a desired servername.

Solution for 6, 7, or 8: Configuring it at CF Startup

If you're on a server where you want to change this for all CFMAIL (and javamail) done from within CF, you can change the JVM.config for your server to add this to the java.args line:

-Dmail.host=desiredservername

That java.args line is one long line. Don't introduce any line breaks. You need to restart the server after making that change. As always, when editing the JVM.config (in the runtim/bin directory of CF), make a backup first, because if you get it wrong, your server won't restart. (Or leave the file open and be prepared to do an Undo in your editor.)

But if you don't want to (or can't) mess with the startup file, or more important, if you are on a server where each app may need to do its own setting, you're not out of luck.

Solution for 6, 7, or 8: Changing it programatically

You can also make the needed change on the fly, programatically. You could issue this set of code just before your CFMAIL:

<cfscript>
sysObj = CreateObject("java", "java.lang.System");
sysObj.setProperty("mail.host", "desiredservername");
</cfscript>

I tried this per-page setting in in 6 and 7 and it worked fine. For CF8, the code will run, but it won't affect the CFMAIL for reasons I explain in the next section. But there's a real problem with this approach, if you're on a shared CF server. You're still setting the value for the entire CF server, but you're just doing it on the fly. It has at least a few problems you need to think about:

  • you'll affect all who use CFMAIL on this server (at least in 6, and 7, since CF8 ignores this, as discussed below)
  • you'll also affect anyone who uses Javamail on this instance of CF
  • and if someone else on the server issue the same code with a different server name, that of course would override your setting

There are a few things you can do to mitigate the problem, but they're not entirely perfect.

First, you could get the current mail.host value (in case it's different), then change it to what you want, and then change it back. You could do that with this code:

<cfscript>
   sysObj = CreateObject("java", "java.lang.System");
   // if there is no mail.host set already, this next variable simply won't be created. need to know that for later.    oldmailserver = sysObj.getProperty("mail.host");
</cfscript>

Note that if the mail.host property has not been set in the startup config or by someone else running such code, then the getproperty will return nothing, and the oldmailserver variable literally will not be created (a curiosity of working with some java methods).

Once you have the oldmailserver (or know that it did not exist), you can do the set of the property and the cfmail as above, then you could set it back with the following:

<cfscript>
// reverse the setting of the mail.host, so as not to affect others on this server if (not isdefined("oldmailserver")) {
   // if there was no previously set mail.host, remove the property
   sysObj.getProperties().remove("mail.host");
}
else{
   // set it back to what it was before
   sysObj.setProperty("mail.host", oldmailserver);
}
</cfscript>

There's still a problem. Because of CF's multi-threaded nature, it's entirely possible that between your setting the host and doing your CFMAIL, someone else could also set the mail server to something different. (This is called a "race condition".)

Now, Mr. Smith in the threads above suggested that you could wrap the code doing the change and the CFMAIL in a named lock, but that's an incomplete solution. It will prevent other other code (that ALSO does the same named lock) from running until yours is complete, but it won't help if others do the set of the property without bothering to use the same named lock (or use a differently named lock).

That's a frequent misconception about locks. They don't prevent other code doing what you're doing in the lock: they only tell other code that IS using the same lock not to run while this lock is held--and even then, only if you use an EXCLUSIVE lock, which he didn't indicate.

So the bottom line is that as useful as the feature is to set the property dynamically, it's fraught with peril in an environment where multiple apps may try to use the code. Only if you can guarantee that all use the same named lock will you be able to protect against this problem is held up while you have the value changed:

<cflock name="setmail-servername" timeout="10" type="EXCLUSIVE">

   <cfscript>
   sysObj = CreateObject("java", "java.lang.System");
   // if there is no mail.host set already, this next variable simply won't be created. need to know that for later.    oldmailserver = sysObj.getProperty("mail.host");
   sysObj.setProperty("mail.host", "desiredserver");
   </cfscript>

   <cfmail ...>
   ...
   </cfmail>
   
   <cfscript>
   // reverse the setting of the mail.host, so as not to affect others on this server    if (not isdefined("oldmailserver")) {
      // if there was no previously set mail.host, remove the property
      sysObj.getProperties().remove("mail.host");
   }
   else{
      // set it back to what it was before
      sysObj.setProperty("mail.host", oldmailserver);
   }
   </cfscript>
</cflock>

Now, someone may propose that all this could be wrapped up into UDFs or CFC methods, and perhaps it could, but since you need to do the setting of the value, and the mail, and the resetting of the value, all within a CFLOCK, it would be challenging to wrap all this up into a generically callable method (unless you wanted to pass in as well all the CMAIL attribute values and the body). Just seems not worth it, since this is a pretty esoteric problem and solution. But others can comment if they feel differently.

Solution for CF8 is much easier

All this is obviated on CF8, because it now properly uses the name of the server specified in the CF Admin mail server setting (or CFMAIL SERVER attribute, which overrides the CF Admin setting.) This is AWESOME news for those challenged by this, and hasn't gotten much press.

Now, what was the caveat I mentioned above? Well, if you use the approach of setting the mail.host servername in the java property, CF8 no longer pays attention to that. It JUST uses the CF Admin mail server setting, or the CFMAIL SERVER attribute. So that code above "won't work". It will work, but it won't affect CFMAIL. But that was a hack to work around CF not honoring these other attributes. I'm not surprised (or bummed myself) to see that it no longer regards the mail.host property for CFMAIL.

(I should say I'm saying this as of CF8. I've not tested it on 8.01.0

Some related notes

If you add the SERVER attribute, you may need to add the PORT, Username, and Password as well.

Here's a little bonus tip, in case you try to use the CFMAIL SERVER attribute for the first time on an existing CFMAIL tag to check this out. Note that using the SERVER attribute on CFMAIL requires you then to specify any other attributes required for the mail connection such as USERNAME and PASSWORD (if needed) and PORT (if not 25). What I mean is that it will no longer pick up the values in the CF Admin. If you override the SERVER, then you override the other config settings as well and need to specify them.

I'm not getting the emails now. Where are they?

If you make a mistake in your setting of the mail server arguments, then CF will move the failed emails to the [coldfusion]/mail/Undelivr directory where CF is installed. They're just plain text files. You can open them to see what got set and perhaps can figure out why they failed. You may also find information in the [coldfusion]/logs/mail.log.

When I look at mail in the spool, it looks fine (if I use CFMAILPARAM). Why doesn't it get through?

Don't be misled. The email you generate in CFMAIL may look fine in the [coldfusion]/mail/spool directory, but when it gets sent to the mail server, CF will change that message-id (and some other headers). You really need to look at the email as it's RECEIVED. You can't even look at the message in the Undelvr folder as an indication. It doesn't have those added headers.

So how DO I observe the mail headers?

You need to receive the email and then look at its headers. I'll show you how to do that in Outlook and gmail in a moment. Let me point out that it can be very englightening to view the message headers: not only the message-id but possibly also other headers as well as messages that CF, your mail server, or the mail server of your recipient may have added, which may include indications of whether your email was detected to be spam (perhaps by tools like SpamAssassin).

In Outlook, you can use View>Options while reading an email, to see the values in the "Internet Headers" box of the window that opens. Scroll down in that to find the message-id and other headers.

In gmail, when you open the message, look to the top right of the pane showing your email, to the right of the indicator of the time the message was received. There's a drop-down box, with options like "reply" and "forward". Choose "show original". That will show the complete message including all the headers at the top.

Other CFMAIL alternatives/replacements

Finally, I'll point out that if you run into other problems with CFMAIL, there are always alternative CF mail server solutions, like those I list in the "CFMAIL replacement/enhancement tools" section of my "Tools to Consider for CFML developers" page.

Hope that helps some.

BlogCFC was created by Raymond Camden. This blog is running version 5.005.

Managed Hosting Services provided by
http://www.edgewebhosting.net/
Managed Dedicated Hosting