Note: This blog post is from 2014. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

But as is often the case in a lot of the CF server troubleshooting consulting I do, I find the causes to be far less often what most people seem to suspect. So what would I look for when someone reported high CPU in ColdFusion (or Lucee or Railo )? Read on.

[....Continue Reading....]

Note: This blog post is from 2014. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

I was wanting to point out to someone the various ColdFusion security resources, and while I have a category on them in my CF411 site, I thought this was a list worth pulling out into its own blog entry and expanding a bit.

You may be surprised to find that there are more to CF security guidelines than just the venerable server "lockdown guide" (for those administering and configuring CF, the OS, and the web server, among other things).

Did you know that there have been "developer security guidelines" as well, focused instead on coding? This latter guide has gone through three iterations, including just recently, as I'll discuss along with the lockdown guides, below.

[....Continue Reading....]

Note: This blog post is from 2014. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

And your operation fails. You're then prompted to "Click here to login", but even if you back up or client another link, you'll be prompted with the CF Admin login.

What gives? Why is it happening? And how can you fix things? Is CF broken? No, not in the sense that you need to reinstall or anything. The good news is that there is a quite simple solution. Well, there are several, depending on your goals.

The simple solution: delete the duplicate cfid/cftoken or jsessionid cookies that you will find your browser is sending to CF. But there is much more to this, as well as other solutions, which would be worth most readers taking a few minutes to read on here.

BTW, the same root problem can be the cause of your own application's users finding that they can't stay logged in. More on that in a moment.

[....Continue Reading....]

Note: This blog post is from 2012. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

With a title like Beginner's Guide, one may wonder what to expect. As a contributor myself to several CF books (carehart.org/contact.cfm#books), I appreciate the challenge in deciding the intended audience for a book, and keeping it in mind as we are writing. And having been a co-contributor on now 10 multi-author books (not all on CF), I appreciate the challenge in keeping a consistent voice among all those authors throughout a book, as well as each keeping that intended audience front and center. It's not easy, although it can be aided greatly by careful editorial control and editing.

What to expect from the book?

[....Continue Reading....]

Note: This blog post is from 2012. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

You can find the recording here. Note that you need to login with an Adobe ID, just like when you download Adobe software or participate in their forums. (I have no control over that.)

Since that link just goes right to the recording, here is the description I'd used for the session, to help decide if the recording may interest you. BTW, I clarify on the session that FR is useful for more than just ColdFusion, in that FusionReactor can be used for Railo, BlueDragon, and OpenBlueDragon, as well as in fact any Java server (Tomcat, JBoss, Jetty, Glassfish, Websphere, etc.), and the session applies just as well to folks using those.

My session: Monitoring ColdFusion with FusionReactor

Recording
Session Description:

[....Continue Reading....]

Note: This blog post is from 2012. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

Yet people will reasonably want to be able to have some mechanism that "watches" CF "from the outside", to know when it's gone down. How can you do that? That's what I'll point out in this entry.

And beyond talking about what goes along with the CF Enterprise Server Monitor, I'll also point out options for those who are NOT running CF 8, 9, or 10 Enterprise and therefore do not have the Enterprise Server Monitor. This also includes those CF 6 or 7. There are solutions for you, and also for those running Lucee, Railo, BD, or indeed any Java server. More on all that in a moment.

This is part 4 of an unexpected series of entries today on the CF Enterprise Server Monitor. :-) I got on a roll, and each seemed deserving of its own topic. See the "Related Blog Entries" below this entry for links to those.

What the CF Server Monitor is, and is not

[....Continue Reading....]

Note: This blog post is from 2010. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

I hear people raise concerns with memory problems quite often, whether in my CF Server Troubleshooting practice, or just in my participating in many mailing lists. Indeed, addressing this issue more than a few times the past couple of weeks has motivated me to create this, which will be a series of blog entries.

The series parts are expected to be:

• Step 1: Determine if indeed you are getting "outofmemory" errors (this entry)
• Step 2: Realize that having high memory usage is not necessarily a problem (entry to come)
• Step 3: Realize that OutOfMemory does not necessarily mean "out of heap" (entry to come)
• Step 4: Diagnose why you really are running out of heap (if you are) (entry to come)
• Step 5: Realize that CF is maybe suffering because you set the heap too large (entry to come)
• Step 6: If CF is hanging up but NOT due to memory, what could it be? (entry to come)

Common refrains about memory issues

The common complaints about memory issues (and my quick responses, to give you a sense of where I'll be going in this series) are:

• "CF is crashing. Is it running out of memory?" (there's a log that could/should prove that, and that's what we'll discuss in this part)
• "CF's use of memory is high" (which may not be a problem. If you're looking at memory from the OS perspective, it may not matter as much as heap use within CF. This will be covered in part 2)
• "CF's use of heap memory is high" (to which you'd think, "ah, well that's got to be a problem, right?", but no, not necessarily, as I'll explain in part 3)
• "CF has a memory leak" (to which I'd retort, no, generally, it does not. There's nearly always some other explanation. We'll cover that among things in part 4)
• "CF is running at 100% CPU before it crashes" (this could be related to memory problems, and more a consequence rather than a cause, or it could be entirely unrelated to memory issues. See part 5)
• "CF is crashing all the time" (well, is it really crashing, or just hanging up and not responding? That's not a good thing, but it's very different from it crashing on its own. See part 6)

So what if we really are suffering a problem?

I'm not saying there's never a real problem of really "running out of memory". It's just that often things are not at all what they seem (or what most presume them to be, from my experience helping people), and that's going to be the bulk of what I'll talk about in this series. But what if your server is really crashing (or simply not responding), and you think/swear/know that it's a memory problem....

What should you do? Increase the heap size? Increase the permspace? Change the GC algorithm?

Sacrifice a chicken?

I'd say, none of them (though if you're in a rural setting, then perhaps cooking and eating the chicken might help settle your blood sugar so you can stay calm). Really, I know that goes against conventional wisdom, which seems always to suggest diving into the JVM settings. I'd say "hold on there, pardner."

Step 1: Determine if indeed you are getting "outofmemory" errors

This is one that surprisingly few people consider when faced with their server crashing or not responding. They go with whatever conveys to them a sense of there being a memory problem, perhaps adding their own experience or what they read, and they start chasing solutions.

I can't tell you how often I hear people lament that they've googled and found all manner of conflicting and confusing recommendations. And it doesn't help at all that they may be running on CF 8 or 9 (with Java 1.6) while reading about a "solution" written in the time of CF 6 or 7, when it ran on Java 1.4. Of course, the writer often won't have thought ahead to clarify that.

Instead, I'm saying, "stop, drop, and roll".

"Stop" the tail-chasing, "drop" into the pertinent logs directory in CF, and "roll" through them looking for an occurrence of "outofmemory".

Look first in the Console/Runtime/JRun logs

Let me be more explicit: the logs you want to look at for these outofmemory errors are NOT (necessarily) the ones you see in the CF Admin Log Files page. Those are the [cf]\logs directory (or are buried deep within an instance on Multiserver).

Instead, you want to see the "console" or "runtime" logs. Where those are depends on how you are running CF:

• If you're running CF from the console, then look at the console display of logging info. (And if you started CF within CFBuilder, and did not set it to start as a Windows Service, then look in the console view of CFB for this info.)
• If on Linux, look at the cfserver.log in the main CF logs directory.
• If on Windows, running CF as a service, look instead at the -out.logs, found in the [cf]\runtime\logs directory (or [jrun]\logs on Multiserver, in which case there will be a prefix for each instance name in the log file names). You can generally ignore the -event logs there, as they typically just have a subset of what's in the -out logs. (Update for CF10: these logs ARE in fact now in the same directory as other CF logs.)

Some refer to these as the runtime logs, or the jrun logs, or perhaps the jvm or console logs. Whatever you may call them, their location is above and the explanation of their value follow will become clear n this series of posts.

Bonus topic: you can increase the max log size

(I will note that the way these -out.log files work, by default, in CF9 and before is that they fill at 200k increments. Yep, not 200mb, but 200kb!, and you may blow through dozens of them in a few minutes if things are going nuts. That size is configurable, but not through means you'd normally expect. See the blog entry I recently published, CF911: How to control the size of CF's -out.logs.)

Look next in the hotspot/pid/jvm abort logs

Separately, there are some other potentially important logs that may relate info concerning memory problems: what some call the "pid", "hotspot", or jvm abort logs. The filename is in a form like hs_err_pidnnnn.log, with some number in place of the n's.

These logs are found in a very unexpected place (for logs): in the directory where CF stores the jvm.config. So on Standard/Server deployments, that's [cf]\runtime\bin. For Multiserver, that's [jrun]\bin. (As of CF10, it's in the [cf10]\cfusion\bin, or the [cf10]\instance\bin for multiple instances.)

Look in that folder for any .log files. There would be one such "pid" log for each time that the jvm "crashes" due to certain kinds of problems. It could be a crash in the hotspot compiler, in hotspot compiled code, or in native code. (To be clear, many "crashes" of CF are not of the sort that will create such logs, so again, it's only certain kinds of crashes that will lead to them.)

What matters most, for this post, is that again you look at least look for such logs in this folder (occurring around the time of some crash you're interested), and if there look in the logs for any reference to the phrase "outofmemory". Of course, there may be such PID logs that don't refer to "outofmemory", but such crashes are beyond the scope of this post. And while the pid logs have lots of information in then, explaining all that is also beyond the scope of this entry.

The point here (of this section of this post) is that when you have a crash and you suspect it's a memory issue (or if you don't know the cause and want to learn more about what it may have been), you want to look in these two log directories mentioned above. Many never do, and this is part of why they end up chasing their tails, going instead on gut feelings or trying out various alternative "solutions". I say instead: find the diagnostic info, and act on it.

Searching through the log files (console or pid logs), the easy way

But rather than "look" at all these logs in these directories, one at a time, I suggest instead that you automate the process and search them (I was tempted to say "stop, drop, and mole" in my quip above, since you're "ferreting" through the logs, but that seemed a stretch.)

If you're on *nix, I don't need to give you any more info on how to search the files. Just grep it and rip it. :-)

If you're on Windows, though, you'll perhaps be tempted to think I'm referring to the built-in Windows Search tool to search the directory. I am not. Indeed, let me plead, for the sake of all things decent, please use something better. It's just not a completely reliable (or fast) tool.

I have blogged about a wonderful free alternative, called FileLocator Lite. Use that instead. (If you have another tool or editor you favor, that's fine. No need to bring that up here. I recognize those other options in the other blog entry.)

The beauty of FLL is that once installed (which is fast, itself), if you right-click on the log directory (or any directory) and choose "FileLocator Lite" from the menu, which will open the FLL UI. You can then just put the string outofmemory in the search box and hit enter (or click the "start" button in the UI). You could use *.log to the files searched, though since this directory is so small it's not critical.

In just moments it will show any found files in the lower left pane. You can sort the list by the "modified" column, to focus on files from around the time of your crash.

Then, here's the real beauty of this file search tool over others: to look inside the found file(s), you don't need to double-click the files to open them. Just single-click each file, and in the right pane it shows any lines in the found file which had the string that was searched. Brilliant, and again, a really fast way to find things.

So we're using this to find if any files in either of the folders above has any outofmemory (oom) errors. And if there are any such files, then we're looking at the occurrences of the oom errors within each log.

Don't stop at the last outofmemory error before a crash

This feature of the File Locator Lite tool, to see all the lines in the file with the given string, is especially useful in this case, because when searching for outofmemory errors, you also want to be able to quickly see the time for *all* the error messages you may find.

And you *do not* want to focus solely on the last error prior to the crash (or the slowdown, that made you want to restart CF).

(I should add that when it comes to the console logs (as opposed to the PID logs), sometimes those found oom error lines may not have a date/time stamp, to help you readily assess if the error is occurring at the time you were interested in. You may need to go ahead and open the files with an editor and search for the oom string, and then look at other nearby lines to find a timestamp.)

Once you find one (or more) preceding the time of the crash, you want to look for any occurring prior to it. It may be that the problem started several minutes before the crash (or your restarting CF). Further, it may be that the outofmemory error just prior to the crash is different from the one that started things out.

The point here is that you want to find out a) if there WERE any outofmemory errors around the time of your crash of interest, and then b) what information if any appears in the logs around that time. We'll then use that to proceed in evaluating the nature of your situation.

And of course it could be that there WERE no outofmemory errors in the logs, which would indicate that your problem was NOT likely an outofmemory error (or not a heap error after all). We would need to proceed to doing more investigation and assessment of whatever diagnostics you DO have.

Step 1 down, 5 more to go

OK, that's step 1 in determining whether memory problems are really at all what they seem. As I mentioned at the outset, the planned parts in the series are:

• Step 1: Determine if indeed you are getting "outofmemory" errors (this entry)
• Step 2: Realize that having high memory usage is not necessarily a problem (entry to come)
• Step 3: Realize that OutOfMemory does not necessarily mean "out of heap" (entry to come)
• Step 4: Diagnose why you really are running out of heap (if you are) (entry to come)
• Step 5: Realize that CF is maybe suffering because you set the heap too large (entry to come)
• Step 6: If CF is hanging up but NOT due to memory, what could it be? (entry to come)

After I publish them, I'll update the lists here to link to them.

As always, I look forward to your feedback (pro, con, or indifferent).

Note: This blog post is from 2009. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

Well, there are several ways you can watch them, as this entry will discuss, and some may be better suited to the job than others. It can be very interesting to discover what's going on, especially if you're having any suspected performance problems which you think may be related to ORM processing (or just if you wonder what all it does for you).

As for spying on the SQL, of course ORM support is just a different way that the CFML engine (through the ORM framework) sends SQL to a database via a regular DSN, just like any other request, so there's nothing really "tricky" about this. It's just about realizing that while you don't write the SQL yourself, it's still generated by the CFML engine/ORM framework, and you may not realize/consider the available tools which can spy on it, just like any other DB processing from within CF. Indeed, some people may not even realize how many options exist to spy on JDBC interactions from their CFML engine to the database engine.

The good news is that there are several approaches, some included in CF (some depending on the edition), and some available separately which would work in any edition of CF or the other CFML engines (Open BlueDragon, Railo, etc.), and with any of the ORM frameworks. And again, some may be better than others for certain challenges.

(FWIW, besides the aforementioned Transfer and Reactor, there are still other ORM solutions for CFML, which I mention in my CF411 list as CFML ORM Frameworks. Indeed, note that you can run Hibernate on CF prior to CF9, if you want to. This is a recovery of a blog entry that no longer exists, recovered via archive.org.)

Built-in ORM Logging Option

First, note that for those using CF9+ ORM, there is indeed a built-in option in the CF ORM setup where one can enable logging, settable in the application.cfc: see the this.ormsettings option and its available key/value pair, logSQL="true".

There are several resources where you can learn more on that (and a related log4j property file approach to logging this). Besides the CF9 docs page on the ORM settings, there is also a blog entry by Adobe engineer Rupesh Kumar.

The default is to log this information to the console, but you can manipulate those log4j settings to tell it to use a file (see the links above). Even so, this will result in quite a lot of data being logged, which you will then need to connect back to your specific requests. The following approaches may be preferable.

Using FusionReactor or SeeFusion

Users of any CF edition (6+) or any CFML engine (Railo, OpenBD, or BD 7+) can use tools like SeeFusion and FusionReactor, which have always had the ability to monitor database interactions by "wrapping" the datasource to be monitored. FusionReactor engineer John Hawksley has posted a recent article specifically on monitoring CF9's ORM interaction, in the FR Devnet site, Using FusionReactor's JDBC Driver Wrapper With ColdFusion 9 ORM. Its concepts would apply to any ORM, of course.

Similarly, I've written generically about FusionReactor's database monitoring feature in What is the FusionReactor datasource monitoring feature? Why would I use it? Powerful stuff. As I point out in that article, the concepts discussed apply as well to SeeFusion's ability to monitor queries by wrapping datasources.

That said, it's worth noting that FusionReactor does have a couple of advantages, in that it provides for the display of all queries for a given request (while viewing the details of that request), whereas SeeFusion only lets you see the slowest query in a given request. FusionReactor also provides a separately available display of all the slowest queries (across all requests). It also logs every query (connecting it to a given request as well), while SeeFusion (Enterprise, at least) can also log the slowest queries to a database.

And note that both of these track any requests coming out of CF, not just those associated with a given request. So if there is ORM SQL that is associated with the startup of CF, that's tracked too. (And for those aware of issues with CF's Client Variables, such DB activity is also tracked, even that done by the hourly purge, which takes place on a background, non-jrpp thread.)

CF Enterprise Server Monitor

Those running CF 8 or 9 (Enterprise only) will find that its available Server Monitor does offer built-in monitoring of the SQL executed against CF datasources, at least, as long as you enable "Start Profiling" (which also enables other features, and overhead, as well). In this way, the Enterprise Server Monitor can monitor database interactivity, including ORM interactions.

Unlike FusionReactor (and like SeeFusion), it focuses only on showing queries that exceed certain limits, and at that it shows them only in a "Slowest Queries" interface, tracking the slowest queries among all requests. The CF Enterprise Server Monitor also has no logging ability at all.

Being able to see every single DB interaction for a given request (or across all requests) may be all the more interesting for discovering/observing what's happening with ORM interactivity.

Another alternative CF feature

Still another little-known feature for spying on JDBC interactions in CF is by way of the JDBC "spy" feature, which does in fact allow logging of all JDBC interactions mde from within CF. This feature was first enabled by way of the DataDirect 3.5 driver update which was made available (as an optional upgrade for 6 and 7) in the CF 7.02 timeframe. I wrote about the Spy feature back back in Aug 2006.

Since then, CF 8 (and now 9) offer it instead as a new "log activity" option in the "advanced settings" for a datasource definition in the CF Admin (which is disabled by default). I pointed this out in another entry from 2007 as one of many easily missed changes for the CF 8 Admin.

This "log activity" output is not as easy to interpret as FusionReactor's logs, and can indeed be voluminous (moreso than FR's), so be careful. Anyway, it's one of the several ways you can monitor JDBC interactions between CFML and your DB engine. Again, any of these may be useful for monitoring any of your CFML/database interactions.

Generic DB Monitoring tools

Indeed, it's worth noting finally that while the focus here has been watching the DB interaction from CF (and the ORM framework) to the database (by watching the JDBC traffic going out of CF and returning), you could just as well watch the DB interactivity from the DB's perspective instead (watching it coming and and being returned).

There are many tools that can monitor database processing, available for each of the major databases (free and commercial). I list several such tools in one of my CF411 section, Database/SQL Monitoring Tools.

Hope all that's helpful, whether you use ORM or not.

Note: This blog post is from 2009. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.

(If you're not familiar with the value of thread dumps or stack traces, read on. The resources I point to get help you to appreciate their usefulness.)

[....Continue Reading....]