[Looking for Charlie's main web site?]

Spying on ORM database interactions: Hibernate, Transfer, etc. on any CFML engine

As people use CF9's ORM feature (or other ORMs like Transfer and Reactor, or indeed Hibernate, on any CFML engine), they may be left wondering what sort of SQL interactions happen "under the covers" between the ORM framework and the database engine. Well, there are several ways you can watch them, as this entry will discuss, and it may be interesting to discover what's going on.

Of course, ORM support is just a different way that the CFML engine (through the ORM framework) sends SQL to a database via a regular DSN, just like any other request, so there's nothing really "tricky" about this. It's just about realizing that while you don't write the SQL yourself, it's still generated by the CFML engine/ORM framework, and you may not realize/consider the available tools which can spy on it, just like any other DB processing from within CF. Indeed, some people may not even realize how many options exist to spy on JDBC interactions from their CFML engine to the database engine.

The good news is that there are several approaches, some included in CF (some depending on the edition), and some available separately which would work in any edition of CF or the other CFML engines (Open BlueDragon, Railo, etc.), and with any of the ORM frameworks.

(FWIW, besides the aforementioned Transfer and Reactor, there are still others, which I mention in my CF411 list as CFML ORM Frameworks. Indeed, note that you can run Hibernate on CF prior to CF9.)

Built-in ORM Logging Option

Update: After posting this entry, I learned that there's yet another option, so I'm adding it here as the first to consider. There is a built-in option in the CF ORM setup where one can enable logging, settable in the application.cfc: see the this.ormsettings option and its available key/value pair, logSQL="true".

There are several resources where you can learn more on that. Besides the CF9 docs page on the ORM settings, there is also a blog entry by Adobe engineer Rupesh Kumar. I've not yet used it, so don't know how it compares to the other options below. I will leave them for all to consider.

Using FusionReactor or SeeFusion

Users of any CF edition (6+) or CFML engine can use tools like SeeFusion and FusionReactor, which have always had the ability to monitor database interactions by "wrapping" the datasource to be monitored. FusionReactor engineer John Hawksley has posted a recent article specifically on monitoring CF9's ORM interaction, in the FR Devnet site, Using FusionReactor's JDBC Driver Wrapper With ColdFusion 9 ORM. Its concepts would apply to any ORM, of course.

Similarly, I've written generically about its database monitoring feature in What is the FusionReactor datasource monitoring feature? Why would I use it? Powerful stuff. As I point out in that article, the concepts discussed apply as well to SeeFusion's ability to monitor queries by wrapping datasources.

CF Enterprise, with a caveat

I also mention in that article that those running CF 8 (or now 9) Enterprise will find that it has built-in wrapping of datasources, such that its Enterprise Server Monitor reports database interactivity, so that can also monitor the ORM interactions.

A key difference for FR and SeeFusion over that, though, is that while they all have means to highlight queries that exceed certain limits, only FR and SeeFusion have the built-in ability to log every DB interaction (FR to text logs and SF to a database, and they both do so with low overhead). The CF Enterprise Server Monitor has no logging ability at all, and while you can set its interface to lower the threshold for what active queries to show, it has no means to show queries which have executed in the past.

It's just that it can be useful sometimes to see every single DB interaction for a request (or all requests), and it may be all the more interesting for discovering/observing what's happening with ORM interactivity.

Another alternative CF feature

Still another little-known feature for spying on JDBC interactions in CF is by way of the JDBC "spy" feature, which does in fact allow logging of all JDBC interactions mde from within CF. This feature was first enabled by way of the DataDirect 3.5 driver update which was made available (as an optional upgrade for 6 and 7) in the CF 7.02 timeframe. I wrote about the Spy feature back back in Aug 2006.

Since then, CF 8 (and now 9) offer it instead as a new "log activity" option in the "advanced settings" for a datasource definition in the CF Admin (which is disabled by default). I pointed this out in another entry from 2007 as one of many easily missed changes for the CF 8 Admin.

This "log activity" output is not as easy to interpret as FusionReactor's logs, and can indeed be voluminous (moreso than FR's), so be careful. Anyway, it's one of the several ways you can monitor JDBC interactions between CFML and your DB engine. Again, any of these may be useful for monitoring any of your CFML/database interactions.

Generic DB Monitoring tools

Indeed, it's worth noting finally that while the focus here has been watching the DB interaction from CF (and the ORM framework) to the database (by watching the JDBC traffic going out of CF and returning), you could just as well watch the DB interactivity from the DB's perspective instead (watching it coming and and being returned).

There are many tools that can monitor database processing, available for each of the major databases (free and commercial). I list several such tools in one of my CF411 section, Database/SQL Monitoring Tools.

Hope all that's helpful, whether you use ORM or not.

Determining Your JDBC Driver Version: Pick from 3 ways

Following onto my 2 previous entries related to updating JDBC drivers in CFMX 6 and 7, some readers may appreciate a little more help on the matter of determining just what version of the drivers they do now have installed.

This news has been shared in various blog entries over the past couple of years. I'd like to reprise them here for your edification. Just pay close attention to whether you mean to have them look at CFMX or CFMX 7. They each point to a directory such as C:\CFusionMX\lib or C:\CFusionMX7\lib. Make sure you use the value appropriate to your configuration:

Hope that helps someone as much as it did me.

More on the new SpyAttributes feature of the"new" 3.5 JDBC drivers for CFMX

In my last entry, I mentioned briefly the new "JDBC Spy" feature that's available in the new 3.5 JDBC drivers for CFMX. I want to point out some more info for those interested in investigating the Spy feature further.

First, notice that there is a site at the DataDirect site (from whom Adobe purchases the JDBC drivers):

http://media.datadirect.com/download/docs/jdbc/alljdbc/reference/spy.html

It discusses some additional attributes. And still more useful info is offered at another page on their site:

http://media.datadirect.com/download/docs/jdbc/alljdbc/reference/diagnostics2.html

This actually shows what the log of info would look like, if you want to decide first if its info is worth the bother setting up.

This other page also talks about ways to turn the spy logging on and off at runtime, using Java method calls. Perhaps someone with more familiarity with the underlying Java classes for these drivers can translate the code there into something we can use in CFML. (It's late, and I don't have the energy.)

As such, be aware that the spy feature could generate a lot of data, so you may want to consider creating two datasource for each DB you may want to monitor. Either rename them to enable whichever you want to be in effect, or change if your code lets you specify a DSN via a variable, you can change the value to the "spying" version when you need it.

I did test and determined that also you can just add/remove the connection string from the end of the JDBC URL (see the technote for more), and immediately the logging stopped against the datasource. As a trick, I stuck the connectionstring value into the "description" field of the datasource definition, to keep it readily accessible if I wanted to add it back. (See my previous entry for a warning about putting it into the ConnectionString field instead.)

Finally, I should clarify that this Spy feature is not to be confused with the much older (and more powerful) p6Spy project, which has gone from open source to commercial to kind of stagnant. It still has great use, and in fact has a useful related SQL Profiler that's worth looking at. Still more at http://www.p6spy.com/, and maybe I'll write more about it later.

Did you see the 3.5 JDBC update for CFMX 6.1 and 7: Anyone tried it? Seen the new features it adds

Had you noticed the JDBC updater at http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=42dcb10a? It applies updates to both the Merant DataDirect and Sequelink (ODBC) drivers.

I have a few questions/observations for others who have run it or may consider it, and they apply to 7.02 and 6.1 users, such as whether you need to apply it manually and about the various new features it adds, like integrated NT authentication for SQL Server, and a new JDBC Spy feature for any DBMS? More on them in a moment.

Do you need to update this for 7.02? 6.1?

First, do you need to apply the update manually if you're running 7.02, or even one of the later cumulative hotfixes (CHFs)? It doesn't say specifically. While it may seem we should presume it was included in 7.02, I think we should not. Some features it adds (below) are not in the C:\CFusionMX7\lib\macromedia_drivers.jar (whose date is also from 2004, at least on my 7.02 install.)

(UPDATE: In fact, it appears that the updated drivers were NOT in fact included in 7.02, or any CHF. First, as mentioned in the comments below a discussion on the Adobe Forums confirmed that these drivers were NOT included in 7.02. Also, note that CHF 1 for 7.02 lists the link for the driver updates as something you need to do yourself, under "additional information". There's also no mention of the JDBC drivers in the technotes for 7.02 CHF's 2 or 3, so it seems safe to conclude that 7.02 users do need to apply the update individually to get the updated JDBC drivers.

But better still, if you have 7.02 installed and wonder what version you have, you can find out for yourself. Check out this later entry of mine which helps you determine what version of the DB drivers you are indeed now running.)

As for 6.1, it's not clear but do we need to apply it if we have the latest 6.1 updater (6,1,0,83762 )? Again that build's macromedia_drivers.jar also lacks some of the files added by this updater, so I would assume so. I applied the update to my CFMX 6.1 updater release including also the important updating of the Sequelink ODBC Server as indicated in the technote. Don't miss that, if you're running on 6.1 (not needed on 7). After restarting, things are working.

Warning for those with 6.1 and 5 installed

I should note that I did find that my CF5 server (also running on this same dev box) suddenly, and for the first time ever since this box was created a few months ago, started raging with a persistent 55% cpu utilization for ntconsolejava.exe. I certainly would have assumed that the problem was in my CFMX 6.1 services, but by process of elimination it was the stopping of "ColdFusion Management Repository Server" (from CF5) that did the trick. That runs a JRun instance in the CF5 server.

I can't see how this would be connected, but I point it out in case it happens to others. I don't know what to do about it. I may just stop my CF5 server until I ever need it. I suppose I could also just set the management service to manual. I don't recall what the mgt service did in CF5. A quick review of the docs shows it had to do with archiving and admin reporting.

New Features in the Update Driver

But perhaps more important, as for the features it adds, did anyone notice the "sql server nt authentication" it enables? It seems that would solve a useful problem, but I could not get it to work. I followed all the steps including extracting the DLL (it's in the zip) and creating an "other" datasource type with the provided "authenticationmethod" in the JDBC URL. Has it worked for anyone?

Note also the interesting new spy feature, which can provide some useful diagnostics (think SQL Profiler, but for any DB and all the info tracked in a log file on the CF server). I was able to get it working successfully with a SQL Server database (using the "other datasource type" as explained in the technote), but when I tried to add it to an odbc dsn against an Access DB, using the connection string approach listed, nothing happened (no spy log created, until I used the "other datasource approach).

Note as well that the updates offer various fixes for SQL Server, Oracle, and Informix. It's also an update that pure JRun 4 users can/should run. I will point out that one blogger has found a change in behavior regarding multiple statements on a single SQL line: http://www.feed-squirrel.com/blog/index.cfm/2006/6/8/CREATE-TABLE-from-CF-7.

See the technote (linked at the top here) to learn more. I've also opened a question in the Adobe forums with this same info: http://www.adobe.com/cfusion/webforums/forum/messageview.cfm?catid=143&threadid=1180864&enterthread=y, as I can't assume as many people read my blog as will read that.

I'll be curious to hear (here or there) what others have experienced and what they think of these new features.

I should warn that the technote indicates a couple of times that you should not try to put these new connection string values into the "connection string" field of a DSN, instead creating an "other" datasource type and providing them on the JDBC URL for that DSN. When I made the mistake, I found that I could not query any ODBC datasources nor could I even start the CFMX Admin or get debugging on any page. All complained that the "DataSource service is not available.". I found no neo-query.bak to recover from, so was quite stuck. A careful analysis of the new-query file found that inappropriate CRLF character codes had made their way into the file (As 0a and 0d values that made for bad xml). I removed them (not trivial to do correctly) to get things back in order, so FOREWARNED IS FOREARMED.

BlogCFC was created by Raymond Camden. This blog is running version 5.005.

Managed Hosting Services provided by
http://www.edgewebhosting.net/
Managed Dedicated Hosting