[Looking for Charlie's main web site?]

CF2016: What's deprecated and/or 'no longer supported' (note: nothing 'removed')

Continuing a series of posts I started last week on ColdFusion (2016 Release), aka CF 2016, I'd like to highlight the items that Adobe has chosen to deprecate and/or declare as "no longer supported", as of CF 2016.

[....Continue Reading....]

Sending HTTP headers in a CFHTTP request? Name them correctly

If you ever try to use CFHTTPPARAM (inside a CFHTTP) to set HTTP headers for the request you're calling, be aware that you need to be careful to specify the name of the headers as they're known in the HTTP specification. Don't be misled by what you see in a CFML dump of the CGI scope.

(This is a reprisal and update of an old blog entry I'd done back in 2003, on a blog site that will soon no longer exist. I'll be reprising a few such blog entries in coming days/weeks, to keep them around for posterity [and to save some having to dig for them in the archive.org site] since often the info offered then may be just as valuable now. I hope that in time these new versions would come up if people do searches that would have found the old entry.)

So, about these http headers, while CFML exposes them in a dump of the CGI scope, such as the user-agent field which shows up as cgi.http_user_agent, the issue is that you would not use that name, nor even "user_agent", when specifying it in a CFHTTPPARAM. The proper way to pass the user agent in a CFHTTPPARAM would be as follows:

[....Continue Reading....]

Struggling with using the XML features of CFML? Here's where to learn more

Are you struggling with using the XML features in CFML? Or do you help people who are?

It's not something most people use often, and there is far more to the feature set than meets the eye. If you don't use the right resources to help you, you could waste time trying to piece together a solution using only scant examples you may find.

In this entry, I'll point to several resources you should consider to help get you quickly up to speed in using--and appreciating--the power of the XML features in CFML.

Even if you don't need them today, you (or someone you know) may need them some day, so keep this in mind. (Or if you find this page doing a google search some years down the road, drop a note to let me know if it helped then!)

Background

[....Continue Reading....]

Bug with Query of Query: using LIKE incorrectly finds records with null values

Someone reported on a mailing list that they'd noticed a mistake in the results of a Query or Query operation when using a LIKE. It finds a record whose value for the column is null, which is of course wrong (that doesn't match the LIKE pattern--indeed a null never equals or doesn't equal anything).

The problem can be worked around by adding "and column is not null", but it really surprised me (and him, obviously) to discover the bug. Maybe someone else has written of it before.

But I tested it in 8.01 and can confirm it's still an issue. While I've recommended that he file it as a bug, I wanted to share it here until it's resolved. Hope this may help someone. (If we're missing something, feel free to comment.)

Here's some code you can run which uses the example app database that comes with CF. See the comments for what's wrong:

<CFQUERY NAME="demoq" DATASOURCE="cfdocexamples">
   select * from employees
</CFQUERY>

<!--- note that by default, all records in the example employees table have values in all columns --->

<cfdump var="#demoq#" label="before insert">

<!--- insert a record with only one column (causing the others to be null)
--->

<CFQUERY NAME="insert" DATASOURCE="cfdocexamples" >
   insert into employees (firstname) values ('charlie') </CFQUERY>

<CFQUERY NAME="test2" DATASOURCE="cfdocexamples">
   select * from employees
</CFQUERY>

<cfdump var="#test2#" label="after insert">

<!--- do a Q of Q of that query, for a column which would have a null for that last added records, so you'd expect it not to find it --->

<cfquery dbtype="query" name="test3">
   select * from test2
   where department like 'Sales%'
</cfquery>

<!--- that record with the null value still shows up! --->
<cfdump var="#test3#" label="dump of Q of Q">

Again, the problem is that the record with the null value still shows up in the Q of Q result, and it definitely should not. A solution for now is that if you change the Where clause to "department is not null and department like 'Sales%'", then it properly does not show the null-valued record(s) anymore.

(If anyone plays with this on their own code, note as well that Q of Q is case-sensitive, so no records would be found at all if you mistakenly used 'sales%' rather than 'Sales%', above.)

CFMAIL being detected as spam? Some solutions for CF 6, 7, and 8

Someone asked on a list about how to resolve the problem where messages sent via CFMAIL have a message-id value that can cause mail to be flagged as spam. I mentioned in an earlier entry how to solve this in CF8. In this entry, I offer a couple of solutions for CF 6 and 7.

To backup and explain the problem, some have lamented that in CFMX 6 and 7, CFMAIL used the name of your server where CF is installed, which might be something like "server1" or "bingo", as in:

Message-ID: <23070863.1197039960343.JavaMail.SYSTEM@Server1>

This might cause a recipient mail server to flag the mail as spam, if the mail server was a different domain name (like "yourcompany.com"). The bummer was that many found no way to fix this. Sure, in CF5 you could set it in a CFMAILPARAM to set a mail header, but CF 6 and above ignores that.

The good news for those on CF 8 is that a change makes the problem go away, which I talked about in a blog entry back in December.

But for those still on 6 or 7, I'd mentioned that there's an available code or (preferably) configuration fix to resolve the problem. Apologies for not posting it at the time, so here it is. I also explain what the issue is all about, for those not aware of it.

Simple CF5 solution doesn't work in 6, 7, or 8

First, let me point out that some may propose that one can just use CFMAILPARAM to set that message-id header, as was mentioned in this FAQ:

http://www.developer.be/index.cfm/fuseaction/faqDetail/FaqId/201.htm

Unfortunately, that's a very old FAQ and it no longer applies after CF5.

CFMX 6 (and 7 and 8) ignore that header you set and set the message-id themselves to the name of the physical server from which the mail's sent. Here's an example, from my laptop (which is named charlied620):

Message-ID: <23070863.1197039960343.JavaMail.SYSTEM@CharlieD620>

On your server, it could be that the problem is that the server is a real domain name, but it's not the same name as the SMTP server through which you're sending email via CFMAIL, and some servers won't let mail through with FROM addresses having a different domain name.

Why doesn't CF just use the SERVER set in CFMAIL or the CF Admin?

You might wish/expect it would just use name of the mail server as specified in the CF Admin mail server setting or the CFMAIL SERVER attribute, but it does not (at least, not prior to CF8, as discussed in my earlier entry.)

But while researching this problem for the person, I found this, where a very compelling solution was proposed. I tested it, and it does indeed work.

It turns out that you can get CFMAIL to use a specific mail server, either of 2 ways, for CF 6, 7, or 8 (again, on CF8, the easier solution is that I discussed in the earlier entry). Ken Smith of Adobe offered it also in this Adobe forum thread. If you make this change, either server-wide or per page/app, you will now find that the CFMAIL sets the message-id to use a desired servername.

Solution for 6, 7, or 8: Configuring it at CF Startup

If you're on a server where you want to change this for all CFMAIL (and javamail) done from within CF, you can change the JVM.config for your server to add this to the java.args line:

-Dmail.host=desiredservername

That java.args line is one long line. Don't introduce any line breaks. You need to restart the server after making that change. As always, when editing the JVM.config (in the runtim/bin directory of CF), make a backup first, because if you get it wrong, your server won't restart. (Or leave the file open and be prepared to do an Undo in your editor.)

But if you don't want to (or can't) mess with the startup file, or more important, if you are on a server where each app may need to do its own setting, you're not out of luck.

Solution for 6, 7, or 8: Changing it programatically

You can also make the needed change on the fly, programatically. You could issue this set of code just before your CFMAIL:

<cfscript>
sysObj = CreateObject("java", "java.lang.System");
sysObj.setProperty("mail.host", "desiredservername");
</cfscript>

I tried this per-page setting in in 6 and 7 and it worked fine. For CF8, the code will run, but it won't affect the CFMAIL for reasons I explain in the next section. But there's a real problem with this approach, if you're on a shared CF server. You're still setting the value for the entire CF server, but you're just doing it on the fly. It has at least a few problems you need to think about:

  • you'll affect all who use CFMAIL on this server (at least in 6, and 7, since CF8 ignores this, as discussed below)
  • you'll also affect anyone who uses Javamail on this instance of CF
  • and if someone else on the server issue the same code with a different server name, that of course would override your setting

There are a few things you can do to mitigate the problem, but they're not entirely perfect.

First, you could get the current mail.host value (in case it's different), then change it to what you want, and then change it back. You could do that with this code:

<cfscript>
   sysObj = CreateObject("java", "java.lang.System");
   // if there is no mail.host set already, this next variable simply won't be created. need to know that for later.    oldmailserver = sysObj.getProperty("mail.host");
</cfscript>

Note that if the mail.host property has not been set in the startup config or by someone else running such code, then the getproperty will return nothing, and the oldmailserver variable literally will not be created (a curiosity of working with some java methods).

Once you have the oldmailserver (or know that it did not exist), you can do the set of the property and the cfmail as above, then you could set it back with the following:

<cfscript>
// reverse the setting of the mail.host, so as not to affect others on this server if (not isdefined("oldmailserver")) {
   // if there was no previously set mail.host, remove the property
   sysObj.getProperties().remove("mail.host");
}
else{
   // set it back to what it was before
   sysObj.setProperty("mail.host", oldmailserver);
}
</cfscript>

There's still a problem. Because of CF's multi-threaded nature, it's entirely possible that between your setting the host and doing your CFMAIL, someone else could also set the mail server to something different. (This is called a "race condition".)

Now, Mr. Smith in the threads above suggested that you could wrap the code doing the change and the CFMAIL in a named lock, but that's an incomplete solution. It will prevent other other code (that ALSO does the same named lock) from running until yours is complete, but it won't help if others do the set of the property without bothering to use the same named lock (or use a differently named lock).

That's a frequent misconception about locks. They don't prevent other code doing what you're doing in the lock: they only tell other code that IS using the same lock not to run while this lock is held--and even then, only if you use an EXCLUSIVE lock, which he didn't indicate.

So the bottom line is that as useful as the feature is to set the property dynamically, it's fraught with peril in an environment where multiple apps may try to use the code. Only if you can guarantee that all use the same named lock will you be able to protect against this problem is held up while you have the value changed:

<cflock name="setmail-servername" timeout="10" type="EXCLUSIVE">

   <cfscript>
   sysObj = CreateObject("java", "java.lang.System");
   // if there is no mail.host set already, this next variable simply won't be created. need to know that for later.    oldmailserver = sysObj.getProperty("mail.host");
   sysObj.setProperty("mail.host", "desiredserver");
   </cfscript>

   <cfmail ...>
   ...
   </cfmail>
   
   <cfscript>
   // reverse the setting of the mail.host, so as not to affect others on this server    if (not isdefined("oldmailserver")) {
      // if there was no previously set mail.host, remove the property
      sysObj.getProperties().remove("mail.host");
   }
   else{
      // set it back to what it was before
      sysObj.setProperty("mail.host", oldmailserver);
   }
   </cfscript>
</cflock>

Now, someone may propose that all this could be wrapped up into UDFs or CFC methods, and perhaps it could, but since you need to do the setting of the value, and the mail, and the resetting of the value, all within a CFLOCK, it would be challenging to wrap all this up into a generically callable method (unless you wanted to pass in as well all the CMAIL attribute values and the body). Just seems not worth it, since this is a pretty esoteric problem and solution. But others can comment if they feel differently.

Solution for CF8 is much easier

All this is obviated on CF8, because it now properly uses the name of the server specified in the CF Admin mail server setting (or CFMAIL SERVER attribute, which overrides the CF Admin setting.) This is AWESOME news for those challenged by this, and hasn't gotten much press.

Now, what was the caveat I mentioned above? Well, if you use the approach of setting the mail.host servername in the java property, CF8 no longer pays attention to that. It JUST uses the CF Admin mail server setting, or the CFMAIL SERVER attribute. So that code above "won't work". It will work, but it won't affect CFMAIL. But that was a hack to work around CF not honoring these other attributes. I'm not surprised (or bummed myself) to see that it no longer regards the mail.host property for CFMAIL.

(I should say I'm saying this as of CF8. I've not tested it on 8.0.1.)

Some related notes

If you add the SERVER attribute, you may need to add the PORT, Username, and Password as well.

Here's a little bonus tip, in case you try to use the CFMAIL SERVER attribute for the first time on an existing CFMAIL tag to check this out. Note that using the SERVER attribute on CFMAIL requires you then to specify any other attributes required for the mail connection such as USERNAME and PASSWORD (if needed) and PORT (if not 25). What I mean is that it will no longer pick up the values in the CF Admin. If you override the SERVER, then you override the other config settings as well and need to specify them.

I'm not getting the emails now. Where are they?

If you make a mistake in your setting of the mail server arguments, then CF will move the failed emails to the [coldfusion]/mail/Undelivr directory where CF is installed. They're just plain text files. You can open them to see what got set and perhaps can figure out why they failed. You may also find information in the [coldfusion]/logs/mail.log.

When I look at mail in the spool, it looks fine (if I use CFMAILPARAM). Why doesn't it get through?

Don't be misled. The email you generate in CFMAIL may look fine in the [coldfusion]/mail/spool directory, but when it gets sent to the mail server, CF will change that message-id (and some other headers). You really need to look at the email as it's RECEIVED. You can't even look at the message in the Undelvr folder as an indication. It doesn't have those added headers.

So how DO I observe the mail headers?

You need to receive the email and then look at its headers. I'll show you how to do that in Outlook and gmail in a moment. Let me point out that it can be very englightening to view the message headers: not only the message-id but possibly also other headers as well as messages that CF, your mail server, or the mail server of your recipient may have added, which may include indications of whether your email was detected to be spam (perhaps by tools like SpamAssassin).

In Outlook, you can use View>Options while reading an email, to see the values in the "Internet Headers" box of the window that opens. Scroll down in that to find the message-id and other headers.

In gmail, when you open the message, look to the top right of the pane showing your email, to the right of the indicator of the time the message was received. There's a drop-down box, with options like "reply" and "forward". Choose "show original". That will show the complete message including all the headers at the top.

Other CFMAIL alternatives/replacements

Finally, I'll point out that if you run into other problems with CFMAIL, there are always alternative CF mail server solutions, like those I list in the "CFMAIL replacement/enhancement tools" section of my "Tools to Consider for CFML developers" page.

Hope that helps some.

Stopping multiple form submissions with CF 7/8 and "submitonce" validation

What happens in your app if a user hits the submit button more than once before the form is processed, such as when the form didn't come back quickly enough, so the user submitted the form again? Would you be interested to know that CF has a feature to prevent them doing that? It's an often-missed hidden gem of CF7.

Have you ever considered this prospect of multiple form submissions? What could happen? It could cause multiple inserts to a database, or multiple charges to a card, or unexpected increases in some session variable counter, etc., which could be real trouble for you or your users.

It's a subject that comes up often in developer circles (even outside of CF). The good news is that there's a very simple solution in the available validate="submitonce" option of CFINPUT, which as of CF7 can be used for submit buttons. You'd use CFINPUT TYPE="submit" VALIDATE="submitonce" NAME="somename", as demonstrated in a complete code example below.

In this entry, I offer more info on the feature. I've not found too many other resources discussing it, so I hope this will help folks. There are some interesting challenges you should understand as well.

Problem already solved for you in Firefox--but do all your users use it?

For instance, if you've read about the feature and tried it on Firefox, you may have been surprised that you couldn't see what it did differently than a normal submit button. That's because it turns out Firefox already solves the problem itself, by preventing double-submission of a given form. It doesn't hurt to use the CF-based feature on FF. It just isn't needed. But you certainly still need the CF-based feature, though, if you may have users visiting your site with other browsers. (I can confirm that even IE 7 does not prevent double form submissions. Haven't tested Safari yet. Any takers want to report in the comments?)

Demonstrating the problem of multiple form submissions

It's not too difficult to demonstrate the problem of multiple form submissions. All you need is a means to track how many times the server-side form processing takes place for a given form's submission(s). In your real application, again, this may lead to multiple inserts in a database or multiple charges to a customer's credit card. To keep things simple, I'll demonstrate it using a session variable that's incremented on each form submission.

But then more important (to demonstrate the effect), we need the form processing to simulate "taking too long" so that a series of rapid submissions of a form will be able to be sent before the processing of the page "completes". We can do that in CF6+ by calling the Java thread sleep method, or in CF8 we can use the new sleep built-in function.

The example code is offered at the bottom here. Before you take a look at that, or try it out, note a few things.

Applies to form, not submit button

Besides the point above about Firefox being immune automatically to the multiple form submission problem, note that this multiple submission protection is enabled on the form itself, not on the submit button, so it does apply just as readily to forms submitted by pressing enter (when permitted by the browser) as by forms submitted by pressing the submit button.

The fact that it's enabled for the entire form (when the CFINPUT above is used) means also that you can't test this by having a "regular" submit button and a "protected" one in the same form. Again, it's not the submit button that gets the protection, but the form in which the CFINPUT type="submit" appears. You'll notice, therefore, that my demo code above uses 2 forms. (It's not at all important whether I use or don't use CFFORM in the "unprotected" form example. It just makes no difference.)

Does require use of CFFORM And Javascript

That makes a point though: the CFINPUT tag does indeed need to be used within a CFFORM tag, as shown in the second form. And further the feature is indeed relying on Javascript (generated by CF) to perform the multiple submit prevention. Neither should be a showstopper for most. You don't need to convert any other HTML tags within a form to their CFML CFFORM equivalents (like CFSELECT or CFTEXTAREA) just to enable the CFINPUT for form submission.

Some may want to point out that you don't need to use this particular approach to solve the problem: there are other Javascript-based approaches, as well as some that assert to work without JS. I'll leave it to commenters to mention them here, if interested.

My goal was to demonstrate the submitonce functionality, since it's not been discussed much. Do let me know if this was interesting to you.

Some final observations about my code example

I've provided comments in my code example code, about things that aren't related to the multiple submissions feature, but which may raise questions for some.

First, note that because I'm using sessions to demonstrate the feature, I've just gone ahead and put the CFAPPLICATION tag right into this template, so it doesn't matter where you put the code (won't be helped or hurt by an existing application.cfm or application.cfc, with respect to the session var created.)

As for the forms, note that I'm using self-posting forms. It doesn't matter if you do or don't for this multiple submission prevention feature to work.

And note that I don't use any means to force a filename into the form ACTION attribute, to make it post back to itself. If you leave the action empty, that makes the form self-posting. This is a legitimate HTML-specified use, if not widely known. No need to force the current file into it using #cgi.script_name# or the like.

And I determine if the form is submitted using a test for cgi.request_method being "post", which it will be on a form submission (versus it being a "get" when the page is first loaded). I don't use a test for whether the submit button is defined (a technique that is taught by some but will fail when the page is run on IE, if the user presses enter rather than the submit button, at least when there's only a single input field). Again, these choices have NO effect on the validity of the tests.

Finally, while I don't *need* to specify the empty action or post method on a CFFORM (since they're the default), I do it to avoid any confusion from those looking at the code and not familiar with that

The example code

You should be able to drop this code, verbatim, into any CFML page in any CF7 or 8 server to see the effect.

<cfapplication sessionmanagement="Yes" name="submitoncedemo">
<cfparam name="session.submitted" default="0">
<cfset interval=2>

<h4>Demonstration of New submitonce validation in CF7+</h4>

Try clicking each of the submit buttons below multiple times, in rapid succession, before the page returns from its #interval# second delay. Wait for that delay before noticing the count indicated after the 2nd form.
<p>
Notice how the "uncontrolled submit" will cause execution of multiple submissions (as tracked by a session variable). (The problem is not apparent in Firefox 1.5 and above, as it includes its own feature to prevent multiple form submissions. Try it on IE, though, even IE 7.)

<form name="test" action="" method="post">
<input type="Submit" value="Uncontrolled Submit">
<input type="button" onclick="location.href='<cfoutput>#cgi.script_name#</cfoutput>'" value="Reset Counter">
</form>
<!--- the use of no value for ACTION, to do a self-post of the form to itself, is legitimate HTML-specified functionality, if not widely known. No need to force the current file into it using #cgi.script_name# or the like. --->

<hr>
The "controlled submit" will not. Regardless of how many times you press it, it will only register a single increment in the session count, demonstrating that CF prevented the form processing from being executed more than once until the form processing page was completed.

<!--- This needs to be in a separate form from above, because the submitonce validation applies to the form, not the button. And while I don't *need* to specify the empty action or post method on a CFFORM (since they're the default), I do it to avoid any confusion from those looking at the code and not familair with that. --->
<cfform name="test2" action="" method="POST">
<cfinput type="Submit" validate="submitonce" name="submit" value="Controlled Submit">
<input type="button" onclick="location.href='<cfoutput>#cgi.script_name#</cfoutput>'" value="Reset Counter">
</cfform>

<cfif cgi.request_method is "get">
   <!--- if the form is requested the first time, or via the HREF below, reset the session variable. --->
   <cfset session.submitted=0>
<cfelseif cgi.request_method is "post">
   <!--- if the form is submitted, process it. (There are other ways to test form submissions, but this is a very good one for many reasons.) --->
   <!--- put thread to sleep for the number of seconds indicated above, to simulate a wait while form submission is being processed --->
   <cfset thread = createObject("java", "java.lang.Thread")>
   <cfset thread.sleep(javaCast("long", 1000*interval))>
   <!--- in CF8, could use sleep function:    <cfset sleep(interval)> --->

   <!--- yes, perhaps we should lock the sesion variable below, but it's not critical to help or hurt this demo--->
   <cfset session.submitted=session.submitted+1>
</cfif>

<cfoutput>
Submitted: #session.submitted# times.

<hr size="3" color="##000000">

<h4>Discussion</h4>
The form above has 2 submit buttons to demonstrate the new SubmitOnce validation in CF7+.
<p>
The first button, "Uncontrolled Submit", is a normal submit button (<input type="submit">).
<p>
The second button, "Controlled Submit", uses (<cfinput type="submit" validate="submitonce" name="somename">) which causes CF to prevent it being clicked twice to cause submission of a single form.
<p>
To demonstrate the effect, the action processing of this form is set to wait #interval# seconds before completing. It also outputs (using a session variable) how many times the button led to submission of a request to the form processing portion of the page.
<p>
Note that clicking the uncontrolled submit multiple times in rapid succession (on other than Firefox--haven't tested on Safari) will lead to multiple submissions. Even though you don't see the output of each submission (because each new submission starts a new request to the server), the increase in the session variable at a rate higher than just one per the #interval# second interval it takes for the form to complete. This demonstrates that multiple submissions are being caused.
<p>
Do the same with the "controlled submit" button, and that doesn't happen. It only ever increases by one per 3 second period.
<p>
</cfoutput>

How can I process Excel files in CFML? Let me count the ways

Most of us have seen over the years many requests and many ways to process (read or create) Excel (.xls) and other spreadsheet files. In fact, there have been so many different ways (some simple, some powerful; some new and some quite old) that I fear some may do a quick Google search and try the first thing they find. Sadly, what works for some may not work for all.

One place to list all the approaches

I don't recall ever seeing any single listing of all the possible ways to do CFML to Excel integration, so I decided to create one. I've created a new section in my Tools to Consider for CFML developers on the topic of "Excel File Processing Tools".

You'll see I've gathered a range of articles and tools from over the years showing how to do CF-Excel integration (both directions) any of several of the following ways (the links go to the section within my tools list on this topic):

  • You can easily create them using HTML tables and CSV generation (as well as read them using CSV)
  • You can create them with the more powerful XML-based approach
  • You can create and read them using Java-based APIs like POI, JExcel, and jXLS
  • You can create and read them using COM and now in CF8, .NET
  • You can read them using ODBC and JDBC drivers
  • Let's not forget also that you can create them using the Report Builder in CF7 and 8, which can output as Excel
  • And finally there's the old Excel Web Query feature for populating a spreadsheet from a CFML request

In the listing, I offer dozens of links to articles, blog entries, downloadable tools, any relevant CF docs, and more, all broken down by the approaches listed above.

Different Strokes for Different Folks

So you see, there are not only different solutions but different needs (create a spreadsheet CFML, read one into CFML, or populate one within Excel from a CFML request). It seems helpful to gather a single resource to organize them all and help people decide.

I need your feedback

So let me know what you think of all this. Did I leave out any other important approaches? I'm not claiming to "know it all". This is just a result of some research I did this morning (and/or recollection from past experiences). I'm sure I could have missed something. Certainly I know I'm not listing *all* possible references on each of the subjects. I just wanted to give folks something to start with, but I'm certainly open to adding any useful references or tools that I left out. Please leave a comment below.

Update: Indeed, already, in response to a comment, I've tried to make more clear the fact that I'm not just talking here about creating spreadsheets from CFML but was referring to reading them as well. That's why I chose "processing" in the title, and I referred several times to "reading them", but to make it stand out, I've bolded all the references to reading them.

More important, because it seems he had missed the 2 links to the lists where I offer more details, I've also changed the words in the list above to link into the discussion of the topic in the tools list. Hope that helps avoid people missing the real point of the listing. I was torn about just offering the listing itself as a blog entry and went with this approach instead of a summary that points to the details. I know some don't like really long blog entries. Also, listing them in the bigger tools list seemed helpful, but I may split it out into its own file. We'll see.

Not just about Excel or even Office

BTW, one may argue that I could have a generic section on processing all sorts of file types (or even just all kinds of MS Office) file types, but there are so many requests for Excel file processing specifically, and so many approaches/tools to suit those needs, that I just figured I'd start out with this and consider other filetypes later (and may rename the section then). It's also certainly true that most of the techniques/tools shown here can be used with any spreadsheet tool that reads xls files or processes CSV/HTML table files in a similar way (like OpenOffice).

Update

Again, note that since creating this blog entry, I may have updated the section on this in my CF411 tools list, "Excel File Processing Tools". Be sure to check that out to see if there are still more new options.

CF8 Hidden Gem: CFMAIL auto-generated message-id uses specified mail server name

Here's a neat hidden gem for CF8 that will delight some: CFMAIL now uses the mail server name you specify in CFMAIL SERVER attribute or the CF Admin mail setup page, in creating the message-id header that's generated when your mail is sent.

Some have lamented that in CFMX 6 and 7, it instead used the name of your server where CF is installed, which might be something like "server1" or "bingo", as in:

Message-ID: <23070863.1197039960343.JavaMail.SYSTEM@Server1>

This might cause a recipient mail server to flag the mail as spam, if the mail server was a different domain name (like "yourcompany.com"). The bummer was that many found no way to fix this. Sure, in CF5 you could set it in a CFMAILPARAM to set a mail header, but CF 6 and above ignores that.

So the good news is that CF8 now does use that CF Admin or CFMAIL SERVER value to create the message-id. If I use CFMAIL SERVER="mail.carehart.org"..., for instance, I might get something like:

What about CF 6 or 7?

But what if you're on 6 or 7? Well, there's a solution for you, too, by way of a tweak in the CF startup script (jvm.config) or (less preferably) in code. Since the change for CF 6 or 7 deserves some explanation to do it justice, I'll take that up in a part 2 message next. I'll also lay out the whole problem with the auto-generated message-id and why it's a concern for some.

For now, I just wanted to get this word out to those who understand the problem already that CF8 solves it. Woo hoo.

CF and OO and blogs, oh my (a flush of CF-OO blog tutorials)

Step right up ladies and gentlemen, get your CF-OO tutorials here. :-)

I've noticed a sudden flush of multipart CF-OO tutorial blog series being done, to help people make the leap to using CFML in a more Object-Oriented way. I thought I'd point them out for those who may have missed them:

Don't know if there's a connection between the rather simultaneous releases, or if it's just pure coincidence. Charlie clarified that there indeed was no connection between the two.

And for those who want to get their CF-OO freak on, there are still more resources on the topic.

And there are certainly many more. I'd welcome hearing of those to add to the list. And there may well be more multi-part CF-OO blog series' created before or after now. Feel free to make note in the comments here and I'll update the list for posterity.

Maybe we're starting to reach a tipping point. Of course, there are some who will lament a push to OO (too complicated for them), while others will say you're a hopeless loser if you don't get with the program.

I don't want to debate that here, nor do I really want to see it blow up in my comments, please! :-) Feel free, instead, to point people to where else it may have taken place so that they can evaluate the discussions there instead.

Resources for getting a much greater understanding of the SQL Server query plan/procedure cache

Ever wanted to understand the SQL Server query plan/procedure cache better? It can be vital to good performance of SQL statements, especially with respect to the concept of "bind variables" or "parameterized queries" (using CFQUERYPARAM in a CFML context, or Parameters.Add() in a C# context, among other ways).

Update: If you may wonder why any of this is important, see some subsequent talks I gave on the topic, that give far more context on the problem. While you rarely hear about it from others, it is a source of potentially significant problems so is worth understanding more completely. See the "My Own Talks" section at the bottom of this entry.

Yet we so often just kind of assume the database will do what's best. Or we blindly use (or recommend use of) bind variables without fully understanding why, nor understanding some implications when it may not always be such a good idea.

And did you know that if you can't (or don't) change your code to cause parameterization, did you know that the database can do it for you, either automatically in some simple cases, or by way of an enforced parameterization, either for an entire table of for queries that meet certain "plan guides"? There are many implications to understand in all this.

And how do you track the plan cache (using DMVs, the profiler, perfmon)? And how does it work (allocation of memory, flushing the cache automatically or manually)?

Fortunately, I've found many great resources to help you understand, and I've not found them listed (all together) anywhere else yet.

The info applies to 2005, 2000, and 7, though some aspects may differ, as the resources indicate. (There are even substantive changes in 2005 SP2 that are important to note over the RTM and SP1 releases.)

Huge Plan Caching Article Series by MS Engineers

First, I'll point out that some MS engineers have put together a long and very resourceful series of article-length blog posts on the topic, with explanations, code, demos, troubleshooting techniques, and lots more.

They do kindly offer a "table of contents" page listing all the topics covered, with bullet points about topics within each entry.

Sadly, that page doesn't offer URLs to the articles, nor have any that I found there, nor do the articles link to each other. You can dig around and find the URLs, even finding a category page that lists them in rather random order. But I've looked around the web and can't find any page that lists them all with their links, in order, so here you go:

  1. Structure of the Plan Cache and Types of Cached Objects
  2. Sql_Handle and Plan_Handle Explained
  3. How Cache Lookups Work
  4. Query Parameterization
  5. Retrieving Query Plans from Plan Cache DMV's
  6. Best Programming Practices
  7. Costing Cache Entries
  8. Factors that affect Batch Cache-ability
  9. Memory Pressure Limits
  10. Plan Cache Flush
  11. Temporary Tables, Table Variables and Recompiles
  12. Plan Cache Trace Events and Performance

They also offer a series of extensions to that, on troubleshooting:

Chapter in "Practical SQL Server 2005 Troubleshooting" book

Next, I'd point out that one of the contributors to that, Bart Duncan, is a Microsoft Support engineer who I saw speak on the topic at the SQL Server PASS 2006 conference, where I first learned a lot of this info.

He identified then that a lot of the info was in a chapter of a new book, SQL Server 2005 Practical Troubleshooting: The Database Engine, which I've since gotten and was indeed very useful.

If you're a member of the O'Reilly Safari service (or join for their trial), you can find the chapter online.

Of course, both the talk and the book came out before the article series above, and certainly before SP2, so I would recommend you consider both. (I've not yet done a comparison to determine what may be in the chapter that's not in the articles.)

SQL Server BooksOnline

Of course, it always pays to read the docs, and there are indeed some discussions of this concept and related features in the SQL Server Books Online, but I honestly found the info above either easier to find or just more complete.

My Own Talks

I've made my own attempts to communicate some of this info myself in a couple of talks I've given to both CFML and SQL Server audiences:

But really, there is just too much to communicate in any one hour. If this topic seems of interest, take a few hours to digest all the above. I think you'll be AMAZED at what you learn.

More Entries

Copyright ©2018 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting