[Looking for Charlie's main web site?]

Solving slow CF startup: my elaborating on an Adobe blog entry on a possible solution

The fine folks at the Adobe CF blog posted a blog entry today, on "Sometimes ColdFusion services refuse to start normally post server restart" (by Rahul Upadhyay), which offers some helpful information on one possible solution to the stated problem of slow CF startup.

That said, there are some concerns I have, with respect to how I fear some may read and take action based on it (especially the notion of deleting the cfclasses files, as a possible solution to the problem).

I'm not contradicting Rahul here, just elaborating on some points, as someone who (like some on the CF team) helps people with CF server troubleshooting every day.

I started to write these thoughts as a comment there, and (as often happens) it grew long so I thought it better to be a blog entry rather than a long comment, and point people here. Once I did that I decided to go further still, hoping to really help those interested to consider the issue more carefully. (It also gives me a chance to highlight again the Adobe CF team blog, something I recommend EVERYONE reading this should follow!)

One quick point (and update) for the TL;DR cloud: My recommendation is that you move the cfclasses folder out of that location, as a temporary test, to see if it makes CF startup happen faster. If it does, I explain why and what the implications are in the choices of renaming, deleting, moving, or disabling the related "save class files" feature. Also, I add an update in E.1 below (since posting this) which you may really want to read: consider turning off your anti-virus software's real-time protection against the cfclasses folder to see if that alone helps with startup.

A) What was recommended, and my initial concern

So you'll see that Rahul indicates that some people have had slowness of CF starting. There can be many reasons for that, but one he notes is if your cfclasses folder (where the compilation of CF templates go) has grown very large. He doesn't indicate how "large" it would need to be to be a problem, but he proposes that one could either delete the files, or move or rename the folder, and/or turn off the "save class files" option in the CF Admin to see if these would help.

I'm on board with the second idea (try renaming the folder). I would want people to be very cautious about either the first (delete the files) or third idea (turn off "save class files"), without making sure first this is the solution to their problem (and doing the move or rename is the easiest way, as I will discuss below).

The thing is, doing any of these could have a negative effect on performance (the old "unintended consequence" of one fix causing a different problem). The saved class files generally IMPROVE performance, holding the first compilation of a CF template after it's created or edited, and the "save class files" option does specifically say it should be turned on in production. (Yes, there have been times over the years when some people have suggested deleting them to fix problems of updates not seeming to take, or some other problem. That's NOT the point Rahul is getting at.)

B) Why do I recommend you NOT delete the cfclasses files? Because it's not reversible

So why do I say to be cautious about deleting the files in that folder? Again, I'm not contradicting Rahul. :-) He offered this as one of a couple of possible solutions, but I fear some may read all that quickly and think "what's the harm?. He doesn't really offer any warning against it."

But note first that he offered this (the issue of a large cfclasses folder) as one *possible* explanation to the problem of slow CF startup times. There can be still other explanations (fodder for another blog entry). Again, you'd really want to make sure this IS the your problem (and solution) before paying the possible price of deleting the class files. (And yes, I do realize that if you delete them, CF will recreate them as needed. But if it was that large to begin with, then there could be a LOT of recompilations that have to take place.)

And I also recommend folks thing twice about turning off the "save class files" to see if that "fixed the problem". While that's at least reversible, you would need to do it, AND remote the cfclasses, and then restart CF, to see if it helps. I just fear some may turn if off, not being fully aware of its impact, and figure "well,it didn't help, but what could it hurt to leave it off?". The problem is that it could have the possible unintended performance impact of CF having compile files ALL the time. What's the issue there?

C) A brief diversion on the matter of CF page compilation, etc.

Before I proceed with why I DO recommend Rahul's suggestion to rename (or better, ove) the folder instead as a temporary test, I realize that some readers may want to point to resources, even going back to the CF6.1 timeframe, suggesting that "CF no longer needs to compile templates to disk", so "what's wrong with deleting them, or telling CF not to bother saving them"?

1) On properly sizing the template cache, in brief

I'll just say briefly that that suggestion presumes that the template cache is properly sized and all needed templates eventually get loaded into memory and would not be compiled again for the life of that CF instance.

Sadly, I help many find that it's woefully undersized and the cause of many ongoing CF performance problems, whether slot page loads or ultimately an outofmemory permgen crash--and how many would connect those dots?! Indeed, the number of saved cfclasses files can be a useful measure (though not perfectly so) to helping know how large the template cache really should be, though that has to be balanced against available space in the heap for the template cache!

This is not really the place to elaborate on the whole template compilation/saving/caching/class loading/permgen process and why it's beneficial, though I do hope to do a blog entry or video about it sometime.

2) On the number of cfclasses created, any why

I will note, related to the size of the cfclasses, that there will be one saved class file for each CFM/CFC template, created the first time it's requested, and recreated after editing the source when it's next requested. (There will also be one for each method/function in that template. More in a moment.)

But do note that there will be one for each location the source may be stored in. If you duplicate code into new directories, that will create new compiled templates for that directory.

And if you move code from one dir to another, the old cfclasses will remain. I think this is what Rahul was getting at when he noted that "Applications dynamically generate ColdFusion files and sometimes also deletes them, but ColdFusion does not remove any old files from the cfclasses folder." So yes, in that case, it would be nice if you COULD somehow delete the "orphaned" cfclasses files for that folder. They don't have any negative impact, other than on this size of the cfclasses, which may or may not be significant.

Finally, note that for every CFC that's compiled on first request, there will be a class file for the CFC and then also one for each method. That may sounds worse to some ears than it should. It just is what it is, but it does explain how some have been confused that they have x number of CFM/CFC files but found x+y cfclass files for them. (Keep in mind also the issue of you moving folders around and having orphaned classes.)

3) On trying to delete a given cfclass for a given CF template

Back to a point made a moment ago, what if you DID want to try to delete the cfclass files for a given CF template source folder? Sadly, it's not as easy as one may hope.

I did a blog entry in 2002 on how (as of CF6) these cfclass filenames connected to their corresponding source CFML files. I'm pretty sure it's still done the same way, which may interest some. But note that it's a partial hash of the directory name, so while the engine can find a class for a given template, it's not as easy for you to reverse the process and connect a given cfclass files to its original source template (though it is possible). This is indeed why some DO punt and just delete all the files when they feel the need.

(I'll note that my post in 2002 didn't mention how there was also a class created for each method/function. That was new as of CF7, after I wrote that entry),

D) So why I DO recommend folks in this situation just MOVE (or at least rename) the cfclasses folder

So back to the problem at hand (CF being slow to startup, and someone thinking that the size of the cfclasses folder is the culprit), here again I state what I think is the best solution/approach.

Like Rahul mentioned (in passing, kind of), I'd simply recommend folks should try first his suggestion to just rename the cfclasses folder (rather than deleting it/it files). At least that IS reversible.

To do that, you would just stop CF, rename the cfclasses folder (name doesn't matter, "cfclasses-old" will work), then start up CF again, and see if that helps with startup time.

(And note that you don't really need to create a new empty cfclasses directory, as Rahul had suggested. I've confirmed (and long noted) that CF will create it for you upon startup if it doesn't exist.)

If it does NOT help with startup time, then it may seem this was NOT (or maybe it is not alone) the solution and you could consider reverting things (stop CF, delete that new cfclasses folder, and rename the old one back to its original name). This way, if having your previous compilations of CF templates WAS helpful you can easily get them back.

But finally, if just renaming the folder doesn't work, try moving it out of the CF folder (I mean for the sake of keeping it only temporarily, until you decide if you will just go ahead and delete it.) Here's why: I've found (through disk i//o analysis) that if the folder remains in that same WEB-INF folder, it IS still at least looked at in some way by CF during startup. It doesn't seem to be the same kind of access the files in cfclasses, but it was at least accessed, and was still contributing to CF being slow to start for one client. When we moved it out (like to the root of the drive, temporarily) then CF did start in a reasonable time.

E) Some closing observations

Here are a few more random thoughts on this matter:

a) A possible impact of virus scanning, especially real-time

(Update: Adding this point E.1 over a year after the original post)

Related to all the above, I'll note that another impact of the large number of files often found in cfclasses is the impact of their being scanned by any virus scanning tool you have. It's not just about scheduled cans but especially "real-time" scans, as many a/v tools enable. With those, any access to a given file causes it to be checked.

I have helped many people where after trying the rename option above, the renamed folder was not only still being accessed by CF for some reason (see my last point in the previous section), but also their anti-virus real-time scan was ALSO scanning each attempt by CF to access the files! And THAT is why moving the old folder out was the answer.

(If you think that I must be mistaken that such realtime scanning scans files on ANY access at all, I'm not. Sure, it would be logical to scan files that are created or updated, but some a/v tool's real-time scans do indeed scan ALL accesses, either by default or as an option that you may find configured. See the docs for the feature in just a few random tools such as Sophos, Avast, and BitDefender, to name just a few.)

OF course, even if you move the old cfclasses folder out (or delete it), in time you may find that you again have a large number of files start accumulating in the "new" cfclasses folder, and again startup times may slow.

In such a case, consider configuring your anti-virus software to have an exclusion for the realtime scanning of the cfclasses folder. I did this with one client and indeed we saw startup time increase dramatically.

Before someone squalks that turning off such virus protection seems dangerous, a) I'm not saying "turn off your a/v" (as in disable it, nor even disabling "realtime protection" entirely). b) I'm also not proposing you prevent the a/v from EVER scanning those folders. I'm proposing that you just disable the realtime scanning for this one directory. You could still let a scheduled scan run occasionally if you feared that somehow a virus could end up in that folder (I can't see any reasonable way it can happen via CF, as it's not a web-accessible directory as configured by default.)

Of course, this idea of real-time a/v scanning of "any read" may cause your mind to race about what other "frequently accessed" files on your server may be being "scanned all the time", and I leave you to consider that possibility. I'll just note that I've found on more than one occasion someone had such a/v real-time scanning happening on a DB server, and they had not excluded the folders holding the DBs themselves (such as the mdf and ldf files in SQL Server), which was wreaking havoc since they are "accessed" all the time.

Indeed, this topic of a/v real-time scanning probably deserves its own post. One of these days.

2) Short-term impact on initial requests, post change

I'll remind you again that while you ARE trying the folder move/rename approach or the "delete all files" approach, whether they help startup time or not, you MAY well notice some impact on request processing, once CF's up, as you/users process requests, and CF has to compile at least the first time each newly requested template. This should only be a modest impact "per request", and should only last a brief time (hours/days) as your templates that are used are requested for their first time since the move/rename/delete.

Then again, if you may be tempted to try the option to disable "save class files" entirely, I'll remind you also that if your template cache is not properly sized, then this impact of recompilation could last all day/permanently while CF may be constantly recompiling them because they've been recently kicked out of the template cache.

3) Beware to manipulate cfclasses, not classes; and about the location of the folder

And let me warn readers to avoid a very common mistake when people go down this path of manipulating the cfclasses folder (perhaps also for other reasons where folks may suggest deleting cfclasses files): we are referring here to the directory called cfclasses, not the one called classes. Don't rename that or delete its files! :-)

And finally, FWIW, Rahul pointed you to the location as {cf.root}/cfusion/wwwroot/WEB-INF/cfclasses. That would indeed be the location for the default/cfusion instance CF 10/11 create. If one is running the Enterprise, Trial, or Dev edition of CF 10 or 11, they could create other instances, and that location would then be {cf.root}/instancename/wwwroot/WEB-INF/cfclasses. This is important, because if one IS running instances and removes the folder in the cfusion instance, it would have NO effect on startup of their troubled instance.

(And in CF9 and earlier, it's still within the wwwroot/WEB-INF folder, though its location varies depending on whether you're running a Standard/Server or Multiserver install of CF.)

Conclusion

Hope all that's helpful. I know some people appreciate the briefest possible explanations of things, and so may feel that these elaborations are "needless melodrama". Believe that at your own risk. And perhaps Rahul was trying also to keep things brief. :-) But since I do help people recover from mistakes like those I've discussed above and many others like them, daily, I just wanted to share that little bit of extra info in case it may help someone.

I'd appreciate your thoughts.

Comments
Thanks for writing this in detail. This is very helpful for new CF admins who would want to know more about cfclasses.
# Posted By Rahul Upadhyay | 1/30/15 12:35 PM
@Rahul, thanks so much for the reply, and again for starting the conversation with your entry. :-)
# Posted By charlie arehart | 1/30/15 12:45 PM
TBH Charlie, a clear-out of the cfclasses occasionally is, I think, good advice.

It's also completely reversable... just run cfcompile over your app's directory and it'll recompile them all. With the added bonus of pointing out any files which have syntax errors in them, preventing compilation.

This has the benefit of only compiling the *current* files in the app. As CF never tidies-up after itself, it's difficult to tell what percentage of the files in that dir are a complete waste of space, and only serve to slow the system down (I think Windows struggles when there's more than 1000-odd files in a single directory)

Bottom line: I think Rahul's advice is perhaps incomplete, but it's still reasonable advice.

Cheers

--
Adam
# Posted By Adam Cameron | 1/30/15 1:43 PM
@Adam, yep, that's what I meant when I said:

"(Yes, there have been times over the years when some people have suggested deleting them to fix problems of updates not seeming to take, or some other problem. That's NOT the point Rahul is getting at.) "

Trust me, I am always keeping in mind ALL the readers of my blog when I wrote these blog entries, which is why they tend to get long. I'm trying to cover multiple bases, and address audiences of varying experience levels.

And that's why I added, "That's NOT the point Rahul is getting at." I do know there may be some who think even "do it regularly". Again, that's not the point of his blog entry. But if it's a side-benefit for those who would do it, fair enough. To that I stand by my later point, though, that "you MAY well notice some impact on request processing, once CF's up, as you/users process requests, and CF has to compile at least the first time each newly requested template. "

As I tried to convey throughout, there are a lot of moving parts to this, and different people know different things, and have vastly different environments, and pressures, and implications of choices.

What i do lament is any attempt to convey a suggestion that leaves the impression that "everyone should do x", if there are implications that are counter to that (but which may not be obvious).

As for whether the advice (from him or others) to clear out the entire CFClasses directory is "reasonable", well, I stand by my contention from the start: if you're doing this to see if it makes CF startup go faster, then don't delete the files. Rename the cfclasses and see if it starts up faster. If it does NOT, then this is NOT the solution to that problem.

But I hear you: there will be some side-benefit to clearing out the classes, to get rid of orphans. I mentioned that in passing.

And while I didn't mention how one can recompile templates, it was in fact discussed in the blog entry I pointed to from 2002, which opens, "In previous bog entries I've talked about compilation and precompilation of CFMX code into Java." I didn't bother to point out the links here, as again to me this was getting beyond the scope of this particular Adobe blog entry, and my reply.

But FWIW, I not only discussed the idea of precompiling templates early on, but I even offered an early form of a precompile.bat file, which presaged the cfcompile.bat which was ultimately included in CF starting CF7. For any interested, you can read here (http://cfmxplus.blog...) and the 2 articles I did on the topic in the CFDJ that year also, the second one at http://www.carehart.... (and linking to the first).

But I appreciate that you, too, Adam, were just trying to be more complete for people.

It's always a challenge for me to decide how much detail to get into. Part of me wants to be complete (for folks who will press if I do not), while part of me tries to be brief (for folks who will press if I do not). And you know that you're sometimes on either side of that fence yourself. :-)
# Posted By charlie arehart | 1/30/15 4:28 PM
Charlie,

I found this post today while having some startup issues on a multi-instance CF11 box. It turned out our issue had nothing to do with CF11.

Looking in the logs of each failed instance, I had a line like this:

"Error","localhost-startStop-1","07/09/15","06:01:59",,"Unable to initialise Monitoring service: coldfusion.server.ServiceException: Address already in use: bind"


When CF11 instances are created with monitoring enabled, it looks as if the jetty.xml file in each intance's /lib directory is configured to try to start the service on port 5050. I tweaked each to use an incrementally different port, rebooted the box, and all came back online happily.
# Posted By Joe Rinehart | 7/9/15 6:59 AM
Hi, Joe. Thanks so sharing that. Sure, it can happen (there are some other situations where ports can be mistakenly shared among instances of CF), and it's a good thing to remind people to consider.

I'm curious about your bringing that up here, though. Are you saying that in your case it led CF to take a long time to start, and since this entry is about Adobe's blog entry that proposed one solution to CF being slow to start?

It's that I'd think that a port conflict would keep CF from starting at all. Is that what happened? If so, sure, again it's good to let people know of how that might be solved. But if it did cause CF to be "slow to start", I would be curious to hear that.

(To any readers who may say that they've experienced things being "slow" when there are port problems, that's indeed so when something as a client is calling something as a server, and that port is blocked. The client will hang quite a while. But Joe's talking here about CF as a server trying to set a port it should listen on. In most cases, when CF tries to listen on a port that is already taken, then CF will generally not start at all.)

Just saying that CF not starting at all is a different problem from the focus of the Adobe blog post I was responding to here. :-) Again, though, Joe, if it may help some reader, thanks for sharing it.
# Posted By charlie arehart | 7/9/15 12:36 PM
Copyright ©2018 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting