If you're trying to get to the bottom of high memory or CPU use or database contention on CFML servers, you may be missing a
seemingly innocuous but deadly invader, especially if you're focusing only on "what are my long running requests?" or wondering "Why does CF have a memory leak?" What if the problem isn't really a "leak" at all but, instead, is due
thousands or millions of small page requests that are unexpectedly creating thousands or millions of sessions and client variables each day? They may even be something you're causing (but more likely not). It's a pernicious problem that many may dismiss too readily.
It may seem innocent enough: search bots or other automated tools may be visiting your site many times a day, due to many different search engines, and perhaps even many times per day per bot. These may be outside search bots or internal ones (like a Google appliance). Or it may be caused by load testing tools, or simple monitoring ping tools. Or you may have hundreds or thousands of folks signed up with RSS readers to watch your RSS feeds. You may have code that uses CFHTTP to request templates on your own site (or others may be scraping your site).
The problem is that if the pages being visited by these tools are CFML-based, then the bots and RSS readers will not likely track the session/client cookies CF sends, which starts a real waterfall of problems. There are two potential impacts, sessions and clients.
Update: Since writing this blog entry in 2006, I have since given a talk on the subject, which was recorded. More here.
Creation of many new sessions
Assuming you have SessionManagement="yes" enabled, each such new request without a cookie will cause the CFML server (CF, BD, Railo, or other) to create a new session. Normally, that's not a problem. The browser for a typical user will usually store these cookies for reuse on a future page request.
But since these bots do not typically track cookies, then this causes the CFML server to create a new session for each page request.
And that's not one new session per bot, but one new session per bot per page requested.
And these new sessions will live as long as your sessiontimeout is set--which could be minutes, hours, or days. That could become a substantial resource for CFML server to manage, even if there's "nothing" in the session.
(Props to Mark Kruger whose blog entry, Sessions and Cookies and Bots (oh my), was the first I saw to point this out. As I pointed out in his comments, the problem could be still worse with respect to client variables also, as I'll explain below. And since then, I've realized that RSS Readers could be another, and different problem, since the number of individuals running them may be far greater than the number of search engines.)
So the challenge is to find out how many sessions you have currently. More in a moment.
Creation of many new sessions with large amounts of memory per session
Further, beyond the number of sessions is the question of how large the sessions are. If you DO for some reason put a lot of data into "new sessions" when they're created, then this could become a huge memory burden. And as memory use increases substantially, so could the cost of garbage collection. Eventually the CPU to manage that GC could become problematic, and your system could become unresponsive or even unstable.
Creation of many client records
Finally, there's even a more pernicious (and more persistent) problem due to client variables. Now, even if you'd say "we don't use them", consider this: if you've got ClientManagement="yes" set in your CFAPPLICATION, and you've not disabled in the CF admin the "Disable Global Client Variable Updates" option for your client repository (and they are enabled by default), then the CFML server will create/update fixed client variables (hitcount, lastvisit, etc.) for every page visit. Besides the possible I/O burden (whether in a database or the Registry), given the problem of bots discussed above, this would also mean a new set of these client tracking records would be created for each visiting bot request!
If you're storing these in the Registry, any great increase in the number of entries is clearly bad enough.
But whether you're storing these auto-created variables there or in a database, the problem is quite different from the wildcat creation of sessions. At least those are removed when their sessions timeout or the server is restarted. With client variables, though, these are typically set to expire in days, weeks, or even months! That means they last that long on your server.
So either your registry or client database tables (or both, since each app can choose its own) could become very large and burdensome to be managed. Note as well that CF wakes up every 67 minutes (by default) to go through the client repositories to find any clients that can be expired.
Between that and the initial creation of client records, the volume of updates (or inserts) into the DB can cause significant contention, preventing legitimate CF queries from processing. Do you see how very pernicious this problem becomes?
Diagnosis
If you think this may be happening to you (and even if you don't), you should set up monitoring to see how many unique new sessions or clients are being created. You have a few ways to do this.
Sadly, there's no documented mechanism for CF to tell you how many new sessions or clients have been created in a day.
Update: since writing this entry in 2006, there are now documented and undocumented solutions for both CF 8 and earlier.
First, if you use J2EE sessions, you can get a report of them in the JRun metrics. Just search for any of many articles on using JRun metrics.
But what if you use regular CF sessions? Well, here's good news. Whether running CF 6, 7, or 8, you can in fact get a count of sessions. See my later blog entry about some simple code by Mark Lynch that uses undocumented functions to report the number of sessions (J2EE or plain CF sessions), in total and per application. Very helpful for this challenge.
And if you use CF 8 Enterprise, its Server Monitor (and Admin API) also report the number of sessions, and you don't need to turn on memory tracking for this. Just see the Statistics tab>Active sessions, and click the chart icon on the right, which will show a screen with the total number of sessions.
That will tell you how many sessions you have, which can be very enlightening. I helped a client recently where we found they had 90,000 active sessions. Yikes!
Of course, the next question, then, is to find out why and how. Is it search engines bots? External or internal? Monitor pings to CF pages? RSS feeds? CFHTTP requests? Load testing tools?
If you can access them, you could analyze your web server logs to find out how many pages are coming in that have no CFID/CFTOKEN values sent in their cookies or query string. (Web server logs can be set to report incoming cookies.) That would be a clue, as it would cause CF to create new ones in response. Of course, it could be legitimate first time users.
Update: You can also view the "user agent" being presented by the browser requests, which can be set to be tracked in web server logs, and is also viewable in CF's CGI scope, as well as in the CF 8 Enterprise Server Monitor and FusionReactor's Request Details page. While bots and other automated request tools can mimic (report) any user agent, some do identify themselves plainly. You may be surprised to find how many visits you get from such requesters. A good log file analysis tool like Log Parser, or any others listed at my CF411 site list of Log Analysis tools.
You could also look at either the registry or client variable database to see how many entries there are. You may be shocked by the number. (Update: Just be careful not to open the Registry location for CF client variables using Regedit or similar tools. If there are a huge number defined there, just trying to view it could bring down your server. A tool like Log Parser, mentioned above, can help, though, as it is in fact able to read and count data in the Registry.)
Remediation
The simplest thing is to ensure that any code that may be hit by bots, search engines, RSS readers, etc. do not use code that has CFAPPLICATION SessionManagement="yes" or ClientManagemenet="yes" (or the equivalent properties in application.cfc). That may be trivial, or it may be a hassle, depending on the complexity of your application.
Update: here's a thought that's come up from discussions on a private mailing list. One solution to consider would be to detect if the request has the CFID cookie--which it wouldn't for bots, and if not, set the session timeout to a short value. Be careful, though, because legitmate first-time visitors will also have no CFID token, so if you set any session vars on the page they request or in the application.cfm/.cfc file, don't set it so short that those will be gone when they go to the next page they visit. Perhaps set it to a minute, but realize the implications.
Update: As far as the global client variables updates are concerned, you should first check if in fact your client variable repositories (in the CF Admin Client Variables page) have been modified to disable global variable updates. If not, it's not an obvious thing to disable that. You need to make sure there's no code trying to use client.hitcount or client.lastvisit, which are the variables created by this feature. Assuming there is none, then you could disable this. That doesn't remove the entries created to this point, but it does stop the inserting and updating of entries there. Update: As far as purging client variables if perhaps a large number have been created, first, for the Registry, there is code available from Adobe to help view/purge client variables in the registry. See the zip file in this technote. And for client variables stored in a database, see an explanation and code offered in this technote.
Other thoughts
Iif you use load testing tools, be sure to enable any option to have them honor cookies. Otherwise your testing results may not be accurate, as you're imposing this burden on the server of it creating new sessions and clients for each user request you're simulating, which would not happen in production (except for the bots and RSS readers, etc.).
Hope this info may help others.
