What if you want it back? In this post, I explain what changed, why, and how to get the functionality back--albeit at your own risk. For more, read on.
I highly recommend that FusionReactor users (and others) register. (Even if you can't attend, you will get access to the recording.) It's about some totally new capabilities:
Note: This blog post is from 2019. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.If your server is behind a load balancer or other sort of proxy, you may have noticed that when you view information about requests in FusionReactor, they all have the same (or nearly the same) IP address. This can be easily fixed, and I show you how in this post.
Note: This blog post is from 2016. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.I'd like to take a diversion from my recent posts focused on CF2016 and talk about something that applies (and should interest) anyone using CF 10, 11, or 2016.
Have you heard of the new "metrics log" option that was enabled in CF10? If you have not, it's worth knowing about (there's precious little documentation, and I'll point to it, and give you still more info to help you use it). It's a useful, low-impact mechanism to get some high-level metrics logged by CF every 60 seconds (by default), and stored along with other CF logs.
If you did know about it, you've probably had some problems with it. Why does it show "nulls"? What do reported metrics really mean? Why do they not jive with what I'd expect to be the numbers reported?
In this post, and a Part 2 to come, I will introduce the metrics log, pointing out some key things you need to know to have it setup to work at all, and then I'll share my observations of things I've come to understand about the reported metrics.
Note: This blog post is from 2010. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.Following on my earlier entry, CF911: Lies, Damned Lies, and CF Request Timeouts...What You May Not Realize, another common source of confusion and misunderstanding for people is when they think their server is "running out of memory", when in fact the problem is often not at all what they think. In this entry, I want to apply the same "cranky" tone :-) and extended explanation to this equally controversial/confusing topic.
I hear people raise concerns with memory problems quite often, whether in my CF Server Troubleshooting practice, or just in my participating in many mailing lists. Indeed, addressing this issue more than a few times the past couple of weeks has motivated me to create this, which will be a series of blog entries.
The series parts are expected to be:
The common complaints about memory issues (and my quick responses, to give you a sense of where I'll be going in this series) are:
I'm not saying there's never a real problem of really "running out of memory". It's just that often things are not at all what they seem (or what most presume them to be, from my experience helping people), and that's going to be the bulk of what I'll talk about in this series. But what if your server is really crashing (or simply not responding), and you think/swear/know that it's a memory problem....
What should you do? Increase the heap size? Increase the permspace? Change the GC algorithm?
Sacrifice a chicken?
I'd say, none of them (though if you're in a rural setting, then perhaps cooking and eating the chicken might help settle your blood sugar so you can stay calm). Really, I know that goes against conventional wisdom, which seems always to suggest diving into the JVM settings. I'd say "hold on there, pardner."
This is one that surprisingly few people consider when faced with their server crashing or not responding. They go with whatever conveys to them a sense of there being a memory problem, perhaps adding their own experience or what they read, and they start chasing solutions.
I can't tell you how often I hear people lament that they've googled and found all manner of conflicting and confusing recommendations. And it doesn't help at all that they may be running on CF 8 or 9 (with Java 1.6) while reading about a "solution" written in the time of CF 6 or 7, when it ran on Java 1.4. Of course, the writer often won't have thought ahead to clarify that.
Instead, I'm saying, "stop, drop, and roll".
"Stop" the tail-chasing, "drop" into the pertinent logs directory in CF, and "roll" through them looking for an occurrence of "outofmemory".
Let me be more explicit: the logs you want to look at for these outofmemory errors are NOT (necessarily) the ones you see in the CF Admin Log Files page. Those are the [cf]\logs directory (or are buried deep within an instance on Multiserver).
Instead, you want to see the "console" or "runtime" logs. Where those are depends on how you are running CF:
Some refer to these as the runtime logs, or the jrun logs, or perhaps the jvm or console logs. Whatever you may call them, their location is above and the explanation of their value follow will become clear n this series of posts.
(I will note that the way these -out.log files work, by default, in CF9 and before is that they fill at 200k increments. Yep, not 200mb, but 200kb!, and you may blow through dozens of them in a few minutes if things are going nuts. That size is configurable, but not through means you'd normally expect. See the blog entry I recently published, CF911: How to control the size of CF's -out.logs.)
Separately, there are some other potentially important logs that may relate info concerning memory problems: what some call the "pid", "hotspot", or jvm abort logs. The filename is in a form like hs_err_pidnnnn.log, with some number in place of the n's.
These logs are found in a very unexpected place (for logs): in the directory where CF stores the jvm.config. So on Standard/Server deployments, that's [cf]\runtime\bin. For Multiserver, that's [jrun]\bin. (As of CF10, it's in the [cf10]\cfusion\bin, or the [cf10]\instance\bin for multiple instances.)
Look in that folder for any .log files. There would be one such "pid" log for each time that the jvm "crashes" due to certain kinds of problems. It could be a crash in the hotspot compiler, in hotspot compiled code, or in native code. (To be clear, many "crashes" of CF are not of the sort that will create such logs, so again, it's only certain kinds of crashes that will lead to them.)
What matters most, for this post, is that again you look at least look for such logs in this folder (occurring around the time of some crash you're interested), and if there look in the logs for any reference to the phrase "outofmemory". Of course, there may be such PID logs that don't refer to "outofmemory", but such crashes are beyond the scope of this post. And while the pid logs have lots of information in then, explaining all that is also beyond the scope of this entry.
The point here (of this section of this post) is that when you have a crash and you suspect it's a memory issue (or if you don't know the cause and want to learn more about what it may have been), you want to look in these two log directories mentioned above. Many never do, and this is part of why they end up chasing their tails, going instead on gut feelings or trying out various alternative "solutions". I say instead: find the diagnostic info, and act on it.
But rather than "look" at all these logs in these directories, one at a time, I suggest instead that you automate the process and search them (I was tempted to say "stop, drop, and mole" in my quip above, since you're "ferreting" through the logs, but that seemed a stretch.)
If you're on *nix, I don't need to give you any more info on how to search the files. Just grep it and rip it. :-)
If you're on Windows, though, you'll perhaps be tempted to think I'm referring to the built-in Windows Search tool to search the directory. I am not. Indeed, let me plead, for the sake of all things decent, please use something better. It's just not a completely reliable (or fast) tool.
I have blogged about a wonderful free alternative, called FileLocator Lite. Use that instead. (If you have another tool or editor you favor, that's fine. No need to bring that up here. I recognize those other options in the other blog entry.)
The beauty of FLL is that once installed (which is fast, itself), if you right-click on the log directory (or any directory) and choose "FileLocator Lite" from the menu, which will open the FLL UI. You can then just put the string outofmemory in the search box and hit enter (or click the "start" button in the UI). You could use *.log to the files searched, though since this directory is so small it's not critical.
In just moments it will show any found files in the lower left pane. You can sort the list by the "modified" column, to focus on files from around the time of your crash.
Then, here's the real beauty of this file search tool over others: to look inside the found file(s), you don't need to double-click the files to open them. Just single-click each file, and in the right pane it shows any lines in the found file which had the string that was searched. Brilliant, and again, a really fast way to find things.
So we're using this to find if any files in either of the folders above has any outofmemory (oom) errors. And if there are any such files, then we're looking at the occurrences of the oom errors within each log.
This feature of the File Locator Lite tool, to see all the lines in the file with the given string, is especially useful in this case, because when searching for outofmemory errors, you also want to be able to quickly see the time for *all* the error messages you may find.
And you *do not* want to focus solely on the last error prior to the crash (or the slowdown, that made you want to restart CF).
(I should add that when it comes to the console logs (as opposed to the PID logs), sometimes those found oom error lines may not have a date/time stamp, to help you readily assess if the error is occurring at the time you were interested in. You may need to go ahead and open the files with an editor and search for the oom string, and then look at other nearby lines to find a timestamp.)
Once you find one (or more) preceding the time of the crash, you want to look for any occurring prior to it. It may be that the problem started several minutes before the crash (or your restarting CF). Further, it may be that the outofmemory error just prior to the crash is different from the one that started things out.
The point here is that you want to find out a) if there WERE any outofmemory errors around the time of your crash of interest, and then b) what information if any appears in the logs around that time. We'll then use that to proceed in evaluating the nature of your situation.
And of course it could be that there WERE no outofmemory errors in the logs, which would indicate that your problem was NOT likely an outofmemory error (or not a heap error after all). We would need to proceed to doing more investigation and assessment of whatever diagnostics you DO have.
OK, that's step 1 in determining whether memory problems are really at all what they seem. As I mentioned at the outset, the planned parts in the series are:
After I publish them, I'll update the lists here to link to them.
As always, I look forward to your feedback (pro, con, or indifferent).
Note: This blog post is from 2010. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.As a CF user or administrator running CF 6-9 on Windows, have you ever wondered how to increase the size of the console logs (-out log files in the [cf]\runtime\logs directory, or [jrun]\logs in Multiserver)? This entry will tell you how. It's quite easy to do, but it's not done using usual log file size control settings in CF's Admin or XML files.
The quick answer is to use either of two approaches: either the jrunsvc.exe in CF's runtime\bin (or [jrun]\bin), or do a manual registry tweak, both of which I show below.
Update for CF10, forward: CF10 and above change to using a new set of xml entries in the neo-logging.xml file, called maxOutLogSize and maxOutFileBackup that should control this (and which should not to be confused with the long-existing maxFileSize and maxFileBackup, both of which correspond to the settings on the CF Admin logging Settings page.) That said, some users had found in the early updates of CF2016 that somehow the xml entries were NOT being honored. It's not clear from them if later updates did fix that problem. I've not heard it as a widely-reported one.
BTW, if you don't know what CF's -out*.log files are about (they're important!), they're technically holding the console output for CF, when it's started as a Windows service. This can be vital information that is NOT logged in the normal [cf]\logs directory or Admin Log Files display. (If you start CF from the command line, then the same info is written to the command line instead and not to the log file.)
(If you're on *nix, pretty much the same info appears instead in the [cf]/logs/cfserver.log. I know some people have wondered about controlling the size of that file. What I discuss here applies only to Windows.)
Note: This blog post is from 2008. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.If you run IIS, are you familiar with its HTTPERR logs? If not, you may be missing out on some useful diagnostic information. Sadly, many (sometimes most) of the entries in the logs are innocuous (you don't need to worry about them), but sometimes they're useful. And if have noticed the logs, perhaps you'd like to know more about them.
There's a useful MS document, Error logging in HTTP API, with more about the HTTPERR logs, including their location, format, and info on the kinds of errors reported within them. Hope that's helpful to my readers.
Here's another (more brief) introduction to the files: in the technical reference section of the IIS 6 docs.
Managed Hosting Services provided by
