[Looking for Charlie's main web site?]

CF Server Monitor: what's the impact on production? you may be surprised

The CF Enterprise Server monitor is more powerful than many may realize, yet naturally some are immediately concerned about its potential overhead, especially if they're considering running it in production. Things aren't quite as obvious as they may seem, or as many may assert.

Update: I've updated this entry now 5 years later, in 2012, to give a bit more context to what I said originally. No change in the message, just a little more info and perspective. (There's also no change in the impact of it, pro or con, in CF 9 or 10.)

First, it's indeed true that depending on what you enable, it *could* be very resource-intensive and could even bring down a server under load. But conversely it also can have ZERO impact--yes, even in production. I'll discuss that in my next entry.

But before that, I just want to take a moment and explain the key 3 features that control what impact, if any, it will have.

Note that there are three buttons at the top of the monitoring page which "start" different monitoring features. (If you don't see these buttons, just refresh the browser again to get them to appear.)

So what are the implications of starting each of the optional monitoring features?

  • start memory tracking: this has the highest potential impact for overhead, which can be very substantial, even to the point of crashing your instance. And even on a low-traffic developer machine, you might see a big hit from running this. More on this in a moment.
  • start profiling: this has much less overhead. It primarily enables tracking of database activity. The help page for the monitor calls its overhead "minimal", but I will note that on a CF server with tremendous DB activity, its overhead could be more substantial.
  • start monitoring: This is the leave impactful button. It's needed to at least see running requests, as well as to have Alerts fire. But even on a busy server I've rarely seen it have a negative impact. That said, you don't need ANY of the 3 buttons enabled to see at least some info. More below.

Definitely check out that help page (from the front page of the monitor) or discussions in my 4-part series of articles on the monitor to learn more about what each button does as well as more about the monitor itself and its many features.

About the Memory Tracking feature

You'll note that I hedged above on the impact of the Memory Tracking. Conventional wisdom is that it is indeed a potential server killer, and I can confirm that I've seen it many times in my CF server troubleshooting consulting practice. But I can also report that I've seen it running on production machines and having virtually no seeming impact. I kid you not.

I suspect it has to do with how many objects are in memory, how complex they are, how busy the server is, etc.

Now, some might propose that you use it only for brief periods (minutes, seconds) to gather some info for analysis, perhaps only in emergency. (I even said this in my initial version of this blog entry.) But many have found that things go horribly wrong on some CF instances the moment it's enabled. So it's probably best not to use it at all on live prod server.

You might be able to do it in a test/dev server, and you may get value form that in looking at the impact at least of individual or small numbers of requests. But beware also that some problems simply don't present themselves except under load (and often only in production, not even with load testing), so using it in dev/test may not help spot/understand/resolve all problems.

So does that mean there's no value if you can't use this feature in prod? Well, no. Remember, this is one of 3 buttons. The other two have less overhead (especially "start monitoring"). More than that, you can get value from the monitor with NONE of the buttons turned on.

Any value if none turned on?

Yes, don't miss this vital point: there is value to using the monitor even with none of the start buttons enabled. That is deserves its own entry. Who says there's no such thing as a free lunch? :-)

So if you hear someone say "don't use the monitor in production", please make sure they're clear on all this. There are 3 features you can enable, or none at all, and each provides different info at different costs--some of it zero.

Postscript: The buttons stay enabled after closing the monitor, and even over restart

I know I've discussed this elsewhere, but while I'm updating this entry let me reiterate the point: it's vital that you understand that if you turn on any of these buttons, they STAY TURNED ON, even if you close the monitor. And EVEN IF YOU RESTART CF. In fact, this is important enough to deserve its own entry. I'll post that now (as I update this in 2012), CF911: Using the #ColdFusion Server Monitor? Be aware that the "Start" buttons remain enabled.

Comments
Charlie, I know you are telling folks to not be scared to use SM on production... but I just want to say that my experience, and this is _just_ my experience, showed that it did seem to impact my performance a bit. Specifically just profiling/monitoring slowed down my blog enough where I felt I needed to turn it off. It wasn't "horrible". But I definitely noticed it and was worried my readers would be affected by it.

I'm just saying - keep it in mind.
# Posted By Raymond Camden | 6/15/07 1:42 PM
We are working with Ray and will try to fix any issue to make it better.

Thanks,
Hemant
# Posted By Hemant | 6/15/07 1:49 PM
Sure, but to be clear, you're talking about enabling the "start monitoring" and "start profiling" features. I don't say that "those" are free. I just say what the docs say, which is that it's minimal". Of course, that's a vague word, so your warning to people on that point should stand.

Still, as the next entry clarifies, there does seem to be an awful lot of good info that can be obtained even without ANY of the "start" features enabled. And that, at a minimum, seems an excellent reason to still use the monitor in production. People just need to be sensitive to what costs what. :-)
# Posted By Charlie Arehart | 6/15/07 1:55 PM
Yep, that is my intent with the warning. Just... be careful I guess. Thats a nice and vague warning as well.
# Posted By Raymond Camden | 6/15/07 2:06 PM
@Ray, I'd really like to see you quantify your experience. I've run monitoring and profiling on a production server with *minimal* performance overhead. In fact the Adobe Hosted Services production servers ran with monitoring and profiling enabled as a matter of course so that we could analyze usage and performance data under real load and tune the system.

The memory tracking is, indeed, fairly resource intensive (I seem to recall about a 20-30% overhead in most situations but higher during extensive object creation, e.g., framework startup). I would definitely advise keeping memory tracking off in production except for very brief periods to debug / analyze specific problems. The main impact seems to be that memory usage increases (presumably to store the data that lets CF8 track memory usage?) and that memory does not always seem to be freed up when memory tracking is turned off.

My experience so far has been that monitoring and profiling truly do have minimal impact and can be left enabled all the time. That has been on Red Hat Linux and Mac OS X, on fairly fast servers. I'm sure it's a YMMV kinda thing tho'...
# Posted By Sean Corfield | 6/15/07 2:40 PM
Quantify? You want me to sit there and do measurements with stuff off/on? Thats a lot of work. ;) I'm just sharing what I saw, that's all. Again - I may have been over reacting to what I perceived as an overly slow response after I had turned things on. Shoot - I'd be happy to turn it on again and let folks comment here if they think the site is responding slower than it does normally.
# Posted By Raymond Camden | 6/15/07 3:28 PM
I am facing some issues on launching the multiserver monitor on Coldfusion8.

I am getting permission denied status line if I add another instance of the CF server .I have modified the multiservermonitor-access-policy.xml file.
Still the error says:Error #2048 url: 'http://wlpv0002.edc....' Ensure that you have allowed access to this server by
changing the multiservermonitor-access-policy.xml file.

Please help me to resolve this issue.
# Posted By Nimi | 3/24/10 1:39 PM
Nimi, sorry for the delay in responding. Did you resolve your problem?

If not, did you see the troubleshooting tips for this problem that I offered in the 4th part of my series on the CF Server monitor? That article (and the others) can be found here: http://www.carehart....

More specifically, the page on which I disuss xml file and some common challenges is here: http://www.adobe.com... (if the line breaks in my comments here, please be sure to join it back together).

Hope that's helpful
# Posted By Charlie Arehart | 4/3/10 9:48 PM
A lot has changed in FlashPlayer policy handling http://www.adobe.com...

So, if you are using FP versions mentioned on the above article, then simply Ignore the multiserver-access-policy file and put the following crossdomain.xml file, with appropriate client machine permissions where you are running multiserver Monitor, and put this crossdomain under the CF Server wwwroot, which you are trying to connect to. Try it, should work.

<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy SYSTEM "http://www.macromedi...">
<!--Generic policy file for flex app access, it should be made more restrictive -->
<cross-domain-policy>
<site-control permitted-cross-domain-policies="all"/>
<!-- <allow-access-from domain="*.example.com"/> -->
</cross-domain-policy>
# Posted By Jayesh Viradiya | 4/9/10 2:21 AM
Thanks for sharing that, Jayesh. Do you know that this will solve Nimi's problem? Or might it have been simply related to the simply approach from the past? Your comment has led me to identify something else for folks to consider about all this.

First, let's be clear for readers: the multiservermonitor-access-policy.xml is just a different name/location for the same crossdomain.xml file Jayesh mentions, as I discuss in the article I pointed to. Adobe puts it in the CFIDE directory during installation.

But as the article Jayesh points to shows, there has been a change in the player security model. This section seems especially significant to this part of the discussion: http://www.adobe.com...

The good news (for those on CF9) is that I see Adobe is putting this crossdomain.xml file in the webroot (in addition to the other in the CFIDE) and it DOES have that new site-control line that Jayesh points out. More on that in a moment.

But for those on earlier editions of CF, it (the crossdomain.xml) wouldn't be there unless you or someone else put it there. And even on CF9, it's possible that it's not there because depending on how one configures CF (whether using the built-in or external web server, and whether one choose an external web server during installation or later uses the web server configuration tool), you may not even find that crossdomain.xml file having been put in the webroot by Adobe. If it's not there, then it's having no influence.

But if it is there, or if you add it, as Jayesh suggests, my read of this Flash security article is that the "all" value of this new site control line should mean that lower-level policy files can override this root one.

Still, I'm finding that the fact that this default root crossdomain.xml has the commented out "allow-access-from" value is blocking access even if the lower level multiserver policy file does uncomment and change it to allow access (again, see my article on the Server monitor for discussions of the change you need to make in this multiserver xml files).

So I needed to uncomment and change that there, AS WELL AS in the multiserver.xml file.

If I've got this right (that we need to change both the CFIDE multiserver xml and the root crossdomain xml files), it would be nice if Adobe might (in the future) put a reference to that in the multiserver xml file (as a comment) in case it may help someone facing a problem.

And with regard to changing these files, please note this VERY IMPORTANT point I make in my server monitor article, in the section on the multiserver xml file: even when you change it (or the crossdomain.xml file), since it's just a static XML file, it's likely the browser will have cached the page. You need to force the browser to go get the new file version (using techniques I discuss there). Even after doing that, you may still need to close and reopen the browser for the flash-based multiserver monitor to pick up the changed xml.

Then again, it may be that all this is still not at the root of Nimi's problem. :-) Let us know. Even so, it should have been helpful for readers. Thanks, Jayesh, and I hope my additional comments get people still further along.
# Posted By Charlie Arehart | 4/9/10 10:57 AM
Copyright ©2017 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting