[Looking for Charlie's main web site?]

Solving metaspace errors, once and for all

I have a really simple solution to offer here, for a problem that has been nagging people running ColdFusion for the past few years. (I've also just filed a bug report asking Adobe to address this.) This post may also benefit those NOT running CF, especially if they have found confusing/conflicting information about the Java metaspace error and jvm argument that relates to it.

Perhaps you're getting errors referring to "metaspace" or "OutOfMemoryError: Metaspace", whether in your web sites, error logs, or even the CF Admin, and you wonder "what to do". Or you may be getting odd occurrences of blank pages, and if you look in your coldfusion-error.log you are finding such metaspace errors.

TLDR; In all these cases, the solution is simple (and may seem contrarian to some ears): REMOVE the maxmetaspace element from your JVM arguments. Indeed, I would go so far as to say everyone should simply remove it, even BEFORE you may get errors.

In the post that follows, I will explain how to remove it, including how you need to be VERY careful when doing that. You may also wonder why I recommend removing it, versus raising it. I cover that, as well as that bug report, below.

Update: I also created an abbreviated version of this post, on the Adobe CF portal, if that may interest some readers.

How to know if you are having the problem

I mentioned how the problem may exhibit itself (as pages coming back blank or failing), but of course the real proof will be that you will see errors in your logs. You may find in your coldfusion-out.log or application.log, perhaps looking like this:

Feb 23, 2020 17:06:46 PM Error ... - Metaspace The specific sequence of files included or processed is...

or more specifically, in your coldfusion-error.log, you may see lines like this:

SEVERE: Servlet.service() for servlet [CfmServlet] in context with path [/] threw exception [ROOT CAUSE:
java.lang.OutOfMemoryError: Metaspace

That's the absolute proof that this is your problem (you're running out of metaspace), especially if you find them happening around the same time you experienced odd behavior from CF (or your app server. Those on other app servers should look to the logs for this info.)

Do beware that the CF logs are set to rotate when they fill, so even if you may not find these errors in the specific logs I named, look at their archived versions, which would have a -1 and so on, reflecting when they filled and were archived.

The solution

And if the CF settings for maxmetaspacesize are indeed the default (192m), or even if you have tried to "raise" it, I'm proposing here that for most people, you just need to REMOVE that argument and stop "chasing that rabbit".

More below. First, some precautions.

Be careful about modifying JVM arguments

First and foremost, whether you're experienced in modifying the JVM arguments underlying CF (or other Java app servers), you do need to be VERY cautious about making this modification. If you make certain easy mistakes, the configuration will be broken and CF (or your other app server) will not be able to start (heck, it may not even be able to STOP, as the same JVM args are used in the process that STOPS CF as starts it).

Before proceeding, you should backup the file that holds the arguments (before making any changes), and then do be very careful about editing it, as I detail below.

Where to find the JVM args in CF

In the case of ColdFusion at least, this and other JVM args will be found in the CF Admin, in its "Java and JVM" page, and then its "JVM arguments" field.

Changing that page ends up changing the underlying jvm.config file, where you can also find the java arguments appearing on its java.args line (you may see slightly different args which appear on that line than in the "JVM arguments" field of the Admin).

If you have only one CF instance, it's in your cfusion/bin folder, while if you have multiple instances, it will be found in a sibling folder to cfusion with the name of your instance, and then in its bin folder.

Backing up the JVM config file before changing things

Whether you will be modifying the Admin or that file, you should make a backup of this underlying jvm.config file.

You can call the backup jvm.config.bak, for instance. (Note that if you change the CF admin JVM page, it will automatically make its own backup as jvm.bak, and you may be well-served to name your backup something else, as I have proposed.)

How to remove the argument, carefully

Finally, whether changing the args in the CF Admin or in jvm.config, you will find the argument looking like this:

-XX:MaxMetaspaceSize=192m
listed among other java arguments.

You may also find an -XX:MetaspaceSize argument, which represents the "initial" size for the metaspace. CF does not set one by default, but you or someone else may have set one. I would argue also to remove that.

When removing either, you want to be sure to remove the opening dash before that XX: prefix. Be sure also to leave a space between the args currently found before and after this one.

If you make either mistake (leave the dash, or fail to leave a space between args), CF (your Java app) may not start or stop. Notice in the screenshot, where I point to the leading dash, and how it can appear on the end of line to the argument, so could easily be missed and "left" by mistake.

Restart CF, after making the change

After removing the argument, be sure to restart CF. (And don't make the change unless you are prepared to restart CF right away, lest you be gone when CF is restarted unexpectedly, and perhaps won't start because of a mistake you made.)

If somehow CF won't start, be prepared to restore the file you changed, and try again. If you had opened an editor to make the change to the file, and it's still open, look closely at what your changed (and if your editor supports an undo feature, see what happens if you undo your last change).

With this maxmetaspace argument removed, you should no longer ever get any outofmemory errors about the metaspace.

You don't need to change anything about the "failing" pages

And to be clear, there's nothing about whatever pages GOT the error that needs to change. They did not CAUSE the error: they were merely victims of it. Any CF restart would "resolve" the problem, at least temporarily... until CF ran out of metaspace again--in which case some OTHER page would likely fail. Thus the need to solve the REAL problem, by removing this artificially low maxmetaspace setting.

Still more, including the related bug report I have just filed

Again, I have filed a bug report today asking Adobe to STOP setting this maxmetaspacesize value for us. If you agree, please add your vote here: https://tracker.adobe.com/#/view/CF-4207269. I have provided below what I wrote.

Note that it offers some perspective on how it came to be that Adobe does set it, and why I think they should not (and why I recommend removing it vs raising the argument's value):

ColdFusion should not be setting a maxmetaspace argument. It shouldn't set the default that it does, and it shouldn't set it equal to the old maxpermsize argument if it finds on (on migration). Many CF users run into problems of CF failing or CF pages returning blank, or errors referring to the metaspace, and they don't know what to do.

The problem is that CF has been setting the maxmetaspace size to a default of 192m since creating installers for CF11 and above which came out on Java 8 or above. Or if the migration of old settings found a maxpermsize, then CF has presumed to set the maxmetaspacesize to that value.

Both are understandable but mistaken presumptions on Adobe's part. It's true that Java 7 and earlier required that CF set maxpermsize arg, because the default was under 100mb, and CF would not run long without it. And it's true that Java 8 and above has replaced the "permspace" with the "metaspace".

But they are not EXACTLY equivalent, in purpose or design. First and most important, in Java 8 and above, there IS no longer a "default max" for maxmetaspacesize (like there was for maxpermsize). This is why CF no longer NEEDS to set it. Second, the metaspace now uses memory from available OS memory (whereas before, there were debates even in Java circles about whether the permspace was a subset of the heap or not).

This problem has plagued CF shows for a few years now. Let's stop the madness!

And again, as the above demonstrates, I think everyone should remove the argument. It's simply no longer needed, in the way that the older maxpermsize was. (I have added this sentence and a similar one at the top, prompted by a helpful comment from "Wouter". And it also led me to think to add these following two additional sections.)

Then why does Java even offer the argument?

Fair question. It's there if you ever wanted to keep the user of the metaspace from growing WILDLY large. Again, metaspace is taken from available memory on the box (not from within the heap), so it's less of a concern for most to bother limiting it. But maybe you may want to say, "ok, feel free to grow large. I won't set an arbitrarily low limit--like CF's default 192m setting--but don't go using more than 1gb". In that case, feel free to set the maxmetaspace to 1g (and you can say 1024 or 1g. Java will figure it out).

But I'm saying that I've never seen anyone NEED to. But that leads to another question...

How to watch the use of metaspace within CF/the JVM

Another reasonable question. If you have CF2018 or above (Standard or Enterprise), you can view it within the PMT (Performance Monitoring Toolset), if you set that up. One of the things it can track is the use of memory in various JVM memory spaces, including the Metaspace. See the "jvm" page, and the "non-heap" graph there, which breaks down the heap spaces that are NOT in the heap. An article including screenshots is here.

And the same can be viewed with FusionReactor, which tracks memory spaces via the Resources menu on the left, then "Memory spaces", and then you can select a desired memory space in the top right corner of that graph, as shown here. FR also tracks such memory space usage in its logs (written every 5 seconds and kept for 30 days), viewable in the Metrics>Archive Metrics page, and the "Memory section of logs it shows.

Finally, of course, various jvm tools (command line or gui's, or GC logging) can show JVM memory spaces, including the metaspace.

Indeed, if you were to watch the metaspace and see it rising substantially and over time, you would want to really address WHY it is rising (rather than merely "limit" it)--as well as understanding then why and when it is that SOME but not other pages experience the error. But that's getting well beyond the scope of this post, of how to readily solve the maxmetaspace problem itself. Perhaps I'll do another on that broader, background topic.

Conclusion

In the meantime, I do hope that my suggestion at the top (for YOU to remove the offending argument if you get these errors) will help many folks, and I do realize that this post may lead to debate and questions, so fire away.

And as always, if you may prefer help either implementing this change (or somehow further assessing it), or recovering from problems if you try it yourself, I am available for online remote consulting, with satisfaction guaranteed. More at carehart.org/consulting.

For more content like this from Charlie Arehart: Need more help with problems?
  • If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
  • See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
Comments
Charlie, if I understand your reasoning correctly, then you don't really need to see error messages about the MetaSpace in order to benefit from the deletion of the MaxMetaSpace parameter - every CF runtime would benefit from it (every CF runtime after CF8 that is). Correct?
# Posted By Wouter | 2/25/20 9:40 AM
Yes. That's why I also say adobe should not even set it.
# Posted By Charlie Arehart | 2/25/20 9:42 AM
But you make a good point that I don't specifically assert that. I will tweak the post to make that very point. Thanks.
# Posted By charlie arehart | 2/25/20 11:11 AM
I'll go update my 6 servers then! Thanks for pointing this out!
# Posted By Wouter | 2/25/20 11:11 AM
OK, I added a mention of that at the top and the bottom (that really, there's no reason for most people to leave that maxmetaspace arg at all).

And then I went on to add two new sections: "Then why does Java even offer the argument?", and "How to watch the use of metaspace within CF/the JVM".

Hope those may be as helpful to those who already read the post, as to future readers. :-)
# Posted By charlie arehart | 2/25/20 11:37 AM
The question is this: absurdly not setting it, the memory could grow more than the physical one by crashing?
# Posted By Paolo | 2/25/20 11:35 PM
Should we still set the -XX:MetaspaceSize? Or can we remove that too?
# Posted By Dave | 2/26/20 12:50 AM
Paolo, please read the final sections. My suggestion is not "absurd". There's reasoned logic behind it.

Dave, it's usually not necessary to set that min level, no. CF does not set it, so I did not address it. Bottom line, unless you have specific requirements to have set either, don't bother
# Posted By Charlie Arehart | 2/26/20 6:03 AM
Thank you for this info, Charlie.
# Posted By pmascari | 2/26/20 7:20 AM
Thanks for the support and encouragement, Paul, here and on my CF portal post.
# Posted By Charlie Arehart | 2/26/20 7:23 AM
I should have added a mention of this the other day: I posted also a far more brief version of this post (the "tldr version") at the Adobe CF Portal. In case folks reading this may prefer to share that with some folks, it's here:

https://coldfusion.a...

And of course it points here for the additional "detail" that I offer. :-)
# Posted By charlie arehart | 2/27/20 1:50 PM
Oddly, CF2018 was crashing strait out of the box on a brand new MS Server 2016.

Removing this did the trick:
-XX:MaxMetaspaceSize=192m

Thank you for all your great blogs.
# Posted By G. Melanson | 3/11/20 8:56 AM
Thanks, g. That is indeed what motivated me to write. Glad to have helped.
# Posted By Charlie Arehart | 3/11/20 9:00 AM
Hey, Charlie! This was the exact error I was getting on a new client machine. I'm removing the MaxMetaspaceSize, which will hopefully take care of the issues we have been seeing!
# Posted By Kyle Shiflett | 3/11/20 9:30 AM
Great to hear, Kyle. If you've been getting that error, this change will indeed fix it. Glad to have helped.
# Posted By charlie arehart | 3/11/20 11:38 AM
Thanks, Charlie. CF2016 wouldn't restart for us after Update 15. This solved it. Thanks!
# Posted By Chris Simmons | 4/16/20 8:11 AM
Great to hear. I am surprised that the update would have any impact, but what matters is your server is running.
# Posted By Charlie Arehart | 4/16/20 8:41 AM
Awesome explanations and directions. Worked!
# Posted By Marion Andrin | 6/8/20 2:16 PM
Glad to help, Marion.
# Posted By Charlie Arehart | 6/9/20 3:53 PM
Copyright ©2020 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting