[Looking for Charlie's main web site?]

Breaking change in CF2021, new date format mask of D may be serious problem for old code

Note: This blog post is from 2020. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.
Wow. Beware of this subtle breaking change in CF2021, something discovered since its release (was not documented as one of the "new" things, nor was it documented at all in the beta). It can be a devastating problem that might lurk for days or weeks, corrupting your data.

In this post, I discuss the problem, as well as two solutions you can choose between: a) finding and changing the incompatible code, or (new since Dec 2020) b) implementing a new JVM arg that Adobe offered to revert this behavior back to how things worked before CF2021. You must do one or the other of these things, if you are affected by this issue. The update does not REVERT the behavior.

[....Continue Reading....]

Related Blog Entries

Comments
The post here got discussed today on the wonderful Modernize Or Die podcast (https://cfmlnews.mod...). In the chat, Scott Steinbeck and I worked up a list of more tags and functions that use date masks. He found these functions:

dateadd, datecompare, dateformat, datediff, dateformat, datepart, datetimeformat, parsedatetime

And I found these tags: cfinput and its mask attribute, cfgridcolumn and its mask attribute

Of course, we can decry people using such UI tags, and it's not clear if their masks are affected by this change, but I'm noting them for the sake of completeness.

And Brad Wood wondered about a regex to help people find such mask use and Scott also started working on one, including to handle the variants above. He will hopefully post here if he comes up with one. Thanks, Scott!

Or if anyone else is up to the challenge, feel free. :-)
I first noticed this too when I installed MuraCMS 7.1 under CF2021...
Hi, Peter. Hope you'll let them know. They may be in a good position to resolve this in their code base, for the sake of all their users.
I have a feeling CF2018 might have done the same but can no longer confirm...
Peter, are you wondering if/asserting that this problem I raise here might have happened in CF2018? No, definitely not. I wouldn't have written the post as I did.

More to the point, if you look at the cffiddle.org link, you can actually RUN the code against CF2021 AND CF2018 AND CF2016, and the problem happens ONLY in CF2021.
As an update to my last comment, I find that somehow my post did NOT have the cffiddle link that I just indicated. To be clear, I HAD meant to offer the link, and had even referred to it in my post.

But I just noticed that somehow the link didn't make it in. I have corrected that, now under the new "Running Example" section near the top.
Hi. Thanks for this.
However, I believe there are some typos in your sample code.
This should be better:
https://cffiddle.org...
# Posted By GunterO | 11/25/20 11:22 AM
Gunter, those are not typos. The things you changed, like M to mm and Y to YY, etc were referring to the secondary observation made in my post, that Adobe had added also a new M and Y, and my observation (in the blog post and the code comment) that I wasn't seeing them doing anything different, even in CF2021.

But again, the crux of the article--and what's demonstrated in that code, whether changed as you have it or not--was about the D (or DD), and the surprising change in CF2021.

And more important is the matter that in prior releases, the D or DD was treated the same as d, and now no longer is.

But I appreciate that you were trying to help. Make sense now?
Thanks for your quick reply.
When I looked to the output, it made no real sense, because the "description" in front does not match the executed code.
dateformat("11-23-20","MM-DD-YY"): 11-328-2020
dateformat("11-23-20","MM-DD-YY"): 11-328-20
dateformat("11-23-20","MM-DD-YY"): 11-23-20
dateformat("11-23-20","MM-DD-YY"): 11-328-2020

That's why I changed it to :
dateformat("11-23-20","M-D-Y"): 11-328-2020
dateformat("11-23-20","MM-DD-YY"): 11-328-20
dateformat("11-23-20","m-d-y"): 11-23-2020
dateformat("11-23-20","M-D-Y"): 11-328-2020

Where the output matches the description in front.
But, I could be wrong of course.
# Posted By GunterO | 11/25/20 11:35 AM
Ah, got it. Sorry. Copy/paste error! When I created it, affecting the output site so I'd not noticed. I will correct it later today.  Thanks very much.
Thanks Charlie. The CF team is looking at it. We are exploring the possibility of a flag to define this behavior.
Wonderful to hear, Ashish, not only for considering the that but for responding directly. Thanks.
Hi Charlie,

As Ashish mentioned, to cater to the requirement of those users who would want the behavior of D to be similar to what was there in earlier versions, we will be introducing a new flag. We already have a fix ready and are currently testing it. We are also working on updating the documentation to call out this change.
# Posted By Vamsee | 11/27/20 8:49 AM
Again, so great to hear. Thanks.
Hi Charlie,

Thank you for flagging this. As discussed, we've provided a fix, the details of which are documented in the following.

1. https://helpx.adobe.... (also contains the hotfix location and installation details)

2. DateFormat- https://helpx.adobe.... (The history section has been updated)

3. New and changed functions in CF 2021- https://helpx.adobe....

Once again, a ton of thanks for your continuous help and support.

Thanks,
Saurav
# Posted By Saurav Ghosh | 12/2/20 2:36 AM
That's great news, Saurav. Thanks for sharing it. And I trust that you'll get a post out soon on it (on the forums or portal), though I may do another one also (on the portal), for those not seeing this.

A couple of things, though: while I highlighted the issue with D vs d, there would seem to be potential for issues with the other case-sensitive variants (Y vs y and M vs m). I would have thought the fix would have been to control case-sensitivity of masks rather than just about the D alone. Or do you have info from the engineers to suggest that's not an issue?

As for the DateFormat reference page, I see the history indicating this new JVM arg. I don't see it indicating that the D was added. I know some people are saying that this is not NEW in CF2021, but I really thought it was--or else we would have seen this problem sooner. Anyway, can you get someone to clarify WHEN it was added (and if it was removed and added back, perhaps), and get THAT indicated in the history? Same for Y and M.

Finally, you may want to get mention of this added to this doc:

https://helpx.adobe....
Fantastic and fast response from the Adobe CF team! <legends>
Quick check...
5974 lines w/ DateFormat
3648 of those with DD instead of dd.

Soo... thanks for being on top of this :-/
# Posted By Joseph Gooch | 12/6/20 5:17 PM
Hi, Joe. First, somehow your comment ended up in the post about the CF2021 new features. Given what you said, I assumed instead you were referring to this post on the Dateformat issue.

And did you happen to catch the comment from Adobe above, that they had addressed the problem with a new update? While all that code WOULD have failed, it need not if you apply the hotfix and JVM arg change.

I'll update this post to clarify that available fix and point to the Adobe comment, and then I plan to do a blog post also to help make the available resolution standout (and I will point to that here at the top of this post).
After looking at the possible arguments for all of the different functions I have narrowed it down to just dateformat and datetimeformat as well as their `LS` counterparts. The others don't fall victim to the same issue.

the regex pattern I have created is here:

((?<=l?s?(dateformat|datetimeformat)).+[DY]+[^\)]+)

This uses regex lookbehind to check for the existence of a D or Y in the dateformat/datetimeformat/lsdateformat/lsdatetimeformat functions.

should work both in script and tags

https://trycf.com/gi...
Scott, I appreciate the regex, but I feel like I'm missing something.

While your demonstration of the use of the java regex pattern matcher is interesting, and the 40k test cases compelling, can you help clarify how you'd see people using it to find and address the problem in their code? :-) That was your original intent, right?

It's just that I found issues with the more common ways I'd expect people might try to use it:

- when I used the regex (in your comment) in vsCode or CFBuilder, it matched all the dateformat examples above, whether using the upper or lowercase D (of course, using the setting in the Find to search on a regex). So that's curious.

- when I tried to use it in notepad++, its regex search flat said it was an invalid regex

- and even when I tried it in an refind function, it got "Malformed regular expression" and "Sequence (?<...) not recognized." Here's a gist of that:
https://trycf.com/gi... (which failed in all the CF versions or Lucee)

I tried the last example both to see if the CF regex engine would be different, and in case someone somehow might want to try to do such a scan of their code using CFML but maybe would not be allowed to do the createobject/cfobject to pull in the java regexer.

But of course it would seem the biggest concern is what I experienced in the editors. I tried to do a little digging to see if I could get it to work but I had to give up and wanted to get this comment out for you or others to consider.

Again, I do appreciate that you're wanting to help folks get a regex to help find code that would be troubled by this issue, especially if they can't use the new Adobe-provided JVM arg "fix".

Thanks, and looking forward to any thoughts.
Yes you can go ahead and edit my first comment. I think the plan was to implement it into the code checker.

I imagine the reason its failing in vscode and notepad++ would be due to parts of the regex expression not being recognized by their respective regex engine.

the tricky bit is the (?<... portion which is a 'lookbehind'). I can follow up with a compliant regex that would work for vs code, ill try notepad++ as well if it supports a different form of lookbehind. This part is required because otherwise, I can't confirm that the date mask is actually a part of the ColdFusion function dateformat.
Ok a little bit more testing, and here we go. A vscode/cfcodecheker/ possibly notepad++ (dont have it installed) compliant regex. I realized part of my need for the lookbehind was returning data vs just matching.

Here is the link https://trycf.com/gi...

Here is the regex:
l?s?(dateformat|datetimeformat)(.*?)[''"](.*?)[DY]{1,10}(.*?)[''"]

Pattern Breakdown:
- l?s?(dateformat|datetimeformat) --> find lsdateformat, lsdatetimeformat, dateformat, datetimeformat
- (.*?)[''"] --> lazy search until you find a single or double qoute
- (.*?)[DY]{1,10} --> lazy search until you can find a string containing between 1 - 10 characters of D and or Y
- (.*?)[''"] --> lazy search until you find a single or double qoute
FYI for VSCode you HAVE TO click the case sensitive "Aa" button in the search for it to use case sensitive
Hey, Scott, thanks for the regex and the explanation. It does indeed work, and in both VSCode and NotePad++. Hope that (and your code to search via CFML) may help those wanting to find and address this issue that way.

And as you say, if using either editor, we DO have to tick the option to make the find be case-sensitive. That may surprise some, if they figure that the point of the regex is that IT can itself control case-sensitivity in what it's matching. While that's true, the find features in the editors really won't pay attention to the case of what they find and match against (even if the regex has case-specificity) unless we do check that option.

Finally, sorry for the delay in responding to the comments here from a couple of weeks ago.
FYI: the regex that worked for me...
(date)(.*)\(.*(DD\/).*\)|(date)(.*)(date)(.*)\(.*(DD-).*\)|(date)(.*)\(.*(\/DD).*\)|(date)(.*)\(.*(-DD).*\)|(date)(.*)\(.*(D\/).*\)|(date)(.*)\(.*(D-).*\)|(date)(.*)\(.*(\/D).*\)|(date)(.*)\(.*(-D).*\)
# Posted By curtis fraser | 4/6/21 5:17 PM
Curtis, can you clarify for us all:
- what editors are you saying yours works in? And did you confirm it works in cfml itself?
- did you create that because for some reason the previous (simpler) one from Scott somehow did not serve you as well? How so?
- finally, it seems worth noting here that the bug fix mentioned above, from Adobe, is now rolled into Update 1 of cf2021, though that does not mean the error no longer happens. Rather, it means merely that we don't have to implement the special fix/jar before adding the jvm arg that reverts cf to the previous way of using D for dateformat.

Looking forward to your thoughts, since you did push on to find and share this alternative regex. Thanks.
Hi Charlie.

I used the regex in sublime, my current favorite editor.

The previous supplied regex definitely works to get any instance of dateformat or datetimeformat. You could also add in parsedatetime, dateadd, datepart if you wanted.

I created this regex to attempt to list any method that contained 'date', which in turn had capital D or DD as part of the mask.

previous regex would include items like DateFormat(defaultEnd,'dd')

after running my regex through a few systems I support, I realized that it was incomplete. The method text should be case insensitive, as some coders like to yell, others do not. This modified version seems to work to get them all.

(?i)date(?-i)(.*)\(.*(DD\/).*\)|(?i)date(?-i)(.*)(?i)date(?-i)(.*)\(.*(DD-).*\)|(?i)date(?-i)(.*)\(.*(\/DD).*\)|(?i)date(?-i)(.*)\(.*(-DD).*\)|(?i)date(?-i)(.*)\(.*(D\/).*\)|(?i)date(?-i)(.*)\(.*(D-).*\)|(?i)date(?-i)(.*)\(.*(\/D).*\)|(?i)date(?-i)(.*)\(.*(-D).*\)
# Posted By curtis fraser | 4/7/21 12:00 PM
OK, great. Thanks for the clarifications, and the code. :-)
I just got burned with this and replaced all instances of uppercase in my date masks. This is an excellent presentation of this problem. I am glad I didn't miss some obvious warning.
Thanks, Scott. And in case you missed it (or I add this for others subscribed to comments here also), Adobe did indeed provide a "fix" for this problem. I had created a new blog post in Dec 2020 when they did (and I updated this post to link to that), and that fix was incorporated into Update 1 of CF2021 in Mar 2021 (and I have updated both posts to mention that).

But as I clarify in both places: the "fix" Adobe offered is a JVM arg that one can add which changes the behavior BACK to letting the D mean the same as d. The fix does not ITSELF change the behavior back, but allows one to control it for a CF instance. And before update 1, one also had to get and implement a special hotfix jar file (discussed in my Dec 2020 post), but update 1 now incorporates the fix (and obviates the need of that jar).

That said, even as of Update 1, one still does need to enable that JVM arg to "revert the behavior" to how things worked before CF2021 (treating the D like a d). So this "breaking change" will remain in CF2021 going forward, it seems, unless one adds that special JVM arg.

If one does NOT want to (or cannot for any reason) rely on that jvm arg and prefers instead to find and replace all their occurrences of D in dateformat, some very good news is that previous comments here from Scott Steinbeck and Curtis Fraser offer regex's that could be used in editors/IDEs to find them. Perhaps you, Scott, were referring to them when you said you "replaced all instances of uppercase in my date masks".

I updated both my blog posts to offer a specific mention of and link to those comments, to help others use such a find/replace operation if they prefer that over relying on the new JVM arg. Again, thanks guys!
Hi Charlie - it gets worse! CF21 has another breaking (distorting) change in the DateDiff function.

In earlier versions of Coldfusion the mask "w" would return the number of weeks but now it returns the number of weekdays! If we want weeks we now need to use "ww".

https://helpx.adobe....

So we need to pick through our code for this one. Using "w" won't throw an error but it will give you unexpected results if you're expecting weeks your return will be out by a factor of about 7!

If Adobe wanted to introduce a weekdays mask they surely could have used "ww" for that and left the legacy "w" mask as was - seems rather inconsiderate to me, unless I've missed something.

Hope this helps someone.

Best wishes,

Nick
# Posted By Nick Way | 5/27/22 7:52 AM
* factor of 5, not 7
# Posted By Nick Way | 5/27/22 7:53 AM
Interesting indeed, Nick. But it seems it's not "new to CF2021". Note that the docs page you link to says:

"Note: The mask w as weekdays is only valid for the 2018 release of ColdFusion. For 2016 and earlier versions of ColdFusion, w returns the number of weeks."

It's kind of surprising to hear that no one seems to have noticed that for 4 years (that I've seen). Then again, most running it probably only switched to it (or later) in the years since, so such changes only become more widely known over time.

And FWIW, I'll confirm that I myself had not come across this change nor had I documented it for my "hidden gems in cf2018" talks back in 2018 and 2019.

So, at least it is documented. But I agree that such a change is significant and seems it should have been avoided. Thanks for sharing it.
As an update on this, Scott later offered a slightly different variant than his regex from Dec 2020 above. Here it is:

l?s?(dateformat|datetimeformat)(.*?)[''"](.*?)[DY]{1,10}(.*?)[''")]

Pattern Breakdown:
- l?s?(dateformat|datetimeformat) --> find lsdateformat, lsdatetimeformat, dateformat, datetimeformat
- (.*?)[''"] --> lazy search until you find a single or double quote
- (.*?)[DY]{1,10} --> lazy search until you can find a string containing between 1 - 10 characters of D and or Y
- (.*?)[''")] --> lazy search until you find a single or double quote or a ending parenthesis
Copyright ©2024 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting