[Looking for Charlie's main web site?]

Breaking change in CF2021, new date format mask of D may be serious problem for old code

Wow. Beware of this subtle breaking change in CF2021, something discovered since its release (was not documented as one of the "new" things, nor was it documented at all in the beta).

[Update: the week after I posted this, Adobe posted an optional fix for this. See my more recent post as well as the comment below from Sauarav, where they have offered both a hotfix jar and a jvm argument that you can use, if you want to revert CF2021 to treating the "D" datemask format as it had in prior releases. Read on for an explanation of the problem, to appreciate the available solution. In time I'll do a separate post with more on the update.]

TLDR; a simple demo of the problem

Consider this fragment, which could exist in similar form in millions of CFML templates: 

dateformat("11-24-20","MM-DD-YY")

See anything wrong? Probably not. It will indeed "work fine" in CF2018 and before, producing 11-24-2020, as most would expect.

But that same code in CF2021 will produces instead 11-329-2020, which virtually no one would expect! Because D now means "day in year". It's a Java-standard datemask, but until now CF didn't complain if you used D. It treated it like d.

And the disaster is that you may have old code that's worked like this for years, but now in CF2021 that code will produce these WRONG RESULTS, which is bad enough if it's just a date shown on a page; but it's catastrophic if that date is then used in code to make decisions, to perform calculations, to store in databases, to pass to other systems, and so on.

And of course it applies if you use 2 D's, as in MM/DD/YYYY, or any variation like DD-MM-YY, and so on. It also applies to lsdateformat, and any functions or tags that take a date mask, such as dateadd, datecompare, dateformat, datediff, dateformat, datepart, datetimeformat, parsedatetime.

So yeah, this seems a huge deal!

What changed, why is this a problem, and what can you or Adobe do? what about Lucee?

What's changed is that a new dateformat mask of D was added (to mean "day in year"). Of course, the lower-case d mask has long existed and meant of course "day in month", as in dateformat(now),"m/d/yyyy". 

The problem is that it turns out CF was loose in its enforcement of this casing before, so you got the same "day in the month" with a "d" OR a "D", such as M/D/YYYY.  Until CF2021, that is. So what now?

For more, read on where I discuss what you can do, what Adobe might want to consider, and how the same "loose casing" of masks happens in Lucee as well (but Lucee has no D for "day in year", so does not have the "same" problem, yet). I also explain how this D for "day in year" is indeed a Java-standard mask value.

Running example

Here's a running example, first at cffiddle.org, where you can change the version to run it either as CF2021, CF2018, or CF2016.

And as an update, since first posting this, I had offered a link at the bottom to run the same code at trycf.com, to see it running on Lucee. But as I had noted then, that site didn't offer CF2021, but I have learned later today that it's being added.

How were we to know?

Again, this was not mentioned in the 500-page beta release notes, nor was it on the what's new page, both of which I discussed in my blog post on, "The many new and improved features in CF2021".

I only found this issue yesterday myself while working with a client, helping him find and understand what was going on, and how things were unexpectedly different.

And while the new mask value of D is indeed documented in the CFML Dateformat reference, it's NOT (currently) indicated there as being something new. I did a comparison with old versions of the docs to discern that it was indeed new. (There's also a new M and Y value, but I have not been able to tell how they differ in their results from the older m and y.)

The Java standard dateformat also supports "D" vs "d"

FWIW, I will note also that the use of the D for "day in year" is not something Adobe just "created out of thin air". It's been in the Java dateformat for years, both per the Java 5 docs (2004 timeframe). 

Again, the problem is not so much Adobe supporting such a "standard" dateformat mask--it's that until now, CF ignored the case and so folks have current apps that use "D" but mean (and found that CF acted as if they meant) merely "d". A HUGE difference (which perhaps Java never suffered if back in earlier versions they had not blithely allowed a "D" where only "d" was defined, if there was such a time).

What can you do?

So, that's the problem. Now you know. But what can you do at this point? And what might Adobe want to consider?

First, if you are concerned with or have been hit by this problem, please register a vote here, the ticket I encouraged my client to create, which he did today: https://tracker.adobe.com/#/view/CF-4210921

Second, you could certainly TRY to search your codebase to find such formatting masks, so as to change any "D" references to "d" instead. And there are of course ways to use either editors or command line tools or search tools to search for any string in code (my favorite, for Windows, is File Locator Lite), and they can all be told to do a case-sensitive search.

But a challenge here is that there are so many mask variants you may have, such as M-D-Y, MM-DD-YY, MM/DD/YY, DD/MM/YY, and so on. It will be hard to pick a single string to be sure to find ALL such occurrences.

And you can't even try to limit it to references to dateformat, as you may also use dateformat, or still other CFML functions that use formatting masks, such as these (added after the original post): dateadd, datecompare, dateformat, datediff, dateformat, datepart, datetimeformat, and parsedatetime. See the comment to the post, below. And there are also tags that accepts a date mask, like cfinput or cfgridcolumn which both have a mask attribute.

Bottom line: I think it's going to be TOUGH for us to easily find, let alone rectify, any and all occurrences of such use of a capital D in a date mask. 

But forewarned is forearmed. Go forth and at least try.

What might Adobe want to consider?

Now, what might Adobe want to do about this?

Of course, it's probably too late to expect Adobe to change this behavior, such as to pick a NEW mask value to refer to "day in year"...but then since CF2021 is so new, perhaps they could.

Or perhaps they could add a switch (in the admin, app-level, or an jvm arg) to allow turning OFF this new D mask, while perhaps also adding a new one. That way those who want to keep the "new" behavior can, but those who don't want to have so many compatibility issues (which may well be major business impacts) can choose to "keep the old way" for some period of time. (And yes, I realize a problem there is that many use such a crutch to then "do nothing"...but it's not like this is some security vulnerability if the "old way" was allowed to remain in perpetuity.)

What about Lucee?

For those who may wonder, I'll note that Lucee works the same as CF did BEFORE 2021--it DOES allow and treat "D" the same as "d".  And according to the Lucee docs for dateformat, it has no current mask for "day in year".  

I'll note also that it too makes no mention of how case is ignored (and none of the masks are capitalized).  So whether we can say "Lucee handles things better" seems an open question, but at least they can learn from this debacle and should NOT implement support for a "D", given how they also were loose in allowing a "D" to be left to mean the same as "d".

Finally, if it may interest anyone, here is a gist to run against Lucee the same code from the cffiddle above, using trycf.com.


So, quite a pickle.

What do you think of all this?

For more content like this from Charlie Arehart: Need more help with problems?
  • If you may prefer direct help, rather than digging around here/elsewhere or via comments, he can help via his online consulting services
  • See that page for more on how he can help a) over the web, safely and securely, b) usually very quickly, c) teaching you along the way, and d) with satisfaction guaranteed
Related Blog Entries

Comments
The post here got discussed today on the wonderful Modernize Or Die podcast (https://cfmlnews.mod...). In the chat, Scott Steinbeck and I worked up a list of more tags and functions that use date masks. He found these functions:

dateadd, datecompare, dateformat, datediff, dateformat, datepart, datetimeformat, parsedatetime

And I found these tags: cfinput and its mask attribute, cfgridcolumn and its mask attribute

Of course, we can decry people using such UI tags, and it's not clear if their masks are affected by this change, but I'm noting them for the sake of completeness.

And Brad Wood wondered about a regex to help people find such mask use and Scott also started working on one, including to handle the variants above. He will hopefully post here if he comes up with one. Thanks, Scott!

Or if anyone else is up to the challenge, feel free. :-)
# Posted By Charlie Arehart | 11/24/20 1:02 PM
I first noticed this too when I installed MuraCMS 7.1 under CF2021...
# Posted By Peter Tilbrook | 11/24/20 5:46 PM
Hi, Peter. Hope you'll let them know. They may be in a good position to resolve this in their code base, for the sake of all their users.
# Posted By Charlie Arehart | 11/24/20 5:56 PM
I have a feeling CF2018 might have done the same but can no longer confirm...
# Posted By Peter Tilbrook | 11/24/20 6:43 PM
Peter, are you wondering if/asserting that this problem I raise here might have happened in CF2018? No, definitely not. I wouldn't have written the post as I did.

More to the point, if you look at the cffiddle.org link, you can actually RUN the code against CF2021 AND CF2018 AND CF2016, and the problem happens ONLY in CF2021.
# Posted By charlie arehart | 11/24/20 6:54 PM
As an update to my last comment, I find that somehow my post did NOT have the cffiddle link that I just indicated. To be clear, I HAD meant to offer the link, and had even referred to it in my post.

But I just noticed that somehow the link didn't make it in. I have corrected that, now under the new "Running Example" section near the top.
# Posted By charlie arehart | 11/24/20 7:18 PM
Hi. Thanks for this.
However, I believe there are some typos in your sample code.
This should be better:
https://cffiddle.org...
# Posted By GunterO | 11/25/20 11:22 AM
Gunter, those are not typos. The things you changed, like M to mm and Y to YY, etc were referring to the secondary observation made in my post, that Adobe had added also a new M and Y, and my observation (in the blog post and the code comment) that I wasn't seeing them doing anything different, even in CF2021.

But again, the crux of the article--and what's demonstrated in that code, whether changed as you have it or not--was about the D (or DD), and the surprising change in CF2021.

And more important is the matter that in prior releases, the D or DD was treated the same as d, and now no longer is.

But I appreciate that you were trying to help. Make sense now?
# Posted By Charlie Arehart | 11/25/20 11:28 AM
Thanks for your quick reply.
When I looked to the output, it made no real sense, because the "description" in front does not match the executed code.
dateformat("11-23-20","MM-DD-YY"): 11-328-2020
dateformat("11-23-20","MM-DD-YY"): 11-328-20
dateformat("11-23-20","MM-DD-YY"): 11-23-20
dateformat("11-23-20","MM-DD-YY"): 11-328-2020

That's why I changed it to :
dateformat("11-23-20","M-D-Y"): 11-328-2020
dateformat("11-23-20","MM-DD-YY"): 11-328-20
dateformat("11-23-20","m-d-y"): 11-23-2020
dateformat("11-23-20","M-D-Y"): 11-328-2020

Where the output matches the description in front.
But, I could be wrong of course.
# Posted By GunterO | 11/25/20 11:35 AM
Ah, got it. Sorry. Copy/paste error! When I created it, affecting the output site so I'd not noticed. I will correct it later today.  Thanks very much.
# Posted By Charlie Arehart | 11/25/20 12:38 PM
Thanks Charlie. The CF team is looking at it. We are exploring the possibility of a flag to define this behavior.
# Posted By Ashish Garg | 11/27/20 2:24 AM
Wonderful to hear, Ashish, not only for considering the that but for responding directly. Thanks.
# Posted By Charlie Arehart | 11/27/20 6:18 AM
Hi Charlie,

As Ashish mentioned, to cater to the requirement of those users who would want the behavior of D to be similar to what was there in earlier versions, we will be introducing a new flag. We already have a fix ready and are currently testing it. We are also working on updating the documentation to call out this change.
# Posted By Vamsee | 11/27/20 8:49 AM
Again, so great to hear. Thanks.
# Posted By Charlie Arehart | 11/27/20 8:52 AM
Hi Charlie,

Thank you for flagging this. As discussed, we've provided a fix, the details of which are documented in the following.

1. https://helpx.adobe.... (also contains the hotfix location and installation details)

2. DateFormat- https://helpx.adobe.... (The history section has been updated)

3. New and changed functions in CF 2021- https://helpx.adobe....

Once again, a ton of thanks for your continuous help and support.

Thanks,
Saurav
# Posted By Saurav Ghosh | 12/2/20 2:36 AM
That's great news, Saurav. Thanks for sharing it. And I trust that you'll get a post out soon on it (on the forums or portal), though I may do another one also (on the portal), for those not seeing this.

A couple of things, though: while I highlighted the issue with D vs d, there would seem to be potential for issues with the other case-sensitive variants (Y vs y and M vs m). I would have thought the fix would have been to control case-sensitivity of masks rather than just about the D alone. Or do you have info from the engineers to suggest that's not an issue?

As for the DateFormat reference page, I see the history indicating this new JVM arg. I don't see it indicating that the D was added. I know some people are saying that this is not NEW in CF2021, but I really thought it was--or else we would have seen this problem sooner. Anyway, can you get someone to clarify WHEN it was added (and if it was removed and added back, perhaps), and get THAT indicated in the history? Same for Y and M.

Finally, you may want to get mention of this added to this doc:

https://helpx.adobe....
# Posted By Charlie Arehart | 12/2/20 11:29 AM
Fantastic and fast response from the Adobe CF team! <legends>
# Posted By Peter Tilbrook | 12/2/20 2:01 PM
Quick check...
5974 lines w/ DateFormat
3648 of those with DD instead of dd.

Soo... thanks for being on top of this :-/
# Posted By Joseph Gooch | 12/6/20 5:17 PM
Hi, Joe. First, somehow your comment ended up in the post about the CF2021 new features. Given what you said, I assumed instead you were referring to this post on the Dateformat issue.

And did you happen to catch the comment from Adobe above, that they had addressed the problem with a new update? While all that code WOULD have failed, it need not if you apply the hotfix and JVM arg change.

I'll update this post to clarify that available fix and point to the Adobe comment, and then I plan to do a blog post also to help make the available resolution standout (and I will point to that here at the top of this post).
# Posted By Charlie Arehart | 12/6/20 6:46 PM
After looking at the possible arguments for all of the different functions I have narrowed it down to just dateformat and datetimeformat as well as their `LS` counterparts. The others don't fall victim to the same issue.

the regex pattern I have created is here:

((?<=l?s?(dateformat|datetimeformat)).+[DY]+[^\)]+)

This uses regex lookbehind to check for the existence of a D or Y in the dateformat/datetimeformat/lsdateformat/lsdatetimeformat functions.

should work both in script and tags

https://trycf.com/gi...
# Posted By Scott A Steinbeck | 12/8/20 1:37 PM
Scott, I appreciate the regex, but I feel like I'm missing something.

While your demonstration of the use of the java regex pattern matcher is interesting, and the 40k test cases compelling, can you help clarify how you'd see people using it to find and address the problem in their code? :-) That was your original intent, right?

It's just that I found issues with the more common ways I'd expect people might try to use it:

- when I used the regex (in your comment) in vsCode or CFBuilder, it matched all the dateformat examples above, whether using the upper or lowercase D (of course, using the setting in the Find to search on a regex). So that's curious.

- when I tried to use it in notepad++, its regex search flat said it was an invalid regex

- and even when I tried it in an refind function, it got "Malformed regular expression" and "Sequence (?<...) not recognized." Here's a gist of that:
https://trycf.com/gi... (which failed in all the CF versions or Lucee)

I tried the last example both to see if the CF regex engine would be different, and in case someone somehow might want to try to do such a scan of their code using CFML but maybe would not be allowed to do the createobject/cfobject to pull in the java regexer.

But of course it would seem the biggest concern is what I experienced in the editors. I tried to do a little digging to see if I could get it to work but I had to give up and wanted to get this comment out for you or others to consider.

Again, I do appreciate that you're wanting to help folks get a regex to help find code that would be troubled by this issue, especially if they can't use the new Adobe-provided JVM arg "fix".

Thanks, and looking forward to any thoughts.
# Posted By charlie arehart | 12/8/20 2:29 PM
Yes you can go ahead and edit my first comment. I think the plan was to implement it into the code checker.

I imagine the reason its failing in vscode and notepad++ would be due to parts of the regex expression not being recognized by their respective regex engine.

the tricky bit is the (?<... portion which is a 'lookbehind'). I can follow up with a compliant regex that would work for vs code, ill try notepad++ as well if it supports a different form of lookbehind. This part is required because otherwise, I can't confirm that the date mask is actually a part of the ColdFusion function dateformat.
# Posted By Scott A Steinbeck | 12/14/20 12:28 PM
Ok a little bit more testing, and here we go. A vscode/cfcodecheker/ possibly notepad++ (dont have it installed) compliant regex. I realized part of my need for the lookbehind was returning data vs just matching.

Here is the link https://trycf.com/gi...

Here is the regex:
l?s?(dateformat|datetimeformat)(.*?)[''"](.*?)[DY]{1,10}(.*?)[''"]

Pattern Breakdown:
- l?s?(dateformat|datetimeformat) --> find lsdateformat, lsdatetimeformat, dateformat, datetimeformat
- (.*?)[''"] --> lazy search until you find a single or double qoute
- (.*?)[DY]{1,10} --> lazy search until you can find a string containing between 1 - 10 characters of D and or Y
- (.*?)[''"] --> lazy search until you find a single or double qoute
# Posted By Scott A Steinbeck | 12/14/20 1:00 PM
FYI for VSCode you HAVE TO click the case sensitive "Aa" button in the search for it to use case sensitive
# Posted By Scott A Steinbeck | 12/14/20 1:05 PM
Hey, Scott, thanks for the regex and the explanation. It does indeed work, and in both VSCode and NotePad++. Hope that (and your code to search via CFML) may help those wanting to find and address this issue that way.

And as you say, if using either editor, we DO have to tick the option to make the find be case-sensitive. That may surprise some, if they figure that the point of the regex is that IT can itself control case-sensitivity in what it's matching. While that's true, the find features in the editors really won't pay attention to the case of what they find and match against (even if the regex has case-specificity) unless we do check that option.

Finally, sorry for the delay in responding to the comments here from a couple of weeks ago.
# Posted By charlie arehart | 12/29/20 3:16 PM
FYI: the regex that worked for me...
(date)(.*)\(.*(DD\/).*\)|(date)(.*)(date)(.*)\(.*(DD-).*\)|(date)(.*)\(.*(\/DD).*\)|(date)(.*)\(.*(-DD).*\)|(date)(.*)\(.*(D\/).*\)|(date)(.*)\(.*(D-).*\)|(date)(.*)\(.*(\/D).*\)|(date)(.*)\(.*(-D).*\)
# Posted By curtis fraser | 4/6/21 5:17 PM
Curtis, can you clarify for us all:
- what editors are you saying yours works in? And did you confirm it works in cfml itself?
- did you create that because for some reason the previous (simpler) one from Scott somehow did not serve you as well? How so?
- finally, it seems worth noting here that the bug fix mentioned above, from Adobe, is now rolled into Update 1 of cf2021, though that does not mean the error no longer happens. Rather, it means merely that we don't have to implement the special fix/jar before adding the jvm arg that reverts cf to the previous way of using D for dateformat.

Looking forward to your thoughts, since you did push on to find and share this alternative regex. Thanks.
# Posted By Charlie Arehart | 4/6/21 9:35 PM
Hi Charlie.

I used the regex in sublime, my current favorite editor.

The previous supplied regex definitely works to get any instance of dateformat or datetimeformat. You could also add in parsedatetime, dateadd, datepart if you wanted.

I created this regex to attempt to list any method that contained 'date', which in turn had capital D or DD as part of the mask.

previous regex would include items like DateFormat(defaultEnd,'dd')

after running my regex through a few systems I support, I realized that it was incomplete. The method text should be case insensitive, as some coders like to yell, others do not. This modified version seems to work to get them all.

(?i)date(?-i)(.*)\(.*(DD\/).*\)|(?i)date(?-i)(.*)(?i)date(?-i)(.*)\(.*(DD-).*\)|(?i)date(?-i)(.*)\(.*(\/DD).*\)|(?i)date(?-i)(.*)\(.*(-DD).*\)|(?i)date(?-i)(.*)\(.*(D\/).*\)|(?i)date(?-i)(.*)\(.*(D-).*\)|(?i)date(?-i)(.*)\(.*(\/D).*\)|(?i)date(?-i)(.*)\(.*(-D).*\)
# Posted By curtis fraser | 4/7/21 12:00 PM
OK, great. Thanks for the clarifications, and the code. :-)
# Posted By Charlie Arehart | 4/7/21 1:44 PM
Copyright ©2021 Charlie Arehart
Carehart Logo
BlogCFC was created by Raymond Camden. This blog is running version 5.005.
(Want to validate the html in this page?)

Managed Hosting Services provided by
Managed Dedicated Hosting