Breaking change in CF2021, new date format mask of D may be serious problem for old code
Note: This blog post is from 2020. Some content may be outdated--though not necessarily. Same with links and subsequent comments from myself or others. Corrections are welcome, in the comments. And I may revise the content as necessary.Wow. Beware of this subtle breaking change in CF2021, something discovered since its release (was not documented as one of the "new" things, nor was it documented at all in the beta). It can be a devastating problem that might lurk for days or weeks, corrupting your data.
In this post, I discuss the problem, as well as two solutions you can choose between: a) finding and changing the incompatible code, or (new since Dec 2020) b) implementing a new JVM arg that Adobe offered to revert this behavior back to how things worked before CF2021. You must do one or the other of these things, if you are affected by this issue. The update does not REVERT the behavior.
dateadd, datecompare, dateformat, datediff, dateformat, datepart, datetimeformat, parsedatetime
And I found these tags: cfinput and its mask attribute, cfgridcolumn and its mask attribute
Of course, we can decry people using such UI tags, and it's not clear if their masks are affected by this change, but I'm noting them for the sake of completeness.
And Brad Wood wondered about a regex to help people find such mask use and Scott also started working on one, including to handle the variants above. He will hopefully post here if he comes up with one. Thanks, Scott!
Or if anyone else is up to the challenge, feel free. :-)
More to the point, if you look at the cffiddle.org link, you can actually RUN the code against CF2021 AND CF2018 AND CF2016, and the problem happens ONLY in CF2021.
But I just noticed that somehow the link didn't make it in. I have corrected that, now under the new "Running Example" section near the top.
However, I believe there are some typos in your sample code.
This should be better:
https://cffiddle.org...
But again, the crux of the article--and what's demonstrated in that code, whether changed as you have it or not--was about the D (or DD), and the surprising change in CF2021.
And more important is the matter that in prior releases, the D or DD was treated the same as d, and now no longer is.
But I appreciate that you were trying to help. Make sense now?
When I looked to the output, it made no real sense, because the "description" in front does not match the executed code.
dateformat("11-23-20","MM-DD-YY"): 11-328-2020
dateformat("11-23-20","MM-DD-YY"): 11-328-20
dateformat("11-23-20","MM-DD-YY"): 11-23-20
dateformat("11-23-20","MM-DD-YY"): 11-328-2020
That's why I changed it to :
dateformat("11-23-20","M-D-Y"): 11-328-2020
dateformat("11-23-20","MM-DD-YY"): 11-328-20
dateformat("11-23-20","m-d-y"): 11-23-2020
dateformat("11-23-20","M-D-Y"): 11-328-2020
Where the output matches the description in front.
But, I could be wrong of course.
As Ashish mentioned, to cater to the requirement of those users who would want the behavior of D to be similar to what was there in earlier versions, we will be introducing a new flag. We already have a fix ready and are currently testing it. We are also working on updating the documentation to call out this change.
Thank you for flagging this. As discussed, we've provided a fix, the details of which are documented in the following.
1. https://helpx.adobe.... (also contains the hotfix location and installation details)
2. DateFormat- https://helpx.adobe.... (The history section has been updated)
3. New and changed functions in CF 2021- https://helpx.adobe....
Once again, a ton of thanks for your continuous help and support.
Thanks,
Saurav
A couple of things, though: while I highlighted the issue with D vs d, there would seem to be potential for issues with the other case-sensitive variants (Y vs y and M vs m). I would have thought the fix would have been to control case-sensitivity of masks rather than just about the D alone. Or do you have info from the engineers to suggest that's not an issue?
As for the DateFormat reference page, I see the history indicating this new JVM arg. I don't see it indicating that the D was added. I know some people are saying that this is not NEW in CF2021, but I really thought it was--or else we would have seen this problem sooner. Anyway, can you get someone to clarify WHEN it was added (and if it was removed and added back, perhaps), and get THAT indicated in the history? Same for Y and M.
Finally, you may want to get mention of this added to this doc:
https://helpx.adobe....
5974 lines w/ DateFormat
3648 of those with DD instead of dd.
Soo... thanks for being on top of this :-/
And did you happen to catch the comment from Adobe above, that they had addressed the problem with a new update? While all that code WOULD have failed, it need not if you apply the hotfix and JVM arg change.
I'll update this post to clarify that available fix and point to the Adobe comment, and then I plan to do a blog post also to help make the available resolution standout (and I will point to that here at the top of this post).
the regex pattern I have created is here:
((?<=l?s?(dateformat|datetimeformat)).+[DY]+[^\)]+)
This uses regex lookbehind to check for the existence of a D or Y in the dateformat/datetimeformat/lsdateformat/lsdatetimeformat functions.
should work both in script and tags
https://trycf.com/gi...
While your demonstration of the use of the java regex pattern matcher is interesting, and the 40k test cases compelling, can you help clarify how you'd see people using it to find and address the problem in their code? :-) That was your original intent, right?
It's just that I found issues with the more common ways I'd expect people might try to use it:
- when I used the regex (in your comment) in vsCode or CFBuilder, it matched all the dateformat examples above, whether using the upper or lowercase D (of course, using the setting in the Find to search on a regex). So that's curious.
- when I tried to use it in notepad++, its regex search flat said it was an invalid regex
- and even when I tried it in an refind function, it got "Malformed regular expression" and "Sequence (?<...) not recognized." Here's a gist of that:
https://trycf.com/gi... (which failed in all the CF versions or Lucee)
I tried the last example both to see if the CF regex engine would be different, and in case someone somehow might want to try to do such a scan of their code using CFML but maybe would not be allowed to do the createobject/cfobject to pull in the java regexer.
But of course it would seem the biggest concern is what I experienced in the editors. I tried to do a little digging to see if I could get it to work but I had to give up and wanted to get this comment out for you or others to consider.
Again, I do appreciate that you're wanting to help folks get a regex to help find code that would be troubled by this issue, especially if they can't use the new Adobe-provided JVM arg "fix".
Thanks, and looking forward to any thoughts.
I imagine the reason its failing in vscode and notepad++ would be due to parts of the regex expression not being recognized by their respective regex engine.
the tricky bit is the (?<... portion which is a 'lookbehind'). I can follow up with a compliant regex that would work for vs code, ill try notepad++ as well if it supports a different form of lookbehind. This part is required because otherwise, I can't confirm that the date mask is actually a part of the ColdFusion function dateformat.
Here is the link https://trycf.com/gi...
Here is the regex:
l?s?(dateformat|datetimeformat)(.*?)[''"](.*?)[DY]{1,10}(.*?)[''"]
Pattern Breakdown:
- l?s?(dateformat|datetimeformat) --> find lsdateformat, lsdatetimeformat, dateformat, datetimeformat
- (.*?)[''"] --> lazy search until you find a single or double qoute
- (.*?)[DY]{1,10} --> lazy search until you can find a string containing between 1 - 10 characters of D and or Y
- (.*?)[''"] --> lazy search until you find a single or double qoute
And as you say, if using either editor, we DO have to tick the option to make the find be case-sensitive. That may surprise some, if they figure that the point of the regex is that IT can itself control case-sensitivity in what it's matching. While that's true, the find features in the editors really won't pay attention to the case of what they find and match against (even if the regex has case-specificity) unless we do check that option.
Finally, sorry for the delay in responding to the comments here from a couple of weeks ago.
(date)(.*)\(.*(DD\/).*\)|(date)(.*)(date)(.*)\(.*(DD-).*\)|(date)(.*)\(.*(\/DD).*\)|(date)(.*)\(.*(-DD).*\)|(date)(.*)\(.*(D\/).*\)|(date)(.*)\(.*(D-).*\)|(date)(.*)\(.*(\/D).*\)|(date)(.*)\(.*(-D).*\)
- what editors are you saying yours works in? And did you confirm it works in cfml itself?
- did you create that because for some reason the previous (simpler) one from Scott somehow did not serve you as well? How so?
- finally, it seems worth noting here that the bug fix mentioned above, from Adobe, is now rolled into Update 1 of cf2021, though that does not mean the error no longer happens. Rather, it means merely that we don't have to implement the special fix/jar before adding the jvm arg that reverts cf to the previous way of using D for dateformat.
Looking forward to your thoughts, since you did push on to find and share this alternative regex. Thanks.
I used the regex in sublime, my current favorite editor.
The previous supplied regex definitely works to get any instance of dateformat or datetimeformat. You could also add in parsedatetime, dateadd, datepart if you wanted.
I created this regex to attempt to list any method that contained 'date', which in turn had capital D or DD as part of the mask.
previous regex would include items like DateFormat(defaultEnd,'dd')
after running my regex through a few systems I support, I realized that it was incomplete. The method text should be case insensitive, as some coders like to yell, others do not. This modified version seems to work to get them all.
(?i)date(?-i)(.*)\(.*(DD\/).*\)|(?i)date(?-i)(.*)(?i)date(?-i)(.*)\(.*(DD-).*\)|(?i)date(?-i)(.*)\(.*(\/DD).*\)|(?i)date(?-i)(.*)\(.*(-DD).*\)|(?i)date(?-i)(.*)\(.*(D\/).*\)|(?i)date(?-i)(.*)\(.*(D-).*\)|(?i)date(?-i)(.*)\(.*(\/D).*\)|(?i)date(?-i)(.*)\(.*(-D).*\)
But as I clarify in both places: the "fix" Adobe offered is a JVM arg that one can add which changes the behavior BACK to letting the D mean the same as d. The fix does not ITSELF change the behavior back, but allows one to control it for a CF instance. And before update 1, one also had to get and implement a special hotfix jar file (discussed in my Dec 2020 post), but update 1 now incorporates the fix (and obviates the need of that jar).
That said, even as of Update 1, one still does need to enable that JVM arg to "revert the behavior" to how things worked before CF2021 (treating the D like a d). So this "breaking change" will remain in CF2021 going forward, it seems, unless one adds that special JVM arg.
If one does NOT want to (or cannot for any reason) rely on that jvm arg and prefers instead to find and replace all their occurrences of D in dateformat, some very good news is that previous comments here from Scott Steinbeck and Curtis Fraser offer regex's that could be used in editors/IDEs to find them. Perhaps you, Scott, were referring to them when you said you "replaced all instances of uppercase in my date masks".
I updated both my blog posts to offer a specific mention of and link to those comments, to help others use such a find/replace operation if they prefer that over relying on the new JVM arg. Again, thanks guys!
In earlier versions of Coldfusion the mask "w" would return the number of weeks but now it returns the number of weekdays! If we want weeks we now need to use "ww".
https://helpx.adobe....
So we need to pick through our code for this one. Using "w" won't throw an error but it will give you unexpected results if you're expecting weeks your return will be out by a factor of about 7!
If Adobe wanted to introduce a weekdays mask they surely could have used "ww" for that and left the legacy "w" mask as was - seems rather inconsiderate to me, unless I've missed something.
Hope this helps someone.
Best wishes,
Nick
"Note: The mask w as weekdays is only valid for the 2018 release of ColdFusion. For 2016 and earlier versions of ColdFusion, w returns the number of weeks."
It's kind of surprising to hear that no one seems to have noticed that for 4 years (that I've seen). Then again, most running it probably only switched to it (or later) in the years since, so such changes only become more widely known over time.
And FWIW, I'll confirm that I myself had not come across this change nor had I documented it for my "hidden gems in cf2018" talks back in 2018 and 2019.
So, at least it is documented. But I agree that such a change is significant and seems it should have been avoided. Thanks for sharing it.
l?s?(dateformat|datetimeformat)(.*?)[''"](.*?)[DY]{1,10}(.*?)[''")]
Pattern Breakdown:
- l?s?(dateformat|datetimeformat) --> find lsdateformat, lsdatetimeformat, dateformat, datetimeformat
- (.*?)[''"] --> lazy search until you find a single or double quote
- (.*?)[DY]{1,10} --> lazy search until you can find a string containing between 1 - 10 characters of D and or Y
- (.*?)[''")] --> lazy search until you find a single or double quote or a ending parenthesis