What was so curious was that my local mail client (Outlook) was configured with the exact same servername, username, password, and port, and ran fine from the same laptop on which CF was running. Both were trying to send email via the remote mail server. Why would CFMAIL fail when Outlook would work, both connecting from the same machine to the same SMTP server with the same connection parms?
The short answer: looks like CFMAIL doesn't support Windows Integrated Authentication
I ultimately determined that the mail server (in my case IIS) was set for something other than plain text (basic) authentication. In my case, the properties for the SMTP server in IIS were set to use only "Integrated Windows Authentication". Changing it to also allow Basic Authentication made the problem go away.
Those using other mail servers may find they also offer something other than plain text (Basic) authentication. Hopefully this will still help those folks solve the problem on their systems, too.
So at a minimum, we can say that CFMAIL doesn't seem to support Integrated Windows Authentication. It may be that it doesn't support anything other than plain text (basic) authentication, but I can't confirm that. The docs don't clarify any restriction (they way they do for CFHTTP, which it says also does not support Integrated Windows Auth).
A google search for the error above, with respect to CFMAIL, also didn't find anyone else making this connection, so hopefully this will help. I will add a comment to the livedocs CFMAIL reference page as well.
FWIW, and how this came about, I recently moved to a new hosted, dedicated server (more in a later entry), running my CF, web server, mail server, etc.
While I had long told Outlook to use my network service provider (Comcast cable) SMTP server, even for mail I would send with a "from" address at carehart.org, I feared that some recipient servers might ding the email as spam for not coming from the mail server at carehart.org. So I changed Outlook to send email via my remote server (carehart.org), and that was all fine for some months.
One day recently I decided to change the CF Admin for my local laptop to also use that server (not that I needed to: I obviously don't send email to customers via CF running on my laptop, but I didn't see any reason not to try.) But that's when I got the error. As I said, part of what threw me was that I had been able to successfully send email from Outlook, on the same laptop as the CF trying to send the email, and Outlook was set to use all the same server settings. I could have switched back to my network provider's SMTP server, but I wanted to get to the bottom of this. And I'm glad I did, as this doesn't seem to have been a well-documented problem and solution.
Some other observations
Before solving this I had presented the problem to some fellow CF gurus, one of the first suggestions I got was from the inestimable Dave Watts, who proposed connecting to the SMTP server via telnet from my laptop. When I did that (after finding the commands needed to do that), it reported (in response to Auth Login) "504 5.7.4 Unrecognized authentication type". I had solved the problem by the time I got around to trying that, but it certainly could have been a key clue for me. I offer it here for others to consider. (Another reasonable suggestion, from Russ Michaels, was to look in the server's logs, but there was nothing much in the server's SMTPSVC log that pointed to a solution. It just showed the 530 code as one of the fields on the attempt login.) Thanks anyway, guys, really. We have such a great community.
Also, some may wonder if the CF Admin's "verify mail server config" option (on the Server Settings>Mail page) would have detected the problem. It did not. It would verify, even when the CFMAIL would fail. That tells me that it doesn't actually try to login, which seems a shortcoming.
Finally, going back to my being thrown by Outlook working where CFMAIL did not, I'll add for anyone curious that none of the Outlook account settings (under "more settings", its 4 tabs) showed it using Windows authentication or encryption (yes, I appreciate the risk of that). I guess Outlook uses Windows Auth by default. It was indeed the only authentication that was configured on the mail server. Again, as soon as I changed it to allow basic, the CFMAIL started working.
Hope all this helps someone.