DT3 Bug: "&" converted to "-" when renaming an indexed file in DT3

[mjnnyc]

Hi guys,

I often rename files that I have indexed in DT3. When I do that, DT3 changes the “&” I want into a “-”. For example if I change “Jim and Jane.mp4” into “Jim & Jane.mp4” DT3 actually saves the new name as “Jim - Jane.mp4”.

It does this consistently whether to the internal SSD or to an external HD attached via USB C. I don’t know if it does it to a networked drive, however.

Thanks!

P.S. I think this is the right place to report this minor bug. Feel free to move it if it should be somewhere else.

[schieferk]

I have experienced this behavior as well, but, frankly, I would not consider this to be just a minor bug. A few months ago I moved to indexed files, after I had previously used imported files. When I moved the imported files to newly created indexed folders, DT3 converted “&” and “?” in every single file name to “-“, messing up thousands of file names in the process. I only noticed this weeks later and have since then been painstakingly trying to fix the affected file names. This behavior, which directly affected my data in such a massive way, was one of the reasons for me to stop using DT3 for any data management cases.

[chrillek]

Well, you probably can’t use / nor : in a filename, either. I agree that DT is a tad too conservative regarding allowed characters in filenames, though. But silently changing them is not a very good policy, in my opinion. They should at least give people a choice to go ahead and accept the changes or stop right then and there.

Background: I suppose that they prohibit “&” because that causes problems on the command line if one doesn’t know what they’re doing. So they probably also forbid |, #, <,>. I think it’s noted somewhere in the documentation.

[cgrunenberg]

@chrillek

Background: I suppose that they prohibit “&” because that causes problems on the command line if one doesn’t know what they’re doing. So they probably also forbid |, #, <,>. I think it’s noted somewhere in the documentation.

The main reason is actually to be compatible to as many filesystems and operating systems as possible. Which is at least IMHO a good reason but for Apple-only users this might be a little bit too restrictive indeed.

[chrillek]

Is there a filesystem where & causes a problem? And wouldn’t it be advisable to let the user know about the change before actually performing it?

[cgrunenberg]

@chrillek

Is there a filesystem where & causes a problem?

I can’t remember which one but all the restrictions were actually added due to user requests.

And wouldn’t it be advisable to let the user know about the change before actually performing it?

At least in case of e.g. scripts or smart rules user interactions are not an option.

[mjnnyc]

It would be straightforward to have an option in one’s preference settings for “mac filename settings” or whatever in a way analogous to fuzzy searches or ignore diacritics.

Or I should be flat-out forbidden from using such characters when changing filenames from within DT3. Regardless, it is distressing to have to discover it on my own – it took me a while to figure out that DT3 was the source of the problem… Even saying “it’s in the manual” really isn’t enough because that would require a very close reading of the manual for something I think most users believe they understand well: the basic working of the file system, particularly as DT3 is very tightly integrated with the mac’s file handling as has been discussed in many other threads.

[chrillek]

First, I must retract my earlier demand that the user be asked about changes to filename changes instead of doing them silently. As @cgrunenberg noted correctly, that makes no sense in the context of scripts or smart rules.

Second, and more important for indexed files: The set of permitted characters depends on the file system you use. If your files are, e.g., stored on an CIFS server, you’ll have other restrictions than if they’re stored on a ExFAT drive or on a Linux file system. DT can’t know about the particular restrictions since it uses an abstraction layer to read/write/create/rename files. The & apparently can cause problems with CIFS file systems (cf. /etc/fstab refuses to accept "&" (ampersand)) and proves at least tricky with WebDAV (as do <, > and %), and there’s a whole bunch of characters not allowed on Windows file systems (cf. Filename - Wikipedia).

Also, there are differences in encodings (Apple’s enfuriating UTF-8 variant comes to mind, other filesystems use UTF-16 or even, horribili dictu, ASCII). Windows does not allow PRN as a file name (and not because it might be considered “adult” content). Some file systems are case-agnostic, so you can’t have files “a” and “A” in the same directory. You may use a space in a file name in windows, but you may not have a file name consisting only of spaces. A leading dot is not a good idea in many file systems, and so on.

@mjnnyc

It would be straightforward to have an option in one’s preference settings for “mac filename settings”

Which would do nothing to alleviate the problem, since indexed files need not reside on a mac file system.

@mjnnyc

Even saying “it’s in the manual” really isn’t enough because that would require a very close reading of the manual for something I think most users believe they understand well

That’s what a manual (or a forum: The topic has been discussed here before) is for: close reading. It is mentioned there because most users believe they understand something else.

@mjnnyc

the basic working of the file system,

Which “file system”? Given that we’re talking about indexed files here, the filesystem could be anything: Dropbox, iCloud, Google Cloud, AWS S3, CIFS, NFS, AFS, WebDAV, exFAT, HFS, APFS, HFS+ … I very much doubt that “most users” have an idea about the workings of the particular filesystem they’re using in every case and even which one it is.

@mjnnyc

particularly as DT3 is very tightly integrated with the mac’s file handling

Not so much if we’re talking about indexed files. Having, for example, “tags” for an indexed file on a CIFS will not work outside of DT. It’s the Finder and Spotlight that handle them.

Third: You might want to try scripting to rename the files using Finder or whatever.

[cgrunenberg]

A new preference is indeed a possibility. For some users (especially those using only Apple and APFS/HFS+) this might actually cause less troubles.

[chrillek]

Assuming that the users know which file system they’re using …

[schieferk]

I’m sorry, but I find the reaction from the DT team rather insulting, and I say that as some who used to really like the software and who used DTPO as a major part of my daily workflow for many years. But here, DT3 messes with my files without giving me any warning that this is happening, thereby severely impacting my workflow, and all I get is what feels like a shrug and a pointer saying that I should’ve checked somewhere in the manual.

[rmschne]

@schieferk

This behavior, which directly affected my data in such a massive way, was one of the reasons for me to stop using DT3 for any data management cases.

@schieferk

but I find the reaction from the DT team rather insulting,

I’m confused why you care so much anymore. Oh well. I’m guessing you don’t see the complexity involved and only look through the lens of a Mac OS. Oh well. I think @chrillek nailed it when describing the bigger picture.

[schieferk]

I care, because I want to use the software, but at the same time I’m very reluctant to do so after having been burned. I very much am aware of the benefits of having DT in my workflow.

[chrillek]

And now you know what to be aware of. Frankly, I’ve been burned by lots of file naming issues across different file systems. Occasionally, I’m still stumbling about them.

One can complain about that or accept it as a given fact. In a way, DT tries to prevent harm for their users. I understand that you find that awkward or inappropriate for your situation. I do not understand what could be “insulting” about all that.

I assume that DT swaps out the characters at a very low level in their software stack, presumably just before calling the create, move or rename function of the operating system. At that point, they can’t warn you. On the other hand, it is completely impractical to include the code to exchange the characters (and the possible warning) at the user level: You’d need it in in the “New” method, in the “Rename” functions, in the different actions for smart rules and (the worst!) in the scripting library.

Also, you could import your files and name the records whatever you want. But remember that you or someone else might run into problems if you send those files to them or upload them to whatever service … I had to change the date part in my record names from “2021/07/06” to “2021-07-06” because I could not upload them in the original form to the tax accountants server.

[cgrunenberg]

@schieferk

I’m sorry, but I find the reaction from the DT team rather insulting … and all I get is what feels like a shrug and a pointer saying that I should’ve checked somewhere in the manual.

Please let me clarify a few things. I work indeed for DEVONtechnologies, all the other users in this thread don’t. But I didn’t mention the manual, only offered to add a new preference to upcoming releases to disable this.

[schieferk]

Thanks for the explanation, chrillek, your patience is appreciated.

[Blanc]

@schieferk

I’m sorry, but I find the reaction from the DT team rather insulting

I personally feel that Criss’ (@cgrunenberg) comments here were both nuanced and showed willingness to reconsider the issue. Criss works for DEVONtech, as you can tell from the green nautilus on his icon (or a brief tap on that same icon). Others in this thread to date are users of DEVONthink, rather than employees. I’m pretty sure none we’re trying to be insulting though, and you got a pretty detailed explanation of the possible rationale behind what DEVONthink is doing. That doesn’t change the fact that you tripped up when indexing your files (and for the record: so would I have done, even though I am actually generally aware of the issue), which must be annoying; I don’t think you can automate renaming you files back to what they were either - as all illegal characters are replaces by the same character, going back is a guessing game. Even if it’s “your fault”, I find it very helpful when users report behaviour which for them was unexpected; on the one hand that can help others avoid the same mistake and on the other let DEVONtech check whether behaviour of the app can be optimised (always considering each and every one of us may have different views on what is ideal).

So thanks for sharing your story, and please don’t feel insulted.

(Re “your fault”: I’m aware of the Swiss cheese model, and the risk of the last slice of cheese being blamed.)

[BLUEFROG]

Not piling on with anyone’s point of view, but something to consider about people who may be Apple only… that would have to include things like "You’re not using Dropbox:, as an example.

Dropbox

Even if you don’t get an error message when you name a file on your operating system, if you don’t follow the guidelines below, it could cause a problem when you try to open or sync the file on another operating system.

File and folder name recommendations

Use less than 260 characters in file or folder paths.

File and folder paths must be less than 260 characters long. For example, the file path “C:\Users\Panda\My Documents\Dropbox\Creative Nonfiction\My Autobiography\Favorite Things\Favorite Foods\Bamboo\Family Recipes\Fresh Leaves.doc” has 142 characters.

Don’t use special characters in file names.

Don’t use the following characters in file or folder names:

/ (forward slash)
\ (backslash)
< (less than)
> (greater than)
: (colon)
" (double quote)
| (vertical bar or pipe)
? (question mark)
\ * (asterisk)
. (period)
emojis or emoticons

Don’t add a space at the end of a file or folder name. **

For example, don’t name a file or folder “Panda .png”.

From: Naming Dropbox files and folders - Dropbox Help

OneDrive

_(Restrictions and limitations in OneDrive and SharePoint - Microsoft Support)

Google Drive File Stream has this behavior…

Just some things to consider.

[mdbraber]

@cgrunenberg

A new preference is indeed a possibility. For some users (especially those using only Apple and APFS/HFS+) this might actually cause less troubles.

It would be great if this could be user configured - would you still consider this?

[BLUEFROG]

User configured, why and in what way?

PS: The preference is already in place: Preferences > General > General > More compatible naming of files.