File encoding problem

Hello,

I have a problem with file encoding regarding Markdown files. Scenario: I generate a new Markdown file in Devonthink (right click → new → Markdown-Text). I edit the file including German umlaute (äöüÄÖÜ). I export the file to the file system (export → document). Opening the file in Obsidian shows that all umlaute are not properly shown, which means the file encoding is not UTF-8. I checked the file encoding using the command ‘file -I filename’ in the macos terminal. It shows me that the exported Markdown file is not properly encoded. It should be UTF-8, however the file command says ‘charset=unknown’.

How can I make sure, that the markdown file is correctly encoded when exported from Devonthink?

Thanks for any help

Goetz

I just checked this and the file is saved using UTF-8 over here. Which encoding is selected in Preferences > Files > Import?

Automatic

Same over here. Are other text editors able to open the document using UTF-8?

Out of interest I just checked this (created new markdown document in DEVONthink as described, copied in the offending characters) and then exported to the file system. The file opens perfectly in Typora with all characters as expected.

Stephen

Yes, I am able to open the file in other editors, such as Apple’s TextEdit or TextMate. They both show the umlaute correctly, however, these programs do not show me the file encoding type.

Interesting side effect: If I create a new markdown file in an indexed folder, the generated markdown file is encoded correctly in UTF-8 as indicated by the system ‘file -I filename’ command and Obsidian also displays it correctly.

Just as a matter of interest, what happens if you export the file using the Export > Files and Folders… command?

Stephen

This results in similar behavior. In fact I first exported multiple Markdown files using this menu option and stumbled over the incorrect file encoding.

Ah OK—sorry for the false lead: I thought the problem might in some way be related to a problem I had last November.

Stephen

Markdown is pure text (as you can verify by using cat in the terminal). It has no inherent text encoding. Or rather: I doubt that any software would use anything but UTF-8 nowadays.

Also, I never experienced any of this with md files in a lot of different programs.
So the interesting question is: what exactly do you see in Obsidian instead of umlauts?

For what it’s worth.

I created a markdown file in DEVONthink 3.8. Exported as text to the desktop.

 ä, ö, ü

I opened using Obsidian and it rendered correctly:

Screenshot 2022-01-25 at 12.49.26

2 Likes

This is what the markdown file looks like in Obsidian. The file was originally created in Devonthink and then exported to the file system.
Bildschirmfoto 2022-01-25 um 14.23.12.pdf (18.3 KB)

Well, more context might have been helpful, like what you’re expecting to see. What is the output of

od -c filename

in the terminal?

FYI, in support of my post above, for the test file I made, the “od” command shows:

(base) iMac:desktop rmschne$ od -c hjdbdbd.txt
0000000      212   ,     232   ,     237  \n  \n  \n                    
0000013

Exporting the markdown file seems to change the file content regarding the special characters (umlaute), at least in my installation. In the attached screenshot you see a markdown file opened from within Devonthink (Open with…) in a hex editor (top). The same file exported from Devonthink to the file system and opened in the hex editor is shown in the bottom. As one can see both the file length and the file content are different. Obviously Obsidian has problems to display the desired content properly.

Bildschirmfoto 2022-01-25 um 14.46.41.pdf (76.9 KB)

The file content should be:

MD-Datei

Devonthink: äöüÄÖÜß

And this is the result of the ‘od -c filename’ command for the exported markdown file of my previous post:

goetz@lovebird Downloads % od -c MD-Datei_exportiert.md
0000000 # M D - D a t e i \n \n D e v o
0000020 n t h i n k : 212 232 237 200 205 206 247
0000037

this link about a year old but looks same as you report about Obsidian.

i did not dive into the details as i am not a user of Obsidian.

I saw that, too, but it shouldn’t be related. The poster there stored their data in Latin-1, and they were on a Windows system.

oh. like i said did dig into it. should have. still, all very odd.

How exactly do you export the file? Both drag & drop and File > Export > Files & Folders… just copy the files.