# Duplicate content detection?

**URL:** <https://discourse.devontechnologies.com/t/duplicate-content-detection/60891>\
**Category:** DEVONthink\
**Tags:** devonthink3\
**Created:** [January 11, 2021, 10:20pm UTC](https://discourse.devontechnologies.com/t/duplicate-content-detection/60891 "2021-01-11T22:20:33Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![JohnAtl](https://discourse.devontechnologies.com/user_avatar/discourse.devontechnologies.com/johnatl/32/7593_2.png) [@JohnAtl](https://discourse.devontechnologies.com/u/JohnAtl)\
**Post date:** [January 11, 2021, 10:20pm UTC](https://discourse.devontechnologies.com/t/duplicate-content-detection/60891/1 "2021-01-11T22:20:33Z")

</div>

Would it be possible to add duplicate detection based on contents, rather than type+size?

It would be nice if this included metadata as well. For example, if I have two files and one has a URL and the other doesn’t, they wouldn’t be considered duplicates.

---

<div class="post-metadata">

**Author:** ![BLUEFROG](https://discourse.devontechnologies.com/user_avatar/discourse.devontechnologies.com/bluefrog/32/135_2.png) [@BLUEFROG](https://discourse.devontechnologies.com/u/BLUEFROG)\
**Post date:** [January 12, 2021, 2:50am UTC](https://discourse.devontechnologies.com/t/duplicate-content-detection/60891/2 "2021-01-12T02:50:06Z")

</div>

Duplicate detection doesn’t consider size and type unless you’ve enabled stricter duplicate detection in **Preferences \> General**.

Metadata isn’t part of the content of a file so it wouldn’t factor into a contents-based duplicate detection. Development would have to assess extensions to the detection mechanism.

---

<div class="post-metadata">

**Author:** ![JohnAtl](https://discourse.devontechnologies.com/user_avatar/discourse.devontechnologies.com/johnatl/32/7593_2.png) [@JohnAtl](https://discourse.devontechnologies.com/u/JohnAtl)\
**Post date:** [January 12, 2021, 3:18am UTC](https://discourse.devontechnologies.com/t/duplicate-content-detection/60891/3 "2021-01-12T03:18:25Z")

</div>

I must not understand how duplicate detection works then. I have four files, all different, and they are being marked as duplicates.

The contents of each is

```auto
[bookends](bookends://sonnysoftware.com/77131)

```

and the number at the end varies, depending on the reference.

---

<div class="post-metadata">

**Author:** ![BLUEFROG](https://discourse.devontechnologies.com/user_avatar/discourse.devontechnologies.com/bluefrog/32/135_2.png) [@BLUEFROG](https://discourse.devontechnologies.com/u/BLUEFROG)\
**Post date:** [January 12, 2021, 3:42am UTC](https://discourse.devontechnologies.com/t/duplicate-content-detection/60891/4 "2021-01-12T03:42:01Z")

</div>

Check the **Instances** dropdown in the Info inspector to see where the other duplicates of each file are

---

<div class="post-metadata">

**Author:** ![rfog](https://discourse.devontechnologies.com/user_avatar/discourse.devontechnologies.com/rfog/32/7487_2.png) [@rfog](https://discourse.devontechnologies.com/u/rfog)\
**Post date:** [January 12, 2021, 9:18am UTC](https://discourse.devontechnologies.com/t/duplicate-content-detection/60891/5 "2021-01-12T09:18:25Z")

</div>

I have some little amount of files (PDF) that are completely different inside and they are marked as duplicates if I have disabled “strict” checking. However, I’m interested in this “soft” way to check duplicates because it is able to find very similar files that really are a duplicate that I want to get rid.

My solution is to have a tag called .false\_match and then modify the duplicate smart group to ignore dulicates with that tag.

(BTW, I use point-starting-tag to indicate that is a “system” tag and not a normal one).

---

<div class="post-metadata">

**Author:** ![JohnAtl](https://discourse.devontechnologies.com/user_avatar/discourse.devontechnologies.com/johnatl/32/7593_2.png) [@JohnAtl](https://discourse.devontechnologies.com/u/JohnAtl)\
**Post date:** [January 12, 2021, 5:12pm UTC](https://discourse.devontechnologies.com/t/duplicate-content-detection/60891/6 "2021-01-12T17:12:09Z")

</div>

I know where they are. The content is different. That is the issue.

 ![Screen Shot 2021-01-12 at 12.09.39 PM](https://devontech-discourse.s3.dualstack.us-east-1.amazonaws.com/uploads/original/3X/8/5/85431ff6419e99d78ffeaeeb9044267d98d72742.png)

![Screen Shot 2021-01-12 at 12.10.03 PM](https://devontech-discourse.s3.dualstack.us-east-1.amazonaws.com/uploads/original/3X/a/3/a385b21cbae587e5932030c2e848cd29cf429cfd.png)

 ![Screen Shot 2021-01-12 at 12.10.23 PM](https://devontech-discourse.s3.dualstack.us-east-1.amazonaws.com/uploads/original/3X/e/d/ed8508381ac33eb513841590b3b53a6271a21897.png)

 ![Screen Shot 2021-01-12 at 12.10.33 PM](https://devontech-discourse.s3.dualstack.us-east-1.amazonaws.com/uploads/original/3X/e/f/ef9fceb9fb311c1f08914ee5da16d29c5ba07ecb.png)

 ![Screen Shot 2021-01-12 at 12.10.41 PM](https://devontech-discourse.s3.dualstack.us-east-1.amazonaws.com/uploads/original/3X/b/a/ba1f8c37d0d89b764698f65fb30d682b0086a9dc.png)

---

<div class="post-metadata">

**Author:** ![BLUEFROG](https://discourse.devontechnologies.com/user_avatar/discourse.devontechnologies.com/bluefrog/32/135_2.png) [@BLUEFROG](https://discourse.devontechnologies.com/u/BLUEFROG)\
**Post date:** [January 12, 2021, 5:19pm UTC](https://discourse.devontechnologies.com/t/duplicate-content-detection/60891/7 "2021-01-12T17:19:12Z")

</div>

The **rendered content** is not different in this case. Both documents contain a single word: **bookends**. If the source of the document was not showing, it would certainly appear the files have the same content.

Development would have to assess modifying this behavior.

---

<div class="post-metadata">

**Author:** ![JohnAtl](https://discourse.devontechnologies.com/user_avatar/discourse.devontechnologies.com/johnatl/32/7593_2.png) [@JohnAtl](https://discourse.devontechnologies.com/u/JohnAtl)\
**Post date:** [September 5, 2022, 12:54pm UTC](https://discourse.devontechnologies.com/t/duplicate-content-detection/60891/8 "2022-09-05T12:54:54Z")

</div>

I still think this is ridiculous, and is one of the reasons I stopped using DEVONthink.

---

<div class="post-metadata">

**Author:** ![BLUEFROG](https://discourse.devontechnologies.com/user_avatar/discourse.devontechnologies.com/bluefrog/32/135_2.png) [@BLUEFROG](https://discourse.devontechnologies.com/u/BLUEFROG)\
**Post date:** [September 5, 2022, 1:38pm UTC](https://discourse.devontechnologies.com/t/duplicate-content-detection/60891/9 "2022-09-05T13:38:05Z")

</div>

We have users who expect this specific behavior, so I wouldn’t call it _ridiculous_ on their behalf.

In the **Help \> Appendix \> Hidden Preferences** , click the **On** link for **IndexRawMarkdownSource** , then do a **File \> Rebuild Database**.