Bookends workflow problems

[astro]

Hello,

I am a newbie who decided to start working with DT3 and Bookends.
I´ll do it for my professional work with some sort of academic research.

Therefor I do have a lot of documents to handle each week.

I wanted to have / wished / imagined a workflow like this:

First:

• Dropping pdf into Bookends.
• Referencing pdf with BookEnds. (author, title, etc.)

Second:

• Indexing the attachment folder of BE with / in the global inbox of DEVONthink
• label them (automatically?) orange = not read / highlighted yet
• sort the pdf to the correct database and folder

My problem is now:

Whenever I sort a pdf in the correct database and folder, it appears (of course) again in the indexed inbox folder.

So when I add new pdf to Bookends, I loose oversight wich ones I have sorted already and wich ones aren’t yet in there place.

Anyone any idea how to manage that?

I am sorry to bother, but I am a complete newbie and have no idea what scripting is, since I read in the forum a lot about scripting the imports from bookends to DT3.

Take care!

[mbbntu]

I use Bookends with DEVONthink, and my practice is NEVER to move attachment pdfs within DEVONthink.

I use DEVONthink to index the folder where Bookends keeps attachments – I never import the attachments (I suspect you misunderstand what the Global Inbox does). Bookends renames pdfs that end up in the attachments folder, and does all the housekeeping there. I regard it as territory belonging to Bookends, and while DEVONthink has “permission” to keep an eye on it, I wouldn’t use DEVONthink to do anything to the pdfs there, except tag the files or annotate them in some way. Do anything else, like moving files around or renaming them, and the link with Bookends will break.

If you need the pdf to appear in a different group, you should probably replicate it there, not move it.

So, my advice would be: do not go near the Inbox with attachments (putting stuff in it automatically imports it, rather than indexing it), leave them where they are and index the Bookends attachments folder, and replicate pdfs that you would like to see in other groups.

Good luck with it.!

(See screenshot of my Library database, with group dedicated to Bookends attachments.)

Screenshot 2020-07-14 07.26.37|690x488

[astro]

Thank you very much for your reply.

Maybe I didn’t explain my problem accurately so I have been misunderstood.

What I do:

I am actually indexing just the attachment folder of Bookends.

But then I sort the index (?) in DT3, or indexed pdf if you want, into a database and the folder I want it to be.

So far so good. The link doesn’t break.

But since the attachment folder is indexed, the Inbox folder in DT3 will refresh itself and I end up having again the same indexed pdf in the inbox.

I end up with a new “replicant”.

I will put hundreds of files each month into Bookends, so I will loose oversight wich ones I already sorted and wich ones I still need to put into the right project folder.

The indexed attachment folder of Bookends never empties!

I don’t see how to get around this problem.

[mbbntu]

Don’t put the Attachments folder in the Inbox. Put it in the main section of your database.

Or perhaps to be more accurate, in Bookends configure your attachments folder so that it is in somewhere like your Documents folder on your hard drive, or perhaps Dropbox or iCloud, wherever you want, and then in DEVONthink choose to index that folder so that it appears in the main body of your database, not in the Inbox. The indexed folder will refresh itself automatically when you add new items in Finder (or via Bookends). You do not need to put it in the Inbox for it to update with new files.

[astro]

Thank you. I tried.

It doesn’t solve the problem.

As soon as I sort, a fresh “replicant” appears in the indexed attachment folder. (By definition this is what a indexed folder is supposed to do)

So I never know what files I have sorted already and wich ones are the new ones.

I want them to “disappear” from the indexed folder as soon as they are in their place.

Like a to-do-list.

[mbbntu]

@astro

So I never know what files I have sorted already and wich ones are the new ones.

In my view (based on over ten years of using DEVONthink and Bookends together in academic work - for an MSc and a PhD) you are taking the wrong approach. You should just leave your attachments in their indexed Attachments folder and largely ignore it. Do not worry about what is in it. Instead of replicating items to groups, it is probably more useful to use tags and smart groups. The way to tell whether you have “sorted” a file (not sure what you mean by sorted here) is to give it the tag “sorted”. Then you can have a smart group which displays all the files that do not have the tag “sorted”. This approach is far more flexible than using static groups, and ultimately is less work. It is far quicker to tag files than it is to replicate them or move them. And there is the advantage that the tags show up in the Finder, and in other places as well (I also use HoudahSpot, which I find a very useful tool). You will also be able to see the tags in Bookends.

That is the approach that has worked for me, and many others, I believe. Forget about moving files into folders and groups. Use tags and smart groups instead. There are undoubtedly disadvantages in this approach, as with any other, but I have found it the most useful in my work.

PS: have a look at this – http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.164.320&rep=rep1&type=pdf&utm_source=zapier.com&utm_medium=referral&utm_campaign=zapier

[astro]

@mbbntu

The way to tell whether you have “sorted” a file (not sure what you mean by sorted here) is to give it the tag “sorted”. Then you can have a smart group which displays all the files that do not have the tag “sorted”. This approach is far more flexible than using static groups, and ultimately is less work. It is far quicker to tag files than it is to replicate them or move them.

By sorting I mean using the DT sorter:

In other words to put it into the right group.

If I only tag them I would loose a lot of the advantages DT gives me (imo).

So maybe it is possible to create a smart folder?

Like:

“Show me all items/files/pdf of the group Attachments in the database Research that do not have a duplicate or replicant in any of the other databases.”

But I don’t get anywhere like this…

Edit: Just found out that DT doesn’t find duplicate or replicant outside of its own instance e.g. database…

[mbbntu]

Software has its limitations, and you have to work within them. In my experience it often creates problems when people have a plan and then try to impose it on the software rather than finding out what the software will do, and exploiting what it offers.

But given that it is not possible to “empty” the Attachments folder like a “to do list”, what I might do in your position is to replicate everything that lands in the Attachments folder, as soon as it arrives there (you can look for the date added, and set up a smart rule to do the replication) into a sort of “holding” group, and then process those replicants. When that “holding” group is empty, you know that you have processed everything. Moreover, you can check the originals in the Attachments folder/group in DT, and if any of them do not have replicants, you will know that they have not yet been processed.

I would advise you to familiarise yourself with smart groups and rules so that you can get an idea of what they can and cannot do. Always go one step at a time. In the screenshot above, I’d be surprised if any of your files had a URL containing “Attachments”. A URL is more likely to look like “http://www.website.com” or something of the sort.

Anyway, try to be adaptable and don’t have too rigid a plan. Work with the software, not against it. Exploit what it can do, rather than trying to bend it to do something that it can’t.

[astro]

Thank you for your advice. In the essence you are totally right.

I found a solution.

The boundary seems to be that DT cannot find replicants or duplicates outside of its own instance.

At first glance this seems to be a little bit odd since you can easily duplicate or replicate a file to any database.

The effect/conclusion is that I can only use the referenced pdf from Bookend in one Database.

Wich is a kind of a pity since I really wanted to use the Bookends pdf for a lot of different fields in my life that I want to keep separated. For example: One Database for academic journals in my job and one for academic research in my university.

But when I place the indexed attachment folder in just one database and creat a smart group “To-Do” inside it like this:

it works.

But what I find pretty strange is that nobody in the forum experienced the same problem before.

I just started with bookends 3 days ago and my workflow seemed to be pretty natural:

• Create a BE Watch folder on the desktop.
• Throw the pdf in there and BE grabs them automatically
• Create the references in BE
• Index the attachment folder of BE with DT
• Sort it in DT

It looked nice to me…

Because even when one goes another way by importing them manually for example the question will always be how to separate the ones you imported already from the ones you still need to.

But thanks a lot. Learned a lot.

[BLUEFROG]

• You can duplicate to other databases, but it is merely a copy versus a move. We could have just calls it Copy to > … instead when used across databases.

• Duplicates are not detected across databases.

• You cannot replicate across databases.

[astro]

@BLUEFROG

You cannot replicate across databases.

Ah, sorry. You are totally correct. My bad.

May I ask why Duplicates are not detected across databases?
Is there a specific reason for it?

[mbbntu]

The problems you mention are also problems I have faced. There is no easy solution. The way I look at it is this: the University Library here where I live (Cambridge) is huge and has works on every field you could imagine. But I can still find the books that I want when I look in the catalogue. And I cannot know in advance that a book or article on psychology will not be relevant at some point to my occasional studies in history. So I have a database called “Library” in which I keep material that might be used in my research and writing. This contains an indexed Attachments folder which is controlled by DEVONthink. In truth, because I work in psychology mainly nowadays, almost anything I find might be a source work and need to be cited, so much material goes into Bookends.

There are advantages and disadvantages in having such a Library database. One advantage is that you never have to remember which database you put something in, because they are all in the Library. Moreover, you never have to decide which one to put them in because there is one main place to put them.

It is true that I have other databases, but they are for matters that are not work-related or academic. And if I want to work on a small project, I can always create a database specifically for that, and collect in it material that particularly relates to that subject. In that case I can index individual pdfs from the Attachments folder if I need to. That gives me a focused environment to work in for that project, but I can always go to the “Library” if I need to. When the project is finished, I can archive that database if I won’t need it again.

Ultimately, however, you have to find a method that works for you.

PS: you might find it valuable to have a look at this article – https://zettelkasten.de/posts/collectors-fallacy/

[astro]

@mbbntu

PS: you might find it valuable to have a look at this article – https://zettelkasten.de/posts/collectors-fallacy/

Wow! What a great text. Thank you for that one.
It’s so true, I don’t know even where to start, because it is kind of mind blowing for me atm.

Let’s stick first to the practical/technical side:

My setup isn’t very scientific.

For one simple reason: I don’t have the time.

That’s why I like DT so much. In my job it’s a life saver.

I don’t need to “know” or even need to “read” the stuff. When I need to have it, it’s there.

I never had the time to sit down and contemplate or develop anything with the suggestions DT gives to me on a particular document. The search function was enough.

I don’t even have the time to go to a certain database and look for a specific folder. I never did that manually.

But still my workflow was like this: throwing everything into the Inbox during the week and try to structure my world on the weekend in some general topics and sort the files accordingly.

Since I don’t have in-depth technical knowledge, I hoped to train the A.I. so one day it will pay out.

And it was a funny experience because the suggestions DT made me, sometimes weren’t where I saw the document.

Remember: I don’t really read the documents - I just “scan” them quickly and then map them to the place. But DT didn’t even show me this place in its suggestions. I hope on the long run this is what they mean by “training the A.I.” (but maybe I am totally wrong about the concept…)

Psychologically there is maybe another extrem on the other end I can tell you about:

I never imported my e-mails into devonthink.

I got a work colleague who sorts his Outlook inbox, i.e. he creates for nearly every e-mail its own folder.

Since there is a team with access to the Inbox: two days later nobody finds the e-mail anymore.

It’s hidden somewhere, not accessible with the search function.

I leave my e-mail folder as it is and I am much faster to find any e-mail with a 3 year old (90 Euro) Samsung phone.

So you are right:

Everybody needs to find his own way.

May I ask you:
Do you do anything in Bookends with your files?
and
Are you using the Zettelkasten System?

Have you heard of Tinderbox?

I watched Beck Tench

Turning Reading Notes into a Tinderbox Map (Part 1)

Do you use Tinderbox?

Me personally somehow don’t get along with “Mind-Mapping” that is why I need groups in DT.

(Maybe it’s because I studied Latin too long… )

Take care!

[mbbntu]

I’m glad you found the article worth looking at. I find many of the posts and discussions on the site to be very thoughtful and thought-provoking. I visit quite regularly and often take away some ideas.

Unless I have misunderstood how DEVONthink’s “See also and classify” works, it operates on word frequency (it is probably a bit more complex than that, but let us not be too technical). In other words, it reads the text of your files, and if some of them mention New York quite frequently, it takes the view that these might belong together. This is all very well, but the problem arises that you may have files in which the main theme is child poverty, but those words never actually appear in the text. Then again, if a human researcher is interested in skyscrapers, the human might think that files on Shanghai and New York belong together, while someone interested in city parks might put New York and London together. In other words, DEVONthink’s evaluations might be “logical” in one sense, but completely useless to a researcher who wants to categorise material in a way that DEVONthink’s algorithms are incapable of detecting. In truth, in the time I have used DEVONthink I have rarely used the automated filing, and when I have it has usually not been very satisfactory for me. I generally prefer to do things by hand. Then again, I do less of that than I used to. Search functions and my own memory are what usually gets the best results.

As to Bookends, I import pdfs, let Bookends rename them with the format Author_Date_Title, and rarely look at them in Bookends again. I read them with either PDF Expert or Highlights. For me, Bookends is basically for making bibliographies. Formatting 350 references by hand is not really something you want to do – unless you are very different from me.

I dabble with the Zettelkasten system, but I don’t use it extensively. I came upon it late in life. If I were starting out now, I might make more use of it.

I do use Tinderbox. It is a unique program, quite unusual, and in my view it is a mistake to think of it as a mind-mapping program. It has a bewildering array of capabilities, and many people (myself included) find it quite hard to get used to. You have to throw a lot of assumptions out of the window. For example, I am presently using it to build two websites. These are sort of hypertext resources, rather like Wikis, with lots of linking from individual words and phrases to other pages in the database. All of the HTML for the websites is generated by Tinderbox, and all I have to do is upload the files on to the server. The documentation for Tinderbox is one huge Tinderbox file that is exported to a website (https://www.acrobatfaq.com/atbref8/index.html). You can even download the file yourself to see how it works.

Roughly speaking, I would say that DEVONthink is where I store things, Tinderbox is where I try to make sense of them and analyse them, and Scrivener is where I write about them (if I manage to understand anything). Bookends handles the bibliography.

[jongilizwe]

Just saw this thread. I have a similar setup with BibDesk. But my general advice would be the same - don’t try to use Bookends and DT to track or organize PDFs - do all the organization and tracking in Bookends, and use DT for storage and full text search. Your working unit should be a publication, which may end up having multiple PDFs and other kinds of note files attached to that.

[pete31]

@mbbntu

The documentation for Tinderbox is one huge Tinderbox file that is exported to a website (A Tinderbox Reference File). You can even download the file yourself to see how it works.

Offtopic but maybe useful: “A Tinderbox Reference File” can be exported as single HTML files (Link) which can then be indexed in DEVONthink. If the indexed files are locked in Finder it’s possible to rename them in DEVONthink to the names that are used when they are displayed in a browser which makes it easier to find what you’re looking for in DEVONthink’s search results.

Exported HTML pages can also be used in a Fluid app by encoding and prefixing (file://) the path. Fluid 2 apps don’t have the option “restore windows and tabs from last time” so I use Fluid 1 which opens the pages I often need to look up automatically on startup.

Edit:

Ha, I knew I read about this somewhere: Displaying exports in their own dedicated app - Inter-app workflows - Tinderbox Forum

[mbbntu]

Thanks, I’d missed that one. Paul Walters often provides very useful material. The Tinderbox forum is a great place to learn.

[mbbntu]

Just to enlarge on the issue of tagging versus placing items in folders/groups, you could see this as adding a piece of information to the item: “it belongs in this category”. But what if it belongs in more than one category? Where do you put it? Libraries have been aware of this problem for decades, which is why their catalogues use key words as an additional piece of information in their entries. So I often find series of key words like “second world war, social issues; second world war, poverty” and so on. Indeed, if I download catalogue entries from my local library catalogue to Bookends, I get those key words which the librarians have provided as part of the data. Pdfs of articles, particularly fairly recent ones, will also come with key words as part of their metadata, or included in the text of the article. And, of course, DEVONthink is able to convert some of those into DEVONthink tags. So, the point is that if you have a load of articles that are about New York that are in a folder/group called “New York” and a load of articles about London in a folder/group called “London”, but what you are interested in is city parks, you can tag the relevant files with “parks” and they will show up together in smart searches for the relevant tag.

The big point here is that adding tags is incremental, while moving items into a folder/group is not. If you move an item you “destroy” the information about what category it was in previously. If you tag files, you can add as many as you like to a file, which can be a way of building a picture of what your data is like. And it doesn’t matter what part of the database an item resides in – with search/smart group/rules you can retrieve anything, with any combination of tags you like. And when you get to writing something, tagging items with “chapter1”, “chapter2” can be a help in keeping track of what goes where.

However, building a methodology that works for us is part of the work we do, in my view, and to some extent it will depend on the unique combination of user+work-to-be-done+software. It is a kind of system.

[lpalladino]

Hi @mbbntu, I have a question based on the above: you said that when you apply a tag to a file in the indexed Bookends Attachments folder, that tag will also show up in Bookends itself?

I am trying to set up the best system for both content organization (using Keywords in Bookends currently) and process sorting (to read, priority, etc)-- and can’t find the best way to get tags/ keywords/ labels what-have-you to sync between DT and Bookends on the attachment itself (assuming, as others have clarified, that the Bookends reference is internal to Bookends, so its metadata cannot sync).

Thanks for any advice.

[mbbntu]

Hi @lpalladino – I’ve never tried to sync the keywords field in Bookends with tags in DT or the Finder. I do most of my work in DT these days, with Bookends used mainly for formatting references. I certainly don’t try to organise or categorise items in Bookends. Nowadays I keep notes in DT in markdown, with links to items in Bookends. I put keywords or tags on the markdown notes rather than in the Bookends keywords field.