Organizing an investment database

I’ve been reading through “How to organize database for archives of local newspaper?” from the forums to get ideas about designing a new database that’s very different from the organization of all my other ones like my financial db (copies of all bills, statements, etc - a very simple organization relying exclusively on group hierarchy). I want to store a myriad of articles on investment topics - company reports, analyses, economic studies, investment strategies, etc.

I thought I’d post to see if I can get more ideas for this specific topic, as the article above covers many ideas but for a different use case. I suspect this investment database is not a simple one like my other ones that primarily rely on the folder/group hierarchy. Investment articles have many relationships - there is the subset that focuses on a specific company, another that relates to a company’s sector, and general economic articles that pair a company or sector or country with a cycle.

I have used several apps in the past (most recently is TheBrain in concert with DTPO) but I never seem to get a workflow/data management system that feels good. I thought TheBrain could be used to add structure while DTPO would reference external files where the AI (using See Also) could augment the visual organization in TheBrain. At this point, I’m going back to DTPO (but still plan to use another app for additional data and structure), but I think I need to begin relying of features that I haven’t needed in the past nor used in my other databases.

I am frequently gathering PDF files of articles I find at a variety of investment sites and documents received from investment services that cover the current state of individual companies, sector analyses that include companies within a sector and the business climate in those sectors, posts of current economic/political news (e.g. having to do with the Fed and other central banks, and anything that includes analyses of economic conditions in the US and abroad), and any other pieces of news/facts/opinions. I also add txt/rtf notes that condense topics for quick reference and other types of files such as images and spreadsheets.

The database will also include sections that provide a dashboard (e.g. actionable trades, current news that I’m following - basically anything that I am currently working on), reference articles (those that don’t become stale, hence I want them available when needed - most other files in the database have a relatively short shelf life), a glossery of terms (e.g. AFFO vs FFO, BDCs, REITs and rules that goven them) - yes these are reference articles/notes, but I prefer to have them in their own folder/subfolder). Hopefully you get the idea.

I already have these files in Finder, used in previous attempts at organizing. I have used tags with other apps to help filter searches, like the ticker symbols for companies mentioned in articles (e.g. $ibm, $ge, $aapl) and sectors (e.g. %utilities, %tech). Other tag sets have been used (e.g. /reits, /bdc; /fed, /inflation). These tags were defined in TheBrain, and prior to that, in other apps, but I haven’t used them in Devonthink, since I didn’t use DTPO’s See Also.

My hope is to use DTPO to search for articles given a particular topic, like investing in $csco at the current time. I would want to find recent articles on Cisco as well as general articles on the tech sector and reports on the current market and sector cycle (overvalued, undervalued, …).

So I’d like to rely on DTPO using more of the underpinnings to help its AI do a lot of the work. I’m thinking of defining a few main folders for storing articles, when I index them to DTPO - I have a design that I don’t want to waste time explaining that requires all files to be indexed. Without a cumbersome folder hierarchy, it will be much simpler to acquire the files and sort them to the right folder. Hazel might be used here, as I use it for the financial db. I then rely on this “black box”, DPTO, to get the articles I want without my traversing a folder hierarchy to find them manually. (My design will rely an another app to provide a visual way of finding files, similar to TheBrain. So let’s not worry about that issue here.)

So here are my questions -

  • Do I use tags for things like tickers, or do I use groups (where groups are not excluded from taging - something the forum article referred to above gets into). This certainly implies the group structure in DTPO can be more complex than the folder structure in Finder. Using group tags are needed for See Also, so using standard tags may not be wise.

  • The groups in DTPO - any suggestions as to what they should represent? Some of them depend on the answer to the first question. I could have a “Sector” group with a subgroup for each sector, e.g. “%tech”, and replicate articles to another group hierarchy for each ticker (“Tickers” > “$csco”). Another hierarchy might list subtopics such as “Economic cycle” and “Rate sensitivity” and articles here may also be replicated to multiple ticker groups and sector groups if appropriate. But now I’m seeing a possibility of quickly expanding “other group” hierarchies. So…

  • Would standard tags also be useful in addition to tag groups on tickers and sectors, when more and more relationships develop? (As long as the AI would not be expected to search for these tags when using See Also.)

  • And speaking of a short shelf life for most articles, I’ve used TheBrain’s calendar to set a review date with alert to determine whether to delete the file or not. Any ideas on implementing this in DTPO?

And distilling all this, does it come down to using group tags instead of standard tags for anything that I want included in a See Also search, everything else can be a standard tag that I can do a simple search on (this assumes I don’t exclude groups from tagging)?

Just a quick note: You won’t be able to use your prefixes as search terms. The $ will be ignored. (Also, this practice of prefixing is outdated. You can use it if you’d like. Some peope do.)

It seems that you have a very good handle on the types of things that you want to collect and curate, and their categories and relationships. (Tags are good for adding additional dimensions of categorization to complement your groups.) However, I’m curious about how “work flow” applies to your situation. You mention work flow, but I don’t see it fleshed out. The design, organization and management of the database (and other programs you use in conjunction with DEVONthink) are driven mainly by how you are going to use the database – a factor as important as what’s inside the database.

Examples of work flows I think could apply in the situation range from very formal: you have a defined sequence of daily, monthly, weekly, annual tasks that you want to accomplish with your data and you want the database to keep track of and facilitate accomplishing those tasks. Or the very informal: you are following research threads, articles, hints, tips, advice wherever it may lead and you want to capture what comes up and then separate the cruft from the gems. Or something that mixes those two.

Where do you see your work with the database falling along the formal-informal spectrum? If you have a set of repetitive tasks in mind, I suggest your toolbox should include something outside DEVONthink that can record DEVONthink URLs for documents or groups – ranging from simple tools like TaskPaper to more complex like 2Do or its cousins. (The latter can be useful on iOS and open links in DTTG if you are syncing your database to a different platform. I also use Curio for work flow and it plays very well with DEVONthink.

I rarely use tags to manage steps in a process or to-dos. You have to be constantly adding and removing tags to make this work well, it’s easy to lose focus and forget to turn off a tag when a step is finished, and it’s very hard with a large database to find tagging errors. That’s another reason to use an external task manager. I’ve tried to use TheBrain for this, and ran into the same problems – plus it is very unlikely that next release of TheBrain will support calendar sync, so it’s not worth the effort to start tracking tasks in TheBrain.

On a different note – I have never found it useful to plan out the structure of a database in great detail before I create it and start populating it. DEVONthink is great for experimenting with structures, tearing them down, and reorganizing. If you focus your group making on structure (relationship, hierarchy, which are what groups are), and your tagging on concepts and attributes, then I find it a lot easier to restructure the group hierarchy whenever needed. I would also “exclude from tagging” almost all groups (this can be turned off for the whole database) because it can get really confusing if you have a mix of normal-group tags and Tags-group tags.

Oh, right - I remember now.


Workflow -
My use of “workflow/data management” is more concerned with how smooth a process is. It’s part of that “look and feel” issue.

But I do have some defined workflows as you mean them, like quarterly reviews of each current investment. Or periodic pruning of out-of-date articles. More often, it’s turning to the database as I read, for example, a glowing review of a company only to find a past note I saved indicating that one of their clients is nearing banckruptcy. I want that process to be quick, not to involve a traversal of part of the group hierarchy in the database, searching for some old note/article that helps me reach a quick decision about whether an possible investment is good. (Your second paragraph of your post above captures a good part of my processes.)

As for tags, I think the basic ones of “ticker” and “sector” are very easy to maintain. And, yes, I’ve read about the next iteration of TheBrain excluding calendar. Another strike against it.

As for your last paragraph, having already creating a DTPO/TheBrain database, I’ve learned what works and what doesn’t, so this new attempt will be cleaner (like rewriting a software program from scratch after having written version 1 and returned to patch it a few times - I’ve done that countless times). My first step is to redefine the finder folders that will store the various articles - here I want to streamline the process by having fewer folder and fewer sublevels. Using indexing, I can then design the structure of the groups differently if I want, and I can also redesign it in the future as I see fit.

Question: (Using indexing) In the extreme, is it okay to put virtually all files in one finder folder (this would behave as an inBox), but then parse them out into more appropriate groups in DTPO, would the AI still work or is the finder folder added to the AI algorithm?

BTW, there’s been a thread in the forum about one user insisting on DTPO maintaining the same stucture as the finder folders, when using indexing. My description above shows how DTPO’s current implementation of indexing serves to my advantage.

Though I’m a portable database kinda guy :smiley: , yes - you could have an “Inbox” in your filesystem and index it. Then you could sort and move stuff in the database as you see fit.

ABSOLUTELY. The current behavior lets you sort in-database, in whatever way you see fit, and it will not disturb the Finder folder.

Note that I suggest having a Smart Group showing Instance is Indexed, to help keep track of items that are indexed but moved about. I find this helpful anyways. :smiley:

Yes this works well. You can distribute indexed files all over your database into groups – including replication if you need them. The tags will get written down to the file in the file system, which can be useful if you ever need to use Spotlight on that folder to locate something. The AI does not care where the file is in the file system – it works off of metadata that is collected in the database when the file is indexed. Be sure if you have PDFs that they are OCRd.

The AI will be a big help on this, as will the Search box, and saved searches.

Having your personal data on disk in a non-hierarchical folder or a limited number of folders, and using search tools of varying capabilities – or a data manager like DEVONthink – is not an uncommon practice. I think the worthies who recently argued heavily here to have DEVONthink mirror a hierarchy in Finder missed the point. If DEVONthink is going to do your organizing and reorganizing, then you don’t need organization in the file system. If you use DEVONthink to organize your data in an intelligent manner, then seeing it organized the same way in Finder has no added value.

It’s sort of like cord-cutting – and sounds treacherous but it works well. If a time comes that you really need hierarchical file storage then you can quickly export your DEVONthink hierarchy in whole or in part to folders that DEVONthink’s export feature will create for you in the destination in the file system. There are several threads in the forum history with use cases like this.

Do I spy a backhanded compliment here?

Speaking as one of the “worthies,” or at least a less worthy person who agrees with the more worthy members of our community, I can see the benefits of having both systems (having it mirrored and not having it mirrored), and I don’t think the current one (only names of Finder folders get changed, and nothing gets moved) ought to be dropped in favor of a different model. Instead, I think an option in Preferences for allowing indexed items to move and get renamed in Finder would be a great solution.

Yes, I know that we should try as much as possible not to have an excess of options, but I would say that indexing is one of DT’s extremely compelling features, and something that clearly distinguishes it from a lot of other apps, so having an option like this for one of the core features seems worth the effort. There are plenty of us out there who greatly benefit from having things mirrored in Finder (to a point – I realize there is plenty that can’t and shouldn’t be mirrored, and DT hasn’t been and won’t be a Finder replacement).

Getting down into the weeds, I often import massive amounts of data into an “unsorted” group / indexed folder in DT. Then, I use the fantastic AI (it could do a better job with Asian characters, but that is a discussion for another thread) to sort things into their destination groups / folders. Let’s say that I am on my iPad in class, though, I am standing in front of a room of about 100 students, and I want to open up something in class that I forgot to put into Mobile Sync. I just open up SpiderOak (my cloud storage of choice) and get the file if it is where it ought to be in the hierarchy (it might take 30 seconds or less), but if it hasn’t been moved in Finder, it will be impossible to find in a timely manner (it could take minutes or hours to track it down). There are a bunch of other examples I could give.

I’ve been living with the current system for years now. I can put up with it indefinitely, if needed, but I think there is a much better way, at least for my workflow. I would suggest, though, that newbies would greatly benefit from this option as well.

Oh, you’re worthy, but not one of the “worthies” – they know who they are :laughing:

You have a good point about indexing / hierarchy / mobile device. I was actually just thinking about the desktop, my local volume, and DEVONthink OS X. I’ll admit I use small ad-hoc Dropbox hierarchies for documents I need on the road and sync those with GoodReader. Some of that hierarchy is occasionally indexed with DEVONthink on the desktop.

I see :slight_smile:

Actually, today I am teaching classes, and I’ll take an indexed folder in Finder, drag it into AirDrop a few seconds before I got to class, and it will automagically get opened up in GoodReader so that when I get to my classroom, I open up the iPad, and there are all of my materials. It’s amazing. If stuff doesn’t get moved around in the Finder folders, this becomes impossible, and it is a major problem in my workflow. Sure, I could have everything prepared weeks in advance for each class! But,I have only managed this a handful of times over the years during my time as an undergraduate, graduate, and now a faculty member, so I don’t think that is a viable solution (even if it would solve a lot of my pain points on the iPad and Mac). Do such people actually exist?

“Do such people actually exist?”

I’d say 99% of the time I’d be updating my course website (last minute editing of the lecture and/or handout materials) during the last 5 minutes before the start of class.

But I suppose there must be those who fully prepare weeks ahead - there must be!

I expect I am the aforementioned “worthy”: Is that an insult, and if so, do you always insult new users in such a nasty way? I didn’t miss the point; I just don’t use DEVONthink the same way that you do. I do need organization in the file system and DEVONthink is not doing my organizing and reorganizing; I do the organizing and reorganizing myself because that’s the best way in my use case. The primarily value of DEVONthink for me is the see also list.

I can’t speak for korm’s intentions, but speaking for myself, I did not infer that it was an insult at all. In general, sarcasm, subtlety, and nuance is tough to get across to other people on Internet forums, so I assume the best meaning possible and don’t worry too much about it. My post about the “worthies” was only to poke at korm with some friendly banter, and not meant to imply in any way that I took offence.