Files Missing

I just took the plunge and imported my working section of my documents folder, about 60K large. I had It delete the folders and files as It imported. Yes, I do have a backed up folder on another drive which I can copy over if this goes bust - but It should have worked?

It took a long time and imported my folders and their contained files. The file structure on my hard drive that was imported is now deleted.

But now when I search for files in DTPO, I see the line that describes the file, but the lower window that should be showing me the file now says “File missing” and shows the old file path.

I imported this time, I did not index. I am looking at the files and DTPO shows them as in the folders and It does show each file in the row in DTPO. But file is missing on display.

I tried rebuilding the DB in case that was needed but nothing changed? It reported that 56,xxx files are missing - the number that I had imported into my inbox and which are now in my open DB.

I did import everything to my global inbox and then dragged that huge file group to my one open DB.

What am I Missing here?

Thanks

ADDED: OK, I see what I did, what an idiot! Oh well. I ingested my own DTPO DB and then trashed that folder too. I found the DB still in the trash and put It in a higher level of Documents which will never get imported again.

I am now verifying but in case that doesn’t work, how to I get DTPO to recognize the new path to It’s own DB?

Get the human to pay attention 8)

Sorry, no disrespect, but this sounds like a distant edge case – I doubt you’ll find many (any?) postings in the forum where the same thing has happened.

When you import a mass of files, be patient and take it slow. Don’t import 60,000 documents in one job. Don’t import them to the Global Inbox then move them. Decide on the destination database, make a plan for how you’ll set up the destination, then import in reasonable chunks (~1,000 documents max, IMO), and verify after each tranche. If it takes a week, then it takes a week. As RR said: trust but verify.

You’ve described your computer as being a messy mass of data, so why “take the plunge” to cram it all together and delete the originals before you’ve verified the copies? (Personally, I only use “Move Originals to Trash” when I’m copying a handful of new documents – never when I’m reorgnizating a disk that has years of use and documents I’ve long forgotten.) Patience is your friend when it comes to curating your data. :slight_smile:

No, that’s OK, I deserved It. :unamused:

Going forward with best practices, I thought DTPO was all about importing to the global inbox - then going through the inbox and sorting later? I thought It didn’t matter but with so many files - AND deleting where they came in from, I guess I should have decided where to import them first. Honestly, I didn’t know for sure that DTPO imported exactly from where I was when I selected IMPORT. Now I do.

I am letting It chew on rebuilding for now. I have some errands to run, church and all that. When I get back if It’s still not making much progress I will delete and reappoint my documents folders and try again, putting It in the area of the open database this time instead of the global inbox.

Although I will say It again, I thought I could move files around in DTPO once imported? I guess not so much. Oh well.

Thanks.

I didn’t intend to say you cannot use the Global Inbox for importing, and then move to other databases. You can certainly do that. What I mean to say is that if you are going to import 60,000 files it’s a good idea to have a plan – at least have some idea how those 60,000 files are going to chunk out in your database(s). There’s no right way or wrong way to use the Global Inbox, but sometimes there can be a more efficient way.

korm (once again) gives great advice here - advice that Bill and I also chant continuously in Support (including email import): When it comes to bulk import/indexing, have a plan and do it in reasonable increments.

I know some people want to cry, “But it’s 2015 and I have a new Mac Pro with 128GB RAM and an 32 Petabyte disk and a 108” Apple UltraSuperRetina Display and… and… AND 'no other program has this problem.", but the process DEVONthink uses to index (and then organize the data internally) is far more intensive than most applications. Remember, how our parents would tell us, “Don’t take such big bites! And slow down! You’re going to choke!!” :mrgreen:

Also, remember that DEVONthink is an information manager. It is often best to manage information on an as need basis, instead of doing data dumps. This was you can create more focused databases and decrease false positives in AI from irrelevant documents. I’m not saying you can’t use some less focused ingestion as well, but remember that DEVONthink not only indexes but you also teach the AI as you work. The AI “sees” the organization you make and its understanding evolves to make better and better choices in its suggestions. (It’s very cool! :smiley: )

Just in case It wasn’t clear, I Have been working very hard at putting my external folders into a meaningful order. I still have two folders that will need sorting through but I now have a clear idea of how my structure works. And that is, HUGE for me and my past organizational challenges.

SO I am not trying to import files willy-nilly. I felt that my external structure was starting to be strong enough to start to do meaningful associations within DTPO.

I have also been using DTPO for daily tasks and not fully associating that with the actual folder structure that is external. So I Wanted to start working with just the one file sooner rather than later.

AS far as importing my subsections, I can give that more thought.

Thanks

While comments on best practices have been useful to a point, I am not hearing what I am needing to hear, how to fix this? I find that after I exported again, not all my files are there anymore. I had custom icons for my main folder structure and those are gone as well.

I can still see all the thumbs of my files in the DTPO DB but each one has a file missing and a path to file above It. Apparently, this makes exporting files problematic.

I do have an older version of all this on a second drive and I will have to start over and play catch up in an area or two.

Apparently I decided to ingest a large amount of files and DTPO was not up to the task. IF dragging to global inbox was not wrong, then what? I did attempt to inject the DB It’s self as I forgot what level I had It on. I suppose then that this was something I messed up and that It was all my fault.

But now that my data is restored form a DTPO backup, how to I tell DTPO to reacquire the data? Or is this not possible? How to I tell DTPO that, a file with this path to file is now restored so, please try again?

Did I Permanently mess this up my not having the working DB in a higher level? If there is a way to fix this - I have tried telling the DB to re-index spotlight and any other commands I could find to repair, restore and optimize - what is that way?

I seem to have really messed this up and I am asking if there is away to fix things.

Not feeling very confident about using DTPO at this point. :angry:

This forum is useful to a point, but it is not the helpdesk. None of us can see your machine or know what steps led to the problems. It’s time to take this offline, contact Support, and work one-on-one to resolve.

I thought other experienced users might have come across this problem before? You never know if you do not ask. I also thought that DTPO developers did read these forums and have in fact weighed in on questions? In some development communities It is the preferred method of solving problems (asking them on a forum such as this one).

Perhaps, in this instance, asking for help from tech support directly really is the better option. If asking a direct question to tech support is the better way, I can certainly do that.

Thanks.

Obviously from this forum and the comments above you can see that it is not normal for DEVONthink to report just-imported files as “missing”, or to report files moved from the Global Inbox to a database as “missing”. It is not possible from the information above to accurately determine the steps necessary to reproduce the problem. So, please open a ticket with Support. Describe the steps you followed that resulted in the current situation. Describe steps you have taken to attempt a resolution (e.g., Verify & Repair, Rebuild, etc.) Support can then dialog with you for necessary information and suggest steps to take to accomplish the import.

:smiley: Thanks for the link, I have a support ticket in now.