Further indexing problems

In 1.3b2, in addition to getting lots of failures on importing images and PDFs, I’m finding that it’s taking it a VERY long time to fail on an image (an image which looks fine under the Finder, btw). It can take it a full minute to finally fail on an image that has no problems. Any idea what’s going on? Anything I can do to help debug?

John

Ok, I have a simple test case which shows this problem in both 1.2 and 1.3b2:

  1. Create a new database.
  2. Cmd-Opt-Drag the “images” directory in the following disk image into your database. It should take forever to ultimately tell you that all the images failed to import.

johnwiegley.com/Images.dmg.gz

Note: In my preferences, I have everything checked as “to be imported” except for Applescripts.

John

John, I didn’t time it, but it took less than ten seconds to index the 43 images into my database. (I’d say about 10 seconds elapsed between the time I hit the File > Index command and then looked for the new group.)

All of the captures were successful; I looked at five or so of the images, and the Log contained no error reports.

I’m running DTPO 1.3beta2 on a MacBook Pro 2.0 GHz with 2 GB RAM, under OS X 10.4.8.

You might want to run some maintenance operations on your OS and disk directory.

I’ve been having image import problems for about 2 weeks now. In fact, I found a PDF which causes my DTP to hang and never return:

johnwiegley.com/bpl.pdf

OK, I really need to get this working again. I can believe that it’s not a problem in DTP, but how can I debug it? I’ll try doing full disk maintenance. Perhaps I have some kind of addon in the system that’s colliding. Do you have clue what might be causing a hang-up like this?

Note that if I turn off image importing, I have almost no problems at all – save for the PDF given above.

John

p.s. Thanks again for the quick response.

It now appears that attempting to Index any PDF causes DEVONthink to go into an infinite loop (no CPU time). I gave it about 30 minutes and it wouldn’t finish, on a simple PDF that I know it’s indexed before.

I’ll start doing my best to analyze it here.

John

John, that PDF that you can’t capture was captured into my database essentially instantly.

You might think about any software you have downloaded within the past few weeks. Christian commented in a similar case that third-party QuickTime plugins could cause problems.

Try removing such items to see if the problems go away, then perhaps (if you need them) add them back in one at a time to see if you can identify the trouble-maker.

I’ve disabling the following so far, to no effect:

/Library/
Internet Plug-Ins
LaunchDaemons
PDF Services
PreferencePanes
QuickTime
StartupItems

I don’t use personal QuickTime addons.

I’ll just keep narrowing it down. I also have another PowerBook, so hopefully I can do a system comparison that will lead some answers.

I also repaired both volumes on my machine. One thing I did find: that during the enormous pause before failing, there is no CPU activity at all.

John

A few other notes:

I am now unable to import any PDF or image file. Also, I’ve proved that I can open these same images in QuickTime instantly; they also import into iPhoto instantly.

I have the following sample data, taken for 20 seconds during the 28 seconds it takes for an image import to fail (and this amount of time is consistent).

John


Analysis of sampling pid 303 every 10.000000 milliseconds
Call graph:
    2000 Thread_0f07
      2000 0x274d
        2000 0x2826
          2000 NSApplicationMain
            2000 -[NSApplication run]
              2000 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]
                2000 _DPSNextEvent
                  2000 BlockUntilNextEventMatchingListInMode
                    2000 ReceiveNextEventCommon
                      2000 RunCurrentEventLoopInMode
                        2000 CFRunLoopRunInMode
                          2000 CFRunLoopRunSpecific
                            2000 __NSFireDelayedPerform
                              2000 0x1d259
                                2000 0x1158c4
                                  2000 0x114514
                                    1992 0x1186fa
                                      1992 _objc_msgForward
                                        1992 -[NSObject(NSForwardInvocation) forward::]
                                          1992 -[NSConnection sendInvocation:]
                                            1992 CFRunLoopRunInMode
                                              1992 CFRunLoopRunSpecific
                                                1992 mach_msg_trap
                                                  1992 mach_msg_trap
                                    8 0x118645
                                      8 -[HelperTask launch]
                                        5 +[NSThread(NSThreadDevonExtensions) sleepTimeInterval:]
                                          5 usleep
                                            5 mach_wait_until
                                              5 mach_wait_until
                                        2 _objc_msgForward
                                          2 -[NSObject(NSForwardInvocation) forward::]
                                            2 -[NSConnection sendInvocation:]
                                              2 CFRunLoopRunInMode
                                                2 CFRunLoopRunSpecific
                                                  2 mach_msg_trap
                                                    2 mach_msg_trap
                                        1 -[NSConcreteTask launch]
                                          1 fork
                                            1 fork
    2000 Thread_1003
      2000 _pthread_body
        2000 __ape_agent
          2000 mach_msg_trap
            2000 mach_msg_trap
    2000 Thread_1103
      2000 _pthread_body
        2000 forkThreadForFunction
          2000 +[NSURLCache _diskCacheSyncLoop:]
            2000 CFRunLoopRunInMode
              2000 CFRunLoopRunSpecific
                2000 mach_msg_trap
                  2000 mach_msg_trap
    2000 Thread_1203
      2000 _pthread_body
        2000 CMMConvTask(void*)
          2000 pthreadSemaphoreWait(t_pthreadSemaphore*)
            2000 semaphore_wait_signal_trap
              2000 semaphore_wait_signal_trap
    1322 Thread_1303
      1322 _pthread_body
        1322 wait4
          1322 wait4
    678 Thread_1403
      678 _pthread_body
        678 wait4
          678 wait4

Total number in stack (recursive counted multiple, when >=5):
        5       _pthread_body

Sort by top of stack, same collapsed (when >= 5):
        mach_msg_trap        5994
        semaphore_wait_signal_trap        2000
        wait4        2000
        mach_wait_until        5

Oh, and I perhaps failed to mention: I tried back-versioning my system down to DTP 1.2, but the same problem exists. So it’s not something new in 1.3b2, but some kind of evil interaction between newly installed additions to my own system. :frowning:

John

Guess which directory I had to disable in order for everything to work:


~/Library/Caches/ShapeShifter

Because apparently having ShapeShifter UI mods active (which get deactivated if this directory is disabled) is causing a bad conflict with DTP, and the importing or indexing of images and PDFs.

What’s strange is that this problem did not appear as soon as I installed ShapeShifter. In fact, it worked for a while, but only gradually started to fail, until today things failed completely. Also, I know other people who use DT with ShapeShifter, and have no problems (yet).

Since I don’t have the tools to figure out exactly why these two are conflicting, ShapeShifter is going out the window.

Thanks for trying out the images I posted, Bill, you’re great.

John

Over time ShapeShifter has been involved in a variety of problems, and we really wish people would not install it, as it does nothing useful.

I’m cautious about things that deliberately modify Apple’s carefully designed operating system. Modifications involve potential stability risks; I don’t think modifying appearance justifies the risk, especially if one needs to get work done on the computer.