DEVONthink Server stops responding

First, thanks for the help in getting started with Apple Script! This is something I’ve not much tinkered with before, and it’s always great to have a new tool in my kit.

Second, I went with a JavaScript bit of code to shut down and restart DT3, which works nicely. The code itself is as follows for anyone else who wants to use this procedure as I will:

(() => {
	/* First, find and quit the app. */
	console.log("About to close DEVONthink 3...");
	var app = Application("DEVONthink 3");
	app.quit();

	/* Now wait a few seconds for it to close, then restart it. */
	console.log("Waiting...");
	delay(5.0);	
	
	console.log("About to restart DEVONthink 3...");
	app = Application("DEVONthink 3");
	app.activate();
})()

I saved this to a file named “RestartDT3.scpt” and can run this from the command line via:

osascript RestartDT3.scpt

And for the final bit of automation, I used the following command line to create a new crontab:

env EDITOR=vim crontab -e

Along with the following contents to execute the necessary command at 04:00 hrs. every morning:

0 4 * * * cd ~/ && osascript RestartDT3.scpt

We’ll see how this works when I’m next out of town. So far, it always takes longer than a day or so to stop responding, so I’m hoping this is the end of it.

2 Likes

Nice work. Let us know how it behaves, especially on its schedule.

1 Like

Just as an aside, this might be one of the longest running support questions I’ve seen people trying to debug a problem that seems very hard to pin down. Kudos both on the support side @BLUEFROG and on the user side @Phileosophos

4 Likes

Thanks. And @cgrunenberg is a big part of this as well on our side of things.

1 Like

The next release will add a new hidden preference (ServerDebugLog) for extended logging by the server. Maybe this will help to figure out what’s going on.

2 Likes

Sounds good! Perhaps it’s serendipitous timing that I had the server stop responding to me last week while I was typing a note. That hasn’t happened before. I’m attaching the zipped log files to this message in the hope that they tell you something. I can’t tell you the exact time to the minute, because it took me five or ten minutes to figure things out, but it was within 5 - 10 minutes of 2022-17-06 10:30 hrs. PDT. The DT app had been successfully restarted that morning by my cron job, so I could easily use it remotely, and I was taking a note about how to colorize a black bitmap in Photoshop when it stopped responding to me.
DevonThink 3 WebServer.log.zip (529.7 KB)

1 Like

Thanks for the log! At that time an HTTP GET request to view a certain file was repeated again and again (214 times in 87 seconds). And most likely that’s what’s making the server unresponsive.

Could you please execute the following script in the Script Editor.app? What kind of document does it open and how large is it?

tell application id "DNtp"
	set theUUID to "94351832-5AEC-4E8F-8CAB-FFB20E9360AD"
	set theRecord to get record with uuid theUUID
	open window for record theRecord
end tell

Sure. It opens the markdown document on which I was working at the time, entitled “Photoshop - Colorizing Line Art”. Today it’s 1.4k, because I’ve since added to it, but I can tell you at the time it was only 505 bytes in length if that helps.

Were both the edit and preview panes visible in the web interface?

Yes. And I type extremely fast if it helps (160 WPM+).

It seems the new release of DEVONthink has changed the behavior of the whole server-stops-responding problem. To wit, rather than simply not responding to requests anymore, the server gives me a login page that I can never log into successfully. That is, I now get the login page that asks for my user ID and password, and no matter how many times I give it the correct ID and password it just keeps giving me that login page. Does that help? Or make sense in light of recent changes? Is there any way for me to debug this new variation?

Does the browser matter or which one did you use?

As a matter of fact, it does. I forgot to check before, but I find any other browser accepts my login. That makes me wonder if there are two different issues and my preferred browser (Firefox) is keeping a cookie too long or something. I’ll clear them and see if that fixes the login issue.

UPDATE: Yup, clearing the 23 cookies Firefox had saved for the site fixed the login issue. I’m back to using Firefox. I’ll keep monitoring and see if the old stops-responding behavior or the login issue returns.

UPDATE 2: And here we are a few hours later with Firefox having 15 cookies on file and can’t log in until I clear them again. It would seem something isn’t right with the way login is being handled. Or with Firefox I guess, given that I can log in again with Microsoft Edge. Any suggestions?

1 Like

Does this happen after logging in via Firefox, using the web interface and then not using the web interface at least 30 minutes but keeping the tab open?

No. I’ve tried it twice so far this morning. Here’s what I observed:

  1. Open a tab to my DT3 server and make a new markdown note just fine.
  2. Stop using the tab for 30+ minutes.
  3. Come back and click a different entry in the UI and get a 403 forbidden error in the content pane.
  4. Refresh the tab and log in again to access DT3 successfully again.

So far it hasn’t given me the login weirdness I was seeing yesterday. For the record, the Firefox settings/privacy section shows only six cookies stored presently. I don’t know if that number is significant or not, but I’ll keep an eye on it as the day passes.

I did not see the login issue again today. But the server did stop responding completely in the more usual manner (i.e., no response at all) somewhere between the last time I used it earlier and now. I understand this new version maybe provides more log information? Should I perhaps look for that somewhere? Or enable some flag to produce more log information or something?

Please quit DEVONthink 3, delete the current logs (~/Library/Application Support/DEVONthink 3/Console.log & ~/Library/Logs/DevonThink 3 WebServer.log) and then enter the following command in the Terminal.app:

defaults write com.devon-technologies.think3 ServerDebugLog -bool TRUE

As soon as you’ll notice the issue again, the server & console logs would be useful plus the time of the incidence. Thanks!

Ok, I don’t have a precise time. I know I was successfully using the DT3 web server from my PCs last Friday, July 8, and I know it wasn’t responding when I tried again this morning, Tuesday, July 12. I had to restart DT3 to get it working again. Attached are the log files over that time period.
Archive.zip (41.8 KB)

Did you or anyone else use the web interface on July 10 too? Several requests on this day were actually logged. But no issues were unfortunately logged and no more requests received after July 10.

Hmm… I didn’t think I did, but it’s entirely possible. I’m sorry to hear it generated nothing additional. Perhaps the next failure will be more precise and useful.