From your description it appears this is related to the “fallback routing feature” in OpenRouter. For reasons of operational efficiency or availability not completely clear, OpenRouter may at times use a fallback model.
This “feature” can be disabled via their API - but I believe when using DT4 that would need to be part of the DT4 app and could not be chosen by a user.
OK. It really seems it wasn´t a “fall-back” mechanism on OpenRouter side.
Below the answer after some detailed reporting and feedback from OR-side. – Basically they say, it must have originated within DEVONthink… which kind of brings me/us back to square one, it seems.
The question remains, why/how would DT on its own cause the call to an Amazon model when this was never selected anywhere in the presets?
At least this is my understanding/take-away from the OR answer. But maybe there is something else to it, that is not so obvious…?
The OR reply to my inquiry:
”Hi Oliver,
Thanks for the detailed info — that really helps.
I checked your logs, and there were indeed requests made using the API key ending in dt2. Based on the data we have, these requests came from Germany.
It’s possible that something in DEVONthink was still running, which could explain the continued requests. OpenRouter does not automatically switch models without an explicit API request specifying them, so this behavior is unusual.
Could you please check with DEVONthink support?”
It doesn’t on its own. But scripts, templates, smart rules, batch processing, summarization, transcription of images & PDF documents and even the chat assistant can use different models on demand.
The only possible scenarios are either right after entering your API key and loading the model list in the settings, in this case the first model is the default model. Or that your formerly preferred model wasn’t available anymore (and then the first one is again the default) but I have never experienced this on my own.
I think the question flowing from this for all people using OpenRouter (or any other multi-provider LLM in the future): is there any way to know/control the fallback-model, as to exclude the possibility it’s an umwanted one?
I think one way of course is excluding model use on the side of provider. (Though this also not very “in your face” as ordinary user, looking at the OR-dashboard etc)
This seems to be able to solve this technically; but also it doesn’t seem/feel to address the question of (random) opacity that seems to be the case here (after all, nowhere did I ever choose Amazon anywhere, nor do I - consciously - use scripts that use AI at all)
Maybe also worthwhile, otherwise, to include some kind of flagging somewhere in the documentation?
Just some individual/personal thoughts as to address issues of system-opacity and involuntary server connections; which might make people wary of using AI altogether otherwise.
So far it’s still absolutely unclear what exactly caused this issue and whether it’s working as expected or not. Therefore additional details would be welcome in case that you should be able to reproduce this.
OK. I´ll definitely let you know whenever something likewise happens again!
For now I have blocked NovaLite at OR, so I guess it would take another form.
Thing is that it´s mainly the issue of not being able to pin this to anything, and I wouldn´t know where to look beyond the infos already shared. Only thing I positively know is I never chose the model by intention, nor was I aware it´s “in the (active) mix, here. So, maybe it´s good it´s up here, as it seems this will (only) be relevant in case others make similar experiences. As remarked earlier, it seems the pooling of experiences given all possible pitfalls seems to me the way to go, as to AI use in DT.
For me the two positive ‘takeaways’, even beyond the special case/riddle, nevertheless and all other possible circumstances aside are two:
there is a very helpful log of the chat, that makes things more transparent and allows a kind of auditing. So I think it would be good everyone into more complex LLM setups is aware things (connections) are actually logged in ~/Library/Application Support/DEVONthink/Chat.log
second ‘news’ to me is
… so, I was/am thinking it might be good the “default model” is exposed somewhere to the user. So, people can be aware, check things and make informed decisions.
… but other than that, I will certainly get back should a similar problem appear, and/or should there be more hints as to possible sources and/or patterns.