What else is it going to do? It’s not like it can actually create anything.
I think it’s potentially useful as a “smarter” search tool and research assistant. I have no interest in using it as a “writing assistant” or “editorial advisor.”
I was specifically responding to your comment about using it for “big-picture guidance and input.”
(1) I’m a better writer than it is and know the material I’m writing about better than it does.
(2) I refuse to support companies that rely on stolen data and exploitative labor practices to build their training sets, that create environmental catastrophes wherever they go, and that openly celebrate their intent to displace the very creators responsible for the material they stole.
(3) Most of the markets I’m interested in have strong “no AI” policies.
I totally agree with you. I’d also add that I’m French, speak British English (probably with regional elements because the UK still has strong regional accents and dialects), and work in a field with its own defined way of speaking (as many fields do of course). On top of that, I also have to include my employer’s brand voice. I neither want nor need my writing to be turned into a homogenised American English soup, which tends to be the result from LLMs regardless of the prompt (which makes sense, because the bulk of the writing it’s been trained on is American English), even before I add the complexity of brand voice.
(Worth noting I also work in a country where my employer is one of many who automatically rejects American English writing. Even using something like Grammarly sometimes causes issues depending on the intended audience for the writing.)
I forget which firm it is, but one of the biggest global marketing firms did an interview where they said this is how their teams use it too. They’ve built their reputation on their originality and skill, and argued that if an LLM has come up with the same idea then it’s not original and needs crossing off their list of ideas. I found this a really useful thought exercise. (I suppose most people might not be seeking novelty in a creative endeavour, but I can recognise how a high-profile “cutting-edge” firm can benefit from automatically eliminating anything that’s been done before.)
As a novel writer, a certain degree of originality is a must. You don’t have to take it to extremes, but a novel made up entirely of clichés is certainly not something I want to spend a year of my life on.
On the other hand, it can actually be helpful to see what you want to avoid. Setting constraints and boundaries often stimulates creativity.
In that sense, AI can be helpful. But all I need to do is chat with one. I don’t need it in DEVONthink.
Can an LLM create sequences of words that do not appear in its training set? Sure.
Can it create sequences of words that fall outside the probabilistic space defined by its training set? No.
It’s quite informative to read discussions of LLM performance in languages other than English. The non-English training sets are much smaller, and the performance is catastrophically worse. If it had some kind of inherent “understanding” of language, it wouldn’t matter.
I am writing a novel, and I created several “personas” to act as beta readers. I defined a reader: age, gender, marital status, geographic location, education, profession, type of books usually read, etc. The instructions were to read each chapter and comment on their impression of the story, the characters, etc. The results were fascinating; each persona was different in what they focused on about the story, what they liked or didn’t like. Most important was I got feedback on things they didn’t understand, and I rewrote some scenes and dialogues to give a reader a better picture of what was happening.
It was easier than finding real, human beta readers, cost me nothing, and I got instant, very interesting feedback that helped improve my story.
I am writing an historical novel and I use ChatGPT extensively for editing and research. With Chat’s help, I developed a set of specific, detailed instructions for doing a Line Edit, a Structural Edit, and an Historical Accuracy Edit. The instructions include; “DO NOT WRITE. DO NOT REWRITE. When you flag a paragraph, say only what you consider to be the problem. My book/my words.”
Since AI was trained to write, it is tempted to escape my constraints now and then and offer a suggestion for a paragraph or something. I remind it that I cannot use any writing it suggests, it apologizes and we move on. (It isn’t a very good writer, so I’m never tempted, in any case.)
It has become an invaluable tool. All the historical queries are answered with citations that I can check myself for accuracy. I challenge the AI assistant frequently about responses, saying something is not correct, or asking them to dig deeper. Many of these queries would take me days to investigate myself and I would no doubt have overlooked some of them; the answers are returned to me in a nano-second and this AI research assistant is available 24/7.
Basically, I agree with what many of you have said: this is a tool, and you need to learn how to use it for what you want to do. Write strict instructions and remind it to stay on task if it wanders off into the weeds. If it hallucinates, ask: “Where did you get that information.” “Show me the evidence.” “Please recheck that; I don’t believe it’s accurate.” It learns what you want and its usefulness increases with time (and patience). By now, my Chat assistant hardly ever gives me inappropriate responses … I can’t remember the last time I had to correct it for something like that.
And an amusing anecdote: I went to Claude and uploaded the same instructions and tried it out. I liked Claude’s voice, but it constantly failed on historical research. So I opened a discussion about what I was doing – the kind of book, etc., and in the end, Claude said: I think you’d be better off using ChatGPT; it was trained better in history than I was".
Or you could give some examples where it succeeded. Without a clear definition, there’s no point either way.
Personally, I’ve got better things to do with my time than argue with a slop machine.
Edit to add: Measuring creativity is inherently subjective. Still, there are tools than can measure, for instance, the uniqueness of ideas generated in a brainstorming session. I don’t know if anyone has applied those tools to LLM output.
I do know that people have tried to use LLMs to generate synthetic data for psychological studies. In principle, that would let you test an intervention in a simulation before going to the expense of testing it with humans. In practice, it didn’t work, because the distribution of LLM responses was too narrow relative to human responses. I don’t see why any other domain would be different, given the same underlying software.
AI can be extremely helpful for all sort of tasks - including critiquing professional reports, critiquing academic papers, finding specific data from within a large document, offering suggestions pro/con regarding a persuasive letter, solving long-unsolved math problems, designing custom software, debugging software, and so much more.
In the end if you do not wish to try it then so be it. If you try sometime and want advice or guidance in how to improve your results then I am sure you can find that help from many users here.
That’s interesting. I could have benefited from a fact-checker for some of my novels; I wouldn’t have made a few embarrassing mistakes then. You research things you don’t know… but you don’t double-check things you’re (mistakenly) sure you know.
Line-editing – not so much. I use Papyrus Author, which has a function to highlight the most common pitfalls (a simple algorithm, no AI needed); that’s enough.
But, DCBerk, you do all this while chatting with ChatGPT, correct?
Yes, it’s an active working method. I upload a chapter, tell the assistant to do a Line Edit, or whatever, and the instruction says to enumerate the paragraphs, write a comment between paragraphs, and pause after a scene (usually set off by a dingbat of some kind in the text), for me to respond, contest, ask questions, etc. With that scene settled, I ask it to continue to the next scene. When it has finished the chapter, I ask it to summarize which paragraphs need work so I don’t have to look through the whole chat to find what I need to deal with.
Then I use a Firefox extension to download the html, convert it to rtf, and bring it into Scrivener. I open the chapter, open the chat in a split window so I can see both, then make the changes.
Depending upon how extensive the change are, I reopen the chat in my ChatGPT Project, paste in paragraph revisions to ask if there are remaining problems, or maybe upload the revised chapter for a complete look.
The History Mode assessment often leads to conversations about some historical subtlety (would she be wearing gloves in that scene might lead to a whole conversation on sartorial expectations).
I have a subscription that allows me to do deep dives for the research, and I get all the citations for everywhere the assistant has searched so I can check accuracy, download things, etc.
It also does rapid translations when needed.
Extraordinary help for doing research for any historical novel.
I don’t trust the big cloud AI companies because they’re a bunch of thieves and liars. In fact, I’m not keen on cloud companies in general — a key reason why I use DEVONthink is that I can have sync run with end-to-end encryption using my servers.
I forget exactly which Qwen3 I used, one of the ones recommended for RAG document search, but after failing to get good results I deleted it. I tried llama3:latest as well. Since I have something that runs locally in under a GB using Apple Intelligence and does the job adequately, I’m not very interested in spending a lot of time messing with multi-gigabyte downloadable local models and the clunky software to run them. I just think it would be nice if DEVONthink offered the same functionality, a way to select a Group and chat across the whole group.
Now that I think about it, I guess I could keep the documents outside DEVONthink and index them for separate search, but I like the convenience of having everything in DEVONthink.
Certainly that is your prerogative. If you choose not to use Cloud AI and you also choose not to set up meaningful local AI infrastructure, then perhaps that explains why you have so far been underwhelmed with AI capabilities.
I’ve said more than once that I have a working local solution. I’m just underwhelmed with the free downloadable models from the big names, and the clunky software to run them.