Gemini Pro 2.5 rolled out a new capability over the weekend, this one is called “Deep Research”, but shouldn’t be confused with notebookLM, which summarises an uploaded PDF. “Deep Research” returns an in-depth research report based on a single prompt.
To test this new capability, I provided the following prompt: Tell me about the book “The Extended Mind”, by Annie Murphy Paul.
The response was a circa 12k word, extensively sourced report, that took 5 to 10 minutes to be generated.
In much the same manner as with my notebookLM tests, I specifically choose a book that I know well, to help me judge the quality of the results. And those results were excellent, but you can judge that for yourself, as I’ve published the unedited results at the link below.
Don’t worry if you don’t know the book in question, if the analysis provides you with a decent flavour of Paul’s arguments and their rigour, I’d judge that a success alone. One interesting note ref the sources used to compile the analysis, there’s a smattering of video sources as well as text sources.
That’s very interesting - thank you. I very regularly reseaerch and cite peer-reviewed literature in my work, enough that I currently subscribe to OpenAI’s Deep Research offering.
At first glance Gemini Pro Deep Research at $20/month is very stiff competition to OpenAI at $200/month. But in the past I have noted considerably more hallucination by other Gemini versions compared with competitors. With that in mind these are comparative reports from the two -
Gemini Deep Research:
OpenAI Deep Research:
Gemini’s product is the best I have seen from Google’s AI offerings so far. I do note two significant differences however:
I asked both to restrict responses to “credible research data.” OpenAI clearly did that. Gemini appeared to simply accept whatever sources were in Google - including a reference to Wikipedia, which is entirely inappropriate for deep academic research.
The OpenAI report is overflowing with inline hyperlinks directly wherever they are pertinent. The Gemini report however simply lists non-hyperlinked footnotes; you need to go to the footnotes section and click the link from there. For a detailed report with dozens of references, this profoundly increases the cogniitve work to understand and read the original soruces, which is the whole point of “deep research.”
Thus overall this is clearly Google’s best effort so far on detailed research. The price difference betweeen Googe vs OpenAI Deep Research is quite notable - thus for many (maybe most) readers the Google product is sufficient. But for someone who truly has a need for deep research - such as graduate students, researchers who are writing/publishing academic studies, and other professionals who need to write fully-cited reports- OpenAI Deep Research still wins the # 1 spot.
Gemini Pro 2.5 “Deep Research” will customise its report based on a more detailed initial prompt, or with follow-up prompts detailing preferences for specific sources, e.g. a preference for academic sources. This is of course is within reason, as academic journals mainly sit behind pay walls, and that’s probably part of the massive price differential between the OpenAI and Gemini products. I would expect Google to release a more specific product offering for the academic community as Gemini matures.
But I would also add that I was simply looking to see the results I could get from the Gemini Pro Deep Research model with the simplest of initial prompts.
At this stage, I wouldn’t suggest which product is no.1 as that particular metric is a very fast moving target. I would however state that Gemini Pro has really impressed me with regard to its pace of development since it was made available to the public in May 2023 (especially after the disaster of Bard). The advancements since Gemini Pro 1.5 have been particularly impressive.
With regard to the Gemini Pro subscription cost, it’s actually part of my Google One subscription, which provides the benefits detailed below for £199 annually. For good or ill, I’ve used Gmail since it’s inception and make regular use of most aspects of the Google ecosystem. I also subscribe to Kagi (as I appreciate private advertising-free search) for approximately the same cost as Google One, and this provides me with access to all the other main Ai models, inclusive of those from OpenAi. But much like with Perplexity Ai, the service level for those Ai models isn’t as good as you get from a direct subscription. However, it’s more than good enough for my needs.
I’m a design strategist by trade but mainly work as a consultant these days, so I look to get the most from my subscriptions to Apple One, Google One, Microsoft 365, GitHub Copilot and Kagi. The combined cost is far less than OpenAI’s deep research offering, and I really don’t feel like I’m lacking for choice with regard to the tools I need to feed my “second brain”.
Interesting points - no question the Google offering will be compelling to many.
I don’t think OpenAI Deep Research goes beyond abstracts for paywalled studies but I am not certain.
My biggest concern with the Gemini report is the use of footnotes rather than inline hyperlinks - I am puzzled why they would do that. It does not cost them more either way. But the OpenAI format is immensely more usable given the hyperlinks.
I might even regularly use both - but only if Gemini were to change and include inline hyperlinks. Finding the sourceds is just way too much cognitive friction the way Gemini has formatted it.
[I did see your post about Kagi the other day and it has me pondering if I should give it a try by the way.]
I’m happy with the format of the report, but the format is a fast moving target. What I mean be that is that the Google team are actively making “improvements” to the reports. The reports I’ve generated today are stylistically different to those I created yesterday. Luckily, the changes do actually feel like improvements. But I’m readying myself for changes that won’t appeal to my subjective tastes. All part of the fun and games of working with experimental features.
BTW, I use Obsidian in tandem with DEVONthink as they each have differing strengths and weaknesses. One of the strengths of Obsidian is that I can copy HTML formatted text and paste it into Obsidian as Markdown formatted plain text. This makes it easy to customise the sources between inline and footer links. I have a personal preference for footer links, but each to their own. My Markdown notes in Obsidian are indexed by DEVONthink, as the native Ai that powers the Graph and See Also functionality elicits different insights with regard to associated notes than those I get in Obsidian. One isn’t better than the other, but they each use different statistical models to suggest connections between my notes, and that’s valuable for my use cases. Smart connections in Obsidian are powered by community created plugins, which by their nature are subject to greater change than one gets with the stable features of DT (both a plus and minus, as community created plugins risk becoming abandonware).
Ref Kagi, it’s definitely worth exploring. I’d suggest going feet first with the premium plan but stick with the monthly subscription, whilst you assess it’s worth. The free and $10 plans are too restrictive if Ai functionality is your primary use case. I’d pay for the private, advertising-free search capabilities alone, so the Ai capabilities of the premium subscription are very much an added value benefit for my use cases.
Re: footnotes vs in-line hyperlinks - I am going to guess that you primarily author original research or original theories. Thus the footnotes are needed for background but your text basically tells the story on its own? And I am going to guess you have some defined sphere of specialization and are largely aware of most if not all major papers within that field.
In my case I am usually making an argument in court as to what the standard of care is in medicine or what literature says regarding life expectancy. I am mostly quoting a consensus of the published literature rather than publishing my own research. The topic and papers I write about may differ enormously from one report to another - knowing the papers from memory is impossible.
I think this difference in how one uses the academic literature lends itself to a preference for one format of references vs the other.
Ref inline vs footer citations, my preference is driven via an Obsidian plugin. This works as follows:
Insert a new numbered footnote marker (e.g. [^1]) with auto-incremented index in your text
Or, insert a new named footnote marker (e.g. [^Citation]) in your text
This adds the corresponding footnote detail (e.g. [^1]: or [^Citation]: ) at the bottom of your text
Then places the cursor so you can fill in the details quickly
Then jumps from your footnote marker TO the footnote detail
Finally, jumps from your footnote detail BACK to the footnote marker
This effectively means you have the benefits of inline citations with the elegant design benefits of footer citations. It makes both the creation and reading back of citation details a seamless process, with minimal cognitive load.
Obsidian is built on HTML stack technologies, so the vast majority of desktop plugins work in the mobile versions of the App on iOS, iPadOS and Android. When I’m accessing my notes’ database on a mobile device, I tend to mainly to use the rendered Markdown view, so having access to citations in lengthy documents, whilst experiencing the elegance and user-centred design qualities of a decent eBook reader is very valuable to me.
I just subscribed to Kagi Ultimate because of your positive remarks about it in a previous post. I’m enjoying the opportunity to compare the premium plans it includes. I especially like Kagi search. I appreciate the expert insights that you and @rkaplan are sharing with us.
