Twitter: twisting and turning to make archiving a PITA

Archiving tweets to DTPO has generally worked well for me in the past; I could capture a webpage and get a copy of a twitter thread.

Twitter got antsy, so using our old friend Keyboard Maestro and a ‘toggle javascript’ shortcut made it possible to flip to a slightly reduced ‘legacy’ twitter version, which could at least be easily captured.

Twitter is removing this interface as of tomorrow, so it can’t be used any more (https://mobile.twitter.com/, for last-gasp viewers).

This leaves me in a nightmarish scenario where captured tweets are showing up as shredded sign-in pages, rather than the content I am trying to archive (often, threads). I think two things are playing off each other badly here; one is that (I suspect) twitter is requiring either cookie acceptance or a sign-in before showing any content to users, and that the capturing layer in DTPO isn’t using the cookies or settings from system-wide safari.

Does this sound a plausible explanation of what’s happening?

And assuming there isn’t a straightforward work-around on the horizon, does anyone have any suggestions for good Mac twitter/tweet-to-DTPO capture tools? I want to be able to capture straight from Safari, but if there’s a service or piece of software I can use on my Mac to support this, that would also be good; what I don’t want to do is rely on a twitter client which is more interested in streaming data than archiving it with third-party tools.

Applications don’t share cookies any longer and haven’t for some time.

Try adding a bookmark to Twitter and signing in while in DEVONthink. Then try capturing a tweet in DEVONthink.

1 Like

Hey, thanks for the suggestion, Jim.

I tried that - it all makes logical sense - but I am still capturing heavily distorted pages with no actual content when I capture as WebArchive, and a generic Twitter error page saying ‘Something went Wrong’ when I try to capture as PDF.

I think this is because Twitter is hellbent on being a web ‘app’, rather than a site, and the default page is the error before it fetches the tweet in question as a payload to render into the HTML page - and the PDF capture is happening before the quivering stack of JavaScript can step in, flag me up with Google Analytics, then fetch the actual page content. I suspect. I have not been able to step in and confirm this.

I think I’m going to have to suck up screenshotting to get Tweets into DTPO. I wish people would stop treating the browser as a runtime rather than a renderer but I’m in the minority these days…

i also have struggled with this. using twitter web site with safari. i now do a simple screen shot of the tweet to a PNG file saved to the DEVONthink Global Inbox folder. That gets automatically imported into DEVONthink Global InBox. i have a rule that watches for incoming Images. it converts any noticed into one-page PDFs that are then OCR’d, then classified. Eventually I manually rename them to something relevant and if i energetic put in some metadata.

This acreenshotting method also works well when on iphone.

I share your pain. I wrote about my solution at Saving tweets into DTTG via iOS, and my current AppleScript code can be found at https://github.com/mhucka/devonthink-hacks/tree/master/auto-convert-web-page-to-PDF

It’s not perfect, but it works for individual tweets (at least in my environment).

For tweet threads, I pay for https://threadreaderapp.com – it’s cheap, and it is hands-down the best Twitter saving tool I have found.

2 Likes

Hey, thanks @mhucka - I’ll definitely look into this, thanks for sharing. I could see a lower-friction version of this working by assuming all bookmarks in DTPO need to be converted to PDF archives, and that’s a great example of what a Smart Rule can do.

I also hear you about Thread Reader.

grumbles *mutters* I just wish Twitter would have a paid model where I could get my data rather than making my data inaccessible so I can look at the work my adblocker is doing

1 Like

Related to capturing Twitter and other social media, I just remembered another tool: https://gwu-libraries.github.io/sfm-ui/

1 Like

I’ve looked at SFM in the past, but it’s more a server suite than a desktop tool. It’s not helped by Twitter’s rapidly evolving use models and data structure, which I think is a polite way of framing it.

You could try IFTTT, I use an applet there which archives off liked tweets to a Google Sheet.

In case people here haven’t seen it: Twitter announced on Jan. 26 that they’re making changes to the API to allow access to the full archive: Enabling the future of academic research with the Twitter API

It’s limited to people working in academic institutions right now, and this change may not be helpful for people who just want to save the occasional tweet, but anyway, it seemed worth mentioning in this conversation.

1 Like

Come across your set of DEVONthink hacks @mhucka - quite useful, thanks! GitHub - mhucka/devonthink-hacks: Scripts and other things for working with DEVONthink

1 Like