Document, receipts, emails and messages. The heavy stuff, like videos and photo’s would be cloned using a HDD dock and taken off site.
I understand it is possible to sync a NAS with another NAS remotely?
Now I get a feeling you are going to throw a spanner into my machine!
So you want to sync several TB of data with DT3???
I understand it is possible to sync a NAS with another NAS remotely?
Basically yes. You normally go the way to sync NAS systems of the same manufacturer, or even better two of the same products.
This is not something every NAS has out of the box… to sync over the Internet has quite some other problems… make sure the NAS you chose does support this and not only local replication (sort of backup).
I don’t think you can just sync the NAS with a DT3 db on it and expect that you can work with it in two locations! You still need the DT3 sync set up. It would be enough to sync this folder between the locations…
I take your point! It would be a gradual sync, as this new venture has yet to begin and no data exists as yet.
They would both be Synology 920+ 4 bay systems. Thank you for pointing out that this facility does not come as standard, I will make sure it is possible with the system I choose.
I am very new to DT and I have much to learn, so we will keep the Dropbox sync for now until we have this backup storage problem sorted out.
Thank you!
Bear in mind, we do not advocate putting DEVONthink databases on an NAS. If it’s a hard-wired Ethernet connected machine, it’s less of a concern but still I’d advise caution.
Also, this seems to be a bit of a strange situation you’re potentially talking about.
The basic tenet of our sync (and DEVONthink in general) is the decentralized data model where each machine has a copy of the data but syncs to a commonly accessible location.
You are referring to having an NAS at each location. This potentially means you’d have two WebDAV services running, one at each location? This would be an odd situation and also consume much more bandwidth between you two. Would you sync to his NAS and he to yours?? See my point?
I bought DEVONthink to keep track of all the data we are expecting in the future. I am the worlds worst at keeping my computer folders and files tidy (my son often say “how do you work with this mess, dad? I can’t find anything”), so after reading up on DEVONthink, I thought “that’s what I need”. I hope I am right.
It will stay on my iMac and hopefully, keep track of what and where everything is. On the NAS or external raid drive too!
If this plan will not work, then so be it.
I don’t know what WebDAV is, YET!
My problem is a simple to understand, but not simple to rectify.
The new business will generate many hundreds of TB’s of data that must be kept for years to come and protected against mechanical failure, fire damage and theft. It does not matter if my son is able to read and write to these files, but he does need to be able to see them and follow our progress (I’m 71 and if I were to die, he needs to know where we are and be able to take over).
A NAS fixes the mechanical failure but not the theft and fire aspect.
So I need to be able to take whatever is on the NAS and remove the copies to another site.
I keep saying NAS, but I am also looking at raid drives connected via thunderbolt 3, which would give me much faster saving times. I do not need remote access or any of the other stuff that comes with these NAS systems for music and film streaming etc.
Cloud storage would be far too expensive and I would feel happier knowing that everything is safe and sound in both locations.
Many thanks!
Are you referring to creating massive files or are you anticipating the accumulation of the files over time will reach into multi-terabyte territory?
Over time. It will take around 12 months or so to fill 10TB, so 50TB after 5 years and 100TB after 10 years. All of this data will need to be kept forever and we must be able to find it so that my son can use it for many years to come, I hope, DEVONthink 8 will be keeping track of it all.
In addition, I must add that the figures above are based on our business plan as it is now. If things take off, it could be much busier from a data point of view.
With all this new information… I think you need a new technology strategy and plan, and perhaps get some paid consultants involved.
I understand that, and maybe later this will be an option, but surely, for the first 10TB I can work something out.
Be advised though, that most of wat was suggested and the advise that was given, probably didn’t take into account this amount of data. I certainly didn’t take into account,
That doesn’t mean most advise and suggestions are useless, but personally speaking I find it difficult to say whether for example the more regular backup strategies are sufficient for your plans. Large amounts of data usually mean large investments in material but also in knowledge about handling it. Whether you can simply obtain that from a forum and books I’m not sure, but I might be mistaken of course.
If you’re looking at RAID with this level of space, you are definitely looking at quite an expense. This is especially true if you want a redundant set at your son’s place. What RAID level are you shopping for?
Good morning (0730h in the UK)
To begin with raid 5 and then raid 6 as we progress.
However, the progress will be gradual and, by the time we get to that level of need, we should be earning enough that the outlay will be just a business expense rather than a personal expense as it is now.
I second what @anon6914418 said before: with this amount of data, you’re probably looking at an entirely unsuitable technology stack.
Several hundreds of TB on a Mac, in DT and stored on a RAID? are you aware that DT needs to access the data locally? They have to be stored kinda on your Mac. Which means attaching “several hundreds of TB” to every machine. And DT manages files. Which puts the filesystem under a lot of stress, because you probably need some TB of files and folders.
You definitely need a consultant, I think.
How are you anticipating database construction, as 1-100TB databases are infeasible on the face of it.
My son and I discussed this at length yesterday and decided we should go DAS. We do not need remote access, music players, video players etc. and a DAS system, using thunderbolt3 should give us faster read/write speeds over a NAS system. We can also keep all files attached to DEVONthink.
This unit set as raid 5 can have more units attached (up to 5 I think) via the thunderbolt socket on the rear. It is configured for MACos and is sold through the Apple website, so we assume it is good for iMac’s.
This would get us started, keep things organised with DT, is easily expandable to get us up to, maybe, 12 months, after which we can reassess the situation.
How we are going to get a back up off site for security reasons? We have no idea.
I am going to get a small Synology NAS for my personal use though. I’m intrigued by NAS’s and what they can do. I wouldn’t mind getting away from my iCloud subscription if possible.
So you scrapped remote access, but how are you going to provide access for the both of you instead? Carry around the disks? Work on the same machine, but at different times?
Why is “read/write speed” an issue? GByte ethernet delivers up to 80 MByte/s (net), you do not have concurrent access anyway.
Seagate has a short comparison of DAS vs. NAS vs.SAN
As for backup: If you want it fast, you’d probably have to have the same DAS to mirror to. Otherwise, there’s tape.
But all that doesn’t help with other issues:
I still believe that the technology stack is not up to your requirements. Storing all the data as files in the filesystem is most probably not efficient for this amount of data and I doubt that DT is able to handle a TB or even more. @BLUEFROG might have more information on that. Also, it is not clear, what kind of files you’re storing and how big they are.
Some excellent points there, thank you.
We need to think a bit deeper about this storage and organisation problem.
After reading the Seagate article, it does appear to be that a NAS, one that can be expanded, would be more suitable to begin with.
Per week: 6 x 1hr videos in 4K @ 60fps + 12 x 15 minute videos in 4K @ 30fps + each 1hr video will be accompanied by 20 RAW photos of approx 25Mbs each + text files, notes, b-roll film & images of spreadsheets, charts and screenshots. In addition, some films will be in slow motion and time lapse, which I understand are data heavy. This material will be added to week on week, month on month, year on year and must be preserved and reused forever.
We cannot go live until 10 weeks of data is recorded.
All the films and photos have got to be edited on the iMacs, hence the fast read/write requirements and we need to be kept updated on each other’s progress and be able to assist with others work (3 persons) If required, even though we are 20 miles apart.
At present, we are using Notion (moved from Trello) to coordinate things, but we do want to bring it all in house and off cloud based storage.
one hour of RAW 4K content requires close to 110GB of storage; approximately 2GB per minute.
5 Reasons You Might Want to Wait on 4K Ultra HD Video Production (For Now) - Videomaker
My question is what is DEVONthink’s role in this project?
Give me an example of a group structure in the database.
Or play “let’s pretend” and you say, “I have this group for week 1 (or whatever). I add the first movie in its own subgroup. I then make a sheet with some notes about the location, client, t-stop, etc.…”
But this is something that you probably do not want to store in DT.
“Images of spreadsheets” instead of the spreadsheets itself? Why? Images take a lot more space usually, are limited to the first worksheet unless you are going the extra mile there and do not permit searching (unless you OCR them, in which case you’d rather save the spreadsheet as such in the first place).
Which is not related to DT. If you’re serious about “forever”, you should be prepared for some heavy spending: backups in different locations, a strategy for copying the backups regularly on other media,
So you want to edit 120 RAW photos plus 9 hours of video on an iMac per week. Manually. The digital image data alone amounts to roughly 1,5 TB a week. Which means that your G-Speed eS 0g02324 with its 16 TB at Raid 5 will be filled up after three weeks (Raid 5 uses at least 3 disks, which leaves you only a third of the 16 TB. At best). And all that connected to a Mac (an iMac, as you said) with its not overly comfortable tools to manage disks, volumes and disk arrays.
It’s always nice to have fast disks – but in this case, you need RAM more than read/write speed, I suppose. You’re not editing on the disk but in the computer, and it needs to read/write the photos only at the beginning/end of the editing process. Videos are probably processed in batches, but even for them RAM is probably the most important resource.
But, as @BLUEFROG insinuated: All this heavy data lifting is not related to DT at all. And after having read your posts, I assume that more than a document management system, you need a workflow. Some structured process for it all. “Notes” that are flying around probably just don’t cut it. Unless you’re very well organized, which you stated that you aren’t.
And even more than that, you need a clear concept. Maybe a consultant. You are, as you said yourself, not IT savvy, but attempt a hardware and software intensive operation without any professional help? Excuse my language, but that sounds like madness to me. There are probably ready-made solutions out there in the movie industry – you might want to look at them. And maybe get a consultant.
Exactly! My 2 x 1Tb drives on my iMac have filled up very quickly.
I have yet to learn the capabilities of Devonthink and how to set it up and use it (I’m halfway through the first book), I understood that I could use it to keep track of these videos, scripts, b-roll film, photos, notes and so forth and file them in such a way that they can be found easily.
At least, that Is what reviews led me to think.
No grouping planned as yet. I think keywords would be used for each individual video and all of its components, so that Devonthink will throw back everything with the requested keyword.
As I say, I do not know enough about DEVONthink to be able to plan anything as yet.