I am currently testing Wasabi; the following behaviour may or may not be related to Wasabi, and may equally be related to a combination of factors.
First scenario: I set up a bucket on Wasabi, selecting versioning and immutability. I used Arq to backup both my encrypted and my non-encrypted test databases, using an encryption password for the backup process itself. Following a report of successful backup, the backup appeared in the sidebar in Arq. Minutes later the backup was no longer listed in the “Restore” area of the sidebar in Arq; whilst a file tree (name of location > name of Mac > name of Backup > folder) remained, no backups were listed in the folder any more.
Second scenario: I set up a bucket without versioning or immutability. Again, I backup my encrypted and non-encrypted test databases. This works as expected, the backup is listed (and remains) in the sidebar. However, whilst changes to other files in the test set are reflected in the backups, changes to the databases are not; the log lists the databases as unchanged, even though in Finder the modified date is later than the date of the last backup. Several days will pass without any changes to backups being reflected. Manually triggering a backup, however, does cause Arq to put a new (incremental) backup of the databases in the bucket.
I am currently testing to see whether changes to non-encrypted databases are noted by Arq; changes to encrypted databases are ignored in my setup, no incremental backup is performed at scheduled backup times.
If anybody else is using Arq on Ventura (or possibly even on a previous OS) I’d be most grateful if they could check to see whether Arq actually is still backing up their encrypted databases. If anybody is in this specific situation, I’ll happily provide more details if required.
Equally, if anybody has successfully used Arq with Wasabi buckets set to use versioning and immutability, I’d be grateful for feedback re. the exact setup used.
I have just tried giving ArqAgent (in addition to Arq itself) Full Disk Access - at which point the scheduled backup included an incremental backup of databases (note: at no point did I ever reject FDA for ArqAgent). I will continue reporting and strongly recommend anybody using Arq on Ventura check to see whether backups are being performed correctly.
Granting ArqAgent FDA also seems to have solved the behaviour of backups to immutable storage on Wasabi, too - the backups no longer disappear from the list. I am able to confirm that Arq cannot access the backups made to immutable storage whilst ArqAgent was not granted FDA, however.
Note that backups were reported as being successful and no error messages appeared even in the log, despite Arq failing to backup my databases during scheduled backup - instead “No changes to plan or files since last complete backup record. Not creating a new backup record.” was logged.
tl;dr: im you are using Arq on Ventura, you need to check that (incremental) backups of databases are being made during scheduled backups. If my experience is anything to go by, you need to grant ArqAgent Full Disk Access.
This is a developing story; I am not saying Arq is at fault here; it is apparent, however, that a combination of factors can lead to a silent failure to backup.
… If you use a security scanner on your Mac and you update to macOS Ventura, check the program directly to see if it’s flagging an error. The workaround to fix the problem is simple once you know to do it. In System Preferences go to Security & Privacy, then the Privacy tab, and then Full Disk Access. Click the lock icon in the lower-left corner of the screen and authenticate with your system password to allow changes. Then uncheck the box next to any security services that are malfunctioning, to let the system know you want to disable their permission. Click the lock in the lower-left corner again to save the change, then redo the process and recheck the relevant boxes to freshly enable the permission without the flaw…
… Wardle says he has been deluged by bug reports about his free, open source malware monitoring tool, BlockBlock. The Ventura bug even makes it appear that security services like BlockBlock and Malwarebytes have been granted extra system access beyond what these programs request, including the accessibility permission, access to input monitoring, and even screen recording…
… “It shows that when Apple is pushing out security fixes for reported bugs, they’re still struggling to do that comprehensively and successfully without breaking other things. And in this case, they’re shipping a version of their operating system that is breaking security tools for millions, if not tens of millions, of users. It’s frustrating and disheartening.”
Unfortunately, it’s anyone’s guess as to how soon Ventura 13.1 will be released, and many people are in need of a solution immediately. There is a workaround. We have posted an article with details for getting Malwarebytes working again, but the same solution works for any other endpoint security client. Essentially, this boils down to:
Open Security Settings → Security & Privacy → Full Disk Access
Select the security software in the Full Disk Access list
Click the minus (-) button at the bottom of the list to remove it
Try to turn on any real-time protection features in your security software
The security software should reappear in the Full Disk Access list; flip the switch to give it Full Disk Access
Some details may vary slightly about exactly how all this works for the software you’re using, but in general, this should work.
And to reply to myself again: whilst three manually triggered backups to a backup with versioning and immutability worked and were listed, they disappeared a little later, following the first scheduled backup (the timing may be coincidental). Further backups now no longer appear at all.
@chrk I’m not so sure; ArqAgent was not listed in the FDA sections of system settings at all. I did experience the bug (affecting BlockBlock) you described on the Mac which I updated rather than performing a complete fresh install on.
My current thinking is that my problem with immutable backups and my problem with non-immutable backups may be separate issues, only one of which seems to have been solved by granting ArqAgent FDA.
I would probably start again, deleting the buckets in Wasabi (or making new ones and deleting later when object lock expires).
If Arq now has FDA, this confounder would drop out.
Currently, I’m not using Wasabi, but I don’t remember doing anything special settings-wise.
I think I followed the basic instructions from here.
I even found screenshots of my settings (thanks to DEVONthink ).
Object lock (the sum of lock and refresh in days) should always be slightly less (89 days max in this case) than the hourly retention (90 days) to prevent error messages, otherwise the hourly retention might kick in to delete stuff, but the object lock still applies.
I chose 90 days because of Wasabi’s minimum storage policy.
For testing object lock, I’d choose 1 day.
Arq itself has had FDA all along; the change I made was to provide ArqAgent with FDA too.
I’ve done that numerous times along the way; it doesn’t seem to be the solution, unfortunately.
Thanks, I’m using the same settings for Wasabi and compatible settings in Arq. As the only other variable, can you remember how you set the access policy?
Did you contact Arq support yet? They usually answer me within 2 days. Sorry I couldn’t be more helpful.
Keep us posted if you find out more, please.
Quick update: Providing ArqAgent with FDA only seemed to solve the problem; once again, databases are not being backed up by scheduled backups (whilst using the same backup and triggering manually does work).
Interesting also that ArqAgent does not usually need FDA. In Monterey, I haven’t added it to the list.
Maybe snapshots are working differently in Ventura and something about the manual scan keeps it working as before, while the schedule is broken. We’ll see what Stefan has to say. Bad timing with the weekend though.
Since the manual backup is working, it seems unlikely to be a Wasabi issue, but if you want to isolate this further and don’t have another service set up, I could offer a temporary bucket and access key to one of my providers to see if that makes a difference (as long as you encrypt your stuff locally in Arq and don’t test an upload bigger than 2 TB).
I love the logical way you are going through this, thanks.
FDA for ArqAgent only seemed to solve the problem, so correlation but not causation. I’ve switched it away from FDA again.
I have spent most of the afternoon experimenting, amongst other things adding additional non-S3-storage. Thank you so much for your kind offer of temporary storage for testing; I have one further location available currently, so won’t yet take you up on the offer.
The past few hours have been successful - I don’t know why, though. Following a reboot I have been able to access the past three hours worth of immutable backups, and backups have been performed reliably.
I’m wondering whether I dismissed your suggestion this could be an Ventura FDA trick too quickly. I’ve been racking my brain and remember that after initially granting FDA to Arq, Arq displayed FDA in the menu, but didn’t actually have FDA in the system settings. I granted it FDA again before setting up any backups or storage, but wonder whether it didn’t really have FDA after all; that wouldn’t really explain how manual backups worked though, but I don’t know how dependencies (ArqMonitor) are treated by macOS with regards FDA for example. I couldn’t swear to having rebooted since I noticed things not working (I think I did, I usually would when trouble shooting, but times aren’t normal…) but if I didn’t then maybe something was cleared by doing so.
Thank you so much for your time. I will keep this thread updated as things move along.
To keep this up-to-date: having successfully backed up test databases to both immutable and non-immutable storage for the past 24 hours, I set about setting up a full backup again. The result is as before: following seemingly successful backup to immutable storage the backups disappear from the restore section of the Arq sidebar and “View latest backup record” shows “No backup records found”.
I have changed the title of the post, as the problem with scheduling failing silently seems to have resolved itself permanently, leaving only the problem of immutable backups.
Interesting. I’m glad the scheduled backups seem to work now.
I guess it might make sense to revisit the Wasabi settings.
Does your policy for Arq have one of those AdminAccess rights?
And how did you set up Wasabi as a storage location in Arq? As an S3-Compatible location or as Wasabi?
I think I used Wasabi (instead of S3) in the past for immutable backups in Arq, whereas Backblaze B2 only works with object lock when using the S3 setting. Confusing.
No; it uses “AmazonS3FullAccess” as a subuser. That should be sufficient, and works nicely with the test set which does (now) reliably back up to Wasabi immutable.
As Wasabi; and here, too, the the test set works. This conforms to the setup recommendations offered by both Arq and Wasabi.
I will now set up a backup plan which excludes all but one file in the folder to be backed up. I will then add one file after another to see if the problem is being caused by a specific file (or data size).
If you set it up as Wasabi and not S3, the only difference between your current and my old setup is the policy, but it also sounds to me like AmazonS3FullAccess should be enough.
One other thing you could try is hitting the Remove Unreferenced Data… menu item (after selecting Wasabi in the “Back Up” section in the top left of Arq’s sidebar) to see if the log says something interesting.
If Arq only has issues reading the immutable backup records, my guess is that it also can’t do much with that feature, which might end up in the log.
Thanks; that feature finished without error or any action.
It appears that backups to immutable storage (as described above) appear to fail due to one of these occurrences (and I cannot further determine which one it is): Upload of more than approximately 2.5 GB in one backup; upload of a file which is more than approximately 2.5 GB in size; total bucket size greater than approximately 2.5 GB.
In my case, even without errors or actions, I always get a log that fills up slightly, stating how much space is used, etc. If it doesn’t say anything, something is off. Just to clarify the behavior I see here for comparison.
Out of ideas for now. Let me know if you wanna test object lock with B2, maybe it really is a Wasabi problem. I don’t currently use B2 as a provider in Arq, but still have my account set up.
No problem at all, hopefully Stefan/Nina from Arq will know. If they take longer than 2 days to get back to you, they must be swarmed with support requests.
To update this thread again: I have written with Stefan from Arq support; he had a number of questions and has come back to me telling me that the fault lies with Wasabi, has been known for some time, and a timely fix had been promised. I have written to Wasabi support to try and confirm that and get a timeline if possible. Whilst it’s good to know Arq is not at fault here, I’m a little worried that the app provided me with no error message. I’ve asked Stefan whether a future implementation might include more robust verification (although without any knowledge of the technical details, I acknowledge that may not be possible). I’ll report back when I hear back from Wasabi.
).
