docs / guides
Syncing through a folder
Use iCloud Drive, Dropbox, Google Drive, OneDrive, Syncthing or a USB drive as a botdb remote instead of a relay.
Any folder that another tool keeps in sync between machines can be a remote. botdb writes encrypted files into it, and iCloud Drive, Dropbox, Syncthing or whatever you already use moves them around. No server needed.
Add a folder remote
During botdb login, botdb lists the cloud folders it detects and suggests a botdb folder inside each. To add one later, pass a path. Use a dedicated folder:
botdb remote add "~/Library/Mobile Documents/com~apple~CloudDocs/botdb" --default # new databases and the vault
botdb remote add ~/Sync/botdb -d movies # one database
Paths can be written as ~/..., /... or file://.... Anything starting with http:// or https:// is treated as a relay.
Detected folders
botdb login looks for these, relative to your home directory:
| Provider | Paths checked |
|---|---|
| iCloud Drive | Library/Mobile Documents/com~apple~CloudDocs (macOS), iCloudDrive (Windows) |
| Dropbox | Dropbox, Library/CloudStorage/Dropbox |
| Google Drive | Google Drive/My Drive, Library/CloudStorage/GoogleDrive-*/My Drive |
| OneDrive | OneDrive, Library/CloudStorage/OneDrive* |
Anything else works too: a Syncthing folder, a network share, or a USB drive (/Volumes/<name> on macOS, /media/<user>/<name> on Linux).
Layout
Each database gets its own directory, named by its id. Every batch is a separate file that is written once and never changed; only the snapshot is replaced. Files are written atomically (to a temporary name, then renamed), and partially synced files are retried later.
<folder>/<db-id>/batches/<replica-id>/<seq, 20 digits>-e<epoch>.bin one encrypted batch
<folder>/<db-id>/snapshot-e<epoch>.bin latest encrypted snapshot
<folder>/<db-id>/snapshot-e<epoch>.vec sequence numbers it covers
Batch and snapshot files are encrypted and padded the same way as data sent to a relay. The .vec file is plaintext metadata: the highest sequence number the snapshot covers for each replica id.
Once 1000 or more batch files exist for a database, a client writes a new snapshot and removes covered batch files older than 7 days. Devices that are far behind, or new, start from the snapshot.
What the folder provider sees
File names (database ids, replica ids, sequence numbers and key epochs), padded file sizes, modification times and the .vec metadata. Not your data.
Folder or relay?
| Folder | Relay | |
|---|---|---|
| Setup | None if you already use a sync tool | A server or container you run |
| Latency | As fast as the sync tool (seconds to minutes) | Immediate, with long-polling |
| Who can add changes | Holders of the database’s write key | Holders of the database’s write key |
| Read-only invites | Enforced (replicas verify signatures) | Enforced (relay and replicas) |
| Works without network | Yes, with a USB drive or LAN sync | No |
Every batch is signed with the database’s write key and every replica verifies the signature, so a folder doesn’t need to authenticate writers: read-only members and anyone else with access to the folder can’t add changes. They can delete or hide files, which every device repairs by re-uploading what it has.
A database can have several remotes, for example a relay for speed and a cloud folder as a second copy.
Sneakernet
A USB drive works as a remote too. Add it, then run a sync while it’s mounted on each machine:
botdb remote add /Volumes/STICK/botdb -d movies
botdb sync -d movies
Tips
- Automatic sync pulls before reads (at most every 10 seconds) and pushes after writes. For continuous sync on an otherwise idle machine, run
botdb sync --watch; with folders it checks for new files every few seconds. botdb doctorchecks that the folder is writable.- Don’t edit or delete files in the botdb folder by hand.
- Cloud providers that offload files to save space (iCloud “Optimize Mac Storage”, OneDrive “Files On-Demand”) can delay sync. Keep the botdb folder downloaded.