Multi-Cloud Sync Architecture

Date: 2025-01-05

This document outlines architectural approaches for syncing Minuta data to multiple cloud providers (Dropbox, Yandex, Google Drive, OneDrive, iCloud, WebDAV/self-hosted, custom cloud).

Current State

┌─────────────────────────────────────────────────────┐
│                    Minuta App                        │
│  ┌─────────────┐    ┌─────────────────────────────┐ │
│  │  AppState   │───▶│  FileStorageServiceProtocol │ │
│  └─────────────┘    └──────────────┬──────────────┘ │
│                                    │                 │
│                     ┌──────────────▼──────────────┐ │
│                     │  LocalFileStorageService    │ │
│                     │  ~/Documents/Minuta/        │ │
│                     │  - tags.json                │ │
│                     │  - records/YYYY/MM/*.json   │ │
│                     └─────────────────────────────┘ │
└─────────────────────────────────────────────────────┘

Strengths:

• Protocol-based (swappable backends)
• Actor-based (thread-safe)
• Simple file structure (human-readable JSON)

Gaps for multi-cloud:

• No sync state tracking (what’s uploaded, what’s pending)
• No conflict detection/resolution
• No change tracking (which files changed since last sync)
• Tightly coupled to local filesystem paths

Approach 1: Cloud Adapter Pattern

Concept: Keep local storage as source of truth, add cloud adapters that mirror local state.

┌─────────────────────────────────────────────────────────────────┐
│                         Minuta App                               │
│  ┌─────────────┐                                                │
│  │  AppState   │                                                │
│  └──────┬──────┘                                                │
│         │                                                        │
│  ┌──────▼──────────────────────────────────────────────────┐   │
│  │              StorageCoordinator (actor)                  │   │
│  │  - Writes to local first                                 │   │
│  │  - Queues changes for sync                               │   │
│  │  - Handles conflicts                                     │   │
│  └──────┬───────────────────────────────────┬──────────────┘   │
│         │                                   │                   │
│  ┌──────▼──────────┐              ┌────────▼────────┐          │
│  │ LocalStorage    │              │  SyncEngine     │          │
│  │ (source of      │◀────────────▶│  (background)   │          │
│  │  truth)         │   changes    │                 │          │
│  └─────────────────┘              └────────┬────────┘          │
│                                            │                    │
│                        ┌───────────────────┼───────────────┐   │
│                        │                   │               │   │
│                 ┌──────▼─────┐  ┌─────────▼────┐  ┌───────▼──┐│
│                 │CloudAdapter│  │CloudAdapter  │  │CloudAdapter│
│                 │ Protocol   │  │ Protocol     │  │ Protocol  ││
│                 └──────┬─────┘  └──────┬───────┘  └─────┬─────┘│
└────────────────────────┼───────────────┼────────────────┼──────┘
                         │               │                │
                    ┌────▼───┐    ┌──────▼─────┐   ┌──────▼──────┐
                    │Dropbox │    │Google Drive│   │ WebDAV      │
                    │  SDK   │    │    SDK     │   │(Nextcloud)  │
                    └────────┘    └────────────┘   └─────────────┘

Protocol Design

/// Represents a file in the cloud
struct CloudFile: Sendable {
    let path: String           // Relative path: "records/2024/01/record.json"
    let contentHash: String    // For change detection
    let modifiedAt: Date
    let size: Int64
    var conflictVersions: [CloudFile]?
}

/// Sync state for tracking what's been synced
struct SyncState: Codable, Sendable {
    var lastSyncedAt: Date?
    var localVersion: String      // Hash of local file
    var remoteVersion: String?    // Hash of remote file
    var status: SyncStatus

    enum SyncStatus: String, Codable {
        case synced
        case pendingUpload
        case pendingDownload
        case conflict
    }
}

/// Cloud provider adapter protocol
protocol CloudAdapterProtocol: Actor {
    /// Provider identifier
    var providerId: String { get }

    /// Human-readable name
    var displayName: String { get }

    /// Authentication state
    var isAuthenticated: Bool { get async }

    /// Authenticate with the provider (shows UI if needed)
    func authenticate() async throws

    /// Sign out
    func signOut() async throws

    /// List all files in the Minuta folder
    func listFiles() async throws -> [CloudFile]

    /// Download a file
    func download(path: String) async throws -> Data

    /// Upload a file (creates or overwrites)
    func upload(path: String, data: Data) async throws -> CloudFile

    /// Delete a file
    func delete(path: String) async throws

    /// Get file metadata without downloading
    func metadata(path: String) async throws -> CloudFile?

    /// Watch for remote changes (if supported)
    func startWatching(onChange: @escaping ([CloudFile]) -> Void) async throws
    func stopWatching() async
}

Concrete Adapters

// Dropbox - Official SDK
actor DropboxAdapter: CloudAdapterProtocol {
    let providerId = "dropbox"
    let displayName = "Dropbox"
    private var client: DropboxClient?
    // Uses Dropbox SDK with OAuth2
}

// Google Drive - Official SDK
actor GoogleDriveAdapter: CloudAdapterProtocol {
    let providerId = "google-drive"
    let displayName = "Google Drive"
    private var driveService: GTLRDriveService?
    // Uses Google SDK with OAuth2
}

// WebDAV - Generic (Nextcloud, ownCloud, Box, etc.)
actor WebDAVAdapter: CloudAdapterProtocol {
    let providerId: String  // "nextcloud", "owncloud", "box-webdav"
    let displayName: String
    private let baseURL: URL
    private let credentials: WebDAVCredentials
    // Uses standard HTTP with WebDAV extensions
}

// iCloud Documents
actor ICloudAdapter: CloudAdapterProtocol {
    let providerId = "icloud"
    let displayName = "iCloud"
    // Uses NSFileManager ubiquity APIs
}

// S3-Compatible (AWS, MinIO, Backblaze B2)
actor S3Adapter: CloudAdapterProtocol {
    let providerId: String  // "aws-s3", "minio", "backblaze"
    let displayName: String
    private let endpoint: URL
    private let credentials: S3Credentials
    // Uses AWS SDK or raw S3 API
}

// Custom cloud
actor MinutaCloudAdapter: CloudAdapterProtocol {
    let providerId = "minuta-cloud"
    let displayName = "Minuta Cloud"
    private let apiURL: URL
    // Custom REST API
}

Pros:

• Clean separation of concerns
• Easy to add new providers
• Local-first (works offline)
• Each adapter handles provider quirks internally

Cons:

• Need to implement each adapter separately
• Different SDKs have different async patterns
• OAuth flows differ per provider

Approach 2: Sync Engine with Change Tracking

Concept: Add explicit change tracking layer that works with any backend.

/// Tracks all changes for sync
actor ChangeTracker {
    private var pendingChanges: [ChangeEntry] = []
    private let storage: ChangeStorage

    struct ChangeEntry: Codable, Identifiable {
        let id: UUID
        let path: String
        let operation: Operation
        let timestamp: Date
        let localHash: String
        var syncedTo: Set<String>  // Provider IDs that have this change

        enum Operation: String, Codable {
            case create
            case update
            case delete
        }
    }

    /// Record a local change
    func recordChange(_ operation: ChangeEntry.Operation, path: String, hash: String) async {
        let entry = ChangeEntry(
            id: UUID(),
            path: path,
            operation: operation,
            timestamp: Date(),
            localHash: hash,
            syncedTo: []
        )
        pendingChanges.append(entry)
        await storage.persist(pendingChanges)
    }

    /// Get changes not yet synced to a provider
    func pendingChanges(for providerId: String) -> [ChangeEntry] {
        pendingChanges.filter { !$0.syncedTo.contains(providerId) }
    }

    /// Mark change as synced to a provider
    func markSynced(_ changeId: UUID, to providerId: String) async {
        if let index = pendingChanges.firstIndex(where: { $0.id == changeId }) {
            pendingChanges[index].syncedTo.insert(providerId)

            // Clean up fully synced changes
            if pendingChanges[index].syncedTo.count >= activeProviderCount {
                pendingChanges.remove(at: index)
            }
            await storage.persist(pendingChanges)
        }
    }
}

/// Sync engine that coordinates between local and cloud
actor SyncEngine {
    private let localStorage: LocalFileStorageService
    private let changeTracker: ChangeTracker
    private var adapters: [String: any CloudAdapterProtocol] = [:]

    enum SyncResult {
        case success
        case conflicts([ConflictInfo])
        case partialFailure(synced: Int, failed: Int)
    }

    struct ConflictInfo {
        let path: String
        let localVersion: Data
        let remoteVersion: Data
        let remoteModifiedAt: Date
    }

    /// Sync with all enabled providers
    func syncAll() async -> [String: SyncResult] {
        var results: [String: SyncResult] = [:]

        await withTaskGroup(of: (String, SyncResult).self) { group in
            for (id, adapter) in adapters {
                group.addTask {
                    let result = await self.sync(with: adapter)
                    return (id, result)
                }
            }

            for await (id, result) in group {
                results[id] = result
            }
        }

        return results
    }

    /// Sync with a specific provider
    func sync(with adapter: any CloudAdapterProtocol) async -> SyncResult {
        let providerId = await adapter.providerId

        // 1. Get pending local changes
        let localChanges = await changeTracker.pendingChanges(for: providerId)

        // 2. Get remote changes
        let remoteFiles = try? await adapter.listFiles()

        // 3. Detect conflicts
        let conflicts = detectConflicts(local: localChanges, remote: remoteFiles ?? [])

        if !conflicts.isEmpty {
            return .conflicts(conflicts)
        }

        // 4. Upload local changes
        for change in localChanges {
            do {
                switch change.operation {
                case .create, .update:
                    let data = try await localStorage.readFile(at: change.path)
                    _ = try await adapter.upload(path: change.path, data: data)
                case .delete:
                    try await adapter.delete(path: change.path)
                }
                await changeTracker.markSynced(change.id, to: providerId)
            } catch {
                // Handle individual file failure
            }
        }

        // 5. Download remote changes
        // ...

        return .success
    }
}

Pros:

• Explicit change tracking survives app restarts
• Can sync to multiple providers with different schedules
• Conflict detection before sync
• Audit trail of all changes

Cons:

• More complex state management
• Change log can grow large
• Need to handle log compaction

Approach 3: File System Abstraction Layer

Concept: Abstract file operations so cloud providers look like local filesystems.

/// Virtual file system protocol
protocol VirtualFileSystem: Actor {
    /// Read file contents
    func read(path: String) async throws -> Data

    /// Write file contents
    func write(path: String, data: Data) async throws

    /// Delete file
    func delete(path: String) async throws

    /// List directory contents
    func list(directory: String) async throws -> [FileEntry]

    /// Check if file exists
    func exists(path: String) async -> Bool

    /// Get file metadata
    func metadata(path: String) async throws -> FileMetadata

    /// Watch for changes
    func watch(path: String, handler: @escaping (FileChange) -> Void) async throws
}

struct FileEntry {
    let name: String
    let path: String
    let isDirectory: Bool
    let size: Int64
    let modifiedAt: Date
}

struct FileMetadata {
    let size: Int64
    let modifiedAt: Date
    let createdAt: Date
    let contentHash: String?
}

enum FileChange {
    case created(path: String)
    case modified(path: String)
    case deleted(path: String)
}

/// Local filesystem implementation
actor LocalFileSystem: VirtualFileSystem {
    private let rootURL: URL

    func read(path: String) async throws -> Data {
        try Data(contentsOf: rootURL.appendingPathComponent(path))
    }

    func write(path: String, data: Data) async throws {
        let url = rootURL.appendingPathComponent(path)
        try FileManager.default.createDirectory(
            at: url.deletingLastPathComponent(),
            withIntermediateDirectories: true
        )
        try data.write(to: url)
    }
    // ...
}

/// Dropbox as a virtual filesystem
actor DropboxFileSystem: VirtualFileSystem {
    private let client: DropboxClient
    private let rootPath: String = "/Minuta"

    func read(path: String) async throws -> Data {
        try await client.files.download(path: rootPath + "/" + path).data
    }

    func write(path: String, data: Data) async throws {
        try await client.files.upload(
            path: rootPath + "/" + path,
            input: data,
            mode: .overwrite
        )
    }
    // ...
}

/// Layered filesystem: local + remote sync
actor SyncedFileSystem: VirtualFileSystem {
    private let local: VirtualFileSystem
    private let remote: VirtualFileSystem
    private let syncQueue: SyncQueue

    func read(path: String) async throws -> Data {
        // Always read from local (source of truth)
        try await local.read(path: path)
    }

    func write(path: String, data: Data) async throws {
        // Write locally first
        try await local.write(path: path, data: data)
        // Queue for remote sync
        await syncQueue.enqueue(.upload(path: path))
    }
    // ...
}

Pros:

• Uniform API for all storage backends
• Easy to compose (cache layer, encryption layer, etc.)
• Storage service doesn’t need to know about sync

Cons:

• May hide provider-specific optimizations (batch upload, resumable upload)
• Authentication doesn’t fit the filesystem metaphor
• Conflict handling is awkward in this model

Approach 4: Event Sourcing

Concept: Store changes as immutable events, replay to any backend.

/// Immutable event
struct StorageEvent: Codable, Identifiable {
    let id: UUID
    let timestamp: Date
    let type: EventType
    let path: String
    let data: Data?       // For create/update
    let metadata: [String: String]

    enum EventType: String, Codable {
        case fileCreated
        case fileUpdated
        case fileDeleted
        case fileMoved
    }
}

/// Event store
actor EventStore {
    private var events: [StorageEvent] = []
    private let persistence: EventPersistence

    /// Append a new event
    func append(_ event: StorageEvent) async {
        events.append(event)
        await persistence.persist(event)
    }

    /// Get all events after a sequence number
    func events(after sequenceNumber: Int) -> [StorageEvent] {
        Array(events.dropFirst(sequenceNumber))
    }

    /// Replay events to rebuild state
    func replay(to target: VirtualFileSystem) async throws {
        for event in events {
            switch event.type {
            case .fileCreated, .fileUpdated:
                if let data = event.data {
                    try await target.write(path: event.path, data: data)
                }
            case .fileDeleted:
                try await target.delete(path: event.path)
            case .fileMoved:
                // Handle move...
                break
            }
        }
    }
}

/// Modified storage service that emits events
actor EventSourcedStorageService: FileStorageServiceProtocol {
    private let eventStore: EventStore
    private let localStorage: LocalFileSystem

    func saveRecord(_ record: TimeRecord) async throws {
        let path = recordPath(for: record)
        let data = try encoder.encode(record)

        // Write locally
        try await localStorage.write(path: path, data: data)

        // Emit event
        let event = StorageEvent(
            id: UUID(),
            timestamp: Date(),
            type: .fileUpdated,
            path: path,
            data: data,
            metadata: ["recordId": record.id.uuidString]
        )
        await eventStore.append(event)
    }
}

/// Sync by replaying events
actor CloudSyncService {
    private let eventStore: EventStore
    private var lastSyncedSequence: [String: Int] = [:]  // Per provider

    func sync(to adapter: CloudAdapterProtocol) async throws {
        let providerId = await adapter.providerId
        let lastSequence = lastSyncedSequence[providerId] ?? 0

        let newEvents = await eventStore.events(after: lastSequence)

        for event in newEvents {
            switch event.type {
            case .fileCreated, .fileUpdated:
                if let data = event.data {
                    _ = try await adapter.upload(path: event.path, data: data)
                }
            case .fileDeleted:
                try await adapter.delete(path: event.path)
            case .fileMoved:
                break
            }
        }

        lastSyncedSequence[providerId] = lastSequence + newEvents.count
    }
}

Pros:

• Complete audit trail
• Can sync to new providers by replaying all events
• Natural fit for conflict-free sync (events are append-only)
• Can implement undo/redo easily

Cons:

• Event log grows unbounded (need compaction/snapshots)
• Stores data twice (events + current state)
• More complex than simple file sync
• Events with data blobs can be large

Approach 5: Hybrid with Pluggable Sync Backends

Concept: Minimal changes to existing architecture, add sync as optional layer.

/// Sync provider configuration
struct SyncProviderConfig: Codable, Identifiable {
    let id: UUID
    let type: ProviderType
    var isEnabled: Bool
    var lastSyncedAt: Date?
    var settings: [String: String]  // Provider-specific settings

    enum ProviderType: String, Codable, CaseIterable {
        case icloud = "iCloud"
        case dropbox = "Dropbox"
        case googleDrive = "Google Drive"
        case oneDrive = "OneDrive"
        case webdav = "WebDAV"
        case s3 = "S3 Compatible"
        case minutaCloud = "Minuta Cloud"
    }
}

/// Registry of available sync providers
actor SyncProviderRegistry {
    private var factories: [SyncProviderConfig.ProviderType: () -> any CloudAdapterProtocol] = [:]

    func register(_ type: SyncProviderConfig.ProviderType, factory: @escaping () -> any CloudAdapterProtocol) {
        factories[type] = factory
    }

    func createAdapter(for config: SyncProviderConfig) -> (any CloudAdapterProtocol)? {
        factories[config.type]?()
    }
}

/// Sync manager that orchestrates everything
@MainActor
@Observable
class SyncManager {
    private(set) var providers: [SyncProviderConfig] = []
    private(set) var syncStatus: SyncStatus = .idle
    private(set) var lastError: Error?

    private let registry: SyncProviderRegistry
    private let localStorage: LocalFileStorageService
    private var activeAdapters: [UUID: any CloudAdapterProtocol] = [:]

    enum SyncStatus: Equatable {
        case idle
        case syncing(provider: String, progress: Double)
        case error(message: String)
    }

    /// Add a new sync provider
    func addProvider(_ config: SyncProviderConfig) async throws {
        guard let adapter = await registry.createAdapter(for: config) else {
            throw SyncError.unsupportedProvider
        }

        try await adapter.authenticate()
        providers.append(config)
        activeAdapters[config.id] = adapter

        // Initial sync
        await syncProvider(config.id)
    }

    /// Manual sync trigger
    func syncNow() async {
        for config in providers where config.isEnabled {
            await syncProvider(config.id)
        }
    }

    /// Sync a specific provider
    private func syncProvider(_ id: UUID) async {
        guard let adapter = activeAdapters[id],
              let index = providers.firstIndex(where: { $0.id == id }) else { return }

        let name = await adapter.displayName
        syncStatus = .syncing(provider: name, progress: 0)

        do {
            // 1. Upload local changes
            let localFiles = try await localStorage.allFilePaths()
            for (i, path) in localFiles.enumerated() {
                let data = try await localStorage.readFile(at: path)
                _ = try await adapter.upload(path: path, data: data)
                syncStatus = .syncing(provider: name, progress: Double(i) / Double(localFiles.count))
            }

            // 2. Download remote changes
            let remoteFiles = try await adapter.listFiles()
            for file in remoteFiles {
                if await shouldDownload(file) {
                    let data = try await adapter.download(path: file.path)
                    try await localStorage.writeFile(at: file.path, data: data)
                }
            }

            providers[index].lastSyncedAt = Date()
            syncStatus = .idle
        } catch {
            lastError = error
            syncStatus = .error(message: error.localizedDescription)
        }
    }
}

Pros:

• Minimal changes to existing storage layer
• Sync is optional/additive
• Easy to understand
• Users can enable/disable providers individually

Cons:

• No change tracking (full sync each time)
• Less efficient for large datasets
• Conflict handling is simplistic

Comparison Matrix

*Event sourcing needs compaction for large datasets

Recommendation

For Minuta’s use case, I recommend Approach 1 (Cloud Adapter Pattern) combined with elements from Approach 2 (Change Tracking):

Phase 1: Foundation (minimal changes)

// Add to FileStorageServiceProtocol
protocol FileStorageServiceProtocol {
    // Existing methods...

    // New: Get all file paths for sync
    func allFilePaths() async throws -> [String]

    // New: Read raw file data
    func readFile(at path: String) async throws -> Data

    // New: Write raw file data
    func writeFile(at path: String, data: Data) async throws

    // New: File hash for change detection
    func fileHash(at path: String) async throws -> String
}

Phase 2: Add sync state tracking

// Simple sync state (stored in syncstate.json)
struct SyncState: Codable {
    var files: [String: FileSyncState]

    struct FileSyncState: Codable {
        var localHash: String
        var providers: [String: ProviderState]
    }

    struct ProviderState: Codable {
        var remoteHash: String?
        var syncedAt: Date?
        var status: Status

        enum Status: String, Codable {
            case synced, pendingUpload, pendingDownload, conflict
        }
    }
}

Phase 3: Implement adapters incrementally

1. WebDAV first - covers Nextcloud, ownCloud, Box, many self-hosted
2. iCloud - already researched, Apple ecosystem
3. Dropbox - popular, good SDK
4. Custom cloud - custom REST API
5. Others - as demand requires

This approach:

• Keeps local storage as source of truth
• Adds sync incrementally without breaking existing code
• Makes each provider a self-contained module
• Allows users to sync to multiple clouds simultaneously

Appendix: Provider-Specific Notes

WebDAV (Nextcloud, ownCloud, Box)

• Standard HTTP protocol with extensions
• No SDK needed, use URLSession
• Basic auth or OAuth2 depending on server
• Good for self-hosted solutions

iCloud Documents

• Use NSFileManager ubiquity APIs
• NSMetadataQuery for sync status
• Cannot trigger uploads (system controlled)
• Conflict versions as separate files

Dropbox

• Official SwiftyDropbox SDK
• OAuth2 authentication
• Supports webhooks for change notifications
• Content hashing for efficient sync

Google Drive

• Official GoogleAPIClientForREST SDK
• OAuth2 with scopes
• File IDs instead of paths (need mapping)
• Supports change notifications via push

OneDrive

• Microsoft Graph API
• OAuth2 with MSAL
• Similar to Google Drive (IDs vs paths)
• Delta queries for efficient sync

S3-Compatible (AWS, MinIO, Backblaze)

• AWS SDK or raw HTTP with signing
• Access key authentication
• No native change notifications (poll or use SQS)
• Good for large files (multipart upload)

Related

• 101-overview - System architecture
• 301-file-storage - File storage service
• 305-storage-location - Storage location manager