Cloud Backup System: Automated Scheduled Backups with Pro License Enforcement

Overview

JunctionRelay Cloud implements a comprehensive automatic backup system that creates scheduled backups of user databases to S3 cloud storage. The system operates across three distributed components with strict Pro license enforcement, ensuring only paying customers can utilize cloud backup features.

This document describes the backup architecture, scheduling mechanism, license validation, cleanup processes, and security model governing automated cloud backups.

---

Architecture Components

Local Backend Scheduler

• Service: Service_CloudBackup_Scheduler (BackgroundService)
• Location: User's local JunctionRelay instance
• Function: Autonomous backup creation and upload orchestration
• License awareness: Checks Pro license status before every backup attempt
• Initialization: Started by Service_Startup after database and event engine ready

Cloud API Backend

• Controllers: Controller_CloudBackups (ASP.NET Core API)
• Database: Service_Database_CloudBackups (PostgreSQL operations)
• Function: License validation, URL generation, metadata storage
• License enforcement: Validates Pro license on every API endpoint
• Storage: Backup metadata in PostgreSQL, files in S3

Cloud Worker

• Service: Service_BackupCleanupProcessor (Redis-based background worker)
• Function: Daily cleanup of expired and stale backups
• Schedule: Runs once every 24 hours
• Database: Direct PostgreSQL access for cleanup operations

---

License Enforcement Model

Pro License Requirements

Only Pro subscribers can: - Enable automatic cloud backups - Create manual backups via "Backup Now" button - Download existing backups from cloud storage - Configure backup settings (frequency, retention, components) - Access backup status and usage statistics

License Validation Strategy

Free users (no Stripe subscription): - Cannot create backup records (403 Forbidden) - Cannot access backup endpoints - See "requires valid Pro license" error message - Settings show hasValidLicense: false

Pro users (active Stripe subscription): - Full access to all backup endpoints - Scheduler creates backups automatically when due - Can manually trigger backups anytime - Settings show hasValidLicense: true

Validation Points

Every 15 minutes (Scheduler):
  Local Backend → GET /cloud-backups/status
  Cloud Backend → Check Stripe subscription status
  Cloud Backend → Return hasValidLicense boolean
  Local Backend → Skip backup if false

Every API Call (Manual/Automatic):
  Client → POST /cloud-backups/request-upload
  Cloud Backend → ValidateProLicense(userId)
  Cloud Backend → Check Stripe subscription
  Cloud Backend → Return 403 if no active subscription

License check implementation:

private async Task<IActionResult?> ValidateProLicense(string endpoint, string clerkUserId)
{
    var hasLicense = await _subscriptionService.HasValidLicenseAsync(clerkUserId);
    if (!hasLicense)
    {
        return StatusCode(403, new { 
            success = false, 
            error = "Cloud backups require a valid Pro license" 
        });
    }
    return null;
}

Subscription service logic:

public async Task<bool> HasValidLicenseAsync(string userId)
{
    // Check for gifted license in Clerk metadata
    var hasGiftedLicense = await _clerkService.HasGiftedLicenseAsync(userId);
    if (hasGiftedLicense) return true;

    // Check for active Stripe subscription
    var subscription = await _dbManager.GetSubscriptionByUserIdAsync(userId);
    return subscription != null && subscription.Status == "active";
}

---

Automatic Backup Scheduling

Scheduler Operation

Check interval: Every 15 minutes
Startup delay: 30 seconds (allows other services to initialize)
License check: Every cycle before backup decision

Schedule Decision Logic

1. Get cloud access token from Service_CloudSessionStore
   → If no token: Skip and log once

2. Call GET /cloud-backups/status
   → Returns: hasValidLicense, settings, usage, lastBackup

3. Check hasValidLicense
   → If false: Skip and log once

4. Check settings.enabled
   → If false: Skip and log once

5. Calculate if backup is due:
   → Daily: 24+ hours since lastBackup
   → Weekly: 7+ days since lastBackup
   → Monthly: 30+ days since lastBackup
   → No lastBackup: Backup immediately

6. If backup is due:
   → Create local backup with configured components
   → Request S3 upload URL
   → Upload to S3
   → Confirm completion
   → Update lastBackup timestamp

Frequency Configuration

Users configure backup frequency through the UI: - Daily: Creates backup every 24 hours - Weekly: Creates backup every 7 days - Monthly: Creates backup every 30 days

Settings stored in cloud PostgreSQL BackupSettings table per user.

---

Backup Creation Process

Local Backup Creation

1. Component selection (from cloud settings):
2. includeKeys: Encryption keys directory
3. includeIdentity: backend-id.json file

4. includeFrameEngine: Frame engine data files

5. Database preparation:

6. Force WAL checkpoint to flush pending writes
7. Create temporary copy of database file

8. Package into ZIP with selected components

9. Backup packaging: csharp var options = new Service_Backups.BackupOptions { IncludeKeys = settings.includeKeys, IncludeIdentity = settings.includeIdentity, IncludeFrameEngine = settings.includeFrameEngine }; var backupResult = await _backupService.CreateBackupAsync(options);

Cloud Upload Workflow

Step 1: Request upload URL

POST /cloud-backups/request-upload
Authorization: Bearer {cloud_token}

{
  "filename": "junction_backup_keys_identity_20251006.zip",
  "backendId": "5ce38839962549459c0cc3b933d633b1",
  "uncompressedSize": 214279,
  "compressedSize": 214279
}

Response:

{
  "success": true,
  "backupId": "fa3d0f88-54c5-4b99-8f6d-2a00002b90f7",
  "uploadUrl": "https://junctionrelay.s3.us-east-2.amazonaws.com/...",
  "s3Key": "backups/user_xxx/backend_xxx/2025-10-06/...",
  "expiresAt": "2025-10-06T01:24:19Z"
}

Step 2: Upload to S3

PUT {uploadUrl}
Content-Type: application/octet-stream

[binary backup data]

Step 3: Confirm completion

POST /cloud-backups/complete-upload

{
  "backupId": "fa3d0f88-54c5-4b99-8f6d-2a00002b90f7",
  "actualCompressedSize": 214279
}

Cloud backend updates: - Backup status: pending → completed - User's lastBackup timestamp in settings

---

Backup Storage and Metadata

PostgreSQL Schema

Backups Table:

CREATE TABLE "Backups" (
  "Id" UUID PRIMARY KEY,
  "ClerkUserId" VARCHAR(255) NOT NULL,
  "BackendId" VARCHAR(255) NOT NULL,
  "Filename" VARCHAR(255) NOT NULL,
  "S3Key" VARCHAR(500) NOT NULL,
  "UncompressedSize" BIGINT NOT NULL,
  "CompressedSize" BIGINT NOT NULL,
  "Status" VARCHAR(50) NOT NULL,  -- pending, completed, failed
  "CreatedAt" TIMESTAMP NOT NULL,
  "ExpiresAt" TIMESTAMP NULL
);

BackupSettings Table:

CREATE TABLE "BackupSettings" (
  "ClerkUserId" VARCHAR(255) PRIMARY KEY,
  "Enabled" BOOLEAN NOT NULL,
  "Frequency" VARCHAR(50) NOT NULL,  -- daily, weekly, monthly
  "RetentionDays" INTEGER NOT NULL,
  "IncludeKeys" BOOLEAN NOT NULL,
  "IncludeIdentity" BOOLEAN NOT NULL,
  "IncludeFrameEngine" BOOLEAN NOT NULL,
  "LastBackup" TIMESTAMP NULL,
  "UpdatedAt" TIMESTAMP NOT NULL
);

S3 Storage Structure

s3://junctionrelay/backups/
  ├── {userId}/
  │   ├── {backendId}/
  │   │   ├── {date}/
  │   │   │   ├── {backupId}_{filename}.zip

Example:

backups/
  user_2ymxUlEfdmifJ5dCI462JV6zgcf/
    5ce38839962549459c0cc3b933d633b1/
      2025-10-05/
        fa3d0f88-54c5-4b99-8f6d-2a00002b90f7_junction_backup_keys_identity_20251005160728.zip

Backup Lifecycle

Creation → pending (upload in progress)
         ↓
Upload complete → completed (available for download)
         ↓
Retention period expires → deleted (by worker cleanup)

Pending timeout: 24 hours (uploads older than 24 hours deleted as stale)
Retention period: User configurable (1-365 days, default 30 days)

---

Quota and Limits

Per-User Limits

Limit Type	Value	Enforcement Point
Maximum single backup size	50 MB	Request validation
Maximum total storage	500 MB	Pre-upload check
Maximum backup count	500	Pre-upload check
Presigned URL validity	30 minutes	S3 URL generation

Quota Enforcement

public async Task<(bool Allowed, string Reason)> CanUserBackupAsync(string userId, long backupSize)
{
    // Check single backup size
    if (backupSize > MAX_BACKUP_SIZE_BYTES)
        return (false, "Backup size exceeds maximum limit of 50MB");

    // Check total storage usage
    var totalSize = await GetTotalStorageUsedAsync(userId);
    if (totalSize + backupSize > MAX_TOTAL_STORAGE_BYTES)
        return (false, "Would exceed storage limit of 500MB");

    // Check backup count
    var backupCount = await GetBackupCountAsync(userId);
    if (backupCount >= MAX_BACKUPS_PER_USER)
        return (false, "Maximum number of backups (500) reached");

    return (true, "");
}

Quota check timing: - Before generating upload URL - Returns 429 (Too Many Requests) if quota exceeded - User must delete old backups to create new ones

---

Backup Cleanup Process

Worker Implementation

Service: Service_BackupCleanupProcessor
Trigger: Runs automatically every 24 hours
Database: Direct PostgreSQL connection

Cleanup Operations

1. Delete expired backups:

DELETE FROM "Backups"
WHERE "ExpiresAt" IS NOT NULL 
  AND "ExpiresAt" < now();

2. Delete stale pending backups:

DELETE FROM "Backups"
WHERE "Status" = 'pending' 
  AND "CreatedAt" < now() - interval '24 hours';

Stale Detection Logic

A backup is considered stale when: - Status remains pending (upload never completed) - CreatedAt timestamp is older than 24 hours

Rationale: - Presigned upload URLs expire after 30 minutes - Normal upload completes within minutes - If still pending after 24 hours, upload failed or was abandoned - Database record should be cleaned up (S3 file may be orphaned)

Cleanup Schedule

Worker starts → Wait 24 hours → Run cleanup
                     ↓
              Delete expired backups
                     ↓
              Delete stale pending backups
                     ↓
              Log deletion count
                     ↓
              Wait 24 hours → Repeat

Example logs:

🧹 Starting backup cleanup task...
🧹 Cleaned up 3 expired backups
🧹 Cleaned up 1 stale pending backups
✅ Backup cleanup completed. Removed 4 backups.

---

Security Considerations

Authentication and Authorization

Cloud authentication: - Uses Clerk JWT tokens for user authentication - Device tokens explicitly rejected (no device backup access) - Backend ownership validation before any backup operations

Authorization checks:

// Reject device tokens
if (User.Claims.FirstOrDefault(c => c.Type == "type")?.Value == "device")
    return StatusCode(403, new { error = "Device tokens not permitted" });

// Validate backend ownership
var ownershipResult = await _backendOwnership.ValidateOrClaimOwnershipAsync(userId, backendId);
if (!ownershipResult.IsSuccess)
    return StatusCode(403, new { error = ownershipResult.ErrorMessage });

Data Protection

In transit: - All API calls use HTTPS/TLS encryption - S3 uploads use presigned URLs with AWS signature authentication - 30-minute URL expiration limits exposure window

At rest: - S3 server-side encryption enabled - PostgreSQL connection encrypted - Backup files contain encrypted secrets (using user's encryption keys)

Access control: - User can only access their own backups (userId validation) - User can only backup backends they own (ownership validation) - Presigned URLs scoped to specific S3 keys

Threat Model

Compromised cloud backend: - Attacker gains access to backup metadata (filenames, sizes, timestamps) - Cannot decrypt backup contents (user encryption keys not in cloud) - Cannot access S3 files without AWS credentials

Compromised S3 bucket: - Attacker gains access to encrypted backup files - Cannot decrypt without user's encryption keys - Metadata in PostgreSQL still protected

Compromised user account: - Attacker can download user's backups - Can decrypt if user stored encryption keys in backups - Mitigation: Users should not include keys in backups for maximum security

---

API Endpoints Reference

Status and Settings

GET /cloud-backups/status - Returns: License status, settings, usage statistics, last backup - Authentication: Required (Clerk JWT) - License check: No (informational endpoint)

POST /cloud-backups/settings - Updates: Frequency, retention, component settings, enabled status - Authentication: Required - License check: Yes (if enabling backups)

Backup Operations

POST /cloud-backups/request-upload - Returns: Presigned S3 upload URL, backup ID - Rate limit: Backup policy (stricter than dashboard) - License check: Yes (Pro required)

POST /cloud-backups/complete-upload - Marks backup as completed - Updates lastBackup timestamp - License check: Yes (Pro required)

GET /cloud-backups/list - Returns: List of user's backups with metadata - Query params: backendId (optional), limit (max 100) - License check: Yes (Pro required)

POST /cloud-backups/{backupId}/request-download - Returns: Presigned S3 download URL - URL validity: 30 minutes - License check: Yes (Pro required)

DELETE /cloud-backups/{backupId} - Deletes backup metadata from database - Note: S3 file deletion not yet implemented - License check: Yes (Pro required)

Rate Limiting

Dashboard policy (inherited class-level): - Standard rate limits for viewing backups - Applied to: list, status, request-download, delete, settings

Backup policy (endpoint-level override): - Stricter limits for resource-intensive operations - Applied to: request-upload, complete-upload

---

Complete Backup Flow Sequence

AUTOMATIC BACKUP (Every 15 minutes):

1. Scheduler wakes up
2. Get cloud access token from Service_CloudSessionStore
3. Call GET /cloud-backups/status
4. Cloud backend validates token, checks Stripe subscription
5. Cloud returns: hasValidLicense, settings (enabled, frequency, lastBackup), usage
6. Scheduler checks hasValidLicense
   → If false: Log "No valid Pro license" and skip
7. Scheduler checks settings.enabled
   → If false: Log "Backups disabled" and skip
8. Scheduler calculates time since lastBackup
9. Compare against frequency setting:
   → Daily: Need 24+ hours elapsed
   → Weekly: Need 7+ days elapsed
   → Monthly: Need 30+ days elapsed
10. If backup not due: Skip and wait 15 more minutes
11. If backup is due:
    a. Log "Starting scheduled backup"
    b. Call Service_Backups.CreateBackupAsync(options from settings)
    c. Local backup created with configured components
    d. POST /cloud-backups/request-upload with filename and size
    e. Cloud validates license, quota, backend ownership
    f. Cloud creates pending record in PostgreSQL
    g. Cloud generates presigned S3 URL (30-minute expiration)
    h. Cloud returns: backupId, uploadUrl, s3Key
    i. Scheduler uploads backup to S3 via PUT request
    j. POST /cloud-backups/complete-upload with backupId
    k. Cloud marks backup as completed
    l. Cloud updates user's lastBackup timestamp
    m. Log "Backup completed successfully"
12. Wait 15 minutes and repeat from step 1

MANUAL BACKUP (User clicks "Backup Now"):

1. User clicks "Create Backup Now" button in UI
2. Frontend calls POST /api/cloud-backups/create
3. Local backend Controller_CloudBackups.CreateBackup()
4. Same flow as automatic backup steps 11a-11m
5. Return success message to UI
6. UI refreshes backup list

CLEANUP (Worker, daily):

1. Worker wakes up (24 hours since last run)
2. Connect to PostgreSQL
3. Execute DELETE for expired backups (ExpiresAt < now)
4. Log count of expired backups deleted
5. Execute DELETE for stale pending (Status=pending, CreatedAt > 24h old)
6. Log count of stale backups deleted
7. Sleep for 24 hours
8. Repeat from step 1

---

Configuration and Deployment

Environment Variables

Local backend:

CLOUD_BACKEND_URL=https://api.junctionrelay.com
# Cloud session managed by Service_CloudSessionStore

Cloud backend:

POSTGRES_CONNECTION_STRING=postgresql://...
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-2
AWS_S3_BUCKET=junctionrelay

Worker:

POSTGRES_CONNECTION_STRING=postgresql://...
REDIS_CONNECTION_URL=redis://...

Service Registration

Local backend (Program.cs):

// Register as singleton (not hosted service)
builder.Services.AddSingleton<Service_CloudBackup_Scheduler>();

// Started by Service_Startup
public class Service_Startup : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        // ... wait for dependencies ...

        // Start cloud backup scheduler
        _backupScheduler = scope.ServiceProvider.GetRequiredService<Service_CloudBackup_Scheduler>();
        _ = _backupScheduler.StartAsync(stoppingToken);
    }
}

Worker (Program.cs):

// Register cleanup services
builder.Services.AddSingleton<Service_Database_CloudBackups>();
builder.Services.AddSingleton<Service_BackupCleanupProcessor>();

// Start cleanup loop
var backupCleanupProcessor = host.Services.GetRequiredService<Service_BackupCleanupProcessor>();
_ = Task.Run(() => RunBackupCleanup(backupCleanupProcessor));

---

Monitoring and Observability

Logging

Scheduler logs (every 15 minutes):

[CLOUD_BACKUP] 🚀 Cloud Backup Scheduler initializing...
[CLOUD_BACKUP] ✅ Cloud Backup Scheduler started
[CLOUD_BACKUP] ⏰ Backup due (Last: 2025-10-05 16:07:28 UTC, Elapsed: 24.2h)
[CLOUD_BACKUP] 📦 Starting scheduled backup (Frequency: daily)
[CLOUD_BACKUP] 📦 Creating local backup...
[CLOUD_BACKUP] ✅ Local backup created: junction_backup_keys_identity_20251006.zip (214279 bytes)
[CLOUD_BACKUP] 🔗 Got upload URL for backup fa3d0f88-54c5-4b99-8f6d-2a00002b90f7
[CLOUD_BACKUP] ✅ Successfully uploaded backup to cloud storage
[CLOUD_BACKUP] ✅ Scheduled backup completed successfully

Error conditions:

[CLOUD_BACKUP] ⏸️ No valid cloud authentication - skipping backup check
[CLOUD_BACKUP] ⏸️ No valid Pro license - skipping backup check
[CLOUD_BACKUP] ⏸️ Cloud backups disabled - skipping
[CLOUD_BACKUP] ❌ Failed to create local backup: [error]
[CLOUD_BACKUP] ❌ Failed to request upload URL: [error]
[CLOUD_BACKUP] ❌ Failed to upload backup to cloud storage

Worker logs (daily):

🧹 Starting backup cleanup task...
🧹 Cleaned up 5 expired backups
🧹 Cleaned up 2 stale pending backups
✅ Backup cleanup completed. Removed 7 backups.

Metrics to Monitor

Operational health: - Backup success rate (completed / attempted) - Average backup size and duration - Scheduler check failures (authentication, license) - Upload failures and retry count

Resource utilization: - Total storage used per user - Number of backups per user - Cleanup job execution time - PostgreSQL query performance

Business metrics: - Pro users with backups enabled - Total backups created per day - Storage growth rate - Retention distribution (30d, 90d, 365d)

---

Troubleshooting Guide

User Cannot Create Backups

Symptom: 403 Forbidden error when attempting backup

Diagnosis: 1. Check user's Stripe subscription status in PostgreSQL 2. Verify license cache hasn't expired (4-hour TTL) 3. Check if user has gifted license in Clerk metadata

Resolution: - Ensure active Stripe subscription exists - Clear license cache if stale: _subscriptionService.ClearUserCache(userId) - Verify Clerk gifted license metadata if applicable

Backups Not Running Automatically

Symptom: No automatic backups despite enabled settings

Diagnosis: 1. Check scheduler logs for authentication failures 2. Verify settings.enabled = true in cloud database 3. Check lastBackup timestamp vs frequency setting 4. Ensure Pro license is valid

Resolution: - Refresh cloud authentication token - Verify backup settings in database - Manually update lastBackup if stuck in past - Check license status with cloud backend

Stale Pending Backups Accumulating

Symptom: Many pending backups older than 24 hours

Diagnosis: 1. Check worker logs for cleanup execution 2. Verify worker has PostgreSQL access 3. Check for database connection issues

Resolution: - Restart worker process - Manually run cleanup: await _backupDb.CleanupExpiredBackupsAsync() - Investigate upload failures causing pending state

Storage Quota Exceeded

Symptom: 429 error when creating backups

Diagnosis: 1. Query total storage used: SELECT SUM("CompressedSize") FROM "Backups" WHERE "ClerkUserId" = ? 2. Check backup count: SELECT COUNT(*) FROM "Backups" WHERE "ClerkUserId" = ?

Resolution: - User must delete old backups to free space - Consider increasing quota limits for specific users - Implement automatic cleanup of oldest backups

---

Future Enhancements

Planned Features

S3 file deletion: - Currently only database records are deleted - TODO: Implement actual S3 object deletion in DeleteBackupAsync() - Prevents orphaned files accumulating in S3

Backup verification: - Checksum validation after upload - Periodic integrity checks of stored backups - Automated restoration testing

Advanced scheduling: - Custom cron-style schedules - Backup windows (e.g., only backup at night) - Differential/incremental backups

Enhanced monitoring: - Backup success/failure notifications - Email alerts for quota approaching limits - Grafana dashboards for backup metrics

Multi-region support: - Backup replication across S3 regions - Geographic redundancy for disaster recovery - User-selectable storage regions

---

Summary

The JunctionRelay Cloud Backup System provides a comprehensive, fully-automated backup solution with strict Pro license enforcement. The three-component architecture (local scheduler, cloud backend, worker cleanup) ensures reliable, scheduled backups while maintaining security and quota limits.

Key Features: - Automatic scheduling - Daily/weekly/monthly backups without user intervention - Pro license enforcement - Only paying customers can access cloud backups - Configurable components - Users control what gets backed up - Quota management - Prevents storage abuse with hard limits - Automatic cleanup - Worker maintains database health daily - Secure storage - S3 with encryption and access controls

Security Guarantees: - Pro license validated on every API call - Backend ownership verified before operations - Presigned URLs limit exposure to 30 minutes - User encryption keys never transmitted to cloud - Multi-layer authentication and authorization

Operational Excellence: - Distributed architecture prevents single points of failure - Comprehensive logging for troubleshooting - Graceful degradation when cloud unavailable - Resource limits prevent system abuse - Daily cleanup maintains database performance