Skip to content

JunctionRelay Enhanced Payload Architecture: Native USB CDC, Compression, and LLLLTTRR Prefix Format

Overview

JunctionRelay's enhanced payload architecture implements a sophisticated communication protocol supporting multiple data transport methods, intelligent compression, structured prefix formatting, and high-performance Native USB CDC connectivity. This system enables efficient data transmission between devices and backends while maintaining backward compatibility and providing advanced features like compression and routing.

This document describes the complete payload handling system, including the four supported payload types, LLLLTTRR prefix format, Native USB CDC implementation, compression using miniz with C# gzip compatibility, and routing capabilities across all communication channels.

Core Payload Architecture

The Four Payload Types

JunctionRelay supports exactly four payload scenarios, providing complete coverage for all communication needs:

Case Prefix Compression Format Description
Type 1 ❌ ❌ Raw JSON Direct JSON transmission without prefix or compression
Type 2 ✅ ❌ Prefixed JSON JSON with LLLLTTRR prefix containing metadata
Type 3 ❌ ✅ Raw Gzip Gzip-compressed data without prefix
Type 4 ✅ ✅ Prefixed Gzip Gzip-compressed data with LLLLTTRR prefix

Design Principles

  • Universal compatibility - All transport methods support all four payload types
  • Intelligent detection - Automatic payload type recognition without ambiguity
  • Efficient compression - Significant bandwidth savings for large payloads (60-80% typical reduction)
  • Structured metadata - Rich prefix information for routing and processing
  • Backward compatibility - Existing JSON payloads continue to work seamlessly
  • Transport agnostic - Works across USB, HTTP, WebSocket, MQTT, and ESP-NOW
  • Binary transmission - All HTTP traffic now uses raw binary (no Base64 overhead)

LLLLTTRR Prefix Format

Structure Definition

The enhanced prefix format uses exactly 8 ASCII digits with structured meaning:

LLLLTTRR
├── LLLL = Length Hint (4 digits, 0000-9999)
├── TT   = Type Field (2 digits)
│   ├── 00 = JSON (uncompressed)
│   └── 01 = Gzip (compressed)
└── RR   = Routing Field (2 digits)
    ├── 00 = Terminal (process locally)
    └── 01 = Forward (gateway routing)

Example Prefixes

00000000 = No length hint, JSON, terminal processing
12340001 = 1234-byte hint, JSON, forward to gateway
00000101 = No length hint, Gzip, forward to gateway
45670100 = 4567-byte hint, Gzip, terminal processing

Length Hint Behavior

  • Non-zero hint - Indicates expected payload size for pre-allocation
  • Zero hint (0000) - Enables auto-detection mode for variable-length streams
  • Maximum value - 9999 bytes for compatibility with ESP32 memory constraints
  • Overflow handling - Values exceeding payload size use actual payload size
  • Optimization - ESP32 can pre-allocate buffers when hint is provided

Type Field Usage

  • 00 (JSON) - Indicates uncompressed JSON data follows prefix
  • 01 (Gzip) - Indicates gzip-compressed data follows prefix
  • Validation - Only values 00 and 01 are valid; others trigger errors
  • Processing - Determines decompression pathway in payload handler

Routing Field Applications

  • 00 (Terminal) - Data processed locally by receiving device
  • 01 (Forward) - Data forwarded through gateway to destination
  • Gateway detection - Automatically set based on junction type
  • ESP-NOW routing - Enables transparent gateway forwarding
  • Future expansion - Additional routing modes can use values 02-99

Native USB CDC Implementation

Architecture Overview

JunctionRelay implements high-performance Native USB CDC communication, replacing traditional UART-based serial communication with modern USB capabilities:

// Native USB CDC Configuration
Serial.begin();  // Auto-negotiates maximum speed
Serial.setRxBufferSize(4096);  // Large buffer for high throughput

Performance Enhancements

Buffer Management

  • Large static buffers - 2048-byte USB buffer vs. previous dynamic allocation
  • Bounds checking - Prevents buffer overflows with explicit validation
  • Memory tracking - Real-time heap monitoring with low-memory protection
  • Efficient clearing - memset operations for buffer cleanup

High-Speed Processing

  • Reduced yielding - Less frequent task switching due to USB efficiency
  • Chunked reading - Processes data in optimal chunk sizes
  • Progress reporting - Debug output every 200 bytes for large transfers
  • Non-blocking initialization - 5-second timeout prevents indefinite blocking

Implementation Details

USB Buffer Architecture

class ConnectionManager {
private:
    static constexpr size_t USB_BUFFER_SIZE = 2048;
    uint8_t usbSerialBuffer[USB_BUFFER_SIZE];
    uint32_t initialHeap = 0;
    uint32_t minHeap = 0;

    // Stream state variables (moved from static to class members)
    bool streamReadingLength = true;
    int streamBytesRead = 0;
    int streamPayloadLength = 0;
    char streamPrefixBuffer[9] = {0};
    uint8_t* streamPayloadBuffer = nullptr;
    static const size_t MAX_PAYLOAD_SIZE = 8192;
};

Memory Health Monitoring

void ConnectionManager::checkMemoryHealth(const char* location) {
    uint32_t currentHeap = ESP.getFreeHeap();
    if (currentHeap < minHeap) minHeap = currentHeap;

    if (currentHeap < 20000) {
        Serial.printf("HEAP %s: %d (min: %d, used: %d)\n", 
                     location, currentHeap, minHeap, initialHeap - currentHeap);
    }

    if (currentHeap < 15000) {
        Serial.println("*** LOW MEMORY WARNING ***");
    }
}

Efficient Data Reading

void ConnectionManager::handleNativeUSBData() {
    size_t bytesRead = 0;

    while (Serial.available() && bytesRead < (USB_BUFFER_SIZE - 1)) {
        if (bytesRead >= USB_BUFFER_SIZE) {
            Serial.println("USB BUFFER OVERFLOW PREVENTED");
            return;
        }

        uint8_t b = Serial.read();
        usbSerialBuffer[bytesRead++] = b;

        if (bytesRead % 200 == 0 && bytesRead > 0) {
            Serial.printf("USB READING: %d bytes...\n", bytesRead);
        }

        if (bytesRead % 100 == 0) {
            yield();
        }
    }

    if (bytesRead > 0) {
        handleIncomingDataChunkPrefix(usbSerialBuffer, bytesRead);
        memset(usbSerialBuffer, 0, bytesRead);
        checkMemoryHealth("USB_PROCESSING");
    }
}

UART Compatibility

The system maintains full backward compatibility with UART communication:

void ConnectionManager::handleSerialData() {
    // Native USB CDC (preferred)
    if (Serial.available() > 0) {
        handleNativeUSBData();
    }

    // UART TX/RX pins (legacy)
    if (Serial1.available() > 0) {
        size_t len = Serial1.available();
        uint8_t* buffer = new uint8_t[len];
        Serial1.readBytes(buffer, len);
        handleIncomingDataChunkPrefix(buffer, len);
        delete[] buffer;
    }
}

Compression Implementation

miniz Library Integration with C# Compatibility

JunctionRelay uses the miniz library for high-performance gzip decompression with special handling for C# .NET gzip format compatibility:

// Enhanced decompression with C# compatibility
void ConnectionManager::decompressAndProcess(uint8_t* compressedData, size_t compressedLen) {
    // Validate minimum gzip header size
    if (compressedLen < 18) { // 10 byte header + 8 byte footer minimum
        Serial.printf("[DECOMPRESS] ❌ Data too small for complete gzip stream: %d bytes\n", compressedLen);
        return;
    }

    // Validate gzip magic bytes
    if (compressedData[0] != 0x1F || compressedData[1] != 0x8B) {
        Serial.printf("[DECOMPRESS] ❌ Invalid gzip magic bytes: 0x%02X 0x%02X\n", 
                     compressedData[0], compressedData[1]);
        return;
    }

    // Allocate decompression buffer
    const size_t DECOMP_BUFFER_SIZE = 16384;
    uint8_t* decompBuffer = new uint8_t[DECOMP_BUFFER_SIZE];

    if (!decompBuffer) {
        Serial.println("[DECOMPRESS] ❌ Failed to allocate decompression buffer");
        return;
    }

    // Extract raw deflate data (skip gzip header and footer)
    // This method works around C# .NET gzip format differences
    size_t deflateDataLen = compressedLen - 18; // Remove 10-byte header and 8-byte footer
    uint8_t* deflateData = compressedData + 10; // Skip 10-byte gzip header

    // Initialize deflate stream
    mz_stream stream = {};
    stream.next_in = deflateData;
    stream.avail_in = deflateDataLen;
    stream.next_out = decompBuffer;
    stream.avail_out = DECOMP_BUFFER_SIZE - 1;

    // Decompress using raw deflate (no gzip wrapper)
    int result = mz_inflateInit2(&stream, -15); // -15 = raw deflate, no headers
    if (result == MZ_OK) {
        result = mz_inflate(&stream, MZ_FINISH);
        if (result == MZ_STREAM_END) {
            mz_ulong decompSize = DECOMP_BUFFER_SIZE - 1 - stream.avail_out;

            // Null-terminate for safety
            decompBuffer[decompSize] = '\0';

            Serial.printf("[DECOMPRESS] ✅ C# Gzip compatibility: %d bytes -> %d bytes\n", 
                         compressedLen, (int)decompSize);

            // Process the decompressed JSON data
            handleIncomingDataChunk(decompBuffer, decompSize);
        } else {
            Serial.printf("[DECOMPRESS] ❌ Raw deflate failed: %d\n", result);
        }
        mz_inflateEnd(&stream);
    } else {
        Serial.printf("[DECOMPRESS] ❌ Raw deflate init failed: %d\n", result);
    }
}

Buffer Overflow Protection

// Multi-level bounds checking
if (streamBytesRead >= USB_BUFFER_SIZE) {
    Serial.println("USB BUFFER OVERFLOW PREVENTED");
    return;
}

if (streamBytesRead + copyLen <= MAX_PAYLOAD_SIZE) {
    memcpy(streamPayloadBuffer + streamBytesRead, data, copyLen);
    streamBytesRead += copyLen;
} else {
    Serial.printf("[ERROR] Payload buffer overflow: %d + %d > %d\n", 
                 streamBytesRead, copyLen, MAX_PAYLOAD_SIZE);
    resetStreamState();
    return;
}

Binary Protocol Validation

// Enhanced validation for binary HTTP protocol
bool validateBinaryPayload(const uint8_t* data, size_t len) {
    // Check minimum length
    if (len < 1) return false;

    // Validate prefix format if present
    if (isdigit(data[0])) {
        if (len < 8) return false;

        // Check all prefix characters are digits
        for (int i = 0; i < 8; i++) {
            if (!isdigit(data[i])) return false;
        }

        // Validate type field (00 or 01 only)
        int typeField = (data[4] - '0') * 10 + (data[5] - '0');
        if (typeField != 0 && typeField != 1) return false;

        // Validate routing field
        int routeField = (data[6] - '0') * 10 + (data[7] - '0');
        if (routeField > 99) return false;
    }

    return true;
}

Configuration and Customization

Compile-Time Options

// Buffer size configuration
static constexpr size_t USB_BUFFER_SIZE = 2048;      // USB receive buffer
static constexpr size_t MAX_PAYLOAD_SIZE = 8192;     // Maximum payload size
static constexpr size_t DECOMP_BUFFER_SIZE = 16384;  // Decompression buffer

// Debug output configuration
#define DEBUG_PAYLOAD_PROCESSING  // Enable detailed payload debug output
#define DEBUG_MEMORY_TRACKING     // Enable memory usage reporting
#define DEBUG_COMPRESSION_STATS   // Enable compression ratio reporting
#define DEBUG_BINARY_PROTOCOL     // Enable binary protocol debugging

// C# Gzip compatibility mode
#define USE_CSHARP_GZIP_COMPAT    // Enable C# .NET gzip compatibility

Runtime Configuration

// Memory monitoring thresholds
const uint32_t MEMORY_WARNING_THRESHOLD = 20000;   // Warn below 20KB
const uint32_t MEMORY_CRITICAL_THRESHOLD = 15000;  // Critical below 15KB

// Progress reporting intervals
const size_t USB_PROGRESS_INTERVAL = 200;          // Report every 200 bytes
const size_t USB_YIELD_INTERVAL = 100;             // Yield every 100 bytes

// Compression settings
const size_t MIN_COMPRESSION_SIZE = 200;            // Don't compress below 200 bytes
const int MAX_COMPRESSION_RATIO = 100;              // Maximum 100:1 compression ratio

Backend Configuration

// Enhanced payload configuration
public class PayloadConfig {
    public bool EnableCompression { get; set; } = true;
    public int CompressionThreshold { get; set; } = 500;  // Compress payloads > 500 bytes
    public bool IncludePrefixByDefault { get; set; } = true;
    public string DefaultRoutingHint { get; set; } = "00";
    public bool UseBinaryHTTP { get; set; } = true;       // Enable raw binary HTTP
    public bool EnableKeepAlive { get; set; } = true;     // HTTP keep-alive connections
}

Migration and Compatibility

Backward Compatibility

  • Raw JSON (Type 1) - Existing payloads work without modification
  • Gradual migration - Devices can be upgraded independently
  • Detection logic - Automatic handling of legacy and enhanced formats
  • No breaking changes - All existing functionality preserved
  • Binary transition - HTTP clients automatically handle binary protocol

Migration Strategy

  1. Phase 1 - Deploy enhanced payload detection to all devices
  2. Phase 2 - Enable binary HTTP protocol on backend services
  3. Phase 3 - Enable prefix generation for new configurations
  4. Phase 4 - Enable compression for large payloads (>1KB)
  5. Phase 5 - Optimize routing and gateway features

Version Detection and Compatibility

// Enhanced version detection with binary protocol support
if (data[0] == '{') {
    // Type 1: Raw JSON - maintain full compatibility
    Serial.println("[COMPAT] Legacy JSON format detected");
    handleIncomingDataChunk(data, len);
} else if (isdigit(data[0])) {
    // Type 2/4: Enhanced format with prefix
    Serial.println("[COMPAT] Enhanced prefixed format detected");
    // Continue with prefix parsing...
} else if (data[0] == 0x1F && data[1] == 0x8B) {
    // Type 3: Raw compressed format
    Serial.println("[COMPAT] Raw gzip format detected");
    decompressAndProcess(data, len);
} else {
    Serial.println("[COMPAT] Unknown payload format");
    // Error handling...
}

Testing and Validation

Test Coverage

Unit Tests

  • Payload type detection - All four types correctly identified
  • Prefix parsing - LLLLTTRR format validation and extraction
  • C# gzip compatibility - Round-trip compression/decompression with .NET
  • Binary HTTP protocol - Raw binary transmission without encoding
  • State management - Class member variables prevent corruption
  • Buffer management - No memory leaks or overflows
  • Error handling - Graceful handling of malformed data

Integration Tests

  • Transport protocols - All payload types across all transports with binary HTTP
  • Gateway forwarding - Routing field functionality
  • Memory constraints - Operation under low-memory conditions
  • Large payloads - Streaming behavior with multi-KB data
  • Mixed environments - Legacy and enhanced devices interoperating
  • C# backend compatibility - Full compatibility with .NET compression

Performance Tests

  • Binary HTTP benchmarks - Performance vs Base64 encoding
  • C# gzip compatibility - Compression/decompression speed and ratios
  • Memory usage - Peak and steady-state memory consumption
  • Throughput - Native USB CDC vs. UART performance comparison
  • Latency - End-to-end processing time for all payload types
  • State management - No corruption under high load

Validation Tools

Debug Output Analysis

// Comprehensive debug output for troubleshooting
Serial.printf("[PAYLOAD] Type: %s, Size: %d, Compressed: %s, Route: %s, Transport: Binary HTTP\n",
             (typeValue == 0) ? "JSON" : "Gzip",
             streamPayloadLength,
             (typeValue == 1) ? "Yes" : "No",
             (routeValue == 1) ? "Forward" : "Terminal");

Serial.printf("[BINARY] Raw bytes: ");
for (int i = 0; i < min(16, (int)len); i++) {
    Serial.printf("%02X ", data[i]);
}
Serial.println();

Memory and State Tracking

void printSystemStatus() {
    Serial.printf("Memory: Free=%d, Min=%d, Used=%d\n",
                 ESP.getFreeHeap(),
                 minHeap,
                 initialHeap - ESP.getFreeHeap());

    Serial.printf("Stream State: Reading=%s, Bytes=%d, Length=%d\n",
                 streamReadingLength ? "Length" : "Payload",
                 streamBytesRead,
                 streamPayloadLength);

    Serial.printf("Buffers: USB=%p, Stream=%p\n",
                 usbSerialBuffer,
                 streamPayloadBuffer);
}

Future Enhancements

Planned Features

Additional Compression Algorithms

  • LZ4 - Faster compression for real-time applications
  • Brotli - Better compression ratios for large payloads
  • Adaptive selection - Choose algorithm based on payload characteristics
  • Hardware acceleration - ESP32-specific compression optimizations

Enhanced Binary Protocol

  • Protocol versioning - Version field in prefix for future compatibility
  • Checksum validation - Integrity checking for binary transmissions
  • Streaming compression - Real-time compression for large payloads
  • Multi-part payloads - Support for payloads exceeding memory limits

Advanced Routing

  • Multi-hop routing - Support for complex network topologies
  • Load balancing - Distribute traffic across multiple gateways
  • Priority routing - QoS-based message prioritization
  • Encrypted routing - Secure gateway forwarding

Extensibility Points

Custom Payload Types

// Framework for adding new payload types
enum PayloadType {
    JSON_UNCOMPRESSED = 0,
    GZIP_COMPRESSED = 1,
    LZ4_COMPRESSED = 2,    // Future
    BROTLI_COMPRESSED = 3, // Future
    ENCRYPTED = 4,         // Future
    MULTIPART = 5          // Future
};

// Extensible processor registry
class PayloadProcessor {
public:
    virtual bool canHandle(uint8_t typeField) = 0;
    virtual void process(uint8_t* data, size_t len) = 0;
    virtual bool supportsBinary() = 0;  // Binary protocol support
};

std::map<uint8_t, std::unique_ptr<PayloadProcessor>> processors;

Enhanced Binary Protocol Extensions

// Future binary protocol enhancements
struct EnhancedPrefix {
    char version[2];      // Protocol version (VV)
    char length[4];       // Length hint (LLLL)
    char type[2];         // Type field (TT)
    char route[2];        // Routing field (RR)
    char checksum[4];     // CRC32 checksum (CCCC)
    char reserved[2];     // Reserved for future use (RR)
};  // Total: 16 bytes for enhanced prefix

Transport Layer Abstraction

// Unified transport interface with binary support
class TransportLayer {
public:
    virtual bool send(const PayloadData& data) = 0;
    virtual void setPayloadHandler(PayloadHandler* handler) = 0;
    virtual bool supportsCompression() = 0;
    virtual bool supportsBinary() = 0;        // Binary protocol support
    virtual size_t getMaxPayloadSize() = 0;
    virtual bool supportsStreaming() = 0;     // Streaming payload support
};

// Enhanced implementations
class BinaryHTTPTransport : public TransportLayer {
    bool supportsBinary() override { return true; }
    bool supportsStreaming() override { return true; }
    // Raw binary HTTP implementation
};

Security Considerations

Binary Protocol Security

Data Integrity for Binary Transmissions

  • Input validation - Strict validation of binary payload format
  • Bounds checking - Comprehensive buffer overflow protection
  • Type validation - Strict enforcement of payload type values
  • Length validation - Payload size limits and sanity checking

Binary Transmission Security

// Secure binary payload validation
bool validateBinaryTransmission(const uint8_t* data, size_t len) {
    // Minimum length check
    if (len == 0 || len > MAX_SECURE_PAYLOAD_SIZE) {
        return false;
    }

    // Check for malformed binary data
    if (data == nullptr) {
        return false;
    }

    // Validate binary payload structure
    if (isdigit(data[0])) {
        // Prefixed payload - validate prefix structure
        return validatePrefixStructure(data, len);
    } else if (data[0] == '{') {
        // JSON payload - validate JSON structure
        return validateJSONStructure(data, len);
    } else if (data[0] == 0x1F && data[1] == 0x8B) {
        // Gzip payload - validate gzip header
        return validateGzipStructure(data, len);
    }

    return false; // Unknown format
}

Compression Security Enhancements

C# Gzip Compatibility Security

// Enhanced security for C# gzip decompression
bool safeCSGzipDecompress(const uint8_t* compressed, size_t compressedLen, 
                         uint8_t* output, size_t* outputLen) {

    // Validate gzip magic bytes
    if (compressedLen < 2 || compressed[0] != 0x1F || compressed[1] != 0x8B) {
        Serial.println("[SECURITY] Invalid gzip magic bytes");
        return false;
    }

    // Check compression ratio limit (prevent zip bombs)
    if (*outputLen > compressedLen * MAX_SAFE_COMPRESSION_RATIO) {
        Serial.println("[SECURITY] Compression ratio too high - potential bomb");
        return false;
    }

    // Enforce absolute size limit
    if (*outputLen > MAX_DECOMPRESSED_SIZE) {
        Serial.println("[SECURITY] Decompressed size exceeds security limit");
        return false;
    }

    // Use our C# compatible decompression method
    return performCSCompatibleDecompression(compressed, compressedLen, output, outputLen);
}

Binary Protocol Attack Mitigation

// Comprehensive binary protocol security
class SecureBinaryProcessor {
private:
    uint32_t processedPayloads = 0;
    uint32_t rejectedPayloads = 0;
    unsigned long lastProcessTime = 0;

public:
    bool processSecurely(const uint8_t* data, size_t len) {
        // Rate limiting
        unsigned long now = millis();
        if (now - lastProcessTime < MIN_PROCESS_INTERVAL) {
            Serial.println("[SECURITY] Rate limit exceeded");
            rejectedPayloads++;
            return false;
        }
        lastProcessTime = now;

        // Input sanitization
        if (!validateBinaryTransmission(data, len)) {
            Serial.println("[SECURITY] Binary validation failed");
            rejectedPayloads++;
            return false;
        }

        // Memory pressure check
        if (ESP.getFreeHeap() < SECURITY_MEMORY_THRESHOLD) {
            Serial.println("[SECURITY] Insufficient memory for secure processing");
            rejectedPayloads++;
            return false;
        }

        processedPayloads++;
        return true;
    }

    float getRejectionRate() const {
        if (processedPayloads + rejectedPayloads == 0) return 0.0;
        return (float)rejectedPayloads / (processedPayloads + rejectedPayloads);
    }
};

Monitoring and Diagnostics

Enhanced Binary Protocol Metrics

Real-Time Binary Protocol Statistics

struct BinaryProtocolStats {
    uint32_t totalBinaryPayloads = 0;
    uint32_t totalTextPayloads = 0;
    uint32_t compressionSuccesses = 0;
    uint32_t compressionFailures = 0;
    uint32_t csharpGzipCompatibility = 0;
    uint64_t totalBytesProcessed = 0;
    uint64_t totalBytesDecompressed = 0;
    float averageBinaryEfficiency = 0.0;
    uint32_t stateCorruptionPrevented = 0;
    uint32_t bufferOverflowsPrevented = 0;
};

BinaryProtocolStats binaryStats;

void updateBinaryStats(PayloadType type, size_t originalSize, 
                      size_t processedSize, bool wasCompressed) {
    if (isBinaryTransport()) {
        binaryStats.totalBinaryPayloads++;
    } else {
        binaryStats.totalTextPayloads++;
    }

    binaryStats.totalBytesProcessed += originalSize;

    if (wasCompressed) {
        binaryStats.totalBytesDecompressed += processedSize;
        binaryStats.compressionSuccesses++;
        binaryStats.csharpGzipCompatibility++;

        float efficiency = (float)originalSize / processedSize;
        binaryStats.averageBinaryEfficiency = 
            (binaryStats.averageBinaryEfficiency * (binaryStats.compressionSuccesses - 1) + efficiency) / 
            binaryStats.compressionSuccesses;
    }
}

Performance Monitoring with Binary Protocol

class BinaryPerformanceMonitor {
private:
    uint32_t binaryProcessStartTime;
    uint32_t compressionTime;
    uint32_t stateManagementTime;
    uint32_t totalBinaryProcessingTime;
    uint32_t payloadsPerSecond;

public:
    void startBinaryProcessing() { 
        binaryProcessStartTime = micros(); 
    }

    void endBinaryProcessing() { 
        totalBinaryProcessingTime = micros() - binaryProcessStartTime;
        updateThroughputMetrics();
    }

    void startCompression() { binaryProcessStartTime = micros(); }
    void endCompression() { compressionTime = micros() - binaryProcessStartTime; }

    void reportBinaryPerformance() {
        Serial.printf("[PERF] Binary Protocol - Total: %dµs, Compression: %dµs, "
                     "Throughput: %d payloads/sec\n",
                     totalBinaryProcessingTime, compressionTime, payloadsPerSecond);
    }

private:
    void updateThroughputMetrics() {
        static uint32_t lastCount = 0;
        static unsigned long lastTime = millis();
        static uint32_t payloadCount = 0;

        payloadCount++;
        unsigned long now = millis();

        if (now - lastTime >= 1000) { // Update every second
            payloadsPerSecond = payloadCount - lastCount;
            lastCount = payloadCount;
            lastTime = now;
        }
    }
};

Diagnostic Endpoints

Enhanced Statistics API

String getBinaryProtocolStatistics() {
    StaticJsonDocument<1024> doc;

    doc["protocol"]["type"] = "binary";
    doc["protocol"]["version"] = "2.0";
    doc["protocol"]["csharpCompatible"] = true;

    doc["payloads"]["totalBinary"] = binaryStats.totalBinaryPayloads;
    doc["payloads"]["totalText"] = binaryStats.totalTextPayloads;
    doc["payloads"]["compressionSuccesses"] = binaryStats.compressionSuccesses;
    doc["payloads"]["compressionFailures"] = binaryStats.compressionFailures;
    doc["payloads"]["csharpGzipCount"] = binaryStats.csharpGzipCompatibility;

    doc["performance"]["totalBytesProcessed"] = binaryStats.totalBytesProcessed;
    doc["performance"]["totalBytesDecompressed"] = binaryStats.totalBytesDecompressed;
    doc["performance"]["averageBinaryEfficiency"] = binaryStats.averageBinaryEfficiency;
    doc["performance"]["compressionRatio"] = 
        binaryStats.totalBytesProcessed > 0 ? 
        (double)binaryStats.totalBytesDecompressed / binaryStats.totalBytesProcessed : 0.0;

    doc["security"]["stateCorruptionPrevented"] = binaryStats.stateCorruptionPrevented;
    doc["security"]["bufferOverflowsPrevented"] = binaryStats.bufferOverflowsPrevented;

    doc["memory"]["freeHeap"] = ESP.getFreeHeap();
    doc["memory"]["minHeap"] = minHeap;
    doc["memory"]["streamBufferAllocated"] = (streamPayloadBuffer != nullptr);

    String output;
    serializeJson(doc, output);
    return output;
}

Binary Protocol Health Check

GET /api/payload/binary-health
{
  "status": "healthy",
  "binaryProtocol": {
    "enabled": true,
    "version": "2.0",
    "supportedTypes": [1, 2, 3, 4],
    "compressionCompatibility": "csharp",
    "stateManagement": "class-members"
  },
  "performance": {
    "averageProcessingTime": "1.8ms",
    "compressionRatio": "0.68",
    "binaryEfficiency": "25% faster than Base64",
    "throughput": "450 payloads/sec"
  },
  "compatibility": {
    "csharpGzipSupport": true,
    "rawBinaryHTTP": true,
    "legacyJsonSupport": true,
    "backwardCompatible": true
  },
  "security": {
    "bufferOverflowProtection": true,
    "stateCorruptionPrevention": true,
    "compressionBombProtection": true,
    "inputValidation": "comprehensive"
  }
}

Best Practices

Implementation Guidelines

Binary Protocol Best Practices

  1. Start with Type 1 (Raw JSON) for initial implementation and testing
  2. Enable binary HTTP for all new deployments (25% bandwidth improvement)
  3. Add prefixes (Type 2) when metadata and routing are needed
  4. Enable compression (Types 3, 4) for payloads >500 bytes
  5. Use C# gzip compatibility for .NET backend integration
  6. Implement proper state management using class member variables

Memory Management for Binary Protocol

// Best practices for binary payload memory handling
class BinaryPayloadProcessor {
private:
    static constexpr size_t BINARY_BUFFER_SIZE = 2048;
    uint8_t* binaryBuffer = nullptr;
    bool bufferInitialized = false;

public:
    BinaryPayloadProcessor() {
        binaryBuffer = new uint8_t[BINARY_BUFFER_SIZE];
        if (binaryBuffer) {
            memset(binaryBuffer, 0, BINARY_BUFFER_SIZE);
            bufferInitialized = true;
        } else {
            Serial.println("[ERROR] Failed to allocate binary buffer");
        }
    }

    ~BinaryPayloadProcessor() {
        if (binaryBuffer) {
            memset(binaryBuffer, 0, BINARY_BUFFER_SIZE); // Secure cleanup
            delete[] binaryBuffer;
            binaryBuffer = nullptr;
        }
    }

    bool isValid() const { return bufferInitialized && binaryBuffer != nullptr; }

    bool processBinaryPayload(const uint8_t* data, size_t len) {
        if (!isValid() || !data || len == 0 || len > BINARY_BUFFER_SIZE) {
            return false;
        }

        // Safe binary processing with bounds checking
        memcpy(binaryBuffer, data, len);
        return performBinaryProcessing(binaryBuffer, len);
    }
};

Error Handling for Binary Protocol

// Defensive programming for binary protocol
bool processBinaryPayload(const uint8_t* data, size_t len) {
    // Input validation
    if (!data || len == 0 || len > MAX_BINARY_PAYLOAD_SIZE) {
        Serial.println("[ERROR] Invalid binary payload parameters");
        return false;
    }

    // State validation
    if (!isInitialized() || streamPayloadBuffer == nullptr) {
        Serial.println("[ERROR] Binary processor not properly initialized");
        return false;
    }

    // Memory availability check
    if (ESP.getFreeHeap() < MIN_BINARY_PROCESSING_MEMORY) {
        Serial.println("[ERROR] Insufficient memory for binary processing");
        return false;
    }

    // Protocol validation
    if (!validateBinaryTransmission(data, len)) {
        Serial.println("[ERROR] Binary protocol validation failed");
        return false;
    }

    // Process with exception handling
    try {
        return doBinaryProcessing(data, len);
    } catch (...) {
        Serial.println("[ERROR] Exception during binary processing");
        resetStreamState();
        return false;
    }
}

Performance Optimization

Binary Protocol Efficiency

// Optimized binary processing
class OptimizedBinaryProcessor {
private:
    uint8_t* reusableBuffer = nullptr;
    size_t bufferSize = 0;
    uint32_t processCount = 0;

public:
    bool initializeOptimized(size_t maxPayloadSize) {
        bufferSize = maxPayloadSize;
        reusableBuffer = new uint8_t[bufferSize];
        return reusableBuffer != nullptr;
    }

    bool processOptimized(const uint8_t* data, size_t len) {
        // Reuse allocated buffer to avoid malloc/free overhead
        if (len <= bufferSize) {
            memcpy(reusableBuffer, data, len);
            processCount++;
            return true;
        }
        return false;
    }

    uint32_t getProcessedCount() const { return processCount; }

    ~OptimizedBinaryProcessor() {
        if (reusableBuffer) {
            delete[] reusableBuffer;
        }
    }
};

Compression Decision Logic

// Smart compression decisions for binary protocol
bool shouldCompressBinary(const uint8_t* data, size_t size) {
    // Don't compress small payloads (binary overhead not worth it)
    if (size < MIN_BINARY_COMPRESSION_SIZE) return false;

    // Always compress large payloads
    if (size > LARGE_PAYLOAD_THRESHOLD) return true;

    // For medium payloads, check if it's likely JSON
    if (data[0] == '{' || (size > 8 && data[8] == '{')) {
        // JSON data compresses well
        return true;
    }

    // Binary data may not compress well
    return false;
}

Conclusion

The JunctionRelay Enhanced Payload Architecture represents a comprehensive evolution of modern IoT communication protocols. This updated implementation delivers significant improvements across all aspects of device communication:

Key Architectural Achievements

  1. Unified Binary Protocol - All transport methods now use consistent binary transmission
  2. C# .NET Compatibility - Seamless integration with .NET backend services
  3. Enhanced State Management - Class member variables eliminate corruption issues
  4. Performance Excellence - 25% bandwidth improvement with binary HTTP protocol
  5. Robust Error Handling - Comprehensive protection against protocol failures
  6. Universal Compatibility - All four payload types work across all transport protocols
  7. Memory Efficiency - Optimized buffer management for resource-constrained devices
  8. Security Focus - Multi-layered protection against malformed data and attacks

Technical Excellence

  • Standards Compliance - RFC 1952 gzip format with C# .NET compatibility
  • Binary Protocol Innovation - Raw binary transmission without encoding overhead
  • State Management - Proper class-based state prevents corruption
  • Compression Compatibility - Works seamlessly with C# GZipStream
  • Diagnostic Capabilities - Comprehensive monitoring and debugging tools
  • Extensible Architecture - Clean design for future protocol enhancements

Performance Improvements

  • Binary HTTP Protocol: 25% bandwidth improvement over Base64 encoding
  • C# Gzip Compatibility: Reliable decompression of .NET compressed data
  • Enhanced State Management: No more static variable corruption issues
  • Memory Optimization: Efficient buffer reuse and cleanup
  • Processing Speed: Streamlined binary handling without encoding overhead

Future-Ready Foundation

The architecture provides a robust foundation for future enhancements including: - Advanced compression algorithms (LZ4, Brotli) - Enhanced security features (encryption, authentication) - Protocol versioning and evolution - Hardware acceleration support - Advanced routing and mesh networking

Production Readiness

This system is now production-ready for demanding industrial and commercial IoT applications, providing: - Reliability: Comprehensive error handling and recovery - Performance: Optimized for high-throughput applications - Compatibility: Seamless integration with existing and new systems - Security: Multi-layered protection against attacks and failures - Maintainability: Clean architecture with extensive diagnostics

The JunctionRelay Enhanced Payload Architecture positions the platform as a leading solution for next-generation IoT device communication, delivering the performance, reliability, and flexibility required for modern connected systems while maintaining full backward compatibility with existing deployments. { mz_ulong decompSize = DECOMP_BUFFER_SIZE - 1 - stream.avail_out;

        // Null-terminate for safety
        decompBuffer[decompSize] = '\0';

        Serial.printf("[DECOMPRESS] ✅ Decompressed %d bytes -> %d bytes\n", 
                     compressedLen, (int)decompSize);

        // Process the decompressed JSON data
        handleIncomingDataChunk(decompBuffer, decompSize);
    } else {
        Serial.printf("[DECOMPRESS] ❌ Deflate failed: %d\n", result);
    }
    mz_inflateEnd(&stream);
} else {
    Serial.printf("[DECOMPRESS] ❌ Deflate init failed: %d\n", result);
}

delete[] decompBuffer;

}

### Backend Compression (C#)

```csharp
// C# compression generates standard gzip format
private byte[] CompressData(string data) {
    var bytes = Encoding.UTF8.GetBytes(data);
    using var output = new MemoryStream();
    using (var gzip = new GZipStream(output, CompressionMode.Compress)) {
        gzip.Write(bytes, 0, bytes.Length);
    }
    return output.ToArray();
}

Compression Benefits

  • Bandwidth reduction - Typically 60-80% reduction for JSON payloads
  • Memory efficiency - Smaller buffers needed for transmission
  • Network optimization - Reduced transmission time over all protocols
  • Battery savings - Less radio usage on wireless devices
  • C# Compatibility - Works seamlessly with .NET GZipStream output

Magic Byte Detection

Gzip payloads are automatically detected using RFC 1952 magic bytes:

bool isGzipData(const uint8_t* data, int len) {
    return len >= 2 && data[0] == 0x1F && data[1] == 0x8B;
}

Enhanced Payload Detection and Processing

Intelligent Stream Parser with Class Member State

The payload handler implements sophisticated streaming logic using class member variables for proper state management:

void ConnectionManager::handleIncomingDataChunkPrefix(uint8_t *data, size_t len) {
    if (data == nullptr || len == 0) {
        return;
    }

    // Initialize stream buffer if needed
    if (streamPayloadBuffer == nullptr) {
        streamPayloadBuffer = new uint8_t[MAX_PAYLOAD_SIZE];
        memset(streamPayloadBuffer, 0, MAX_PAYLOAD_SIZE);
    }

    // Payload type detection at start of new message
    if (streamReadingLength && streamBytesRead == 0 && len > 0) {
        // Type 1: Raw JSON detection
        if (data[0] == '{') {
            Serial.println("[PAYLOAD] Detected Raw JSON (Type 1)");
            handleIncomingDataChunk(data, len);
            return;
        }

        // Type 3: Raw Gzip detection
        if (len >= 2 && data[0] == 0x1F && data[1] == 0x8B) {
            Serial.println("[PAYLOAD] Detected Raw Gzip (Type 3)");
            decompressAndProcess(data, len);
            return;
        }
    }

    // Prefix parsing for Types 2 and 4
    if (streamReadingLength) {
        size_t prefixCopyLen = min(len, (size_t)(8 - streamBytesRead));
        memcpy(streamPrefixBuffer + streamBytesRead, data, prefixCopyLen);
        streamBytesRead += prefixCopyLen;

        if (streamBytesRead >= 8) {
            streamPrefixBuffer[8] = '\0';

            // Check if the prefix contains only digits
            bool isValidPrefix = true;
            for (int i = 0; i < 8; i++) {
                if (!isdigit(streamPrefixBuffer[i])) {
                    isValidPrefix = false;
                    break;
                }
            }

            if (!isValidPrefix) {
                Serial.println("[ERROR] Invalid prefix format - expected 8 digits");
                resetStreamState();
                return;
            }

            // Parse LLLLTTRR format
            char lengthHint[5] = {0};
            char typeField[3] = {0};
            char routeField[3] = {0};

            memcpy(lengthHint, streamPrefixBuffer, 4);
            memcpy(typeField, streamPrefixBuffer + 4, 2);
            memcpy(routeField, streamPrefixBuffer + 6, 2);

            int lengthHintValue = atoi(lengthHint);
            int typeValue = atoi(typeField);
            int routeValue = atoi(routeField);

            Serial.printf("[PREFIX] Length hint: %04d, Type: %02d, Route: %02d\n", 
                         lengthHintValue, typeValue, routeValue);

            // Validate type field
            if (typeValue != 0 && typeValue != 1) {
                Serial.printf("[ERROR] Invalid type field: %02d (expected 00 or 01)\n", typeValue);
                resetStreamState();
                return;
            }

            // Use length hint or default
            if (lengthHintValue > 0) {
                streamPayloadLength = lengthHintValue;
            } else {
                streamPayloadLength = MAX_PAYLOAD_SIZE;
                Serial.println("[WARNING] Length hint is 0000, using auto-detection mode");
            }

            if (streamPayloadLength <= 0 || streamPayloadLength > MAX_PAYLOAD_SIZE) {
                Serial.printf("[ERROR] Invalid payload length: %d\n", streamPayloadLength);
                resetStreamState();
                return;
            }

            Serial.printf("[PREFIX] Expecting %d bytes, Type: %s\n", 
                         streamPayloadLength, (typeValue == 0) ? "JSON" : "Gzip");

            streamReadingLength = false;
            streamBytesRead = 0;

            // If there's remaining data after the prefix, start accumulating payload
            if (prefixCopyLen < len) {
                size_t remainingLen = len - prefixCopyLen;
                if (remainingLen <= MAX_PAYLOAD_SIZE) {
                    memcpy(streamPayloadBuffer, data + prefixCopyLen, remainingLen);
                    streamBytesRead += remainingLen;
                } else {
                    Serial.printf("[ERROR] Remaining data too large: %d\n", remainingLen);
                    resetStreamState();
                    return;
                }
            }
        }
    } 
    // Payload accumulation
    else {
        size_t remainingBytes = streamPayloadLength - streamBytesRead;
        size_t copyLen = (len < remainingBytes) ? len : remainingBytes;

        if (streamBytesRead + copyLen <= MAX_PAYLOAD_SIZE) {
            memcpy(streamPayloadBuffer + streamBytesRead, data, copyLen);
            streamBytesRead += copyLen;
        } else {
            Serial.printf("[ERROR] Payload buffer overflow: %d + %d > %d\n", 
                         streamBytesRead, copyLen, MAX_PAYLOAD_SIZE);
            resetStreamState();
            return;
        }
    }

    // Process complete payload
    if (!streamReadingLength && streamBytesRead >= streamPayloadLength) {
        // Determine payload type from the original prefix
        char typeField[3] = {0};
        memcpy(typeField, streamPrefixBuffer + 4, 2);
        int typeValue = atoi(typeField);

        if (typeValue == 0) {
            // Type 2: Prefixed JSON
            Serial.println("[PAYLOAD] Processing Prefixed JSON (Type 2)");
            handleIncomingDataChunk(streamPayloadBuffer, streamPayloadLength);
        } else if (typeValue == 1) {
            // Type 4: Prefixed Gzip
            Serial.println("[PAYLOAD] Processing Prefixed Gzip (Type 4)");
            decompressAndProcess(streamPayloadBuffer, streamPayloadLength);
        } else {
            Serial.printf("[ERROR] Unknown payload type: %02d\n", typeValue);
        }

        resetStreamState();
    }
}

Fixed State Management

void ConnectionManager::resetStreamState() {
    streamReadingLength = true;
    streamBytesRead = 0;
    streamPayloadLength = 0;
    memset(streamPrefixBuffer, 0, sizeof(streamPrefixBuffer));
    if (streamPayloadBuffer) {
        memset(streamPayloadBuffer, 0, MAX_PAYLOAD_SIZE);
    }
}

Backend Implementation (C#) - Binary HTTP Architecture

Enhanced HTTP Sender - Unified Binary Protocol

public class Service_Send_Data_HTTP : IDisposable
{
    public Task<(bool Success, string ResponseMessage)> SendPayloadAsync(string payload)
    {
        try
        {
            if (_disposed)
                return Task.FromResult((false, "HTTP sender has been disposed."));

            if (string.IsNullOrEmpty(payload))
                return Task.FromResult((false, "Payload cannot be null or empty."));

            // Convert string to UTF-8 bytes and send as binary
            byte[] payloadBytes = Encoding.UTF8.GetBytes(payload);
            return SendPayloadAsync(payloadBytes);
        }
        catch (Exception ex)
        {
            Console.WriteLine($"[SERVICE_SEND_DATA_HTTP] Error converting string payload: {ex.Message}");
            return Task.FromResult((false, ex.Message));
        }
    }

    public async Task<(bool Success, string ResponseMessage)> SendPayloadAsync(byte[] payloadBytes)
    {
        var result = await SendPayloadWithHealthAsync(payloadBytes);
        return (result.Success, result.ResponseMessage);
    }

    public async Task<HttpSendResult> SendPayloadWithHealthAsync(byte[] payloadBytes)
    {
        if (_disposed)
            throw new ObjectDisposedException(nameof(Service_Send_Data_HTTP));

        if (payloadBytes == null || payloadBytes.Length == 0)
            return new HttpSendResult
            {
                Success = false,
                ErrorType = "invalid_payload",
                ErrorMessage = "Payload cannot be null or empty."
            };

        var stopwatch = Stopwatch.StartNew();
        bool poolRecreated = false;

        try
        {
            // Send raw binary bytes directly
            var content = new ByteArrayContent(payloadBytes);
            content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/octet-stream");

            // Add headers to indicate binary payload format
            content.Headers.Add("X-Payload-Type", "binary");
            if (payloadBytes.Length > 8)
            {
                // Check for ASCII prefix in binary data (8-digit prefix)
                bool hasPrefix = true;
                for (int i = 0; i < 8 && i < payloadBytes.Length; i++)
                {
                    if (payloadBytes[i] < '0' || payloadBytes[i] > '9')
                    {
                        hasPrefix = false;
                        break;
                    }
                }
                if (hasPrefix)
                {
                    content.Headers.Add("X-Payload-Prefix", "true");
                    // Extract and log the prefix for debugging
                    string prefix = Encoding.ASCII.GetString(payloadBytes, 0, 8);
                    content.Headers.Add("X-Payload-Prefix-Value", prefix);
                }
            }

            using var request = new HttpRequestMessage(HttpMethod.Post, _endpointUrl)
            {
                Content = content
            };

            HttpResponseMessage response = await _httpClient.SendAsync(request);
            stopwatch.Stop();

            string responseBody = await response.Content.ReadAsStringAsync();

            if (response.IsSuccessStatusCode)
            {
                return new HttpSendResult
                {
                    Success = true,
                    StatusCode = (int)response.StatusCode,
                    LatencyMs = stopwatch.ElapsedMilliseconds,
                    ResponseMessage = responseBody,
                    KeepAlivePoolRecreated = poolRecreated
                };
            }
            else
            {
                return new HttpSendResult
                {
                    Success = false,
                    StatusCode = (int)response.StatusCode,
                    ErrorType = "http_error",
                    ErrorMessage = $"HTTP {response.StatusCode}: {responseBody}",
                    LatencyMs = stopwatch.ElapsedMilliseconds,
                    ResponseMessage = responseBody,
                    KeepAlivePoolRecreated = poolRecreated
                };
            }
        }
        catch (Exception ex)
        {
            stopwatch.Stop();
            Console.WriteLine($"[SERVICE_SEND_DATA_HTTP] Error sending binary payload to {_endpointUrl}: {ex.Message}");

            return new HttpSendResult
            {
                Success = false,
                ErrorType = "binary_send_error",
                ErrorMessage = ex.Message,
                LatencyMs = stopwatch.ElapsedMilliseconds,
                KeepAlivePoolRecreated = poolRecreated
            };
        }
    }
}

Updated Stream Manager - Binary Support

// Key changes in Service_Stream_Manager_HTTP for binary protocol support

// For compressed byte[] payloads - send raw binary directly
if (rawCompressed is byte[] compressedBytes)
{
    // Extract prefix from binary payload
    string compressedPrefix = ExtractBinaryPrefix(compressedBytes);
    info.UpdateCompressedConfigPayloadPrefix(compressedPrefix);

    // Send raw binary bytes directly (no Base64 conversion)
    var (success, _) = await httpSender.SendPayloadAsync(compressedBytes);
    if (!success)
    {
        Console.WriteLine($"[SERVICE_STREAM_MANAGER_HTTP] Failed to send compressed config payload.");
        info.Dispose();
        _streamingTokens.TryRemove(screen.Id, out _);
        return;
    }
}

// For sensor data streaming - use health tracking method with binary conversion
Stopwatch stopwatch = Stopwatch.StartNew();
var result = await info.HttpSender!.SendPayloadWithHealthAsync(Encoding.UTF8.GetBytes(uncompressedSensorJson));
stopwatch.Stop();

// Update health information
info.Health.UpdateHealth(result);

Automatic Routing Detection

public async Task<Dictionary<string, object>> GenerateConfigPayloadsAsync(
    string screenKey,
    List<Model_Sensor> assignedSensors,
    Model_Device_Screens screen,
    Model_Screen_Layout? overrideTemplate = null,
    string? junctionType = null,
    string? gatewayDestination = null,
    bool compressPayload = false) {

    // ... payload building logic ...

    // Automatic routing detection
    string routingHint = (!string.IsNullOrEmpty(junctionType) && 
                         junctionType.Contains("Gateway", StringComparison.OrdinalIgnoreCase)) 
                         ? "01" : "00";

    object finalPayload = SerializeWithOptionalPrefix(
        payloadDict, 
        template.IncludePrefixConfig, 
        "config", 
        compressPayload, 
        routingHint);

    result[screenKey] = finalPayload;
    return result;
}

Transport Protocol Support

Universal Binary Compatibility

All four payload types now work seamlessly across every transport method with unified binary transmission:

Native USB CDC

  • Type 1-4 - Full support with high-speed processing
  • Streaming - Handles chunked data across USB packet boundaries
  • Performance - Optimized for large payload transfers
  • Class member state - Proper state management prevents corruption

HTTP POST (Enhanced Binary)

  • Type 1-4 - Complete support via raw binary request body
  • No Base64 overhead - Direct binary transmission for all payloads
  • Content-Type - application/octet-stream for all transmissions
  • Prefix detection - Automatic header generation for debugging

WebSocket

  • Type 1-4 - Real-time bidirectional support
  • Binary frames - All payloads sent as binary WebSocket frames
  • Unified protocol - Consistent binary format across all channels

MQTT

  • Type 1-4 - Publish/subscribe with all payload types
  • QoS support - Reliable delivery for compressed payloads
  • Binary publishing - Direct binary payload publishing

ESP-NOW

  • Type 1-4 - Peer-to-peer mesh communication
  • Broadcast - All payload types support broadcast transmission
  • Gateway forwarding - Routing field enables transparent forwarding

Transport-Specific Optimizations

USB-Specific Features

void ConnectionManager::initNativeUSB() {
    Serial.begin();  // Auto-negotiates maximum speed
    while (!Serial && millis() < 5000) {
        delay(10);  // Non-blocking initialization
    }

    if (Serial) {
        Serial.setRxBufferSize(4096);  // Large receive buffer
        Serial.println("[USB] Native USB CDC initialized");
    }

    memset(usbSerialBuffer, 0, USB_BUFFER_SIZE);
}

HTTP Binary Frame Handling

// C# HTTP binary handling
server.on("/api/data", HTTP_POST,
    [](AsyncWebServerRequest* req){ 
        AsyncWebServerResponse *response = req->beginResponse(200, "text/plain", "OK");
        response->addHeader("Connection", "keep-alive");
        response->addHeader("Keep-Alive", "timeout=60, max=1000");
        req->send(response);
    },
    nullptr,
    [](AsyncWebServerRequest* req, uint8_t* data, size_t len, size_t, size_t) {
        if (gConnMgr) gConnMgr->handleIncomingDataChunkPrefix(data, len);
    }
);

Performance Characteristics

Compression Ratios

Typical compression performance for different payload types:

Payload Type Original Size Compressed Size Reduction
Config JSON 1,200 bytes 480 bytes 60%
Sensor JSON 800 bytes 320 bytes 60%
Large Config 3,000 bytes 900 bytes 70%
Matrix Data 2,400 bytes 720 bytes 70%

Memory Usage

ESP32 Memory Profile

  • USB Buffer - 2,048 bytes static allocation
  • Stream Buffer - 8,192 bytes static allocation
  • Decompression Buffer - 16,384 bytes temporary allocation
  • State Variables - ~50 bytes class member variables
  • Total Peak - ~26KB during large compressed payload processing

Processing Performance

  • Native USB CDC - ~10x faster than UART for large payloads
  • Binary HTTP - ~25% faster than Base64 (no encoding overhead)
  • C# Gzip Compatibility - ~150ms for 3KB payload decompression on ESP32
  • Detection - <1ms for payload type identification
  • State Management - No corruption from static variable issues

Bandwidth Optimization

Before Enhancement (Base64 Encoded)

HTTP POST: Content-Type: application/json
Base64 payload: 33% size overhead + JSON structure

After Enhancement (Raw Binary)

HTTP POST: Content-Type: application/octet-stream  
Raw binary: No encoding overhead, direct transmission
25% bandwidth improvement over Base64

Error Handling and Recovery

Robust State Management

// Class member variables prevent state corruption
class ConnectionManager {
private:
    // Stream state variables (moved from static to class members)
    bool streamReadingLength = true;
    int streamBytesRead = 0;
    int streamPayloadLength = 0;
    char streamPrefixBuffer[9] = {0};
    uint8_t* streamPayloadBuffer = nullptr;
    static const size_t MAX_PAYLOAD_SIZE = 8192;

public:
    void resetStreamState() {
        streamReadingLength = true;
        streamBytesRead = 0;
        streamPayloadLength = 0;
        memset(streamPrefixBuffer, 0, sizeof(streamPrefixBuffer));
        if (streamPayloadBuffer) {
            memset(streamPayloadBuffer, 0, MAX_PAYLOAD_SIZE);
        }
    }
};

C# Gzip Compatibility Error Handling

```cpp // Enhanced decompression with fallback method int result = mz_inflateInit2(&stream, -15); // -15 = raw deflate, no headers if (result == MZ_OK) { result = mz_inflate(&stream, MZ_FINISH); if (result == MZ_STREAM_END)