Building a Custom Serialization Framework (Compress, Encrypt, Version Data)

Modern enterprise systems often need to serialize complex objects, transfer them securely across services, and store them in a compact, version-friendly format. While JSON works for most applications, systems that deal with sensitive data, massive payloads, or backward compatibility quickly outgrow default serializers.

This article explains how to design a custom serialization framework that supports:

Compression — reduce payload size
Encryption — secure sensitive fields
Versioning — decode old payloads gracefully
Extensibility — easily plug in new formats (binary, JSON, custom)
Cross-platform compatibility — Angular → ASP.NET Core → SQL → Queues
Performance optimization — streaming, chunking, caching

1. Why Build a Custom Serialization Framework?

Typical enterprise challenges:

Need to serialize millions of records for ETL or data sync jobs
Need backward compatibility when object schema evolves
Need secure transmission over distributed microservices
Need smaller payloads for faster API responses or cheaper storage
Need partial encryption (some fields encrypted, others not)
Need cross-language decoding (Angular, .NET, Python, Go)

A custom framework provides full control over how objects are encoded, compressed, encrypted, and resolved by version.

2. Architecture Overview

Below is a clean modular architecture.

Layers

Serializer
Converts objects to raw bytes (JSON, binary, protobuf, BSON, custom format)
Compressor
Applies compression algorithm: GZip, Brotli, LZ4, Zstd
Encryptor
AES-256, RSA, hybrid encryption, field-level encryption
Version Header Manager
Attaches version identifiers & metadata
Payload Wrapper
Bundles everything into a single transport-safe object

[Object] 
   → Serializer 
      → Compressor 
         → Encryptor 
            → Version Header 
               → Payload Bytes

3. Designing the Payload Structure

A recommended binary payload structure:

| VERSION (int16) | FLAGS (bitmask) | LENGTH (int32) | DATA (byte[]) |

Version — tells decoder which schema to use
Flags — compression type, encryption type
Length — size of the final byte array
Data — compressed + encrypted serialized object

Example flag bitmask

Bit	Meaning
0001	Compressed GZIP
0010	Compressed Brotli
0100	AES-256 Encryption
1000	RSA Encryption

4. Serialization Strategy

Option A: JSON Serialization

Human readable
Good for mixed systems
Slower & larger
Great for debugging

Option B: Binary Serialization (recommended)

Smallest size
Fast encoding/decoding
Best for internal processes
Standard options: MessagePack, Protobuf, FlatBuffers

Recommendation

Use MessagePack — extremely fast, cross-language, lightweight.

5. Compression Layer

Recommended algorithms

Algorithm	Best Use Case
GZip	General compression, moderate speed
Brotli	Highest compression ratio, slower
LZ4	Lightning-fast, moderate compression
Zstd	Best balanced, enterprise recommended

Recommendation

Use LZ4 for real-time APIs
Use Brotli for storage (archives, snapshots)

6. Encryption Layer

Use a hybrid encryption strategy:

1. AES-256 (fast symmetric encryption)

2. RSA (encrypt the AES key)

Payload structure:

{
   aesEncryptedData,
   rsaEncryptedKey,
   initializationVector,
   flags,
   version
}

Partial field-level encryption

Encrypt only critical fields (PII):

{
   "name": "John",
   "secretData_encrypted": "A78DN4K3...",
   "metadata": { ... }
}

This is useful when some consumers don’t require secure fields.

7. Version Management Layer

Versioning is the most important part of enterprise serialization.

Version Strategy

Major Version (Breaking change)
- Database restructuring
- Object property removals/renames
Minor Version (Non-breaking change)
- Adding nullable fields
- Adding metadata
Patch Version
- Bug fixes or encoding adjustments

How to handle multi-version decoding

Implement a version switch:

switch (payload.Version)
{
    case 1:
        return DeserializeV1(payload.Data);
    case 2:
        return DeserializeV2(payload.Data);
    case 3:
        return DeserializeV3(payload.Data);
    default:
        throw new VersionNotSupportedException();
}

Use converter classes like:

V1 → V2 (add fields)  
V2 → V3 (rename fields)

This ensures backward compatibility.

8. ASP.NET Core Implementation (Reference Code)

Serialize

public byte[] Serialize<T>(T obj, short version)
{
    // 1. Serialize
    var raw = MessagePackSerializer.Serialize(obj);

    // 2. Compress
    var compressed = LZ4Pickler.Pickle(raw);

    // 3. Encrypt
    var encrypted = EncryptAES(compressed);

    // 4. Wrap with header
    using var ms = new MemoryStream();
    using var bw = new BinaryWriter(ms);

    bw.Write(version);
    bw.Write((byte)CompressionType.LZ4 | (byte)EncryptionType.AES);
    bw.Write(encrypted.Length);
    bw.Write(encrypted);

    return ms.ToArray();
}

Deserialize

public T Deserialize<T>(byte[] bytes)
{
    using var ms = new MemoryStream(bytes);
    using var br = new BinaryReader(ms);

    var version = br.ReadInt16();
    var flags = br.ReadByte();
    var length = br.ReadInt32();

    var encrypted = br.ReadBytes(length);
    var decrypted = DecryptAES(encrypted);
    var decompressed = LZ4Pickler.Unpickle(decrypted);

    return LoadByVersion<T>(decompressed, version);
}

9. Angular Front-End Integration

Typical flow

User edits a complex object
Angular serializes using MessagePack
Sends compressed + encrypted payload to .NET
.NET stores encoded blob in SQL
Angular receives compressed data for read-only views

Angular can use:

msgpack-lite
lz4js
crypto-js (AES)

10. Storing Serialized Data in SQL

Use VARBINARY(MAX) for storing byte payloads:

ALTER TABLE UserSnapshot  
ADD Payload VARBINARY(MAX),  
    Version SMALLINT,  
    Flags INT;

Indexes on version & flags help during batch migration jobs.

11. Performance Optimization Techniques

1. Chunking large payloads

For > 10 MB data, stream instead of loading whole file into memory.

2. Caching serializers

MessagePack recommend reusing resolvers/formatters.

3. Async streaming APIs

ASP.NET Core supports streaming large payloads via IAsyncEnumerable.

4. Parallel compression

Use ParallelCompression for archives.

12. Use Cases

Offline mobile sync
Ledger or audit entries
ERP snapshots
Attachments stored in compressed binary blobs
High-security document workflows
Cross-language microservice communication

13. Final Recommendations

Use MessagePack as the base serializer
Use LZ4 for live systems, Brotli for cold storage
Use AES + RSA hybrid encryption
Maintain version headers in every payload
Create version resolvers to ensure backward compatibility
Store serialized blobs in VARBINARY(MAX)
Add batch migration scripts when version changes