Building a Custom Serialization Framework (Compress, Encrypt, Version Data) | Angular + ASP.NET Core

Modern enterprise systems often need to serialize complex objects, transfer them securely across services, and store them in a compact, version-friendly format. While JSON works for most applications, systems that deal with sensitive data, massive payloads, or backward compatibility quickly outgrow default serializers.

This article explains how to design a custom serialization framework that supports:

1. Compression — reduce payload size

2. Encryption — secure sensitive fields

3. Versioning — decode old payloads gracefully

4. Extensibility — easily plug in new formats (binary, JSON, custom)

5. Cross-platform compatibility — Angular → ASP.NET Core → SQL → Queues

6. Performance optimization — streaming, chunking, caching

1. Why Build a Custom Serialization Framework?

Typical enterprise challenges:

• Need to serialize millions of records for ETL or data sync jobs

• Need backward compatibility when object schema evolves

• Need secure transmission over distributed microservices

• Need smaller payloads for faster API responses or cheaper storage

• Need partial encryption (some fields encrypted, others not)

• Need cross-language decoding (Angular, .NET, Python, Go)

A custom framework provides full control over how objects are encoded, compressed, encrypted, and resolved by version.

2. Architecture Overview

Below is a clean modular architecture.

Layers

1. Serializer
   Converts objects to raw bytes (JSON, binary, protobuf, BSON, custom format)

2. Compressor
   Applies compression algorithm: GZip, Brotli, LZ4, Zstd

3. Encryptor
   AES-256, RSA, hybrid encryption, field-level encryption

4. Version Header Manager
   Attaches version identifiers & metadata

5. Payload Wrapper
   Bundles everything into a single transport-safe object

[Object] 
   → Serializer 
      → Compressor 
         → Encryptor 
            → Version Header 
               → Payload Bytes

3. Designing the Payload Structure

A recommended binary payload structure:

| VERSION (int16) | FLAGS (bitmask) | LENGTH (int32) | DATA (byte[]) |

• Version — tells decoder which schema to use

• Flags — compression type, encryption type

• Length — size of the final byte array

• Data — compressed + encrypted serialized object

Example flag bitmask

Bit	Meaning
0001	Compressed GZIP
0010	Compressed Brotli
0100	AES-256 Encryption
1000	RSA Encryption

4. Serialization Strategy

Option A: JSON Serialization

• Human readable

• Good for mixed systems

• Slower & larger

• Great for debugging

Option B: Binary Serialization (recommended)

• Smallest size

• Fast encoding/decoding

• Best for internal processes

• Standard options: MessagePack, Protobuf, FlatBuffers

Recommendation

Use MessagePack — extremely fast, cross-language, lightweight.

5. Compression Layer

Recommended algorithms

Algorithm	Best Use Case
GZip	General compression, moderate speed
Brotli	Highest compression ratio, slower
LZ4	Lightning-fast, moderate compression
Zstd	Best balanced, enterprise recommended

Recommendation

Use LZ4 for real-time APIs
Use Brotli for storage (archives, snapshots)

6. Encryption Layer

Use a hybrid encryption strategy:

1. AES-256 (fast symmetric encryption)

2. RSA (encrypt the AES key)

Payload structure:

{
   aesEncryptedData,
   rsaEncryptedKey,
   initializationVector,
   flags,
   version
}

Partial field-level encryption

Encrypt only critical fields (PII):

{
   "name": "John",
   "secretData_encrypted": "A78DN4K3...",
   "metadata": { ... }
}

This is useful when some consumers don’t require secure fields.

7. Version Management Layer

Versioning is the most important part of enterprise serialization.

Version Strategy

1. Major Version (Breaking change)

   • Database restructuring

   • Object property removals/renames

2. Minor Version (Non-breaking change)

   • Adding nullable fields

   • Adding metadata

3. Patch Version

   • Bug fixes or encoding adjustments

How to handle multi-version decoding

Implement a version switch:

switch (payload.Version)
{
    case 1:
        return DeserializeV1(payload.Data);
    case 2:
        return DeserializeV2(payload.Data);
    case 3:
        return DeserializeV3(payload.Data);
    default:
        throw new VersionNotSupportedException();
}

Use converter classes like:

V1 → V2 (add fields)  
V2 → V3 (rename fields)  

This ensures backward compatibility.

8. ASP.NET Core Implementation (Reference Code)

Serialize

public byte[] Serialize<T>(T obj, short version)
{
    // 1. Serialize
    var raw = MessagePackSerializer.Serialize(obj);

    // 2. Compress
    var compressed = LZ4Pickler.Pickle(raw);

    // 3. Encrypt
    var encrypted = EncryptAES(compressed);

    // 4. Wrap with header
    using var ms = new MemoryStream();
    using var bw = new BinaryWriter(ms);

    bw.Write(version);
    bw.Write((byte)CompressionType.LZ4 | (byte)EncryptionType.AES);
    bw.Write(encrypted.Length);
    bw.Write(encrypted);

    return ms.ToArray();
}

Deserialize

public T Deserialize<T>(byte[] bytes)
{
    using var ms = new MemoryStream(bytes);
    using var br = new BinaryReader(ms);

    var version = br.ReadInt16();
    var flags = br.ReadByte();
    var length = br.ReadInt32();

    var encrypted = br.ReadBytes(length);
    var decrypted = DecryptAES(encrypted);
    var decompressed = LZ4Pickler.Unpickle(decrypted);

    return LoadByVersion<T>(decompressed, version);
}

9. Angular Front-End Integration

Typical flow

1. User edits a complex object

2. Angular serializes using MessagePack

3. Sends compressed + encrypted payload to .NET

4. .NET stores encoded blob in SQL

5. Angular receives compressed data for read-only views

Angular can use:

msgpack-lite
lz4js
crypto-js (AES)

10. Storing Serialized Data in SQL

Use VARBINARY(MAX) for storing byte payloads:

ALTER TABLE UserSnapshot  
ADD Payload VARBINARY(MAX),  
    Version SMALLINT,  
    Flags INT;

Indexes on version & flags help during batch migration jobs.

11. Performance Optimization Techniques

1. Chunking large payloads

For > 10 MB data, stream instead of loading whole file into memory.

2. Caching serializers

MessagePack recommend reusing resolvers/formatters.

3. Async streaming APIs

ASP.NET Core supports streaming large payloads via IAsyncEnumerable.

4. Parallel compression

Use ParallelCompression for archives.

12. Use Cases

• Offline mobile sync

• Ledger or audit entries

• ERP snapshots

• Attachments stored in compressed binary blobs

• High-security document workflows

• Cross-language microservice communication

13. Final Recommendations

• Use MessagePack as the base serializer

• Use LZ4 for live systems, Brotli for cold storage

• Use AES + RSA hybrid encryption

• Maintain version headers in every payload

• Create version resolvers to ensure backward compatibility

• Store serialized blobs in VARBINARY(MAX)

• Add batch migration scripts when version changes