Event Driven Architecture backdrop or workflow orchestration

Introduction

Before modern event-driven platforms such as Apache Kafka became mainstream, many large enterprise systems relied on database-driven workflow architectures to process business requests reliably and at scale.

Although newer technologies have introduced more sophisticated approaches to event processing, this architecture remains relevant today for applications with moderate traffic requirements where reliability, fault tolerance, auditability, and simplicity are more important than ultra-high throughput.

Interestingly, many modern workflow orchestration products still implement variations of these same concepts behind the scenes.

This article explains the architecture from first principles rather than focusing on specific products or tools.

Understanding the Server Concept

A server is fundamentally a software component that:

Receives requests
Handles multiple concurrent requests
Processes those requests
Returns results

For example:

An HTTP server such as IIS listens on a network port.
It receives web requests.
It processes those requests concurrently.
It sends responses back to clients.

However, not every server needs to be a web server.

It is possible to build software systems that behave like servers without directly exposing network endpoints.

An Indirect Server Architecture

Consider the following architecture:

Web Application
      |
      V
 Message Queue (Optional)
      |
      V
   Job Table
      |
      V
 Windows Service
      |
      V
 Business Logic Plugins

This architecture behaves like a server even though it is implemented using several components.

Request Capture Layer

A web application or web service receives incoming requests.

Instead of processing them immediately, it creates a record in a Job Table.

This provides several benefits:

Request persistence
Auditability
Reliability
Decoupling between request submission and execution

Because the web application runs on IIS or another web server, concurrency is already handled efficiently.

The application can accept thousands of requests without directly executing long-running business processes.

Adding a Message Queue

In high-volume environments, a Message Queue (MQ) can be introduced between the web application and the Job Table.

Web Application
      |
      V
 Message Queue
      |
      V
   Job Table

Benefits include:

Reliable message delivery
Reduced request loss during traffic spikes
Decoupled processing
Improved fault tolerance

The web application writes XML or JSON messages to the queue.

A background process consumes those messages and inserts jobs into the Job Table.

This ensures that sudden traffic bursts do not overwhelm the database.

Processing Requests Using a Windows Service

The second responsibility of any server is request execution.

In this architecture, that responsibility is handled by a Windows Service.

The service continuously polls the Job Table.

Typical workflow:

Read pending jobs.
Lock the selected job.
Execute business logic.
Update job status.
Move to the next job.

The Windows Service can implement:

Multi-threading
Thread pools
Parallel execution

This enables concurrent processing similar to traditional servers.

Plugin-Based Job Execution

One of the most powerful aspects of this architecture is its plugin model.

Instead of hard-coding all business logic into the Windows Service, individual job implementations can be loaded dynamically.

Configuration Table

Store the following information:

Job Type	Assembly Name	Class Name
Payment	Payment.dll	PaymentProcessor
Shipping	Shipping.dll	ShippingProcessor
Invoice	Invoice.dll	InvoiceGenerator

Common Interface

Every job implementation follows the same contract.

public interface IJob
{
    void Execute();
}

Dynamic Loading Using Reflection

When the Windows Service processes a job:

Read the Job Type.
Lookup Assembly Name and Class Name.
Load the assembly.
Create the object dynamically.
Execute the interface method.

Example:

Type type = assembly.GetType(className);

IJob job = (IJob)Activator.CreateInstance(type);

job.Execute();

This allows new functionality to be added without modifying the workflow engine.

Supporting Enterprise Workflows

Simple jobs are easy.

The real challenge comes when business processes span multiple departments.

Consider an e-commerce order lifecycle:

Order Creation
      |
Order Packing
      |
Order Shipping
      |
Order Delivery
      |
Order Closure

Some steps complete immediately.

Others may take hours or days.

For example:

Order Creation → synchronous
Packing → synchronous
Shipping → asynchronous
Delivery → asynchronous

The workflow engine must support waiting between stages.

The Challenge of Asynchronous Departments

Suppose the Shipping department requires several days to complete processing.

The main workflow cannot remain active for days.

Holding memory and threads for long periods is not practical.

Instead, the workflow state must be persisted.

Workflow State Persistence

After completing the last synchronous step:

Serialize the workflow object.
Store it in a State Database.
Mark the job as Pending.

Example status values:

NEW
PEND
READY
DONE

The workflow engine stops processing the request while preserving its state.

Communicating with Other Departments

When an asynchronous department must perform work:

Send a message to that department.
Store workflow state.
Mark the main job as Pending.

Example:

Main Workflow
      |
      V
 Shipping Queue
      |
      V
 Shipping Department

The shipping department processes the request independently.

Resuming the Workflow

Once the external department finishes:

Store results in an Async Data Table.
Update the main Job Status to READY.
The polling service detects the change.
Reload serialized workflow state.
Continue execution from the next step.

The workflow effectively pauses and resumes later.

Passing Data Back to the Main Workflow

Asynchronous departments often need to return data.

Examples:

Shipping status
Delivery confirmation
Payment amount
Approval status
Rejection reason

A dedicated table can store this information.

Example structure:

Job ID	Step Number	Status	XML Data

When the workflow resumes:

Read the serialized state.
Read asynchronous response data.
Populate workflow fields.
Continue execution.

Insurance Claim Processing Example

Consider an insurance claim system.

Department 1: Claim Submission

Customer submits:

Claim application
Supporting documents

Department 2: Verification

A verification team manually reviews:

Documents
Policy details
Eligibility

This process may take several days.

Department 3: Payment

After approval:

Payment amount is calculated.
Payment is issued.

Workflow Execution

Submit Claim
      |
Validation
      |
Serialize State
      |
Pending
      |
Verification Department
      |
Approval
      |
Ready
      |
Deserialize State
      |
Payment Processing
      |
Completed

The workflow engine never blocks while waiting for human approval.

Instead, it stores state and resumes later.

Reliability Characteristics

This architecture provides several important enterprise qualities:

Fault Tolerance

Jobs survive service restarts because they are stored in the database.

Recoverability

Failed jobs can be retried without losing information.

Auditability

Every request and state transition is stored permanently.

Extensibility

New business processes can be added using plugins.

Department Independence

Each department operates independently while still participating in larger workflows.

Comparison with Modern Event-Driven Architectures

Modern systems often use:

Kafka
Event Hubs
RabbitMQ
Cloud-native workflow engines

However, the underlying concepts remain similar:

Traditional Workflow Engine	Modern Event Architecture
Job Table	Event Log
Polling Service	Event Consumer
Serialized State	State Store
MQ	Event Broker
Plugin Classes	Event Handlers

The implementation changes, but the core workflow concepts remain the same.

Key Takeaways

Database-driven workflow servers were widely used before Kafka and modern event-driven systems.
The architecture separates request capture from request execution.
A Windows Service can act as a workflow engine by polling jobs and executing business logic.
Reflection enables a plugin-based architecture for extensibility.
Long-running workflows require state persistence rather than keeping requests in memory.
Asynchronous departments can participate in workflows using queues, serialized state, and status transitions.
Many modern orchestration platforms still follow these same architectural principles.

Summary

A database-driven workflow server architecture is a proven enterprise pattern for building reliable, fault-tolerant, and extensible systems. By combining a Job Table, polling service, plugin-based execution model, serialized workflow state, and asynchronous department integration, organizations can support complex business processes that span multiple teams and long time periods. Although modern event-driven platforms have evolved the implementation details, the foundational concepts remain highly relevant in enterprise software design today.