Implementing An RNN In C# For Sequence Prediction

Recurrent neural networks (RNNs) are a type of neural network that is particularly good at processing sequential data, such as time-series data or natural language. In a typical RNN, information is passed from one time step to the next through a hidden state vector, allowing the network to maintain a kind of "memory" of previous inputs.

See the example implementation of an RNN with a single hidden layer,

using System;
using System.Linq;
using MathNet.Numerics.LinearAlgebra;

class RNN {
    private int inputSize;
    private int hiddenSize;
    private int outputSize;
    private Matrix<double> Wxh;
    private Matrix<double> Whh;
    private Matrix<double> Why;
    private Vector<double> bh;
    private Vector<double> by;
    private Vector<double> hprev;

    public RNN(int inputSize, int hiddenSize, int outputSize) {
        this.inputSize = inputSize;
        this.hiddenSize = hiddenSize;
        this.outputSize = outputSize;

        // Initialize weight matrices and bias vectors
        Wxh = Matrix<double>.Build.Random(hiddenSize, inputSize);
        Whh = Matrix<double>.Build.Random(hiddenSize, hiddenSize);
        Why = Matrix<double>.Build.Random(outputSize, hiddenSize);
        bh = Vector<double>.Build.Random(hiddenSize);
        by = Vector<double>.Build.Random(outputSize);
        hprev = Vector<double>.Build.Dense(hiddenSize);
    }

    public Vector<double> Forward(Vector<double> x) {
        // Update hidden state
        var h = Sigmoid(Wxh * x + Whh * hprev + bh);

        // Compute output
        var y = Softmax(Why * h + by);

        // Store hidden state for next time step
        hprev = h;

        return y;
    }

    private Vector<double> Sigmoid(Vector<double> x) {
        return x.Map(v => 1 / (1 + Math.Exp(-v)));
    }

    private Vector<double> Softmax(Vector<double> x) {
        var expX = x.Map(v => Math.Exp(v));
        var sumExpX = expX.Sum();
        return expX / sumExpX;
    }
}

To test the RNN on a toy dataset, let's generate a simple sequence of input vectors and their corresponding output vectors. For example,

var inputs = new[] {
    Vector<double>.Build.Dense(new[] { 0.1, 0.2, 0.3 }),
    Vector<double>.Build.Dense(new[] { 0.2, 0.3, 0.4 }),
    Vector<double>.Build.Dense(new[] { 0.3, 0.4, 0.5 }),
    Vector<double>.Build.Dense(new[] { 0.4, 0.5, 0.6 }),
};

var targets = new[] {
    Vector<double>.Build.Dense(new[] { 0.2, 0.3, 0.4 }),
    Vector<double>.Build.Dense(new[] { 0.3, 0.4, 0.5 }),
    Vector<double>.Build.Dense(new[] { 0.4, 0.5, 0.6 }),
    Vector<double>.Build.Dense(new[] { 0.5, 0.6, 0.7 }),
};

Each input vector is a sequence of three numbers, and the corresponding output vector is the next sequence of three numbers. Given the current input vector, the RNN will be trained to predict the next output vector.

To train the RNN, we can use stochastic gradient descent to update the weights and biases based on the difference between predicted and actual output. Here's an example training loop,

// Initialize RNN and hyperparameters
var rnn = new RNN(inputSize: 3, hiddenSize: 5, outputSize: 3);
var learningRate = 0.1;
var numEpochs = 1000;

// Train RNN using stochastic gradient descent
for (var epoch = 0; epoch < numEpochs; epoch++) {
    // Reset hidden state for each epoch
    rnn.Reset();

    // Loop over input sequences
    for (var i = 0; i < inputs.Length; i++) {
        // Forward pass
        var x = inputs[i];
        var y = rnn.Forward(x);
        var target = targets[i];

        // Compute loss and gradients
        var loss = -target.DotProduct(Vector<double>.Build.DenseOfEnumerable(y.Map(Math.Log)));
        var dy = y - target;
        var dh = (rnn.Why.Transpose() * dy + Vector<double>.Build.Dense(rnn.hiddenSize)).PointwiseMultiply(rnn.hprev.Map(SigmoidDeriv));
        var dbh = dh;
        var dby = dy;

        // Update weights and biases
        rnn.Why -= learningRate * (dy.ToColumnMatrix() * rnn.hprev.ToRowMatrix());
        rnn.by -= learningRate * dby;
        rnn.Whh -= learningRate * (dh.ToColumnMatrix() * rnn.hprev.ToRowMatrix());
        rnn.bh -= learningRate * dbh;
        rnn.Wxh -= learningRate * (dh.ToColumnMatrix() * x.ToRowMatrix());
    }

    // Print loss for this epoch
    if (epoch % 100 == 0) {
        var totalLoss = 0.0;
        for (var i = 0; i < inputs.Length; i++) {
            var x = inputs[i];
            var y = rnn.Forward(x);
            var target = targets[i];
            var loss = -target.DotProduct(Vector<double>.Build.DenseOfEnumerable(y.Map(Math.Log)));
            totalLoss += loss;
        }
        Console.WriteLine($"Epoch {epoch}, loss: {totalLoss / inputs.Length}");
    }
}