Understanding Inference Parameters of Transformer Encoders

Introduction

The encoder in a transformer model processes input sequences (like text) into meaningful representations. While training focuses on updating model weights via backpropagation, inference is the phase in which the model is used to generate outputs or embeddings without updating its weights.

During inference, certain parameters can be set or adjusted to control performance, memory, and output characteristics.

During inference, an encoder:

Takes tokenized input sequences.
Processes embeddings through each encoder layer (multi-head attention + feed-forward layers).
Outputs either:
- Contextual token embeddings (for downstream tasks)
- Attention weights (optional, for analysis)
- Pooled representations (sentence or sequence-level embeddings)

Unlike training, no gradients are computed, so parameters affecting learning are ignored.

Tokenizer / Input-Level Parameters

These parameters control how input text is converted into tokens and fed to the encoder:

input_ids – Token IDs representing the input sequence.
attention_mask – Indicates which tokens should be attended to (1 = real token, 0 = padding).
token_type_ids / segment_ids – Distinguishes segments in tasks like question-answering.
position_ids – Optional; manually sets token positions.
padding_token_id – ID used to pad sequences to the same length.
max_position_embeddings – Maximum allowed sequence length; inputs longer than this are truncated.

Purpose: Ensures sequences are represented correctly for attention computation and downstream outputs.

Attention-Level Parameters

These parameters govern the self-attention mechanism inside each encoder layer:

num_attention_heads – Number of attention heads in multi-head attention.
attention_mask – Masks out padding tokens to prevent them from affecting attention.
head_mask – Optionally disables certain attention heads during inference for analysis or efficiency.
output_attentions – If True, returns attention weights for all heads and layers.

Purpose: Controls how the encoder focuses on different tokens and optionally exposes attention weights for interpretability or debugging.

Hidden-State / Layer-Level Parameters

These parameters control what is returned from each encoder layer:

output_hidden_states – If True, returns hidden states from all layers instead of just the final layer.
return_dict – If True, returns a structured dictionary including last_hidden_state, hidden_states, and attentions.
use_cache / past_key_values – For encoder-decoder models; caches previous keys/values to speed up sequential inference.

Purpose: Provides flexibility in extracting embeddings for downstream tasks, e.g., sentence embeddings, intermediate representations, or analysis of layer-wise features.

Output / Post-Processing Parameters

These parameters control how the encoder's output is structured or used:

last_hidden_state – Token-level contextual embeddings from the final layer.
pooler_output – Pooled representation of the sequence (commonly for classification).
return_dict_in_generate – For encoder-decoder generation tasks; returns outputs as a structured dict.

Purpose: Allows the user to select what output is needed, reducing memory use if only some components are required.

Example: Hugging Face Encoder

from transformers import BertTokenizer, BertModel
import torch

# ------------------------------------
# Tokenizer / Input-Level Parameters
# ------------------------------------

# Load tokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

# Encode input text
inputs = tokenizer(
    "Hello world! This is an example of encoder inference parameters.",
    return_tensors="pt",           # Return PyTorch tensors
    padding=True,                 # Pad to max length in batch
    truncation=True,              # Truncate sequences > max length
    max_length=20,                # Maximum token length
    add_special_tokens=True       # Add [CLS] and [SEP] tokens
)

# ----------------------------
# Attention-Level Parameters
# ----------------------------
# Inference attention options
attention_mask = inputs["attention_mask"]
head_mask = [None] * model.config.num_hidden_layers  # No heads masked

# -----------------------------------------
# Hidden-State / Layer-Level Parameters
# -----------------------------------------
# Choose which hidden states to return
output_hidden_states = True    # Return hidden states of all layers
use_cache = False              # Not using cache in pure encoder inference

# ----------------------------
# Output / Post-Processing Parameters
# ----------------------------
output_attentions = True       # Return attention scores
return_dict = True             # Return as dictionary for easier access

# ----------------------------
# Run Encoder Inference
# ----------------------------
outputs = model(
    input_ids=inputs["input_ids"],
    attention_mask=attention_mask,
    token_type_ids=inputs["token_type_ids"],
    position_ids=inputs["position_ids"],
    head_mask=head_mask,
    output_attentions=output_attentions,
    output_hidden_states=output_hidden_states,
    use_cache=use_cache,
    return_dict=return_dict
)

# ----------------------------
# Inspect Outputs
# ----------------------------
print("=== Output Shapes ===")
print(f"Last hidden state: {outputs.last_hidden_state.shape}")  # (batch, seq_len, hidden_size)
print(f"Number of hidden states: {len(outputs.hidden_states)}") # hidden states from all layers
print(f"Number of attention matrices: {len(outputs.attentions)}") # attention weights from all layers

Summary

Encoder inference parameters can be grouped as:

Category	Key Parameters	Purpose
Tokenizer / Input-Level	input_ids, attention_mask, token_type_ids, position_ids, padding_token_id	Controls input representation and sequence handling
Attention-Level	num_attention_heads, head_mask, attention_mask, output_attentions	Controls focus of self-attention and exposes attention scores
Hidden-State / Layer-Level	output_hidden_states, return_dict, use_cache	Determines what is returned from layers for analysis or embeddings
Output-Level	last_hidden_state, pooler_output, return_dict_in_generate	Controls the type and structure of outputs