Understanding __new__ vs __init__ in Python: Allocation, Initialization, and Object Pooling

If you have been writing Python for a while, you are undoubtedly familiar with the __init__ method. It is the go-to place for setting up an object's initial state. However, lurking just behind the scenes is its lesser-known sibling: __new__. Most Python developers go their entire careers without ever needing to touch __new__. But when you do need it, it unlocks powerful design patterns that __init__ simply cannot achieve.

In this article, we will break down the exact differences between the two, explore the specific scenario where overriding __new__ is strictly necessary, and walk through an end-to-end real-world implementation of an Object Pool to manage heavy resources.

The Core Difference: Allocation vs. Initialization

To understand the difference, you must understand that creating an object in Python is actually a two-step process.

Step 1: `new` (The Allocator)

What it does: It allocates memory for the object and creates the raw, empty instance.
First Argument: cls (the class itself).
Return Value: It must return the newly created instance (usually by calling super().__new__(cls)).
Nature: It is implicitly a static method.

Step 2: `init` (The Initializer)

What it does: It takes the raw instance created by __new__ and populates it with data (assigns attributes, opens files, etc.).
First Argument: self (the instance created by __new__).
Return Value: It must return None. (If you return anything else, Python throws a TypeError.)
Nature: It is an instance method.

The Execution Flow

When you call:

obj = MyClass()

Python does this under the hood:

# 1. Allocate memory
obj = MyClass.__new__(MyClass)

# 2. Initialize the object (ONLY if __new__ returned an instance of MyClass)
if isinstance(obj, MyClass):
    MyClass.__init__(obj)

The Analogy

Think of building a house.

__new__ is the construction crew pouring the foundation and building the physical structure.
__init__ is the interior designer coming in to paint the walls and place the furniture.

You cannot furnish a house (__init__) that hasn't been built yet (__new__).

When Is Overriding `new` Strictly Necessary?

There is one specific scenario in Python where you cannot use __init__ and must use __new__: Subclassing Immutable Built-in Types.

Immutable types in Python include:

int
str
float
tuple
frozenset

Because their values cannot be changed after creation, their value is set at the exact moment memory is allocated (during __new__). By the time __init__ is called, the object is already "frozen," and attempting to change its value will fail.

The Wrong Way (Using `init`)

class NonZeroInt(int):
    def __init__(self, value):
        # This will FAIL to change the actual integer value!
        # The integer was already created as '0' or whatever was passed.
        if value == 0:
            value = 1

num = NonZeroInt(0)
print(num)  # Output: 0 (The __init__ modification was ignored)

The Right Way (Using `new`)

class NonZeroInt(int):
    def __new__(cls, value):
        # We intercept the creation and alter the value BEFORE it is frozen
        if value == 0:
            value = 1
        return super().__new__(cls, value)

num = NonZeroInt(0)
print(num)  # Output: 1 (Success!)

Real-World Use Case: Object Pooling & Caching

While subclassing immutables is where __new__ is required, overriding it for Object Pooling (or the Flyweight Pattern) is where it becomes an enterprise superpower.

The Problem

Imagine you are building a backend service that interacts with a heavy resource. This could be:

A Machine Learning Model (for example, loading a 2GB Transformer model into RAM).
A Database Connection pool.
A complex Image Processing pipeline.

If multiple parts of your application request this resource, you do not want to load it into memory multiple times. You want to return the exact same instance every time.

Normally, you would manage this with a separate Cache/Factory dictionary. But by overriding __new__, we can bake the caching logic directly into the class itself, making the API incredibly clean for the end user.

The Solution

By intercepting __new__, we can check a cache. If the resource already exists, we return the cached instance. If it doesn't, we allocate a new one, cache it, and return it.

End-to-End Code Implementation

Let's build a HeavyMLModel class that simulates loading a massive AI model. We will use __new__ to ensure that no matter how many times we "load" the model, it only initializes once per model name.

import time

class HeavyMLModel:
    """
    Simulates a heavy Machine Learning model that takes time and memory to load.
    We use __new__ to implement an internal cache (Object Pool).
    """

    # Class-level dictionary to act as our cache/pool
    _model_cache = {}

    def __new__(cls, model_name: str, *args, **kwargs):
        """
        Step 1: Intercept object creation.
        Check if the model is already loaded in memory.
        """
        print(f"[__new__] Checking cache for model: '{model_name}'...")

        if model_name in cls._model_cache:
            print(f"[__new__] Cache HIT! Returning existing instance of '{model_name}'.")
            return cls._model_cache[model_name]

        # If not in cache, allocate memory for a brand new instance
        print(f"[__new__] Cache MISS. Allocating memory for new '{model_name}' instance.")
        instance = super().__new__(cls)

        # Store the newly allocated instance in the cache
        cls._model_cache[model_name] = instance
        return instance

    def __init__(self, model_name: str, version: str = "1.0"):
        """
        Step 2: Initialize the object.
        Note: Because __new__ might return a CACHED instance, __init__
        will be called AGAIN on an already initialized object.
        We must guard against re-initializing heavy resources.
        """

        # Guard clause to prevent re-initializing a cached object
        if hasattr(self, "_is_initialized") and self._is_initialized:
            print(f"[__init__] Skipping initialization for '{model_name}' (Already initialized).")
            return

        print(f"[__init__] Loading heavy weights for '{model_name}' v{version} into RAM...")
        time.sleep(2)  # Simulating a 2-second heavy I/O or CPU operation

        # Set up the actual attributes
        self.model_name = model_name
        self.version = version
        self.weights_loaded = True
        self._is_initialized = True

        print(f"[__init__] Model '{model_name}' successfully loaded!\n")

    def predict(self, data):
        return f"Predicting on {data} using {self.model_name} v{self.version}"


# ==========================================
# Real-Time Execution Scenario
# ==========================================
if __name__ == "__main__":
    print("--- Request 1: Loading 'bert-base' for the first time ---")
    model_a = HeavyMLModel("bert-base", version="2.1")

    print("\n--- Request 2: Another module requests 'bert-base' ---")
    model_b = HeavyMLModel("bert-base", version="2.1")

    print("\n--- Request 3: Loading a DIFFERENT model 'resnet-50' ---")
    model_c = HeavyMLModel("resnet-50")

    # Verification
    print("--- Verification ---")
    print(f"Are model_a and model_b the exact same object in memory? {model_a is model_b}")
    print(f"Is model_a the same as model_c? {model_a is model_c}")

    print("\n--- Testing Predictions ---")
    print(model_a.predict("Hello World"))
    print(model_c.predict("Image.jpg"))

Output of the Implementation

--- Request 1: Loading 'bert-base' for the first time ---
[__new__] Checking cache for model: 'bert-base'...
[__new__] Cache MISS. Allocating memory for new 'bert-base' instance.
[__init__] Loading heavy weights for 'bert-base' v2.1 into RAM...
[__init__] Model 'bert-base' successfully loaded!

--- Request 2: Another module requests 'bert-base' ---
[__new__] Checking cache for model: 'bert-base'...
[__new__] Cache HIT! Returning existing instance of 'bert-base'.
[__init__] Skipping initialization for 'bert-base' (Already initialized).

--- Request 3: Loading a DIFFERENT model 'resnet-50' ---
[__new__] Checking cache for model: 'resnet-50'...
[__new__] Cache MISS. Allocating memory for new 'resnet-50' instance.
[__init__] Loading heavy weights for 'resnet-50' v1.0 into RAM...
[__init__] Model 'resnet-50' successfully loaded!

--- Verification ---
Are model_a and model_b the exact same object in memory? True
Is model_a the same as model_c? False

--- Testing Predictions ---
Predicting on Hello World using bert-base v2.1
Predicting on Image.jpg using resnet-50 v1.0

Why This Implementation Is Powerful

Zero Boilerplate for the User: The developer using HeavyMLModel doesn't need to know about a ModelManager or a get_model() factory function. They simply call HeavyMLModel("name") and trust that it won't duplicate memory.
Memory Efficiency: Notice how model_a is model_b evaluates to True. We saved gigabytes of RAM by reusing the exact same pointer in memory.
The __init__ Guard: Notice the hasattr(self, "_is_initialized") check in __init__. This is a crucial detail. Because __new__ returned an existing instance of the class, Python automatically calls __init__ on it again. Without this guard, we would waste two seconds "re-loading" the model every time it was requested.

Summary Cheat Sheet

Feature	__new__	__init__
Purpose	Allocates memory and creates the instance.	Initializes the instance (sets attributes).
Execution Order	Called first.	Called second (only if __new__ returns an instance of the class).
First Argument	cls (The Class)	self (The Instance)
Return Value	Must return the instance.	Must return None.
When to Override?	Subclassing immutables (int, str), Object Pooling, Singletons, returning a different class type.	Most standard OOP use cases for setting up object state.

Conclusion

While __init__ is the workhorse of Python object-oriented programming, __new__ is the scalpel. You won't need it every day, but when you are dealing with immutable data structures or when you need to implement memory-saving patterns like Object Pooling and Singletons, __new__ gives you absolute control over the very birth of your objects.