If you have been writing Python for a while, you are undoubtedly familiar with the __init__ method. It is the go-to place for setting up an object's initial state. However, lurking just behind the scenes is its lesser-known sibling: __new__. Most Python developers go their entire careers without ever needing to touch __new__. But when you do need it, it unlocks powerful design patterns that __init__ simply cannot achieve.
In this article, we will break down the exact differences between the two, explore the specific scenario where overriding __new__ is strictly necessary, and walk through an end-to-end real-world implementation of an Object Pool to manage heavy resources.
The Core Difference: Allocation vs. Initialization
To understand the difference, you must understand that creating an object in Python is actually a two-step process.
Step 1: __new__ (The Allocator)
What it does: It allocates memory for the object and creates the raw, empty instance.
First Argument: cls (the class itself).
Return Value: It must return the newly created instance (usually by calling super().__new__(cls)).
Nature: It is implicitly a static method.
Step 2: __init__ (The Initializer)
What it does: It takes the raw instance created by __new__ and populates it with data (assigns attributes, opens files, etc.).
First Argument: self (the instance created by __new__).
Return Value: It must return None. (If you return anything else, Python throws a TypeError.)
Nature: It is an instance method.
The Execution Flow
When you call:
obj = MyClass()
Python does this under the hood:
# 1. Allocate memory
obj = MyClass.__new__(MyClass)
# 2. Initialize the object (ONLY if __new__ returned an instance of MyClass)
if isinstance(obj, MyClass):
MyClass.__init__(obj)
The Analogy
Think of building a house.
You cannot furnish a house (__init__) that hasn't been built yet (__new__).
When Is Overriding __new__ Strictly Necessary?
There is one specific scenario in Python where you cannot use __init__ and must use __new__: Subclassing Immutable Built-in Types.
Immutable types in Python include:
int
str
float
tuple
frozenset
Because their values cannot be changed after creation, their value is set at the exact moment memory is allocated (during __new__). By the time __init__ is called, the object is already "frozen," and attempting to change its value will fail.
The Wrong Way (Using __init__)
class NonZeroInt(int):
def __init__(self, value):
# This will FAIL to change the actual integer value!
# The integer was already created as '0' or whatever was passed.
if value == 0:
value = 1
num = NonZeroInt(0)
print(num) # Output: 0 (The __init__ modification was ignored)
The Right Way (Using __new__)
class NonZeroInt(int):
def __new__(cls, value):
# We intercept the creation and alter the value BEFORE it is frozen
if value == 0:
value = 1
return super().__new__(cls, value)
num = NonZeroInt(0)
print(num) # Output: 1 (Success!)
Real-World Use Case: Object Pooling & Caching
While subclassing immutables is where __new__ is required, overriding it for Object Pooling (or the Flyweight Pattern) is where it becomes an enterprise superpower.
The Problem
Imagine you are building a backend service that interacts with a heavy resource. This could be:
A Machine Learning Model (for example, loading a 2GB Transformer model into RAM).
A Database Connection pool.
A complex Image Processing pipeline.
If multiple parts of your application request this resource, you do not want to load it into memory multiple times. You want to return the exact same instance every time.
Normally, you would manage this with a separate Cache/Factory dictionary. But by overriding __new__, we can bake the caching logic directly into the class itself, making the API incredibly clean for the end user.
The Solution
By intercepting __new__, we can check a cache. If the resource already exists, we return the cached instance. If it doesn't, we allocate a new one, cache it, and return it.
End-to-End Code Implementation
Let's build a HeavyMLModel class that simulates loading a massive AI model. We will use __new__ to ensure that no matter how many times we "load" the model, it only initializes once per model name.
import time
class HeavyMLModel:
"""
Simulates a heavy Machine Learning model that takes time and memory to load.
We use __new__ to implement an internal cache (Object Pool).
"""
# Class-level dictionary to act as our cache/pool
_model_cache = {}
def __new__(cls, model_name: str, *args, **kwargs):
"""
Step 1: Intercept object creation.
Check if the model is already loaded in memory.
"""
print(f"[__new__] Checking cache for model: '{model_name}'...")
if model_name in cls._model_cache:
print(f"[__new__] Cache HIT! Returning existing instance of '{model_name}'.")
return cls._model_cache[model_name]
# If not in cache, allocate memory for a brand new instance
print(f"[__new__] Cache MISS. Allocating memory for new '{model_name}' instance.")
instance = super().__new__(cls)
# Store the newly allocated instance in the cache
cls._model_cache[model_name] = instance
return instance
def __init__(self, model_name: str, version: str = "1.0"):
"""
Step 2: Initialize the object.
Note: Because __new__ might return a CACHED instance, __init__
will be called AGAIN on an already initialized object.
We must guard against re-initializing heavy resources.
"""
# Guard clause to prevent re-initializing a cached object
if hasattr(self, "_is_initialized") and self._is_initialized:
print(f"[__init__] Skipping initialization for '{model_name}' (Already initialized).")
return
print(f"[__init__] Loading heavy weights for '{model_name}' v{version} into RAM...")
time.sleep(2) # Simulating a 2-second heavy I/O or CPU operation
# Set up the actual attributes
self.model_name = model_name
self.version = version
self.weights_loaded = True
self._is_initialized = True
print(f"[__init__] Model '{model_name}' successfully loaded!\n")
def predict(self, data):
return f"Predicting on {data} using {self.model_name} v{self.version}"
# ==========================================
# Real-Time Execution Scenario
# ==========================================
if __name__ == "__main__":
print("--- Request 1: Loading 'bert-base' for the first time ---")
model_a = HeavyMLModel("bert-base", version="2.1")
print("\n--- Request 2: Another module requests 'bert-base' ---")
model_b = HeavyMLModel("bert-base", version="2.1")
print("\n--- Request 3: Loading a DIFFERENT model 'resnet-50' ---")
model_c = HeavyMLModel("resnet-50")
# Verification
print("--- Verification ---")
print(f"Are model_a and model_b the exact same object in memory? {model_a is model_b}")
print(f"Is model_a the same as model_c? {model_a is model_c}")
print("\n--- Testing Predictions ---")
print(model_a.predict("Hello World"))
print(model_c.predict("Image.jpg"))
Output of the Implementation
--- Request 1: Loading 'bert-base' for the first time ---
[__new__] Checking cache for model: 'bert-base'...
[__new__] Cache MISS. Allocating memory for new 'bert-base' instance.
[__init__] Loading heavy weights for 'bert-base' v2.1 into RAM...
[__init__] Model 'bert-base' successfully loaded!
--- Request 2: Another module requests 'bert-base' ---
[__new__] Checking cache for model: 'bert-base'...
[__new__] Cache HIT! Returning existing instance of 'bert-base'.
[__init__] Skipping initialization for 'bert-base' (Already initialized).
--- Request 3: Loading a DIFFERENT model 'resnet-50' ---
[__new__] Checking cache for model: 'resnet-50'...
[__new__] Cache MISS. Allocating memory for new 'resnet-50' instance.
[__init__] Loading heavy weights for 'resnet-50' v1.0 into RAM...
[__init__] Model 'resnet-50' successfully loaded!
--- Verification ---
Are model_a and model_b the exact same object in memory? True
Is model_a the same as model_c? False
--- Testing Predictions ---
Predicting on Hello World using bert-base v2.1
Predicting on Image.jpg using resnet-50 v1.0
![__new__ and __init__]()
Why This Implementation Is Powerful
Zero Boilerplate for the User: The developer using HeavyMLModel doesn't need to know about a ModelManager or a get_model() factory function. They simply call HeavyMLModel("name") and trust that it won't duplicate memory.
Memory Efficiency: Notice how model_a is model_b evaluates to True. We saved gigabytes of RAM by reusing the exact same pointer in memory.
The __init__ Guard: Notice the hasattr(self, "_is_initialized") check in __init__. This is a crucial detail. Because __new__ returned an existing instance of the class, Python automatically calls __init__ on it again. Without this guard, we would waste two seconds "re-loading" the model every time it was requested.
Summary Cheat Sheet
| Feature | __new__ | __init__ |
|---|
| Purpose | Allocates memory and creates the instance. | Initializes the instance (sets attributes). |
| Execution Order | Called first. | Called second (only if __new__ returns an instance of the class). |
| First Argument | cls (The Class) | self (The Instance) |
| Return Value | Must return the instance. | Must return None. |
| When to Override? | Subclassing immutables (int, str), Object Pooling, Singletons, returning a different class type. | Most standard OOP use cases for setting up object state. |
Conclusion
While __init__ is the workhorse of Python object-oriented programming, __new__ is the scalpel. You won't need it every day, but when you are dealing with immutable data structures or when you need to implement memory-saving patterns like Object Pooling and Singletons, __new__ gives you absolute control over the very birth of your objects.