Debugging a production problem is very different from debugging an application on your laptop.
Locally, you can stop the program, add a breakpoint, print variables, restart the process, and try again.
In production, restarting the process may make the problem disappear. Adding debug code may require a deployment. And if the problem happens only under real traffic, reproducing it locally can be almost impossible.
Python 3.15 makes remote debugging more practical by adding support for attaching a debugger to a running Python process.
That opens an interesting possibility:
Production process
↓
Attach debugger
↓
Inspect running state
↓
Understand problem
↓
Detach
But there is an important word in that workflow:
Safely.
Attaching a debugger to a live process gives you powerful access to the application. Depending on how the debugging interface is configured, it can expose application state and allow code execution inside the process.
So remote debugging should not be treated like another logging feature.
It should be treated as a privileged production operation.
This article explains Python 3.15's remote debugging support, how to use it, what you can inspect, and the security precautions you should take before attaching to a production process.
What Is Remote Debugging?
Remote debugging means connecting a debugger to a Python process that is already running.
Normally, the debugging flow looks like:
Start application
↓
Debugger attached
↓
Breakpoint
↓
Application runs
Remote debugging changes the order:
Application already running
↓
Identify process
↓
Enable / attach debugger
↓
Inspect process
↓
Detach
This is useful when restarting or reproducing the application is difficult.
For example, suppose a production worker occasionally gets stuck:
Worker #4
CPU: normal
Memory: normal
Status: processing
Duration: 45 minutes
You may not know what it is doing.
A debugger attached to that process can provide information that logs never captured.
Why This Is Useful in Production
Consider an application with a problem that only occurs once every few hours.
You could add logging:
logger.debug("Processing order %s", order_id)
Then deploy a new version and wait.
But the problem may disappear.
Remote debugging provides another option.
Instead of changing the code, you can inspect the state of the existing process.
For example:
Which function is running?
What are the current stack frames?
What values are in local variables?
Is the process waiting?
Which thread is active?
This can be extremely useful for difficult production incidents.
Python 3.15 Debugging Improvements
Python 3.15 introduces a new remote debugging interface based around the sys.remote_debug module.
The functionality is designed to allow a debugger to attach to a running CPython process.
It builds on the low-level debugger support already present in modern CPython and provides a programmatic interface for initiating remote debugging.
The important point is that this is a CPython feature.
Do not assume that every Python implementation provides exactly the same interface.
Check Your Python Version
Before trying the feature, check:
python --version
You should also check:
import sys
print(sys.version)
For production debugging, record the complete interpreter version.
A debugging workflow that works on one Python build may not behave identically on another.
Finding the Process
The first step is identifying the process you want to inspect.
On Linux:
ps aux | grep python
You might see:
app 18452 82.3 ... python server.py
The process ID is:
18452
Before attaching a debugger, verify that you have selected the correct process.
In a production environment, multiple Python workers may be running:
worker-1 PID 18452
worker-2 PID 18453
worker-3 PID 18454
worker-4 PID 18455
Attaching to the wrong worker can waste time or, worse, affect a healthy process.
Process Selection Should Be Part of the Incident Workflow
Do not simply run:
debugger --attach 18452
because someone posted a PID in a chat message.
Confirm:
Process ID
Application name
Worker identity
Container
Host
Current CPU
Current memory
Current request / job
For example:
ps -p 18452 -o pid,ppid,cmd,%cpu,%mem
This gives you a quick confirmation.
In a containerized environment, also verify the container and service identity.
The Debugging Entry Point
Python 3.15 exposes remote debugging functionality through:
import sys.remote_debug
The module provides functionality for interacting with the running interpreter's debugging state.
The exact debugger integration depends on the debugger being used.
The key architectural idea is:
Python process
↓
remote_debug
↓
Debugger protocol
↓
Debugger client
This separates the running application from the debugger UI.
Do Not Expose a Debugging Port Publicly
This is the most important security rule.
A production debugging interface should never be exposed directly to the public internet.
Avoid architectures like:
Internet
↓
debug.example.com:5678
↓
Python process
A debugger is far more powerful than an ordinary application endpoint.
Instead, use a controlled administrative path.
For example:
Developer
↓
VPN / private network
↓
Bastion host
↓
SSH tunnel
↓
Production process
The exact architecture depends on your infrastructure, but the principle is the same:
The debugging interface should not be directly reachable by untrusted clients.
SSH Tunneling
An SSH tunnel can be useful when a debugging endpoint must be reached from a developer workstation.
Conceptually:
ssh -L 5678:127.0.0.1:5678 production-host
Now the local machine can connect to:
127.0.0.1:5678
while the connection is forwarded through SSH.
This avoids exposing the debugger directly to the network.
The actual port and debugger protocol depend on the debugging tool you use.
Authentication Is Not Optional
A debugging interface must be protected.
Do not rely on:
"This server is private."
Private networks can still contain compromised hosts, misconfigured routing, or unauthorized users.
Use multiple controls:
Authentication
+
Network restriction
+
Authorization
+
Audit logging
If your debugging setup supports authentication or an authenticated transport, use it.
If it does not, isolate the interface so that only trusted operators can reach it.
Why a Debugger Is More Sensitive Than Logs
A log might expose:
Request ID
Status
Duration
Error message
A debugger can expose:
Local variables
Function arguments
Environment state
Object contents
Application configuration
Thread state
Database credentials in memory
Session information
And depending on the debugger, it may also allow expressions to be evaluated inside the process.
That makes the debugging interface extremely sensitive.
Treat it closer to shell access than to application monitoring.
Never Assume Secrets Are Not in Memory
Suppose your application contains:
api_key = os.environ["PAYMENT_API_KEY"]
That secret may exist in memory while the process is running.
A debugger that can inspect local variables may potentially expose it.
Other sensitive values can include:
Database passwords
Access tokens
Session cookies
JWT secrets
Encryption keys
User information
Payment data
Internal URLs
Cloud credentials
Before enabling production debugging, understand what information could be exposed.
Start With Stack Inspection
The safest useful debugging operation is often simply looking at stack information.
You may want to answer:
Where is this thread currently executing?
For example:
worker_loop()
↓
process_job()
↓
generate_report()
↓
calculate_statistics()
If the worker has been stuck in calculate_statistics() for 20 minutes, that immediately gives you a direction.
You may not need to inspect sensitive variables at all.
Use the minimum amount of information necessary to answer the incident question.
Inspect Threads
Production Python services frequently use multiple threads.
For example:
Main thread
Worker thread 1
Worker thread 2
Worker thread 3
Background thread
Monitoring thread
A single stack frame is not enough.
A useful debugging session should help answer:
Which threads exist?
What is each thread doing?
Are any threads blocked?
Are multiple threads waiting on the same resource?
This is especially useful when diagnosing:
Deadlocks
Thread contention
Worker stalls
Unexpected CPU usage
Background-job problems
Debugging Async Applications
Modern Python applications frequently use asyncio.
The execution model is different from a traditional threaded application.
You may have:
Event loop
↓
Task A
Task B
Task C
Task D
The important question becomes:
Which task is currently running?
Which task is waiting?
Where is the event loop spending its time?
Remote debugging can help inspect the state of the running process, but async applications require you to understand the difference between:
Thread
Task
Coroutine
Event loop
Do not assume that a thread stack alone explains the application's async behavior.
Use Debugging Alongside Logs and Traces
Remote debugging should not replace observability.
A production system should already have:
Metrics
Logs
Distributed traces
Error tracking
Profiling
Remote debugging is useful when those signals are not enough.
For example:
Metrics
↓
CPU spike detected
↓
Logs
↓
No useful error
↓
Trace
↓
Request appears stuck
↓
Debugger
↓
Inspect current stack
This is a much better workflow than reaching for a debugger every time something looks unusual.
Debugging a CPU Spike
Suppose monitoring reports:
CPU: 98%
Worker: 18452
Duration: 12 minutes
You attach a debugger and find:
process_batch()
↓
transform_records()
↓
normalize_value()
repeatedly appearing in the active stack.
That tells you where to investigate.
You can then combine this with profiling.
For example:
Debugger
↓
Current execution state
Profiler
↓
CPU distribution
The debugger tells you what is happening now.
The profiler tells you what the application has been doing over a period of time.
Those are different but complementary views.
Debugging a Stuck Worker
Consider a background worker:
def worker():
while True:
job = queue.get()
process(job)
Monitoring shows:
Queue depth: 5000
Worker count: 8
Completed jobs: 0
A debugger can help determine whether workers are:
Waiting for queue
Blocked on database
Stuck in a loop
Waiting for a lock
Processing one unusually large job
That can be much faster than adding new logging and redeploying the service.
Debugging Deadlocks
Deadlocks are particularly difficult to reproduce.
Consider:
with lock_a:
with lock_b:
process()
while another thread does:
with lock_b:
with lock_a:
process()
The application may freeze only under a particular timing sequence.
A debugger can show:
Thread 1 → waiting for lock_b
Thread 2 → waiting for lock_a
That is extremely useful evidence.
Once again, the debugger is not fixing the deadlock.
It is helping you see the state that produced it.
Debugging Long-Running Services
Remote debugging is especially useful for services that run for days or weeks.
These applications can develop state that is difficult to recreate:
Cache growth
Connection pool exhaustion
Unexpected queues
Rare race conditions
Long-running tasks
Resource leaks
Restarting the service may destroy the evidence.
Attaching to the running process allows you to inspect the state before deciding what to do.
Do Not Change State Unless Necessary
A debugger can make it tempting to execute expressions or modify variables.
For example:
variable = new_value
That can be dangerous in production.
You may accidentally:
Corrupt application state
Skip validation
Modify a user's data
Break a transaction
Trigger external side effects
Change authentication state
Make the problem disappear without understanding it
Prefer read-only inspection.
If changing state is absolutely necessary, treat it as a production intervention and follow your organization's change-control process.
Production Debugging Can Change Timing
Attaching a debugger can alter application behavior.
This is especially important for concurrency bugs.
Suppose the problem happens because:
Thread A
↓
Thread B
↓
Race condition
Attaching a debugger may slow one thread enough to change the timing.
The bug may disappear.
This is known as a Heisenbug: observing the problem can change the conditions that produce it.
That is another reason to use profiling and tracing alongside debugging.
Keep the Debugging Session Short
Do not attach a debugger and leave it connected indefinitely.
A better process is:
Identify incident
↓
Attach
↓
Collect required information
↓
Detach
↓
Analyze
The longer a debugger remains attached, the greater the operational risk.
It can also affect application timing and resource usage.
Containerized Applications Need Extra Care
Python services commonly run in containers.
The process might look like:
Kubernetes
↓
Pod
↓
Container
↓
Python process
Your debugger needs access to the correct process namespace.
You also need to consider:
Container permissions
PID namespaces
Security policies
Service accounts
Network policies
Sidecars
Ephemeral debugging containers
Do not grant broad privileges to the application container simply because debugging is inconvenient.
Use the smallest administrative access needed.
Kubernetes Debugging
In Kubernetes, a safer operational model is usually:
Developer
↓
Approved cluster access
↓
Target pod
↓
Controlled debugging environment
Avoid adding a permanent public debugging endpoint to the application.
If your platform provides ephemeral debugging mechanisms, they can be preferable because the debugging environment exists only for the duration of the investigation.
The exact commands depend on your Kubernetes configuration and security policies.
Debugging and Secrets Management
A production debugger should never become a shortcut around your secrets-management system.
Do not copy:
Database password
Cloud access token
Private key
into a local file just because you saw it during debugging.
If a secret is exposed during an incident:
Treat it as potentially compromised.
Follow your incident-response procedure.
Rotate it when appropriate.
Review access logs.
Remove unnecessary copies.
Debugging creates another path through which sensitive information can be observed.
That path needs to be controlled.
Audit Production Debugging
A good production debugging process records:
Who attached?
Which process?
Which host?
When?
Why?
How long?
What actions were performed?
This is particularly important for regulated environments.
The goal is not to make debugging difficult.
The goal is to make powerful access accountable.
Use a Debugging Runbook
Before a production incident happens, create a short runbook.
For example:
1. Confirm incident.
2. Identify affected process.
3. Confirm operator authorization.
4. Establish secure connection.
5. Attach debugger.
6. Inspect stack/thread state.
7. Collect required evidence.
8. Avoid state changes.
9. Detach debugger.
10. Document findings.
This is much safer than inventing the procedure during a major outage.
Test Remote Debugging Before You Need It
Do not wait for a production incident to discover that the debugger does not work.
Create a staging environment that resembles production:
Same Python version
Same container
Same dependencies
Same deployment model
Same security controls
Then test:
Attach
Inspect
Detach
Also test the failure cases:
Unauthorized user
Wrong process
Network unavailable
Debugger unavailable
Process restart
Container restart
A debugging workflow is useful only if it works when you actually need it.
Common Mistakes
Exposing the Debugger to the Internet
Never treat a debugging port like an ordinary application port.
Using Production Debugging Without Authorization
Attaching to a process is a privileged operation.
Inspecting More Data Than Necessary
Start with stack and thread information.
Only inspect variables when necessary.
Modifying Live State
A production debugger is not a shell for experimenting with application state.
Ignoring Secrets
Assume sensitive information may exist in memory.
Leaving the Debugger Attached
Keep the session short.
Debugging Without Observability
Logs, metrics, traces, and profiling should remain the primary diagnostic tools.
Testing Only Locally
Production networking, containers, permissions, and security policies can be completely different.
A Practical Production Workflow
A mature workflow might look like this:
Alert
↓
Metrics
↓
Logs
↓
Trace
↓
Profile
↓
Need live state?
↓
Secure debugger
↓
Inspect
↓
Detach
↓
Fix
↓
Verify
Each step narrows the problem.
The debugger is used only when the previous signals cannot answer the question.
This keeps the production debugging surface small.
When Remote Debugging Makes Sense
Remote debugging is most useful when:
The problem occurs only in production.
Restarting would destroy useful state.
The issue is intermittent.
A worker is stuck.
A process has unexpected state.
You need to inspect thread stacks.
Logs and traces are not enough.
The process is difficult to reproduce locally.
It is less useful when:
The problem is easily reproducible locally.
A profiler already answers the question.
The issue is clearly visible in logs.
You only need application metrics.
In those cases, use the simpler tool.
Final Thoughts
Remote debugging is one of those capabilities that can save hours during a difficult production incident.
Instead of restarting a process and hoping the problem happens again, you can inspect the process while the problem is actually occurring.
Python 3.15's remote debugging capabilities make this workflow more accessible to CPython developers.
But the power of a debugger is also the reason it needs strong controls.
A production debugger should be treated as privileged access. Keep it behind authenticated administrative paths, avoid public exposure, minimize the amount of data you inspect, never assume secrets are absent from memory, and avoid modifying application state unless there is a controlled reason to do so.
The best production debugging workflow is therefore not:
Attach debugger
It is:
Observe
↓
Narrow the problem
↓
Attach only when needed
↓
Inspect safely
↓
Detach
↓
Fix the root cause
That approach gives you the benefits of live debugging without turning the debugging interface into another production security problem.
Summary
Python 3.15 adds better support for inspecting a running CPython process through remote debugging. This can be useful when an application is stuck, a worker is behaving unexpectedly, or a production-only problem cannot be reproduced locally.
The most useful information to start with is usually the current call stack, thread state, and execution path. You can then combine that information with logs, metrics, traces, and profiling to understand the larger problem.
The security side is just as important. A debugger can potentially expose variables, configuration, credentials, and other sensitive information, and some debugging workflows can execute code inside the running process. Never expose a production debugger directly to the internet. Use controlled access such as private networking, VPNs, or secure administrative tunnels, and keep sessions short and audited.
Remote debugging is best treated as an emergency diagnostic tool, not a permanent production feature. Use it carefully, collect only the information you need, and fix the underlying problem after the incident is understood.
Join the conversation! Your thoughts help the community grow.