Python asyncio Task Cancellation: How to Avoid Stuck Background Jobs

Python's asyncio makes it possible to run many I/O-bound operations concurrently without creating a separate thread for every task. It is commonly used for HTTP clients, background workers, database operations, and applications that handle multiple connections.

However, asynchronous tasks do not always finish successfully. A request may time out, a user may disconnect, or an application may shut down while a task is waiting for a response. In these situations, the application may need to cancel work that is no longer useful.

Cancellation can become a problem when a task ignores cancellation requests, fails to release resources, or leaves background operations running after its caller has stopped waiting.

Python's asyncio provides cancellation mechanisms, but they work differently from forcibly terminating a thread or process. Understanding these differences helps developers build background jobs that stop safely and clean up resources correctly.

How asyncio Task Cancellation Works

An asyncio.Task wraps a coroutine and schedules it on an event loop. When cancellation is requested, the task receives an asyncio.CancelledError at an appropriate suspension point, usually when it next resumes from an await.

Consider this example:

import asyncio


async def background_job():
    try:
        print("Job started")

        await asyncio.sleep(10)

        print("Job completed")
    except asyncio.CancelledError:
        print("Job was cancelled")
        raise


async def main():
    task = asyncio.create_task(background_job())

    await asyncio.sleep(1)

    task.cancel()

    try:
        await task
    except asyncio.CancelledError:
        print("Cancellation completed")


asyncio.run(main())

The background job starts and waits for ten seconds. After one second, the main coroutine requests cancellation by calling task.cancel().

When the task resumes, cancellation is delivered as an exception. The job performs its cleanup or logging, then re-raises CancelledError.

The main coroutine catches the exception when it awaits the cancelled task.

Calling cancel() requests cancellation. It does not guarantee that the task has already stopped. The task must get an opportunity to process the request, and its coroutine must not indefinitely prevent cancellation from being delivered.

Why CancelledError Should Usually Be Re-Raised

One common mistake is catching every exception in a background job and treating cancellation as an ordinary failure.

For example:

import asyncio


async def process_job():
    try:
        await asyncio.sleep(30)
        print("Processing completed")
    except BaseException:
        print("An error occurred")

Catching BaseException also catches asyncio.CancelledError. Because the code does not re-raise it, the task can suppress cancellation and continue as if it had handled the situation successfully.

This can cause shutdown problems and make callers believe that cancellation completed normally.

A better implementation handles cancellation separately:

import asyncio


async def process_job():
    try:
        await asyncio.sleep(30)
        print("Processing completed")
    except asyncio.CancelledError:
        print("Processing was cancelled")
        raise
    except Exception as exc:
        print(f"Processing failed: {exc}")
        raise

The cancellation handler performs any required cleanup or logging and then re-raises the exception. Ordinary application errors are handled separately.

In most cases, do not catch BaseException unless you have a specific reason to handle process-level exceptions. Catching Exception is more appropriate for most application error handling, while cancellation should be treated deliberately.

Use try/finally to Release Resources

Background tasks often manage resources such as files, network connections, locks, and temporary state. These resources must be released even when the task is cancelled.

The finally block is useful because it runs when the coroutine exits through normal completion or an exception, including cancellation.

import asyncio


async def background_job():
    resource = await acquire_resource()

    try:
        await process_data(resource)
    finally:
        await resource.close()

This example assumes that acquire_resource(), process_data(), and resource.close() are asynchronous operations provided by the application.

The finally block ensures that closing the resource is attempted when processing finishes or cancellation is delivered.

For resources that support asynchronous context managers, async with is often clearer:

async def background_job():
    async with open_async_resource() as resource:
        await process_data(resource)

The context manager is responsible for releasing the resource when the block exits, provided its cleanup implementation is correct.

Cleanup itself can fail or take time. Avoid placing unbounded work in a finally block, because a slow cleanup operation can delay task completion and application shutdown.

Apply Timeouts to Prevent Indefinite Waiting

A background task may wait indefinitely for an external service if the operation has no timeout. Cancellation can help stop unnecessary work, but timeouts are often a better way to define how long an operation is allowed to run.

Python provides asyncio.timeout() for this purpose.

import asyncio


async def fetch_data():
    await asyncio.sleep(10)
    return {"status": "ok"}


async def main():
    try:
        async with asyncio.timeout(3):
            result = await fetch_data()
            print(result)
    except TimeoutError:
        print("The operation timed out")


asyncio.run(main())

The operation is allowed to run for three seconds. If it does not finish in time, the timeout context cancels the current task's work and raises TimeoutError outside the context manager.

The example uses asyncio.timeout(), available in Python 3.11 and later.

A timeout does not guarantee that every underlying operation has stopped. If a coroutine suppresses cancellation or delegates work to a thread, process, or external service, additional handling may be necessary.

For network clients and database drivers, configure their own operation timeouts where supported. This helps ensure that the underlying operation has a defined limit rather than relying only on a surrounding coroutine timeout.

Use wait_for When You Need to Limit One Awaitable

asyncio.wait_for() is another way to apply a timeout to an asynchronous operation.

import asyncio


async def fetch_data():
    await asyncio.sleep(10)
    return "Data received"


async def main():
    try:
        result = await asyncio.wait_for(
            fetch_data(),
            timeout=3
        )
        print(result)
    except TimeoutError:
        print("Request timed out")


asyncio.run(main())

If the timeout expires, wait_for() requests cancellation of the awaited operation and raises TimeoutError after handling the cancellation.

There is an important difference between the two APIs. asyncio.timeout() applies a timeout context to the current task, while asyncio.wait_for() applies a timeout to a particular awaitable.

Choose the API that best matches the operation you want to limit. In both cases, cancellation handling can affect how quickly the operation actually finishes.

Cancel and Await Background Tasks During Shutdown

Creating a task with asyncio.create_task() does not automatically give it a complete lifecycle. The application must decide when to wait for it, how to handle failures, and how to stop it during shutdown.

Consider a long-running worker:

import asyncio


async def worker():
    try:
        while True:
            print("Checking for jobs")
            await asyncio.sleep(2)
    except asyncio.CancelledError:
        print("Worker is stopping")
        raise


async def main():
    task = asyncio.create_task(worker())

    try:
        await asyncio.sleep(7)
    finally:
        task.cancel()

        try:
            await task
        except asyncio.CancelledError:
            pass


asyncio.run(main())

The worker repeatedly performs work until cancellation is requested. When the main coroutine exits, it requests cancellation and awaits the task.

Awaiting the task gives it an opportunity to execute its cleanup code and ensures that its completion is observed.

The example suppresses CancelledError only in the shutdown code, where cancellation is expected. The worker itself still re-raises the exception.

For applications with multiple background tasks, maintain references to those tasks and define an orderly shutdown procedure. Do not assume that creating a task means it will automatically finish safely when the application exits.

Handle Multiple Tasks with TaskGroup

When several tasks form one logical unit of work, Python's asyncio.TaskGroup can simplify lifecycle management.

A task group waits for its child tasks to finish. If one child raises an ordinary exception, the group cancels the remaining tasks and propagates the failure when the group exits.

import asyncio


async def fetch_user():
    await asyncio.sleep(1)
    return "User loaded"


async def fetch_orders():
    await asyncio.sleep(5)
    return "Orders loaded"


async def main():
    async with asyncio.TaskGroup() as group:
        user_task = group.create_task(fetch_user())
        orders_task = group.create_task(fetch_orders())

    print(user_task.result())
    print(orders_task.result())


asyncio.run(main())

In this example, both operations are started within the same task group. The program continues after the group exits, once both tasks have completed successfully.

If one operation fails with an ordinary exception, the task group cancels the other unfinished operation and waits for the group to finish its cleanup.

TaskGroup is available in Python 3.11 and later. It is useful when child tasks should share a clear lifetime instead of running independently in the background.

Cancellation of the parent task also affects tasks managed by the group. This makes task groups a useful choice for structured concurrency, where child tasks should not outlive the operation that created them.

Avoid Stuck Tasks That Suppress Cancellation

A task can become difficult to stop if it catches cancellation and continues running.

import asyncio


async def bad_worker():
    while True:
        try:
            await asyncio.sleep(1)
        except asyncio.CancelledError:
            print("Cancellation ignored")

When cancellation is requested, this worker catches the exception and starts another loop iteration. Because it suppresses cancellation, the task may continue indefinitely.

A safer approach is to re-raise cancellation after cleanup:

import asyncio


async def worker():
    try:
        while True:
            await asyncio.sleep(1)
    except asyncio.CancelledError:
        await cleanup()
        raise

This example assumes that cleanup() is an asynchronous function provided by the application.

If cancellation must be suppressed for a specific reason, make that decision explicit and ensure that the task has another reliable termination mechanism. Otherwise, shutdown code may wait indefinitely for a task that refuses to finish.

Also avoid long-running synchronous loops inside coroutines. A coroutine cannot process cancellation while it is executing uninterrupted synchronous code. Break large operations into manageable chunks or move CPU-intensive work to a process or worker thread where appropriate.

Common Mistakes to Avoid

Calling cancel without awaiting the task

Calling task.cancel() requests cancellation but does not prove that cleanup has finished. Await the task when its completion matters, and handle the expected cancellation exception.

Catching cancellation as an ordinary error

Broad exception handling can accidentally suppress cancellation. Handle asyncio.CancelledError deliberately and normally re-raise it.

Giving background tasks no owner

Tasks created without a clear lifecycle can continue after the request or operation that created them has ended. Keep track of tasks, use task groups where appropriate, and define shutdown behavior.

Using timeouts without understanding the underlying operation

A coroutine timeout cannot always terminate work already running in another thread, process, or external system. Configure timeouts and cancellation behavior at the appropriate layer.

Performing lengthy cleanup

Cleanup should release essential resources without delaying shutdown indefinitely. Set sensible limits for external operations and make cleanup safe to run after partial failure.

Best Practices for Reliable Background Jobs

Give each background task a clear owner and lifetime. Use TaskGroup for related concurrent work and keep references to independently managed tasks.

Use timeouts for network calls, database operations, and other operations that could otherwise wait indefinitely. Make cancellation handlers release resources and re-raise CancelledError unless suppressing it is an intentional part of the design.

Test cancellation explicitly. Verify that tasks stop when requested, locks and connections are released, and application shutdown does not wait forever. Test failures as well as normal cancellation, because cleanup often behaves differently when an operation has only partially completed.

For jobs that must survive application restarts, in-process asyncio tasks may not be enough. Use a durable job queue or external worker system when the work must be retried, tracked, or recovered after a process failure.

Conclusion

Python asyncio cancellation is a request for a coroutine to stop, not an unconditional termination mechanism. A reliable background job must respond to cancellation, release its resources, and allow its caller to observe completion.

Use try/finally or asynchronous context managers for cleanup, apply timeouts to operations that can wait indefinitely, and await tasks during shutdown. For related concurrent operations, asyncio.TaskGroup provides a clearer way to manage child tasks and propagate failures.

The key is to design cancellation into the task's lifecycle from the beginning. That makes background work easier to stop, easier to debug, and less likely to leave resources or unfinished operations behind.