Watchdog in Python: A Technical Guide to File System Monitoring

Introduction

Modern backend systems often require reactive file processing - whenever a file is created, modified, deleted, or moved, the system should respond automatically. In Python, one of the most reliable libraries for this purpose is watchdog.

This article provides a technical deep dive into:

• What watchdog is

• How it works internally

• Supported events

• Architecture

• Production best practices

• Advanced usage patterns

What is Watchdog?

watchdog is a Python library that monitors file system events and triggers callbacks when changes occur.

It allows you to build:

• File ingestion pipelines

• Automated document processors

• ETL triggers

• Log monitoring systems

• AI workflow triggers

• Event-driven backend systems

Install it:

pip install watchdog

Why Watchdog Instead of Polling?

Without watchdog, you might check for file changes like this:

while True:
    check_folder()
    sleep(5)

This is inefficient because:

• It wastes CPU cycles

• It adds latency

• It doesn't scale well

Watchdog uses OS-native file system notification APIs, making it:

• Event-driven

• Efficient

• Low-latency

• Scalable

How Watchdog Works Internally

Watchdog connects to the operating system’s file notification mechanism:

OS	Native Mechanism
Windows	ReadDirectoryChangesW
Linux	inotify
macOS	FSEvents

Architecture:

Operating System - Native FS Event API - Watchdog Observer - Event Handler - Your Business Logic

When the OS detects a change, watchdog immediately dispatches the event to your handler.

Core Components

Watchdog has three main components:

Observer

Monitors file system events.

from watchdog.observers import Observer

Event Handler

Handles events like create, modify, delete, move.

from watchdog.events import FileSystemEventHandler

Scheduler

Links observer and handler to a specific directory.

Basic Example

import time
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class FolderEventHandler(FileSystemEventHandler):
    def on_created(self, event):
        # Ignore directories
        if not event.is_directory:
            print(f"[EVENT] New file detected: {event.src_path}")
            trigger_message(event.src_path)

def trigger_message(file_path):
    # Your custom logic here
    print(f"[ACTION] Processing file: {file_path}")

if __name__ == "__main__":
    path_to_watch = "./watch_folder"

    event_handler = FolderEventHandler()
    observer = Observer()
    observer.schedule(event_handler, path_to_watch, recursive=False)

    observer.start()
    print(f"Watching folder: {path_to_watch}")

    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()

    observer.join()

Available Event Types

You can override:

Method	Trigger
on_created	File or folder created
on_modified	File or folder modified
on_deleted	File or folder deleted
on_moved	File moved or renamed
on_any_event	Any event

Event object contains:

event.src_path
event.is_directory
event.event_type

For move events:

event.dest_path

Production Considerations

Duplicate Events

When copying a file, you may receive:

• created

• multiple modified

Solution:

• Track last modified timestamp

• Debounce events

File Not Fully Written

Large files may trigger events before writing completes.

Solution:

import time
import os

def wait_until_stable(file_path):
    previous_size = -1
    while True:
        current_size = os.path.getsize(file_path)
        if current_size == previous_size:
            break
        previous_size = current_size
        time.sleep(1)

Ignore Temporary Files

Many systems create temp files like:

• .tmp

• .swp

• ~filename

Filter them before processing.

Monitoring Local vs Remote Systems

Watchdog primarily monitors:

• Local file systems

It may work with:

• Mounted network drives (NFS, SMB)

It does NOT directly monitor:

• AWS S3

• Azure Blob

• Google Cloud Storage

• SharePoint Online

For those, use cloud-native event systems.

Advanced Architecture: Event-Driven File Processing

Watchdog integrates well into event-driven systems.

Example architecture:

File Created - Watchdog Event - Event Bus (Kafka / Redis / RabbitMQ) - Processing Service - Database / AI / Storage

This pattern enables:

• Horizontal scalability

• Loose coupling

• Async processing

• Observability

Async Integration

Watchdog itself is synchronous, but you can integrate it with async frameworks by:

• Publishing events to a queue

• Using background workers

• Dispatching to async tasks

Example concept:

Watchdog - Queue - Async Worker - LLM Processing

When to Use Watchdog

Use it when:

• Monitoring local directories

• Building ingestion pipelines

• Triggering workflows on file changes

• Automating document processing

• Creating event-driven backend systems

Avoid it when:

• Monitoring cloud object storage directly

• You need distributed remote monitoring

• System is very simple and polling is sufficient

Watchdog vs Polling

Feature	Polling	Watchdog
CPU usage	Higher	Low
Latency	Delayed	Immediate
Scalability	Poor	Good
Architecture	Loop-based	Event-driven

Summary

Watchdog is a:

• Lightweight

• Event-driven

• OS-level file monitoring library

• Ideal for backend automation and ingestion systems

It enables reactive systems where file system changes automatically trigger workflows.