Multiprocessing changes the logging problem because separate Python processes do not share one interpreter, one handler lock, or one logging configuration in the same way threads do. If several worker processes write directly to the same log file, records can become mixed, duplicated, lost, or difficult to rotate safely. The reliable pattern is to let workers create log records and send them to one controlled logging path.

This lesson continues from OSPython.037: QueueHandler and QueueListener. It also builds on OSPython.036: Logging Configuration with dictConfig and OSPython.029: Logging to Files. The new goal is to make logging safe when work is distributed across multiple operating-system processes.

Corey Schafer explains Python multiprocessing and ProcessPoolExecutor, providing the process model used throughout this lesson.

Learning Objectives

• Explain why one FileHandler should not be shared casually across multiple processes.
• Use QueueHandler to move LogRecords from workers toward one logging owner.
• Use multiprocessing.Queue for process-safe transport.
• Configure logging separately inside worker processes when required.
• Understand the difference between fork and spawn behavior.
• Use a listener thread or listener process as the single writer.
• Understand the ProcessPoolExecutor queue requirement.
• Shut down workers and listeners in the correct order.

Why Multiple Processes Change Logging

Python logging is thread-safe inside one process, but the official Python 3.15 Logging Cookbook states that direct logging to one file from multiple processes is not supported as a general pattern. The standard FileHandler classes do not use multiprocessing-aware locks to serialize file access across independent processes.

That distinction matters. Threads share process memory, so the logging module can coordinate access to a handler with thread locks. Separate processes have separate memory spaces. Even if two processes point at the same filename, they may each own a different handler object and a different lock.

The engineering rule is therefore simple: many processes may produce log records, but one controlled path should own the final file output.

The Safe Architecture

A common architecture is worker process → QueueHandler → multiprocessing.Queue → listener → final handlers. Each worker sends records to the queue instead of opening the production log file itself. The listener receives those records and writes them through FileHandler, RotatingFileHandler, StreamHandler, or other final destinations.

This design gives the application one place to control formatting, file rotation, handler levels, and output order. It also avoids having several independent processes competing over the same file.

ArjanCodes covers production Python logging design and why logging configuration should be deliberate rather than scattered through application code.

Worker Processes Need QueueHandler

Inside each worker, configure the logger with a QueueHandler that points to the shared multiprocessing queue. The worker should not normally create its own FileHandler for the same centralized log file.

Worker setup:
import logging
from logging.handlers import QueueHandler

def configure_worker(log_queue):
    root = logging.getLogger()
    root.handlers.clear()
    root.setLevel(logging.INFO)
    root.addHandler(QueueHandler(log_queue))

Clearing inherited or previously configured handlers is useful when you want the worker to have one predictable route. The important part is that the worker emits through QueueHandler while the file-owning handler stays somewhere else.

Use multiprocessing.Queue, Not queue.Queue

The regular queue.Queue class is designed for communication between threads inside one process. For process-to-process logging, use multiprocessing.Queue. Python’s current QueueHandler documentation specifically recommends avoiding SimpleQueue for multiprocessing and using multiprocessing.Queue instead.

Create the queue: log_queue = multiprocessing.Queue(-1). A negative or omitted practical size is commonly used for an effectively unbounded queue, but production systems should still think about memory growth and the rate at which logs are produced.

A Listener Thread Can Own the File

One approach is to keep the final handlers in the main process and run QueueListener there. Worker processes send records through the multiprocessing queue, while QueueListener’s internal thread performs the final output work.

Main-process listener:
file_handler = logging.FileHandler("app.log")
file_handler.setFormatter(logging.Formatter("%(asctime)s %(processName)s %(levelname)s %(message)s"))
listener = QueueListener(log_queue, file_handler)
listener.start()

The %(processName)s field is especially helpful because it shows which worker produced each record even though one listener writes the final file.

A Dedicated Listener Process Is Another Option

The Logging Cookbook also demonstrates a separate listener process. In that design, the listener process owns the final handlers and waits for LogRecords arriving on the multiprocessing queue. Workers never touch the production file.

Listener concept:
record = log_queue.get()
if record is None:
    break
logging.getLogger(record.name).handle(record)

The None value in this pattern is a sentinel: it tells the listener that no more worker records are expected and that shutdown can begin. The sentinel must be sent only after workers have finished producing log records.

A learnpython discussion illustrates a common cross-platform problem: logging that appears to work under fork can behave differently under Windows spawn because worker configuration is not inherited the same way.

Fork and Spawn Are Different

On systems that use fork, a child begins with a copy of much of the parent’s process state. That can make inherited logging configuration appear to work. On spawn, the child starts a fresh interpreter and imports the program again. Windows commonly uses spawn, and modern Python applications should not assume that full logger configuration will simply appear inside every child.

The Python Logging Cookbook explicitly notes that on Windows you cannot rely on fork semantics and that worker logging configuration should run when each worker starts. Cross-platform code should therefore configure the QueueHandler inside the worker initialization path.

Protect the Program Entry Point

Multiprocessing code should use the normal entry-point guard: if __name__ == "__main__":. Without it, a spawned child may import the module and execute process-creation code again, producing recursive process creation or other failures.

On Windows and frozen applications, multiprocessing.freeze_support() may also be appropriate. The broader rule is to keep process startup inside an explicit main path rather than at module import time.

ProcessPoolExecutor Needs a Different Queue Pattern

If workers are created with concurrent.futures.ProcessPoolExecutor, the Python 3.15 Logging Cookbook recommends using multiprocessing.Manager().Queue(-1) rather than a plain multiprocessing.Queue(-1) for the shared logging queue used by the executor example.

Executor queue: manager = multiprocessing.Manager()
log_queue = manager.Queue(-1)

This proxy queue can be passed into ProcessPoolExecutor workers. The logging architecture stays the same: workers use QueueHandler, while one listener path owns the final output handlers.

This community troubleshooting thread shows the practical symptom that motivates centralized multiprocessing logging: worker processes can appear to lose or ignore a logging setup that worked in the parent process.

Do Not Let Every Process Rotate the Same File

File rotation makes multi-process direct writing even riskier. Imagine several processes each holding a RotatingFileHandler pointed at app.log. One process may rename the file while another still has it open. Records can then land in unexpected files, rotation can fail, or retention behavior can become inconsistent.

A single listener avoids this conflict. Only the listener owns RotatingFileHandler or TimedRotatingFileHandler, so rotation policy is applied in one place.

Avoid the multiprocessing Internal-Logger Trap

Python’s QueueHandler documentation contains an important warning: multiprocessing.Queue can emit DEBUG messages through the multiprocessing module’s own internal logger. If those internal messages are routed through a QueueHandler that uses the exact same multiprocessing queue, the setup can recurse or deadlock.

Do not point multiprocessing’s internal diagnostics back into the same queue whose activity generated those diagnostics. Keep that internal logger separate or at a level that avoids the cycle.

Shutdown Order Matters

A common failure is stopping the listener too early. If workers are still running when the listener is stopped, final records may remain queued and never reach the destination.

1. Start the logging listener.
2. Start the worker processes.
3. Wait for workers to finish with join() or by waiting for executor futures.
4. After workers are finished, stop QueueListener or send the listener-process sentinel.
5. Wait for the listener to finish.
6. Close final handlers if your application owns their lifecycle explicitly.

Queue Capacity and Backpressure

Centralizing logs does not make throughput unlimited. If workers generate records faster than the listener can write them, the queue grows. A bounded queue can fill; an unbounded queue can consume increasing memory.

Production design should therefore consider log level, destination speed, queue size, record-dropping policy, and whether DEBUG traffic belongs in the production pipeline at all.

Troubleshooting Order

1. Confirm the worker function is actually running.
2. Confirm each worker executes its logging configuration.
3. Confirm the worker has QueueHandler attached.
4. Confirm the same process-safe queue reaches workers and listener.
5. Confirm the listener starts before workers emit records.
6. Include %(processName)s in the formatter to identify producers.
7. Make sure no worker owns a competing FileHandler for the same file.
8. Wait for workers before stopping the listener.
9. Check for the multiprocessing internal-logger recursion warning.
10. For ProcessPoolExecutor, verify that a Manager queue is being used when following the documented pattern.

Hands-On Exercise

Create a program that starts four worker processes. Give each worker the same multiprocessing queue and configure each worker with QueueHandler. In the main process, create one FileHandler and one QueueListener. Include %(processName)s and %(process)d in the formatter.

Have each worker log ten INFO messages. Wait for all workers to finish, then stop the listener. Confirm that one log file contains records from all four workers and that each record identifies its producing process.

For a second test, replace the manually created processes with ProcessPoolExecutor and use multiprocessing.Manager().Queue(). Compare the setup and verify that the logging output remains centralized.

Knowledge Check

1. Why is direct FileHandler output from many processes unsafe as a general design?
2. What handler should worker processes normally use in the centralized-logging pattern?
3. Which queue type should be used for normal multiprocessing?
4. Why must worker logging configuration be explicit on spawn-based systems?
5. Which formatter field identifies the worker by process name?
6. When should the listener be stopped?
7. What queue pattern does the Python Logging Cookbook recommend with ProcessPoolExecutor?
8. What configuration can create recursion or deadlock with multiprocessing’s internal logger?

Answers

1. Separate processes do not share the normal logging handler lock, so independent handlers can compete over the same file and rotation state.
2. QueueHandler.
3. multiprocessing.Queue.
4. Spawn starts a fresh interpreter rather than relying on inherited fork state, so the worker must establish the handlers it needs.
5. %(processName)s.
6. After all workers have finished producing records.
7. multiprocessing.Manager().Queue() in the documented executor pattern.
8. Sending multiprocessing’s own queue DEBUG messages through a QueueHandler that uses that same multiprocessing queue.

The Engineering Rule

Many processes may create LogRecords. One controlled logging path should own the final file.

Once this architecture is clear, multiprocessing logging becomes much easier to reason about. Workers do work and enqueue records. The listener owns formatting, filtering, rotation, and final output. That separation keeps concurrency concerns away from the log file itself.

BitcoinVersus.Tech Editor’s Note: This lesson follows the Python 3.15 documentation current on October 12, 2026. Multiprocessing start methods and application deployment details can differ by operating system, Python version, and runtime environment, so test the complete logging lifecycle on the platforms you actually deploy.

Support independent technology education: Bitcoin donations help fund BitcoinVersus.Tech research and publishing.

Disclaimer: BitcoinVersus.Tech provides technology education and analysis for informational purposes only.