Question Details

No question body available.

Tags

python multithreading multiprocessing

Answers (2)

Accepted Answer Available
Accepted Answer
July 7, 2026 Score: 3 Rep: 77 Quality: High Completeness: 60%

gc.collect() is probably not the source of the deadlock. Garbage collection does not interact with multiprocessing.Lock internally, so it should not block a lock by itself.

Given the symptoms (worker at 0% CPU, no exception, parent stuck in join()), the worker is most likely blocked waiting on an OS synchronization primitive. The first thing I would suspect is the Lock.

If a process acquires the lock and then hangs before releasing it, every other worker will block forever:

with lock:
    shm.buf[0] = i

To find where the worker is stuck, enable faulthandler:

import faulthandler
import signal
faulthandler.register(signal.SIGUSR1)

Then, when the process freezes:

kill -USR1

This will print the current stack trace and show whether it is blocked on the lock or somewhere else.

I would also avoid unbounded lock waits in production:

if lock.acquire(timeout=10):
    try:
        shm.buf[0] = i
    finally:
        lock.release()
else:
    print("Lock acquisition timed out")

For high-throughput processing, consider avoiding a single shared write lock. A common pattern is multiprocessing.Queue for task coordination combined with SharedMemory for large data blocks. Let workers write to separate regions whenever possible.

Manager is usually not a good fit here because it introduces extra IPC overhead and will be much slower for large datasets.

My first debugging step would be to capture a frozen worker stack trace with faulthandler; that will usually identify the exact blocking point.

July 7, 2026 Score: 2 Rep: 82 Quality: Low Completeness: 80%

The behavior you describe is almost certainly a deadlock or a blocked OS resource, not a garbage collector problem.

gc.collect() does not manage multiprocessing.Lock or shared memory synchronization, so it cannot directly deadlock your workers. However, it can trigger object cleanup, so it is still worth removing it from the hot path while debugging.

The most suspicious code is:

with lock:
    shm.buf[0] = i

multiprocessing.Lock is not automatically released if a process is killed or gets stuck while holding it. One worker can acquire the lock and stop, leaving every other worker blocked forever. The parent then waits indefinitely in join().

The first thing I would do is capture the stack trace of the frozen worker. Add this at worker startup:

import faulthandler
import signal
faulthandler.register(signal.SIGUSR1)

When the worker hangs:

kill -USR1

The traceback will tell you exactly where it is waiting (for example, inside lock.acquire()).

Also change lock usage temporarily to detect stuck locks:

if lock.acquire(timeout=30):
    try:
        shm.buf[0] = i
    finally:
        lock.release()
else:
    print("Worker could not acquire lock")

For production workloads, I would avoid multiple workers writing to the same shared buffer protected by a single global lock. A more scalable design is:

  • multiprocessing.Queue for task/result coordination

  • SharedMemory only for large data exchange

  • separate memory regions per worker when possible

Manager is usually not a replacement here because it serializes access through another process and adds significant overhead.

The next step is not changing GC settings — it is finding the exact blocking point with faulthandler or strace. That will usually reveal whether the issue is a lock, shared memory lifecycle, or another IPC wait.