Multithreading becomes much slower than multiprocessing in free-threaded Python

for future paths to better performant multi-threading Python, what we have so far is this draft pep: https://pep-previews--4579.org.readthedocs.build/pep-0805/ - and also this one disputing with it:

https://pep-previews--4468.org.readthedocs.build/pep-0795/

I tend to favor the proposals in 805 much better - the 795 authors just sent a reviewed version, I am still to check - but it were utterly broken before (basically, ANY Python program and set of libraries would have to pick between runtime dynamism, or having data structures with runtime aware imutability)

The discussions are taking place in Python discourse:

https://discuss.python.org/t/pre-pep-safe-parallel-python/103636 (PEP 805)

https://discuss.python.org/t/pep-795-revamped-deep-immutability-for-efficient-sharing-and-concurrency-safety/105356 (latest round for PEP 795)

I didn't investigate the code deeply enough, but you could give a try to sprinkle some ThreadHandles on top of your code to see if you can avoid the bottleneck. Probably you would do something like this?

def worker(group, nloop):
    return sum(kernel(ThreadHandle(g), nloop) for g in group)

But again, I haven't thoroughly checked your code and I might be missing some other bottleneck. The tricky part to this reference counting contention problems is that you need to remove all of them before you see an actual speedup. (Because the lesser contention on one object may result in greater contention on another object.)

Full disclosure: I maintain that library.