dlopen() takes ages when used on a freshly created file

This is code signature caching at the vnode layer.

You can see this by first running:

sudo sysctl vm.csdebug=1
log stream --process 0 --predicate 'sender == "AppleMobileFileIntegrity"'

kernel: (AppleMobileFileIntegrity) AMFI: vnodechecksignature called with platform 1
kernel: (AppleMobileFileIntegrity) AMFI: '/private/tmp/aaa/mylib.dylib' has no CMS blob?
kernel: (AppleMobileFileIntegrity) AMFI: '/private/tmp/aaa/mylib.dylib': Unrecoverable CT signature issue, bailing out.
kernel: (AppleMobileFileIntegrity) AMFI: code signature validation failed.
kernel: (AppleMobileFileIntegrity) AMFI: vnodechecksignature called with platform 1
kernel: (AppleMobileFileIntegrity) AMFI: '/private/tmp/aaa/tmpfile' has no CMS blob?
kernel: (AppleMobileFileIntegrity) AMFI: '/private/tmp/aaa/tmpfile': Unrecoverable CT signature issue, bailing out.
kernel: (AppleMobileFileIntegrity) AMFI: code signature validation failed.

kernel: (AppleMobileFileIntegrity) AMFI: vnodechecksignature called with platform 1
kernel: (AppleMobileFileIntegrity) AMFI: '/private/tmp/aaa/tmpfile' has no CMS blob?
kernel: (AppleMobileFileIntegrity) AMFI: '/private/tmp/aaa/tmpfile': Unrecoverable CT signature issue, bailing out.
kernel: (AppleMobileFileIntegrity) AMFI: code signature validation failed.

As for what you can do about it, you could either unload AMFI in its entirety, if you only care about getting the maximum performance out of your machine and don't need to run any day-to-day apps. To do that, you'd need to boot into recovery, enable boot-args with bputil -a, and then disable AMFI via sudo nvram boot-args='amfigetoutofmyway=1'. But doing so will mark all processes as "platform" (Apple) processes, for which the system applies some tighter restrictions than for 3rd party processes, so this is likely to make some apps crash.

Other than that, you could of course pre-load the code signature cache, either by parsing the Mach-O header, finding the code signature offset, and calling fcntl(F_ADDFILESIGS, ...) with it like dyld does, or by simply calling dlopen like you already do... but of course that only moves the operation elsewhere, and doesn't get rid of it.