Question Details

No question body available.

Tags

r parallel-processing purrr

Answers (2)

May 12, 2026 Score: 4 Rep: 169,044 Quality: Medium Completeness: 80%

Two sections:

  1. Basic use of mirai for inparallel(..)
  2. Passing local objects to other process.

Basic mirai

inparallel() uses a mirai setup of nodes.

# simple, single-process
imap(5:8, ~ runif(.y, max = .x))

potentially parallelized

imap(5:8, inparallel(~ runif(.y, max = .x)))

I say "potentially" b/c it works whether there is a mirai setup of processes or not. In this case, it is the same process, so inparallel(.) here is sequential and provides no true parallelization. And since it doesn't warn, complain, or even comment about this, there is no apparent improvement.

We can show this more easily with this:

map(1:3, ~ Sys.getpid())

[[1]]

[1] 48159

[[2]]

[1] 48159

[[3]]

[1] 48159

map(1:3, inparallel(~ Sys.getpid()))

[[1]]

[1] 48159

[[2]]

[1] 48159

[[3]]

[1] 48159

No difference. However, if we setup 3 processes, we can see that the process id can different between them.

mirai::daemons(3)
map(1:5, ~ Sys.getpid())

[[1]]

[1] 48159

[[2]]

[1] 48159

[[3]]

[1] 48159

[[4]]

[1] 48159

[[5]]

[1] 48159

map(1:5, in
parallel(~ Sys.getpid()))

[[1]]

[1] 71513

[[2]]

[1] 71520

[[3]]

[1] 71533

[[4]]

[1] 71520

[[5]]

[1] 71513

Some interesting things to know:

### number of cores your particular computer has;
### can use logical=FALSE to differentiate between
###   logical and physical cores, varies by chip
parallel::detectCores()

[1] 16

### current state of your mirai persistent processes mirai::status()

$connections

[1] 3

$daemons

[1] "ipc:///tmp/5e75569e6c116e9605b79e24"

$mirai

awaiting executing completed

0 0 5

A timing example, perhaps more palpable:

mirai::status() # default when nothing previously setup

$connections

[1] 0

$daemons

[1] 0

system.time(map(1:3, ~ Sys.sleep(1)))

user system elapsed

0.003 0.001 3.015

system.time(map(1:3, inparallel(~ Sys.sleep(1))))

user system elapsed

0.003 0.001 3.016

mirai::daemons(3) system.time(map(1:3, ~ Sys.sleep(1)))

user system elapsed

0.000 0.000 3.014

system.time(map(1:3, in
parallel(~ Sys.sleep(1))))

user system elapsed

0.001 0.000 1.008


Local Objects

If you take this to your example directly, you'll find another issue:

imap(sampledids, inparallel(~ resampdf(.x, .y, df)))

Error in map2():

ℹ In index: 1.

Caused by error in resampdf():

! could not find function "resampdf"

Run rlang::lasttrace() to see where the error occurred.

The function is clearly defined, but all of the mirai worker processes start with pristine/empty R environments, so none of what you defined (namely resampdf and df itself) are available. We can pass it in a number of ways:

  1. Globally you can pass objects to all objects using mirai::everywhere:

    mirai::everywhere({}, resampdf=resampdf, df=df)
    imap(sampledids, inparallel(~ resampdf(.x, .y, df))) |> bindrows()
    

    The first argument must be some form of expression, all ... args are named arguments of objects to store in all workers' global environment. It's common to name them the same as their object in this environment but that is not required. The first arg is not optional even if we're not using it, so we pass the empty {} as a no-op; we could just have easily passed pi or c or something else innocuous.

    As more demonstration (and a setup for option two), I'll clean up the workers' environments:

    mirai::everywhere(ls(envir = .GlobalEnv))[]
    

    [[1]]

    [1] "df" "resampdf"

    [[2]]

    [1] "df" "resampdf"

    mirai::everywhere(rm(resampdf, df, envir=.GlobalEnv))[]

    [[1]]

    NULL

    [[2]]

    NULL

    mirai::everywhere(ls(envir = .GlobalEnv))[]

    [[1]]

    character(0)

    [[2]]

    character(0)

  2. The ... args of inparallel work similarly to everywhere, though with a small difference: each expression sent to a mirai worker is executed in a non-global environment. While .GlobalEnv is visible in its search path, working in an ephemeral environment means that cleanup is natural, there are no accidental "stale objects". Because of this, when our expressions complete, we no longer see resampdf and df when the expression is complete.

    imap(sampledids, inparallel(~ resampdf(.x, .y, data), resampdf = resampdf, data = df)) |>
      bindrows()
    

    # A tibble: 2,059 × 4

    id outcome predictor id2

    1 406 1 35.3 1

    2 406 0 38.2 1

    3 406 1 32.4 1

    4 406 0 34 1

    5 406 1 36.2 1

    6 406 0 34 1

    7 42 0 36.3 2

    8 42 1 39.3 2

    9 8 1 28.3 3

    10 8 1 36.3 3

    # ℹ 2,049 more rows

    # ℹ Use print(n = ...) to see more rows

    mirai::everywhere(ls(envir = .GlobalEnv))[]

    [[1]]

    character(0)

    [[2]]

    character(0)

    (Notice that I named data = df for that call, that's purely for demonstration of renaming in the call. Normally, most people would likely do df=df and use it as df.)

May 13, 2026 Score: 1 Rep: 21,955 Quality: Medium Completeness: 80%

/../ and although I thought I understood how to pass arguments for parallel execution, but seems that's not the case

Key takeaways of ?inparallel are that it must be called with a fresh formula or (fresh) function, and you must supply named arguments that will be declared in the environment of that formula / function.

In other words, you can't use your resampdf directly, instead you should pass a formula or an anonymous function as an argument of inparallel() and call resampdf() from there. As parallel workers are separate processes that do not know too much about your main R process nor its environment(s), inparallel() prepares a carte object that should include everything that's needed to run your fresh function in a new clean environment, so you should include both your data (df) and declared function (resampdf) as named arguments of inparallel():

# start 2 background processes
mirai::daemons(2)
d2 
  purrr::list_rbind()
mirai::daemons(0)

compare against non-parallel imap() result:

identical(d1, d2) #> [1] TRUE