Package 'crew'

Description

In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The NNG-powered mirai R package is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The crew package extends mirai with a unifying interface for third-party worker launchers. Inspiration also comes from packages future, rrq, clustermq, and batchtools.

Description

Assert that a condition is true.

Usage

crew_assert(value = NULL, ..., message = NULL, envir = parent.frame())

Arguments

Value

NULL (invisibly). Throws an error if the condition is not true.

See Also

Other utility: crew_clean(), crew_deprecate(), crew_eval(), crew_random_name(), crew_retry(), crew_terminate_process(), crew_terminate_signal(), crew_worker()

Examples

crew_assert(1 < 2)
crew_assert("object", !anyNA(.), nzchar(.))
tryCatch(
  crew_assert(2 < 1),
  crew_error = function(condition) message("false")
)

Description

R6 class for mirai clients.

Details

See crew_client().

Active bindings

Methods

Public methods

• crew_class_client$new()

• crew_class_client$validate()

• crew_class_client$set_started()

• crew_class_client$start()

• crew_class_client$terminate()

• crew_class_client$status()

• crew_class_client$pids()

Method new()

mirai client constructor.

Usage

crew_class_client$new(
  host = NULL,
  port = NULL,
  tls = NULL,
  serialization = NULL,
  profile = NULL,
  seconds_interval = NULL,
  seconds_timeout = NULL,
  relay = NULL
)

Arguments

Returns

An R6 object with the client.

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
client$log()
client$terminate()
}

Method validate()

Validate the client.

Usage

crew_class_client$validate()

Returns

NULL (invisibly).

Method set_started()

Register the client as started.

Usage

crew_class_client$set_started()

Details

Exported to implement the sequential controller. Only meant to be called manually inside the client or the sequential controller.

Returns

NULL (invisibly).

Method start()

Start listening for workers on the available sockets.

Usage

crew_class_client$start()

Returns

NULL (invisibly).

Method terminate()

Stop the mirai client and disconnect from the worker websockets.

Usage

crew_class_client$terminate()

Returns

NULL (invisibly).

Method status()

Get the counters from mirai::info().

Usage

crew_class_client$status()

Returns

A named integer vector of task counts (awaiting, executing, completed) as well as the number of worker connections.

Method pids()

Deprecated on 2025-08-26 in crew version 1.2.1.9005.

Usage

crew_class_client$pids()

Returns

The integer process ID of the current process.

See Also

Other client: crew_client()

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
client$log()
client$terminate()
}

## ------------------------------------------------
## Method `crew_class_client$new`
## ------------------------------------------------

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
client$log()
client$terminate()
}

Description

R6 class for controllers.

Details

See crew_controller().

Active bindings

Methods

Public methods

• crew_class_controller$new()

• crew_class_controller$validate()

• crew_class_controller$size()

• crew_class_controller$empty()

• crew_class_controller$nonempty()

• crew_class_controller$resolved()

• crew_class_controller$unresolved()

• crew_class_controller$saturated()

• crew_class_controller$start()

• crew_class_controller$started()

• crew_class_controller$launch()

• crew_class_controller$scale()

• crew_class_controller$autoscale()

• crew_class_controller$descale()

• crew_class_controller$crashes()

• crew_class_controller$push()

• crew_class_controller$walk()

• crew_class_controller$map()

• crew_class_controller$pop()

• crew_class_controller$collect()

• crew_class_controller$wait()

• crew_class_controller$push_backlog()

• crew_class_controller$pop_backlog()

• crew_class_controller$summary()

• crew_class_controller$cancel()

• crew_class_controller$pids()

• crew_class_controller$terminate()

Method new()

mirai controller constructor.

Usage

crew_class_controller$new(
  client = NULL,
  launcher = NULL,
  reset_globals = NULL,
  reset_packages = NULL,
  reset_options = NULL,
  garbage_collection = NULL,
  crashes_max = NULL,
  backup = NULL
)

Arguments

Returns

An R6 controller object.

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
launcher <- crew_launcher_local()
controller <- crew_controller(client = client, launcher = launcher)
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()
controller$terminate()
}

Method validate()

Validate the controller.

Usage

crew_class_controller$validate()

Returns

NULL (invisibly).

Method size()

Number of tasks in the controller.

Usage

crew_class_controller$size(controllers = NULL)

Arguments

Returns

Non-negative integer, number of tasks in the controller.

Method empty()

Check if the controller is empty.

Usage

crew_class_controller$empty(controllers = NULL)

Arguments

Details

A controller is empty if it has no mirai::mirai() task objects in the controller. There may still be other tasks running on the workers of an empty controller, but those tasks were not submitted with push() or collect(), and they are not part of the controller task queue.

Returns

TRUE if the controller is empty, FALSE otherwise.

Method nonempty()

Check if the controller is nonempty.

Usage

crew_class_controller$nonempty(controllers = NULL)

Arguments

Details

A controller is empty if it has no mirai::mirai() task objects in the controller. There may still be other tasks running on the workers of an empty controller, but those tasks were not submitted with push() or collect(), and they are not part of the controller task queue.

Returns

TRUE if the controller is empty, FALSE otherwise.

Method resolved()

Cumulative number of resolved tasks.

Usage

crew_class_controller$resolved(controllers = NULL)

Arguments

Details

resolved() is cumulative: it counts all the resolved tasks over the entire lifetime of the controller session.

Returns

Non-negative integer of length 1, number of resolved tasks. The return value is 0 if the condition variable does not exist (i.e. if the client is not running).

Method unresolved()

Number of unresolved tasks.

Usage

crew_class_controller$unresolved(controllers = NULL)

Arguments

Returns

Non-negative integer of length 1, number of unresolved tasks.

Method saturated()

Check if the controller is saturated.

Usage

crew_class_controller$saturated(
  collect = NULL,
  throttle = NULL,
  controller = NULL
)

Arguments

Details

A controller is saturated if the number of uncollected tasks is greater than or equal to the maximum number of workers. You can still push tasks to a saturated controller, but tools that use crew such as targets may choose not to (for performance and user-friendliness).

Returns

TRUE if the controller is saturated, FALSE otherwise.

Method start()

Start the controller if it is not already started.

Usage

crew_class_controller$start(controllers = NULL)

Arguments

Details

Register the mirai client and register worker websockets with the launcher.

Returns

NULL (invisibly).

Method started()

Check whether the controller is started.

Usage

crew_class_controller$started(controllers = NULL)

Arguments

Details

Actually checks whether the client is started.

Returns

TRUE if the controller is started, FALSE otherwise.

Method launch()

Launch one or more workers.

Usage

crew_class_controller$launch(n = 1L, controllers = NULL)

Arguments

Returns

NULL (invisibly).

Method scale()

Auto-scale workers out to meet the demand of tasks.

Usage

crew_class_controller$scale(throttle = TRUE, controllers = NULL)

Arguments

Details

The scale() method launches new workers to run tasks if needed.

Returns

Invisibly returns TRUE if auto-scaling was attempted (throttling can skip it) and there was any relevant auto-scaling activity (new worker launches or worker connection/disconnection events). FALSE otherwise.

Method autoscale()

Run worker auto-scaling in a later loop in polling intervals determined by exponential backoff.

Usage

crew_class_controller$autoscale(
  loop = later::current_loop(),
  controllers = NULL
)

Arguments

Details

Call controller$descale() to terminate the auto-scaling loop.

Returns

NULL (invisibly).

Method descale()

Terminate the auto-scaling loop started by controller$autoscale().

Usage

crew_class_controller$descale(controllers = NULL)

Arguments

Returns

NULL (invisibly).

Method crashes()

Report the number of consecutive crashes of a task.

Usage

crew_class_controller$crashes(name, controllers = NULL)

Arguments

Details

See the crashes_max argument of crew_controller().

Returns

Non-negative integer, number of consecutive times the task crashed.

Method push()

Push a task to the head of the task list.

Usage

crew_class_controller$push(
  command,
  data = list(),
  globals = list(),
  substitute = TRUE,
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  seconds_timeout = NULL,
  scale = TRUE,
  throttle = TRUE,
  name = NULL,
  save_command = NULL,
  controller = NULL
)

Arguments

Returns

Invisibly return the mirai object of the pushed task. This allows you to interact with the task directly, e.g. to create a promise object with promises::as.promise().

Method walk()

Apply a single command to multiple inputs, and return control to the user without waiting for any task to complete.

Usage

crew_class_controller$walk(
  command,
  iterate,
  data = list(),
  globals = list(),
  substitute = TRUE,
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  seconds_timeout = NULL,
  names = NULL,
  save_command = NULL,
  verbose = interactive(),
  scale = TRUE,
  throttle = TRUE,
  controller = NULL
)

Arguments

Details

In contrast to walk(), map() blocks the local R session and waits for all tasks to complete.

Returns

Invisibly returns a list of mirai task objects for the newly created tasks. The order of tasks in the list matches the order of data in the iterate argument.

Method map()

Apply a single command to multiple inputs, wait for all tasks to complete, and return the results of all tasks.

Usage

crew_class_controller$map(
  command,
  iterate,
  data = list(),
  globals = list(),
  substitute = TRUE,
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  seconds_interval = NULL,
  seconds_timeout = NULL,
  names = NULL,
  save_command = NULL,
  error = "stop",
  warnings = TRUE,
  verbose = interactive(),
  scale = TRUE,
  throttle = TRUE,
  controller = NULL
)

Arguments

Details

map() cannot be used unless all prior tasks are completed and popped. You may need to wait and then pop them manually. Alternatively, you can start over: either call terminate() on the current controller object to reset it, or create a new controller object entirely.

Returns

A tibble of results and metadata: one row per task and columns corresponding to the output of pop().

Method pop()

Pop a completed task from the results data frame.

Usage

crew_class_controller$pop(
  scale = TRUE,
  collect = NULL,
  throttle = TRUE,
  error = NULL,
  controllers = NULL
)

Arguments

Details

If not task is currently completed, pop() will attempt to auto-scale workers as needed.

Returns

If there is no task to collect, return NULL. Otherwise, return a one-row tibble with the following columns.

• name: the task name.

• command: a character string with the R command.

• result: a list containing the return value of the R command. NA if the task failed.

• status: a character string. "success" if the task succeeded, "cancel" if the task was canceled with the cancel() controller method, "crash" if the worker running the task exited before it could complete the task, or "error" for any other kind of error.

• error: the first 2048 characters of the error message if the task status is not "success", NA otherwise. Messages for crashes and cancellations are captured here alongside ordinary R-level errors.

• code: an integer code denoting the specific exit status: 0 for successful tasks, -1 for tasks with an error in the R command of the task, and another positive integer with an NNG status code if there is an error at the NNG/nanonext level. nanonext::nng_error() can interpret these codes.

• trace: the first 2048 characters of the text of the traceback if the task threw an error, NA otherwise.

• warnings: the first 2048 characters. of the text of warning messages that the task may have generated, NA otherwise.

• seconds: number of seconds that the task ran.

• seed: the single integer originally supplied to push(), NA otherwise. The pseudo-random number generator state just prior to the task can be restored using set.seed(seed = seed, kind = algorithm), where seed and algorithm are part of this output.

• algorithm: name of the pseudo-random number generator algorithm originally supplied to push(), NA otherwise. The pseudo-random number generator state just prior to the task can be restored using set.seed(seed = seed, kind = algorithm), where seed and algorithm are part of this output.

• controller: name of the crew controller where the task ran.

• worker: name of the crew worker that ran the task.

Method collect()

Pop all available task results and return them in a tidy tibble.

Usage

crew_class_controller$collect(
  scale = TRUE,
  throttle = TRUE,
  error = NULL,
  controllers = NULL
)

Arguments

Returns

A tibble of results and metadata of all resolved tasks, with one row per task. Returns NULL if there are no tasks to collect. See pop() for details on the columns of the returned tibble.

Method wait()

Wait for tasks.

Usage

crew_class_controller$wait(
  mode = "all",
  seconds_interval = NULL,
  seconds_timeout = Inf,
  scale = TRUE,
  throttle = TRUE,
  controllers = NULL
)

Arguments

Details

The wait() method blocks the calling R session until the condition in the mode argument is met. During the wait, wait() iteratively auto-scales the workers.

Returns

A logical of length 1, invisibly. wait(mode = "all") returns TRUE if all tasks in the mirai compute profile have resolved (FALSE otherwise). wait(mode = "one") returns TRUE if the controller is ready to pop or collect at least one resolved task (FALSE otherwise). wait(mode = "one") assumes all tasks were submitted through the controller and not by other means.

Method push_backlog()

Push the name of a task to the backlog.

Usage

crew_class_controller$push_backlog(name, controller = NULL)

Arguments

Details

pop_backlog() pops the tasks that can be pushed without saturating the controller.

Returns

NULL (invisibly).

Method pop_backlog()

Pop the task names from the head of the backlog which can be pushed without saturating the controller.

Usage

crew_class_controller$pop_backlog(controllers = NULL)

Arguments

Returns

Character vector of task names which can be pushed to the controller without saturating it. If the controller is saturated, character(0L) is returned.

Method summary()

Summarize the collected tasks of the controller.

Usage

crew_class_controller$summary(controllers = NULL)

Arguments

Returns

A data frame of cumulative summary statistics on the tasks collected through pop() and collect(). It has one row and the following columns:

• controller: name of the controller.

• seconds: total number of runtime in seconds.

• tasks: total number of tasks collected.

• success: total number of collected tasks that did not crash or error.

• error: total number of tasks with errors, either in the R code of the task or an NNG-level error that is not a cancellation or crash.

• crash: total number of crashed tasks (where the worker exited unexpectedly while it was running the task).

• cancel: total number of tasks interrupted with the cancel() controller method.

• warning: total number of tasks with one or more warnings.

Method cancel()

Cancel one or more tasks.

Usage

crew_class_controller$cancel(names = character(0L), all = FALSE)

Arguments

Returns

NULL (invisibly).

Method pids()

Deprecated on 2025-08-26 in crew version 1.2.1.9005.

Usage

crew_class_controller$pids(controllers = NULL)

Arguments

Returns

The integer process ID of the current process.

Method terminate()

Terminate the workers and the mirai client.

Usage

crew_class_controller$terminate(controllers = NULL)

Arguments

Returns

NULL (invisibly).

See Also

Other controller: crew_controller()

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
launcher <- crew_launcher_local()
controller <- crew_controller(client = client, launcher = launcher)
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()
controller$terminate()
}

## ------------------------------------------------
## Method `crew_class_controller$new`
## ------------------------------------------------

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
launcher <- crew_launcher_local()
controller <- crew_controller(client = client, launcher = launcher)
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()
controller$terminate()
}

Description

R6 class for controller groups.

Details

See crew_controller_group().

Active bindings

Methods

Public methods

• crew_class_controller_group$new()

• crew_class_controller_group$validate()

• crew_class_controller_group$size()

• crew_class_controller_group$empty()

• crew_class_controller_group$nonempty()

• crew_class_controller_group$resolved()

• crew_class_controller_group$unresolved()

• crew_class_controller_group$saturated()

• crew_class_controller_group$start()

• crew_class_controller_group$started()

• crew_class_controller_group$launch()

• crew_class_controller_group$scale()

• crew_class_controller_group$autoscale()

• crew_class_controller_group$descale()

• crew_class_controller_group$crashes()

• crew_class_controller_group$push()

• crew_class_controller_group$walk()

• crew_class_controller_group$map()

• crew_class_controller_group$pop()

• crew_class_controller_group$collect()

• crew_class_controller_group$wait()

• crew_class_controller_group$push_backlog()

• crew_class_controller_group$pop_backlog()

• crew_class_controller_group$summary()

• crew_class_controller_group$pids()

• crew_class_controller_group$terminate()

Method new()

Multi-controller constructor.

Usage

crew_class_controller_group$new(controllers = NULL, relay = NULL)

Arguments

Returns

An R6 object with the controller group object.

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
persistent <- crew_controller_local(name = "persistent")
transient <- crew_controller_local(
  name = "transient",
  tasks_max = 1L
)
group <- crew_controller_group(persistent, transient)
group$start()
group$push(name = "task", command = sqrt(4), controller = "transient")
group$wait()
group$pop()
group$terminate()
}

Method validate()

Validate the client.

Usage

crew_class_controller_group$validate()

Returns

NULL (invisibly).

Method size()

Number of tasks in the selected controllers.

Usage

crew_class_controller_group$size(controllers = NULL)

Arguments

Returns

Non-negative integer, number of tasks in the controller.

Method empty()

See if the controllers are empty.

Usage

crew_class_controller_group$empty(controllers = NULL)

Arguments

Details

A controller is empty if it has no running tasks or completed tasks waiting to be retrieved with push().

Returns

TRUE if all the selected controllers are empty, FALSE otherwise.

Method nonempty()

Check if the controller group is nonempty.

Usage

crew_class_controller_group$nonempty(controllers = NULL)

Arguments

Details

A controller is empty if it has no running tasks or completed tasks waiting to be retrieved with push().

Returns

TRUE if the controller is empty, FALSE otherwise.

Method resolved()

Number of resolved mirai() tasks.

Usage

crew_class_controller_group$resolved(controllers = NULL)

Arguments

Details

resolved() is cumulative: it counts all the resolved tasks over the entire lifetime of the controller session.

Returns

Non-negative integer of length 1, number of resolved mirai() tasks. The return value is 0 if the condition variable does not exist (i.e. if the client is not running).

Method unresolved()

Number of unresolved mirai() tasks.

Usage

crew_class_controller_group$unresolved(controllers = NULL)

Arguments

Returns

Non-negative integer of length 1, number of unresolved mirai() tasks.

Method saturated()

Check if a controller is saturated.

Usage

crew_class_controller_group$saturated(
  collect = NULL,
  throttle = NULL,
  controller = NULL
)

Arguments

Details

A controller is saturated if the number of uncollected tasks is greater than or equal to the maximum number of workers. You can still push tasks to a saturated controller, but tools that use crew such as targets may choose not to (for performance and user-friendliness).

Returns

TRUE if all the selected controllers are saturated, FALSE otherwise.

Method start()

Start one or more controllers.

Usage

crew_class_controller_group$start(controllers = NULL)

Arguments

Returns

NULL (invisibly).

Method started()

Check whether all the given controllers are started.

Usage

crew_class_controller_group$started(controllers = NULL)

Arguments

Details

Actually checks whether all the given clients are started.

Returns

TRUE if the controllers are started, FALSE if any are not.

Method launch()

Launch one or more workers on one or more controllers.

Usage

crew_class_controller_group$launch(n = 1L, controllers = NULL)

Arguments

Returns

NULL (invisibly).

Method scale()

Automatically scale up the number of workers if needed in one or more controller objects.

Usage

crew_class_controller_group$scale(throttle = TRUE, controllers = NULL)

Arguments

Details

See the scale() method in individual controller classes.

Returns

Invisibly returns TRUE if there was any relevant auto-scaling activity (new worker launches or worker connection/disconnection events) (FALSE otherwise).

Method autoscale()

Run worker auto-scaling in a later loop.

Usage

crew_class_controller_group$autoscale(
  loop = later::current_loop(),
  controllers = NULL
)

Arguments

Returns

NULL (invisibly).

Method descale()

Terminate the auto-scaling loop started by controller$autoscale().

Usage

crew_class_controller_group$descale(controllers = NULL)

Arguments

Returns

NULL (invisibly).

Method crashes()

Report the number of consecutive crashes of a task, summed over all selected controllers in the group.

Usage

crew_class_controller_group$crashes(name, controllers = NULL)

Arguments

Details

See the crashes_max argument of crew_controller().

Returns

Number of consecutive crashes of the named task, summed over all the controllers in the group.

Method push()

Push a task to the head of the task list.

Usage

crew_class_controller_group$push(
  command,
  data = list(),
  globals = list(),
  substitute = TRUE,
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  seconds_timeout = NULL,
  scale = TRUE,
  throttle = TRUE,
  name = NULL,
  save_command = NULL,
  controller = NULL
)

Arguments

Returns

Invisibly return the mirai object of the pushed task. This allows you to interact with the task directly, e.g. to create a promise object with promises::as.promise().

Method walk()

Apply a single command to multiple inputs, and return control to the user without waiting for any task to complete.

Usage

crew_class_controller_group$walk(
  command,
  iterate,
  data = list(),
  globals = list(),
  substitute = TRUE,
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  seconds_timeout = NULL,
  names = NULL,
  save_command = NULL,
  verbose = interactive(),
  scale = TRUE,
  throttle = TRUE,
  controller = NULL
)

Arguments

Details

In contrast to walk(), map() blocks the local R session and waits for all tasks to complete.

Returns

Invisibly returns a list of mirai task objects for the newly created tasks. The order of tasks in the list matches the order of data in the iterate argument.

Method map()

Apply a single command to multiple inputs.

Usage

crew_class_controller_group$map(
  command,
  iterate,
  data = list(),
  globals = list(),
  substitute = TRUE,
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  seconds_interval = NULL,
  seconds_timeout = NULL,
  names = NULL,
  save_command = NULL,
  error = "stop",
  warnings = TRUE,
  verbose = interactive(),
  scale = TRUE,
  throttle = TRUE,
  controller = NULL
)

Arguments

Details

The idea comes from functional programming: for example, the map() function from the purrr package.

Returns

A tibble of results and metadata: one row per task and columns corresponding to the output of pop().

Method pop()

Pop a completed task from the results data frame.

Usage

crew_class_controller_group$pop(
  scale = TRUE,
  collect = NULL,
  throttle = TRUE,
  error = NULL,
  controllers = NULL
)

Arguments

Returns

If there is no task to collect, return NULL. Otherwise, return a one-row tibble with the same columns as pop() for ordinary controllers.

Method collect()

Pop all available task results and return them in a tidy tibble.

Usage

crew_class_controller_group$collect(
  scale = TRUE,
  throttle = TRUE,
  error = NULL,
  controllers = NULL
)

Arguments

Returns

A tibble of results and metadata of all resolved tasks, with one row per task. Returns NULL if there are no available results.

Method wait()

Wait for tasks.

Usage

crew_class_controller_group$wait(
  mode = "all",
  seconds_interval = NULL,
  seconds_timeout = Inf,
  scale = TRUE,
  throttle = TRUE,
  controllers = NULL
)

Arguments

Details

The wait() method blocks the calling R session until the condition in the mode argument is met. During the wait, wait() iteratively auto-scales the workers.

Returns

A logical of length 1, invisibly. wait(mode = "all") returns TRUE if all tasks in the mirai compute profile have resolved (FALSE otherwise). wait(mode = "one") returns TRUE if the controller is ready to pop or collect at least one resolved task (FALSE otherwise). wait(mode = "one") assumes all tasks were submitted through the controller and not by other means.

Method push_backlog()

Push the name of a task to the backlog.

Usage

crew_class_controller_group$push_backlog(name, controller = NULL)

Arguments

Details

pop_backlog() pops the tasks that can be pushed without saturating the controller.

Returns

NULL (invisibly).

Method pop_backlog()

Pop the task names from the head of the backlog which can be pushed without saturating the controller.

Usage

crew_class_controller_group$pop_backlog(controllers = NULL)

Arguments

Returns

Character vector of task names which can be pushed to the controller without saturating it. If the controller is saturated, character(0L) is returned.

Method summary()

Summarize the workers of one or more controllers.

Usage

crew_class_controller_group$summary(controllers = NULL)

Arguments

Returns

A data frame of aggregated worker summary statistics of all the selected controllers. It has one row per worker, and the rows are grouped by controller. See the documentation of the summary() method of the controller class for specific information about the columns in the output.

Method pids()

Deprecated on 2025-08-26 in crew version 1.2.1.9005.

Usage

crew_class_controller_group$pids(controllers = NULL)

Arguments

Returns

The integer process ID of the current process.

Method terminate()

Terminate the workers and disconnect the client for one or more controllers.

Usage

crew_class_controller_group$terminate(controllers = NULL)

Arguments

Returns

NULL (invisibly).

See Also

Other controller_group: crew_controller_group()

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
persistent <- crew_controller_local(name = "persistent")
transient <- crew_controller_local(
  name = "transient",
  tasks_max = 1L
)
group <- crew_controller_group(persistent, transient)
group$start()
group$push(name = "task", command = sqrt(4), controller = "transient")
group$wait()
group$pop()
group$terminate()
}

## ------------------------------------------------
## Method `crew_class_controller_group$new`
## ------------------------------------------------

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
persistent <- crew_controller_local(name = "persistent")
transient <- crew_controller_local(
  name = "transient",
  tasks_max = 1L
)
group <- crew_controller_group(persistent, transient)
group$start()
group$push(name = "task", command = sqrt(4), controller = "transient")
group$wait()
group$pop()
group$terminate()
}

Description

R6 class for sequential controllers.

Details

See crew_controller_sequential().

Super class

crew::crew_class_controller -> crew_class_controller_sequential

Methods

Public methods

• crew_class_controller_sequential$resolved()

• crew_class_controller_sequential$unresolved()

• crew_class_controller_sequential$saturated()

• crew_class_controller_sequential$start()

• crew_class_controller_sequential$launch()

• crew_class_controller_sequential$scale()

• crew_class_controller_sequential$autoscale()

• crew_class_controller_sequential$descale()

• crew_class_controller_sequential$push()

• crew_class_controller_sequential$wait()

• crew_class_controller_sequential$push_backlog()

• crew_class_controller_sequential$pop_backlog()

• crew_class_controller_sequential$cancel()

• crew_class_controller_sequential$terminate()

Method resolved()

Number of resolved tasks.

Usage

crew_class_controller_sequential$resolved(controllers = NULL)

Arguments

Details

resolved() is cumulative: it counts all the resolved tasks over the entire lifetime of the controller session. For the sequential controller, tasks are resolved as soon as they are pushed.

Returns

Non-negative integer of length 1, number of resolved tasks. The return value is 0 if the condition variable does not exist (i.e. if the client is not running).

Method unresolved()

Number of unresolved tasks.

Usage

crew_class_controller_sequential$unresolved(controllers = NULL)

Arguments

Returns

Returns 0 always because the sequential controller resolves tasks as soon as they are pushed.

Method saturated()

Check if the controller is saturated.

Usage

crew_class_controller_sequential$saturated(
  collect = NULL,
  throttle = NULL,
  controller = NULL
)

Arguments

Returns

Always returns FALSE for the sequential controller because tasks run immediately on the local process and there are no workers.

Method start()

Start the controller if it is not already started.

Usage

crew_class_controller_sequential$start(controllers = NULL)

Arguments

Details

For the sequential controller, there is nothing to do except register the client as started.

Returns

NULL (invisibly).

Method launch()

Does nothing for the sequential controller.

Usage

crew_class_controller_sequential$launch(n = 1L, controllers = NULL)

Arguments

Returns

NULL (invisibly).

Method scale()

Does nothing for the sequential controller.

Usage

crew_class_controller_sequential$scale(throttle = TRUE, controllers = NULL)

Arguments

Returns

Invisibly returns FALSE.

Method autoscale()

Not applicable to the sequential controller.

Usage

crew_class_controller_sequential$autoscale(loop = NULL, controllers = NULL)

Arguments

Method descale()

Not applicable to the sequential controller.

Usage

crew_class_controller_sequential$descale(controllers = NULL)

Arguments

Returns

NULL (invisibly).

Method push()

Push a task to the head of the task list.

Usage

crew_class_controller_sequential$push(
  command,
  data = list(),
  globals = list(),
  substitute = TRUE,
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  seconds_timeout = NULL,
  scale = TRUE,
  throttle = TRUE,
  name = NULL,
  save_command = NULL,
  controller = NULL
)

Arguments

Returns

Invisibly returns a mirai-like list where the data element is the result of the task.

Method wait()

Not applicable to the sequential controller.

Usage

crew_class_controller_sequential$wait(
  mode = "all",
  seconds_interval = NULL,
  seconds_timeout = Inf,
  scale = TRUE,
  throttle = TRUE,
  controllers = NULL
)

Arguments

Returns

Always returns TRUE (invisibly) for the sequential controller.

Method push_backlog()

Not applicable to the sequential controller.

Usage

crew_class_controller_sequential$push_backlog(name, controller = NULL)

Arguments

Returns

NULL (invisibly).

Method pop_backlog()

Not applicable to the sequential controller.

Usage

crew_class_controller_sequential$pop_backlog(controllers = NULL)

Arguments

Returns

Always character(0L) for the sequential controller.

Method cancel()

Not applicable to the sequential controller.

Usage

crew_class_controller_sequential$cancel(names = character(0L), all = FALSE)

Arguments

Method terminate()

Terminate the controller.

Usage

crew_class_controller_sequential$terminate(controllers = NULL)

Arguments

Returns

NULL (invisibly).

See Also

Other sequential controllers: crew_controller_sequential()

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
controller <- crew_controller_sequential()
controller$push(name = "task", command = sqrt(4))
controller$pop()
}

Description

R6 abstract class to build other subclasses which launch and manage workers.

Public fields

Active bindings

Methods

Public methods

• crew_class_launcher$new()

• crew_class_launcher$validate()

• crew_class_launcher$poll()

• crew_class_launcher$settings()

• crew_class_launcher$call()

• crew_class_launcher$start()

• crew_class_launcher$terminate()

• crew_class_launcher$launch()

• crew_class_launcher$launch_worker()

• crew_class_launcher$launch_workers()

• crew_class_launcher$scale()

• crew_class_launcher$terminate_workers()

• crew_class_launcher$crashes()

• crew_class_launcher$set_name()

• crew_class_launcher$clone()

Method new()

Launcher constructor.

Usage

crew_class_launcher$new(
  name = NULL,
  workers = NULL,
  seconds_interval = NULL,
  seconds_timeout = NULL,
  seconds_launch = NULL,
  seconds_idle = NULL,
  seconds_wall = NULL,
  seconds_exit = NULL,
  tasks_max = NULL,
  tasks_timers = NULL,
  reset_globals = NULL,
  reset_packages = NULL,
  reset_options = NULL,
  garbage_collection = NULL,
  crashes_error = NULL,
  launch_max = NULL,
  tls = NULL,
  processes = NULL,
  r_arguments = NULL,
  options_metrics = NULL
)

Arguments

Returns

An R6 object with the launcher.

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
launcher <- crew_launcher_local()
launcher$start(url = client$url, profile = client$profile)
launcher$launch()
task <- mirai::mirai("result", .compute = client$profile)
mirai::call_mirai(task)
task$data
client$terminate()
}

Method validate()

Validate the launcher.

Usage

crew_class_launcher$validate()

Returns

NULL (invisibly).

Method poll()

Poll the throttle.

Usage

crew_class_launcher$poll()

Returns

TRUE to run whatever work comes next, FALSE to skip until the appropriate time.

Method settings()

List of arguments for mirai::daemon().

Usage

crew_class_launcher$settings()

Returns

List of arguments for mirai::daemon().

Method call()

Create a call to crew_worker() to help create custom launchers.

Usage

crew_class_launcher$call(worker = NULL)

Arguments

Returns

Character string with a call to crew_worker().

Examples

launcher <- crew_launcher_local()
launcher$start(url = "tcp://127.0.0.1:57000", profile = "profile")
launcher$call()
launcher$terminate()

Method start()

Start the launcher.

Usage

crew_class_launcher$start(url = NULL, profile = NULL, sockets = NULL)

Arguments

Returns

NULL (invisibly).

Method terminate()

Terminate the whole launcher, including all workers.

Usage

crew_class_launcher$terminate()

Returns

NULL (invisibly).

Method launch()

Launch a worker.

Usage

crew_class_launcher$launch(n = 1L)

Arguments

Returns

Handle of the launched worker.

Method launch_worker()

Abstract worker launch method.

Usage

crew_class_launcher$launch_worker(call)

Arguments

Details

Launcher plugins will overwrite this method.

Returns

A handle to mock the worker launch.

Method launch_workers()

Launch multiple workers.

Usage

crew_class_launcher$launch_workers(call, n)

Arguments

Details

Launcher plugins may overwrite this method to launch multiple workers from a single system call.

Returns

A handle to mock the worker launch.

Method scale()

Auto-scale workers out to meet the demand of tasks.

Usage

crew_class_launcher$scale(status, throttle = NULL)

Arguments

Returns

Invisibly returns TRUE if there was any relevant auto-scaling activity (new worker launches or worker connection/disconnection events) (FALSE otherwise).

Method terminate_workers()

Deprecated on 2025-08-26 (crew version 1.2.1.9004).

Usage

crew_class_launcher$terminate_workers()

Returns

NULL (invisibly).

Method crashes()

Deprecated on 2025-01-28 (crew version 1.0.0).

Usage

crew_class_launcher$crashes(index = NULL)

Arguments

Returns

The integer 1, for compatibility.

Method set_name()

Deprecated on 2025-01-28 (crew version 1.0.0).

Usage

crew_class_launcher$set_name(name)

Arguments

Returns

NULL (invisibly).

Method clone()

The objects of this class are cloneable with this method.

Usage

crew_class_launcher$clone(deep = FALSE)

Arguments

See Also

Other launcher: crew_launcher()

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
launcher <- crew_launcher_local()
launcher$start(url = client$url, profile = client$profile)
launcher$launch()
task <- mirai::mirai("result", .compute = client$profile)
mirai::call_mirai(task)
task$data
client$terminate()
}

## ------------------------------------------------
## Method `crew_class_launcher$new`
## ------------------------------------------------

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
launcher <- crew_launcher_local()
launcher$start(url = client$url, profile = client$profile)
launcher$launch()
task <- mirai::mirai("result", .compute = client$profile)
mirai::call_mirai(task)
task$data
client$terminate()
}

## ------------------------------------------------
## Method `crew_class_launcher$call`
## ------------------------------------------------

launcher <- crew_launcher_local()
launcher$start(url = "tcp://127.0.0.1:57000", profile = "profile")
launcher$call()
launcher$terminate()

Description

R6 class to launch and manage local process workers.

Details

See crew_launcher_local().

Super class

crew::crew_class_launcher -> crew_class_launcher_local

Active bindings

Methods

Public methods

• crew_class_launcher_local$new()

• crew_class_launcher_local$validate()

• crew_class_launcher_local$launch_worker()

• crew_class_launcher_local$clone()

Method new()

Local launcher constructor.

Usage

crew_class_launcher_local$new(
  name = NULL,
  workers = NULL,
  seconds_interval = NULL,
  seconds_timeout = NULL,
  seconds_launch = NULL,
  seconds_idle = NULL,
  seconds_wall = NULL,
  seconds_exit = NULL,
  tasks_max = NULL,
  tasks_timers = NULL,
  crashes_error = NULL,
  tls = NULL,
  processes = NULL,
  r_arguments = NULL,
  options_metrics = NULL,
  options_local = NULL
)

Arguments

Returns

An R6 object with the local launcher.

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
launcher <- crew_launcher_local(name = client$name)
launcher$start(url = client$url, profile = client$profile)
launcher$launch()
task <- mirai::mirai("result", .compute = client$profile)
mirai::call_mirai(task)
task$data
client$terminate()
}

Method validate()

Validate the local launcher.

Usage

crew_class_launcher_local$validate()

Returns

NULL (invisibly).

Method launch_worker()

Launch a local process worker which will dial into a socket.

Usage

crew_class_launcher_local$launch_worker(call)

Arguments

Details

The call argument is R code that will run to initiate the worker. Together, the launcher, worker, and instance arguments are useful for constructing informative job names.

Returns

A handle object to allow the termination of the worker later on.

Method clone()

The objects of this class are cloneable with this method.

Usage

crew_class_launcher_local$clone(deep = FALSE)

Arguments

See Also

Other plugin_local: crew_controller_local(), crew_launcher_local()

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
launcher <- crew_launcher_local(name = client$name)
launcher$start(url = client$url, profile = client$profile)
launcher$launch()
task <- mirai::mirai("result", .compute = client$profile)
mirai::call_mirai(task)
task$data
client$terminate()
}

## ------------------------------------------------
## Method `crew_class_launcher_local$new`
## ------------------------------------------------

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
launcher <- crew_launcher_local(name = client$name)
launcher$start(url = client$url, profile = client$profile)
launcher$launch()
task <- mirai::mirai("result", .compute = client$profile)
mirai::call_mirai(task)
task$data
client$terminate()
}

Description

Local monitor R6 class

Details

See crew_monitor_local().

Methods

Public methods

• crew_class_monitor_local$dispatchers()

• crew_class_monitor_local$daemons()

• crew_class_monitor_local$workers()

• crew_class_monitor_local$terminate()

Method dispatchers()

List the process IDs of the running mirai dispatcher processes.

Usage

crew_class_monitor_local$dispatchers(user = ps::ps_username())

Arguments

Returns

Integer vector of process IDs of the running mirai dispatcher processes.

Method daemons()

List the process IDs of the locally running mirai daemon processes which are not crew workers.

Usage

crew_class_monitor_local$daemons(user = ps::ps_username())

Arguments

Returns

Integer vector of process IDs of the locally running mirai daemon processes which are not crew workers.

Method workers()

List the process IDs of locally running crew workers launched by the local controller (crew_controller_local()).

Usage

crew_class_monitor_local$workers(user = ps::ps_username())

Arguments

Details

Only the workers running on your local computer are listed. Workers that are not listed include jobs on job schedulers like SLURM or jobs on cloud services like AWS Batch. To monitor those worker processes, please consult the monitor objects in the relevant third-party launcher plugins such as crew.cluster and crew.aws.batch.

Returns

Integer vector of process IDs of locally running crew workers launched by the local controller (crew_controller_local()).

Method terminate()

Terminate the given process IDs.

Usage

crew_class_monitor_local$terminate(pids)

Arguments

Details

Termination happens with the operating system signal given by crew_terminate_signal().

Returns

NULL (invisibly).

See Also

Other monitor: crew_monitor_local()

Description

R6 class for relay configuration.

Details

See crew_relay().

Active bindings

Methods

Public methods

• crew_class_relay$new()

• crew_class_relay$validate()

• crew_class_relay$start()

• crew_class_relay$terminate()

• crew_class_relay$set_from()

• crew_class_relay$set_to()

• crew_class_relay$wait()

Method new()

Relay constructor.

Usage

crew_class_relay$new(throttle)

Arguments

Returns

A crew_relay() object.

Method validate()

Validate the object.

Usage

crew_class_relay$validate()

Returns

NULL (invisibly).

Method start()

Start the relay object.

Usage

crew_class_relay$start()

Returns

NULL (invisibly).

Method terminate()

Terminate the relay object.

Usage

crew_class_relay$terminate()

Returns

NULL (invisibly).

Method set_from()

Set the condition variable to relay from.

Usage

crew_class_relay$set_from(from)

Arguments

Returns

NULL (invisibly).

Method set_to()

Set the condition variable to relay to.

Usage

crew_class_relay$set_to(to)

Arguments

Returns

NULL (invisibly).

Method wait()

Wait until an unobserved task resolves or the timeout is reached. Use the throttle to determine the waiting time.

Usage

crew_class_relay$wait()

Returns

NULL (invisibly).

See Also

Other relay: crew_relay()

Examples

crew_relay()

Description

R6 class for throttle configuration.

Details

See crew_throttle().

Active bindings

Methods

Public methods

• crew_class_throttle$new()

• crew_class_throttle$validate()

• crew_class_throttle$poll()

• crew_class_throttle$accelerate()

• crew_class_throttle$decelerate()

• crew_class_throttle$reset()

• crew_class_throttle$update()

Method new()

Throttle constructor.

Usage

crew_class_throttle$new(
  seconds_max = NULL,
  seconds_min = NULL,
  seconds_start = NULL,
  base = NULL
)

Arguments

Returns

An R6 object with throttle configuration.

Examples

throttle <- crew_throttle(seconds_max = 1)
throttle$poll()
throttle$poll()

Method validate()

Validate the object.

Usage

crew_class_throttle$validate()

Returns

NULL (invisibly).

Method poll()

Poll the throttler.

Usage

crew_class_throttle$poll()

Returns

TRUE if poll() did not return TRUE in the last max seconds, FALSE otherwise.

Method accelerate()

Divide seconds_interval by base.

Usage

crew_class_throttle$accelerate()

Returns

NULL (invisibly). Called for its side effects.

Method decelerate()

Multiply seconds_interval by base.

Usage

crew_class_throttle$decelerate()

Returns

NULL (invisibly). Called for its side effects.

Method reset()

Reset the throttle object so the next poll() returns TRUE, and reset the wait time interval to its initial value.

Usage

crew_class_throttle$reset()

Returns

NULL (invisibly).

Method update()

Reset the throttle when there is activity and decelerate it gradually when there is no activity.

Usage

crew_class_throttle$update(activity)

Arguments

Returns

NULL (invisibly).

See Also

Other throttle: crew_throttle()

Examples

throttle <- crew_throttle(seconds_max = 1)
throttle$poll()
throttle$poll()

## ------------------------------------------------
## Method `crew_class_throttle$new`
## ------------------------------------------------

throttle <- crew_throttle(seconds_max = 1)
throttle$poll()
throttle$poll()

Description

R6 class for TLS configuration.

Details

See crew_tls().

Active bindings

Methods

Public methods

• crew_class_tls$new()

• crew_class_tls$validate()

• crew_class_tls$client()

• crew_class_tls$worker()

• crew_class_tls$url()

Method new()

TLS configuration constructor.

Usage

crew_class_tls$new(
  mode = NULL,
  key = NULL,
  password = NULL,
  certificates = NULL
)

Arguments

Returns

An R6 object with TLS configuration.

Examples

crew_tls(mode = "automatic")

Method validate()

Validate the object.

Usage

crew_class_tls$validate(test = TRUE)

Arguments

Returns

NULL (invisibly).

Method client()

TLS credentials for the crew client.

Usage

crew_class_tls$client()

Returns

NULL or character vector, depending on the mode.

Method worker()

TLS credentials for crew workers.

Usage

crew_class_tls$worker(profile)

Arguments

Returns

NULL or character vector, depending on the mode.

Method url()

Form the URL for crew worker connections.

Usage

crew_class_tls$url(host, port)

Arguments

Returns

Character string with the URL.

See Also

Other tls: crew_tls()

Examples

crew_tls(mode = "automatic")

## ------------------------------------------------
## Method `crew_class_tls$new`
## ------------------------------------------------

crew_tls(mode = "automatic")

Description

Deprecated on 2025-08-26 in crew version 1.2.1.9006. Please use crew_monitor_local() instead.

Usage

crew_clean(
  dispatchers = TRUE,
  workers = TRUE,
  user = ps::ps_username(),
  seconds_interval = 0.25,
  seconds_timeout = 60,
  verbose = TRUE
)

Arguments

Details

Behind the scenes, mirai uses an external R process called a "dispatcher" to send tasks to crew workers. This dispatcher usually shuts down when you terminate the controller or quit your R session, but sometimes it lingers. Likewise, sometimes crew workers do not shut down on their own. The crew_clean() function searches the process table on your local machine and manually terminates any mirai dispatchers and crew workers associated with your user name (or the user name you select in the user argument. Unfortunately, it cannot reach remote workers such as those launched by a crew.cluster controller.

Value

NULL (invisibly). If verbose is TRUE, it does print out a message for every terminated process.

See Also

Other utility: crew_assert(), crew_deprecate(), crew_eval(), crew_random_name(), crew_retry(), crew_terminate_process(), crew_terminate_signal(), crew_worker()

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
crew_clean()
}

Description

Create an R6 wrapper object to manage the mirai client.

Usage

crew_client(
  name = NULL,
  workers = NULL,
  host = NULL,
  port = NULL,
  serialization = NULL,
  profile = crew::crew_random_name(),
  tls = crew::crew_tls(),
  tls_enable = NULL,
  tls_config = NULL,
  seconds_interval = 0.25,
  seconds_timeout = 60,
  retry_tasks = NULL
)

Arguments

See Also

Other client: crew_class_client

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
client$summary()
client$terminate()
}

Description

This function is for developers of crew launcher plugins. Users should use a specific controller helper such as crew_controller_local().

Usage

crew_controller(
  client,
  launcher,
  reset_globals = TRUE,
  reset_packages = FALSE,
  reset_options = FALSE,
  garbage_collection = FALSE,
  crashes_max = 5L,
  backup = NULL,
  auto_scale = NULL
)

Arguments

See Also

Other controller: crew_class_controller

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
launcher <- crew_launcher_local()
controller <- crew_controller(client = client, launcher = launcher)
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()
controller$terminate()
}

Description

Create an R6 object to submit tasks and launch workers through multiple crew controllers.

Usage

crew_controller_group(..., seconds_interval = 0.25)

Arguments

See Also

Other controller_group: crew_class_controller_group

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
persistent <- crew_controller_local(name = "persistent")
transient <- crew_controller_local(
  name = "transient",
  tasks_max = 1L
)
group <- crew_controller_group(persistent, transient)
group$start()
group$push(name = "task", command = sqrt(4), controller = "transient")
group$wait()
group$pop()
group$terminate()
}

Description

Create an R6 object to submit tasks and launch workers on local processes.

Usage

crew_controller_local(
  name = NULL,
  workers = 1L,
  host = "127.0.0.1",
  port = NULL,
  tls = crew::crew_tls(),
  tls_enable = NULL,
  tls_config = NULL,
  serialization = NULL,
  profile = crew::crew_random_name(),
  seconds_interval = 0.25,
  seconds_timeout = 60,
  seconds_launch = 30,
  seconds_idle = 300,
  seconds_wall = Inf,
  seconds_exit = NULL,
  retry_tasks = NULL,
  tasks_max = Inf,
  tasks_timers = 0L,
  reset_globals = TRUE,
  reset_packages = FALSE,
  reset_options = FALSE,
  garbage_collection = FALSE,
  crashes_error = NULL,
  launch_max = NULL,
  r_arguments = c("--no-save", "--no-restore"),
  crashes_max = 5L,
  backup = NULL,
  options_metrics = crew::crew_options_metrics(),
  options_local = crew::crew_options_local(),
  local_log_directory = NULL,
  local_log_join = NULL
)

Arguments

See Also

Other plugin_local: crew_class_launcher_local, crew_launcher_local()

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
controller <- crew_controller_local()
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()
controller$terminate()
}

Description

The sequential controller runs tasks on the same R process where the controller object exists. Tasks run sequentially rather than in parallel.

Usage

crew_controller_sequential(
  reset_globals = TRUE,
  reset_packages = FALSE,
  reset_options = FALSE,
  garbage_collection = FALSE
)

Arguments

See Also

Other sequential controllers: crew_class_controller_sequential

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
controller <- crew_controller_sequential()
controller$push(name = "task", command = sqrt(4))
controller$pop()
}

Description

Show an informative warning when a crew feature is deprecated.

Usage

crew_deprecate(
  name,
  date,
  version,
  alternative,
  condition = "warning",
  value = "x",
  skip_cran = FALSE,
  frequency = "always"
)

Arguments

Value

NULL (invisibly). Throws a warning if a feature is deprecated.

See Also

Other utility: crew_assert(), crew_clean(), crew_eval(), crew_random_name(), crew_retry(), crew_terminate_process(), crew_terminate_signal(), crew_worker()

Examples

suppressWarnings(
  crew_deprecate(
    name = "auto_scale",
    date = "2023-05-18",
    version = "0.2.0",
    alternative = "use the scale argument of push(), pop(), and wait()."
  )
)

Description

Not a user-side function. Do not call directly.

Usage

crew_eval(
  command,
  name,
  data = list(),
  globals = list(),
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  reset_globals = TRUE,
  reset_packages = FALSE,
  reset_options = FALSE,
  garbage_collection = FALSE
)

Arguments

Details

The crew_eval() function evaluates an R expression in an encapsulated environment and returns a monad with the results, including warnings and error messages if applicable. The random number generator seed, globals, and global options are restored to their original values on exit.

Value

A monad object with results and metadata.

See Also

Other utility: crew_assert(), crew_clean(), crew_deprecate(), crew_random_name(), crew_retry(), crew_terminate_process(), crew_terminate_signal(), crew_worker()

Examples

crew_eval(quote(1 + 1), name = "task_name")

Description

This function is useful for inheriting argument documentation in functions that create custom third-party launchers. See ⁠@inheritParams crew::crew_launcher⁠ in the source code file of crew_launcher_local().

Usage

crew_launcher(
  name = NULL,
  workers = 1L,
  seconds_interval = 0.25,
  seconds_timeout = 60,
  seconds_launch = 30,
  seconds_idle = 300,
  seconds_wall = Inf,
  seconds_exit = NULL,
  tasks_max = Inf,
  tasks_timers = 0L,
  reset_globals = NULL,
  reset_packages = NULL,
  reset_options = NULL,
  garbage_collection = NULL,
  crashes_error = NULL,
  launch_max = NULL,
  tls = crew::crew_tls(),
  processes = NULL,
  r_arguments = c("--no-save", "--no-restore"),
  options_metrics = crew::crew_options_metrics()
)

Arguments

Auto-scaling

crew launchers implement auto-scaling in the scale() method. When the task load increases, the number of workers increases in response to demand. When the task load decreases, the workers start to exit. This behavior happens dynamically over the course of a workflow, and it can be tuned with arguments seconds_interval, seconds_wall, and tasks_max.

tasks_max is special: it determines not only the number of tasks a worker runs before exiting, it also determines how often auto-scaling runs. If tasks_max is finite, then crew uses an aggressive deterministic exponential backoff algorithm to determine how frequently to auto-scale (see crew_throttle()). But if tasks_max is Inf, then crew only scales at equally-spaced time intervals of seconds_interval to allow enough pending tasks to accumulate for job arrays. This last part is important because auto-scaling too frequently could lead to hundreds of separate job arrays with only job per array (as opposed to the desired outcome of 1 or 2 arrays with many jobs each).

See Also

Other launcher: crew_class_launcher

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
launcher <- crew_launcher_local()
launcher$start(url = client$url, profile = client$profile)
launcher$launch()
task <- mirai::mirai("result", .compute = client$profile)
mirai::call_mirai(task)
task$data
client$terminate()
}

Description

Create an R6 object to launch and maintain local process workers.

Usage

crew_launcher_local(
  name = NULL,
  workers = 1L,
  seconds_interval = 0.25,
  seconds_timeout = 60,
  seconds_launch = 30,
  seconds_idle = Inf,
  seconds_wall = Inf,
  seconds_exit = NULL,
  tasks_max = Inf,
  tasks_timers = 0L,
  reset_globals = NULL,
  reset_packages = NULL,
  reset_options = NULL,
  garbage_collection = NULL,
  crashes_error = NULL,
  launch_max = NULL,
  tls = crew::crew_tls(),
  r_arguments = c("--no-save", "--no-restore"),
  options_metrics = crew::crew_options_metrics(),
  options_local = crew::crew_options_local(),
  local_log_directory = NULL,
  local_log_join = NULL
)

Arguments

See Also

Other plugin_local: crew_class_launcher_local, crew_controller_local()

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
client$start()
launcher <- crew_launcher_local(name = client$name)
launcher$start(url = client$url, profile = client$profile)
launcher$launch()
task <- mirai::mirai("result", .compute = client$profile)
mirai::call_mirai(task)
task$data
client$terminate()
}

Description

Create an R6 object to monitor local processes created by crew and mirai.

Usage

crew_monitor_local()

See Also

Other monitor: crew_class_monitor_local

Description

Options for the local crew launcher.

Usage

crew_options_local(log_directory = NULL, log_join = TRUE)

Arguments

Value

A classed list of options for the local launcher.

See Also

Other options: crew_options_metrics()

Examples

crew_options_local()

Description

crew_options_metrics() configures the crew to record resource usage metrics (such as CPU and memory usage) for each running worker. To be activate resource usage logging, the autometric R package version 0.1.0 or higher must be installed.

Logging happens in the background (through a detached POSIX) so as not to disrupt the R session. On Unix-like systems, crew_options_metrics() can specify ⁠/dev/stdout⁠ or ⁠/dev/stderr⁠ as the log files, which will redirect output to existing logs you are already using. autometric::log_read() and autometric::log_plot() can read and visualize resource usage data from multiple log files, even if those files are mixed with other messages.

Usage

crew_options_metrics(path = NULL, seconds_interval = 5)

Arguments

Value

A classed list of options for logging resource usage metrics.

See Also

Other options: crew_options_local()

Examples

crew_options_metrics()

Description

Generate a random string that can be used as a name for a worker or task.

Usage

crew_random_name(n = 8L)

Arguments

Details

The randomness is not reproducible and cannot be set with e.g. set.seed() in R.

Value

A random character string.

See Also

Other utility: crew_assert(), crew_clean(), crew_deprecate(), crew_eval(), crew_retry(), crew_terminate_process(), crew_terminate_signal(), crew_worker()

Examples

crew_random_name()

Description

Create an R6 crew relay object.

Usage

crew_relay(throttle = crew_throttle())

Arguments

Details

A crew relay object keeps the signaling relationships among condition variables.

Value

An R6 crew relay object.

See Also

Other relay: crew_class_relay

Examples

crew_relay()

Description

Repeatedly retry a function while it keeps returning FALSE and exit the loop when it returns TRUE

Usage

crew_retry(
  fun,
  args = list(),
  seconds_interval = 0.25,
  seconds_timeout = 60,
  max_tries = Inf,
  error = TRUE,
  message = character(0),
  envir = parent.frame(),
  assertions = TRUE
)

Arguments