How to pass a custom function inside a ForEach-Object -Parallel

Question

I can't find a way to pass the function. Just variables.

Any ideas without putting the function inside the ForEach loop?

function CustomFunction {
    Param (
        $A
    )
    Write-Host $A
}

$List = "Apple", "Banana", "Grape" 
$List | ForEach-Object -Parallel {
    Write-Host $using:CustomFunction $_
}

Answer 1

The solution isn't quite as straightforward as one would hope:

# Sample custom function.
function Get-Custom {
  Param ($A)
  "[$A]"
}

# Get the function's definition *as a string*
$funcDef = ${function:Get-Custom}.ToString()

"Apple", "Banana", "Grape"  | ForEach-Object -Parallel {
  # Define the function inside this thread...
  ${function:Get-Custom} = $using:funcDef
  # ... and call it.
  Get-Custom $_
}

^{Note: This answer contains an analogous solution for using a script block from the caller's scope in a ForEach-Object -Parallel script block.}

• Note: If your function were defined in a module that is placed in one of the locations known to the module-autoloading feature, your function calls would work as-is with ForEach-Object -Parallel, without extra effort - but each thread would incur the cost of (implicitly) importing the module.

• The above approach is necessary, because - aside from the current location (working directory) and environment variables (which apply process-wide) - the threads that ForEach-Object -Parallel creates do not see the caller's state, notably neither with respect to variables nor functions (and also not custom PS drives and imported modules).

  • Update: js2010's helpful answer shows a more straightforward solution that passes a System.Management.Automation.FunctionInfo instance, obtained via Get-Command, which can be invoked directly with &. The only caveat is that the original function should be side-effect-free, i.e. should operate solely based on parameter or pipeline inputs, without relying on the caller's state, notably its variables, as that could lead to thread-safety issues. The stringification technique above implicitly prevents any problematic references to the caller's state, because the function body is rebuilt in each thread's context.

• As of PowerShell 7.1, an enhancement is being discussed in GitHub issue #12240 to support copying the caller's state to the threads on demand, which would make the caller's functions available.

Note that making do without the aux. $funcDef variable and trying to redefine the function with ${function:Get-Custom} = ${using:function:Get-Custom} is tempting, but ${function:Get-Custom} is a script block, and the use of script blocks with the $using: scope specifier is explicitly disallowed.

• However, ${function:Get-Custom} = ${using:function:Get-Custom} would work with Start-Job; see this answer for an example.

• It would also work with Start-ThreadJob, where you could even do & ${using:function:Get-Custom} $_, because ${using:function:Get-Custom} is preserved as a script block (unlike with Start-Job, where it is deserialized as a string, which is itself surprising behavior - see GitHub issue #11698). However, it is unclear whether this behavior is supported by design, because it is subject to the same potential cross-thread issues noted above.

${function:Get-Custom} is an instance of namespace variable notation, which allows you to both get a function (its body as a [scriptblock] instance) and to set (define) it, by assigning either a [scriptblock] or a string containing the function body.

Answer 2

I just figured out another way using get-command, which works with the call operator. $a ends up being a FunctionInfo object. EDIT: I'm told this isn't thread safe, but I don't understand why.

function hi { 'hi' }
$a = get-command hi
1..3 | foreach -parallel { & $using:a }

hi
hi
hi

Answer 3

So I figured out another little trick that may be useful for people trying to add the functions dynamically, particularly if you might not know the name of it beforehand, such as when the functions are in an array.

# Store the current function list in a variable
$initialFunctions=Get-ChildItem Function:

# Source all .ps1 files in the current folder and all subfolders
Get-ChildItem . -Recurse | Where-Object { $_.Name -like '*.ps1' } |
     ForEach-Object { . "$($_.FullName)" }

# Get only the functions that were added above, and store them in an array
$functions = @()
Compare-Object $initialFunctions (Get-ChildItem Function:) -PassThru |
    ForEach-Object { $functions = @($functions) + @($_) }

1..3 | ForEach-Object -Parallel {
    # Pull the $functions array from the outer scope and set each function
    # to its definition
    $using:functions | ForEach-Object {
        Set-Content "Function:$($_.Name)" -Value $_.Definition
    }
    # Call one of the functions in the sourced .ps1 files by name
    SourcedFunction $_
}

The main "trick" of this is using Set-Content with Function: plus the function name, since PowerShell essentially treats each entry of Function: as a path.

This makes sense when you consider the output of Get-PSDrive. Since each of those entries can be used as a "Drive" in the same way (i.e., with the colon).

Answer 4

If you're a pro, of course you added the -Parallel flag on purpose because you really needed parallel processing (so see the accepted answer)

Newbies, like me, might consider removing the -Parallel flag because you didn't realize the code you copied from somewhere else doesn't really need it.. and then your function calls just work like normal.