Top 3 Highlights from the Education and Certification Lounge at VMworld 2019 US

This post was originally published on this site

With San Francisco behind us and Barcelona ahead, it’s official: VMworld season has arrived again!   Didn’t make it to VMworld US in San Francisco last month? That’s too bad! Nothing comes close to the in-person VMworld experience. But the Education Services team still wants to give you bit of an idea of what you

The post Top 3 Highlights from the Education and Certification Lounge at VMworld 2019 US appeared first on VMware Education Services.

F5 iRule Setup and Notes for VMware vCloud Director Accessibility

This post was originally published on this site

Recently, I’ve had several providers that have had to manage and/or update their F5 iRules for secure accessibility for vCD tenants and providers. F5 has a nice deployment guide here. F5 provides a solid solution for VIPs and Load Balancer capabilities which I see often between many of our global VMware Cloud Providers. With the … Continue reading “F5 iRule Setup and Notes for VMware vCloud Director Accessibility”

The post F5 iRule Setup and Notes for VMware vCloud Director Accessibility appeared first on Clouds, etc..

Deep Dive into VMware Cloud Management at VMworld

This post was originally published on this site

With just over a month to VMworld Europe in Barcelona, we are putting the finishing touches on another dazzling list of fantastic events and sessions. The Cloud Management Business Unit, the home of vRealize Suite of cloud management-focused products, has prepared a great program for you, full of both useful content and networking fun. To help you navigate

The post Deep Dive into VMware Cloud Management at VMworld appeared first on VMworld Blog.

Feature in Five: vSAN Cluster Quickstart

This post was originally published on this site

Welcome to “Feature in Five,” our byte-sized videos of industry-leading VMware product solutions, all in—you guessed—five short minutes! Last time, we video-profiled vSAN Data Placement and Availability; in this edition, we turn our attention to the vSAN Cluster Quickstart wizard. vSAN 6.7 U1 and vSphere 6.7 U1 introduced Quickstart, a cluster creation wizard that guides

The post Feature in Five: vSAN Cluster Quickstart appeared first on Virtual Blocks.

Join Your Customers at VMworld 2019 | 4-7 November in Barcelona

This post was originally published on this site

The VMworld 2019 Europe full conference pass will give you complimentary access to participate in the partner program, Partner Forum @ VMworld 2019. View our top 5 Partner Forum activities and resources below and start building your conference agenda today. Attend both the Partner General Session and Partner Networking Reception on Monday, 4 November. During […]

The post Join Your Customers at VMworld 2019 | 4-7 November in Barcelona appeared first on Partner News.

DSC Resource Kit Release September 2019

This post was originally published on this site

We just released the DSC Resource Kit!

This release includes updates to 15 DSC resource modules. In the past 6 weeks, 160 pull requests have been merged and 68 issues have been closed, all thanks to our amazing community!

The modules updated in this release are:

  • ActiveDirectoryCSDsc 4.1.0.0
  • ActiveDirectoryDsc 4.1.0.0
  • ComputerManagementDsc 7.0.0.0
  • DFSDsc 4.4.0.0
  • NetworkingDsc 7.4.0.0
  • SecurityPolicyDsc 2.10.0.0
  • SqlServerDsc 13.2.0.0
  • xDnsServer 1.15.0.0
  • xExchange 1.29.0.0
  • xFailOverCluster 1.13.0.0
  • xPSDesiredStateConfiguration 8.10.0.0
  • xRemoteDesktopSessionHost 1.9.0.0
  • xSCSMA 2.1.0.0
  • xWebAdministration 2.8.0.0

For a detailed list of the resource modules and fixes in this release, see the Included in this Release section below.

Our latest community call for the DSC Resource Kit was last Wednesday, September 11. A recording of the call is posted on the PowerShell YouTube channel. You can join us for the next call at 12PM (Pacific time) on August 28th to ask questions and give feedback about your experience with the DSC Resource Kit.

The next DSC Resource Kit release will be on Wednesday, October 9.

We strongly encourage you to update to the newest version of all modules using the PowerShell Gallery, and don’t forget to give us your feedback in the comments below, on GitHub, or on Twitter (@PowerShell_Team)!

Please see our documentation here for information on the support of these resource modules.

Included in this Release

You can see a detailed summary of all changes included in this release in the table below. For past release notes, go to the README.md or CHANGELOG.md file on the GitHub repository page for a specific module (see the How to Find DSC Resource Modules on GitHub section below for details on finding the GitHub page for a specific module).

Module Name Version Release Notes
ActiveDirectoryCSDsc 4.1.0.0
  • AdcsCertificationAuthoritySettings:
    • Fix grammar in the resource README.md.
  • Fix minor style issues in statement case.
ActiveDirectoryDsc 4.1.0.0
    • We could not add the change log to the release notes due to the length of the change log. What have change in this release can be found here

https://github.com/PowerShell/ActiveDirectoryDsc/blob/dev/CHANGELOG.md#4100

    .
ComputerManagementDsc 7.0.0.0
  • ScheduledTask:
    • Better compatibility with Group LogonType when passing BuiltIn groups through ExecuteAsCredential
      • Primary use case is “BUILTINUsers”
      • Use the ExecuteAsCredential property to pass the username The PSCredential needs a non-null that is ignored
    • Delay property not handled properly on AtLogon and AtStartup trigger – Fixes Issue 230
    • Changed Get-ScheduledTask calls to ScheduledTasksGet-ScheduledTask to avoid name clash with Carbon module. Fixes Issue 248
    • Cast MultipleInstances value returned by Get-TargetResource to string – fixes Issue 255
  • PendingReboot:
    • Migrated xPendingReboot from xPendingReboot and renamed to PendingReboot.
    • Converted to meet HQRM guidelines – Fixes Issue 12.
    • Changed SkipCcmClientSDK parameter to default to $true – Fixes Issue 13.
    • Fixed Test-TargetResource so that if ConfigMgr requires a reboot then the pending reboot will be set – Fixes Issue 26.
    • Refactored Test-TargetResource to reduce code duplication and move to a data driven design.
    • Refactored Get-TargetResource by adding a new function Get-PendingRebootState so that Test-TargetResource no longer needed to use Get-TargetResource. This eliminated the need to include write parameters in Get-TargetResource.
    • Converted the call to Invoke-WmiMethod to Invoke-CimMethod.
    • Deleted the code that removes the regRebootLocations variable at the end of the resource as it appears to serve no purpose.
  • Correct all tests to meet Pester 4.0 standards.
  • RemoteDesktopAdmin:
    • New resource for configuring Remote Desktop for Administration – fixes Issue 224.
  • Updated common function Test-DscParameterState to support ordered comparison of arrays by copying function and tests from NetworkingDsc – fixes Issue 250.
  • BREAKING CHANGE: ScheduledTask:
    • Correct output type of DaysInterval,StartTime,WeeksDaysOfWeek, and WeeksInterval parameters from Get-TargetResource to match MOF.
    • Refactored Get-TargetResource to remove parameters that are not key or required – fixes Issue 249.
    • Added function Test-DateStringContainsTimeZone to determine if a string containing a date time includes a time zone.
    • Enable verbose preference to be passed through to Test-DscParameterState.
    • Changed Test-TargetResource so that StartTime is only compared for trigger types Daily,Weekly or Once.
  • Fix minor style issues in statement case.
DFSDsc 4.4.0.0
  • Fix example publish to PowerShell Gallery by adding gallery_api environment variable to AppVeyor.yml – fixes Issue 91.
  • Fix minor style issues in statement case.
NetworkingDsc 7.4.0.0
  • Added Comment Based Help for New-NotImplementedException common function – fixes Issue 411.
  • Added common function “Format-Win32NetworkADapterFilterByNetConnectionID” to properly accept wild cards for Win32_NetworkAdapter filters.
  • Updated MSFT_Netbios to use “Format-Win32NetworkADapterFilterByNetConnectionID”
  • Corrected minor style and consistency issues in NetworkingDsc.Common.tests.ps1 and NetworkingDsc.Common.ps1.
  • Changed verbose messages in Test-DscParameterState to include full type name.
  • Fixed bug in Test-DscParameterState that causes it to return true when both the current array and desired array is empty.
  • Fix minor style issues in statement case.
SecurityPolicyDsc 2.10.0.0
  • Changes to SecurityPolicyDsc
    • Opt-in to the following DSC Resource Common Meta Tests:
      • Common Tests – Validate Module Files
      • Common Tests – Validate Script Files
      • Common Tests – Validate Markdown Files
      • Common Tests – Required Script Analyzer Rules
      • Common Tests – Flagged Script Analyzer Rules
      • Common Tests – New Error-Level Script Analyzer Rules
      • Common Tests – Custom Script Analyzer Rules
      • Common Tests – Validate Markdown Links
      • Common Tests – Relative Path Length
      • Common Tests – Validate Example Files
      • Common Tests – Validate Example Files To Be Published
    • Fix keywords to lower-case to align with guideline.
SqlServerDsc 13.2.0.0
  • Changes to SqlServerDsc
    • Fix keywords to lower-case to align with guideline.
    • Fix keywords to have space before a parenthesis to align with guideline.
xDnsServer 1.15.0.0
xExchange 1.29.0.0
  • Enable Script Analyzer default rules
  • Fixed keywords in upper case
xFailOverCluster 1.13.0.0
  • Updated the xCluster test method to return true if a node is joined to the cluster but is in a Paused state.
xPSDesiredStateConfiguration 8.10.0.0
  • Changes to xPSDesiredStateConfiguration
    • Fix keywords to lower-case to align with guideline.
  • Added SMB PullServer support for publishing.
xRemoteDesktopSessionHost 1.9.0.0
  • Changes to xRDRemoteApp
    • Fixing typo in parameter name when calling the function ValidateCustomModeParameters (issue 50).
  • Changes to xRDSessionDeployment
    • When RDMS service does not exist the Get-TargetResource will no longer throw an error (issue 47).
  • Rename Tests/Unit folder to use upper case on first letter.
  • Update appveyor.yml to use the default template.
  • Added default template files .codecov.yml, .gitattributes, and .gitignore, and .vscode folder.
  • xRDSessionCollectionConfiguration:
    • Changed CollectionName variable validation max length to 256
  • xRDSessionCollection
    • Changed CollectionName variable validation max length to 256
  • xRDRemoteApp
    • Changed CollectionName variable validation max length to 256
xSCSMA 2.1.0.0
  • Update appveyor.yml to use the default template.
  • Added default template files .codecov.yml, .gitattributes, and .gitignore, and .vscode folder.
  • Closed issue 29 – Web bindings fail due to hardcoded WSE
  • Switched from Get-WmiObject Win32_Product to Get-ItemProperty for identifer number
xWebAdministration 2.8.0.0
  • Fix multiple HTTPS bindings on one xWebsite receiving the first binding”s certificate 332
    • Added unit regression test
  • Changes to xWebsite
    • Added ServerAutoStart (controls website autostart) and changed documentation for ServiceAutoStartEnabled (controls application auto-initialization). Fixes 325.
    • Fix multiple HTTPS bindings on one xWebsite receiving the first binding”s certificate 332
      • Added unit regression test
    • Changes to xWebAppPool
      • Fix false Test-TargetResource failure for logEventOnRecycle if items in the Configuration property are specified in a different order than IIS natively stores them 434
    • Changes to xIisModule
      • Fixed the parameters specification for the internal Get-IISHandler and Remove-IISHandler function

How to Find Released DSC Resource Modules

To see a list of all released DSC Resource Kit modules, go to the PowerShell Gallery and display all modules tagged as DSCResourceKit. You can also enter a module’s name in the search box in the upper right corner of the PowerShell Gallery to find a specific module.

Of course, you can also always use PowerShellGet (available starting in WMF 5.0) to find modules with DSC Resources:

# To list all modules that tagged as DSCResourceKit
Find-Module -Tag DSCResourceKit 
# To list all DSC resources from all sources 
Find-DscResource

Please note only those modules released by the PowerShell Team are currently considered part of the ‘DSC Resource Kit’ regardless of the presence of the ‘DSC Resource Kit’ tag in the PowerShell Gallery.

To find a specific module, go directly to its URL on the PowerShell Gallery:
http://www.powershellgallery.com/packages/< module name >
For example:
http://www.powershellgallery.com/packages/xWebAdministration

How to Install DSC Resource Modules From the PowerShell Gallery

We recommend that you use PowerShellGet to install DSC resource modules:

Install-Module -Name < module name >

For example:

Install-Module -Name xWebAdministration

To update all previously installed modules at once, open an elevated PowerShell prompt and use this command:

Update-Module

After installing modules, you can discover all DSC resources available to your local system with this command:

Get-DscResource

How to Find DSC Resource Modules on GitHub

All resource modules in the DSC Resource Kit are available open-source on GitHub.
You can see the most recent state of a resource module by visiting its GitHub page at:
https://github.com/PowerShell/< module name >
For example, for the CertificateDsc module, go to:
https://github.com/PowerShell/CertificateDsc.

All DSC modules are also listed as submodules of the DscResources repository in the DscResources folder and the xDscResources folder.

How to Contribute

You are more than welcome to contribute to the development of the DSC Resource Kit! There are several different ways you can help. You can create new DSC resources or modules, add test automation, improve documentation, fix existing issues, or open new ones.
See our contributing guide for more info on how to become a DSC Resource Kit contributor.

If you would like to help, please take a look at the list of open issues for the DscResources repository.
You can also check issues for specific resource modules by going to:
https://github.com/PowerShell/< module name >/issues
For example:
https://github.com/PowerShell/xPSDesiredStateConfiguration/issues

Your help in developing the DSC Resource Kit is invaluable to us!

Questions, comments?

If you’re looking into using PowerShell DSC, have questions or issues with a current resource, or would like a new resource, let us know in the comments below, on Twitter (@PowerShell_Team), or by creating an issue on GitHub.

Michael Greene
Principal Program Manager
PowerShell DSC Team
@migreene (Twitter)
@mgreenegit (GitHub)

The post DSC Resource Kit Release September 2019 appeared first on PowerShell.

PowerShell 7 Preview 4

This post was originally published on this site

We continue to make progress towards our PowerShell 7 release which currently is targeting December 2019 for a Release Candidate and January 2020 for General Availability and will be our first LTS (Long Term Servicing) release!

Please see the previous blog post on Preview 3 for more details about LTS and also Windows PowerShell compatibility.

Preview 4 contains a number of bug fixes, but also new features which I’ll cover in this blog post.

New Features in Preview 4

This is just a small part of the entire changelog. New experimental features in this preview from the community and also the PowerShell team:

Ternary Operator

The ternary operator is popular among C# developers due to its terseness which can improve readability if you are familiar with this operator.

This operator is completely opt-in so if you prefer to use if..else instead, you can certainly continue to do that.

gif

Start-Job -WorkingDirectory

Those of you familiar with the Start-Job cmdlet will have encountered that the new PowerShell process started to handle the job will have different working directory on Windows PowerShell and PowerShell Core and it can sometimes be not what you expected. This parameter was added to allow you to specify the working directory of the new job process before your script block runs!

gif

$ErrorActionPreference = “Break”

This feature comes from a well known PowerShell MVP Kirk Munro. Basically, if you set $ErrorActionPreference to Break, then when there is an error it will drop you into the debugger immediately!

gif

Invoke-DscResource

With this change, you can now leverage DSC Resources while by-passing the LCM (Local Configuration Manager). This means that you can author your own LCM or simply leverage existing DSC Resources within your scripts and this also works cross platform!

Note that binary DSC Resources are not supported!

gif

DSC Configuration Compilation

Previously if you authored a DSC Configuration script, you would need to use a Windows machine to compile it to a mof file to deploy onto your managed node. Starting with Preview4, you can now perform DSC compilation on non-Windows systems.

Note that this is work in progress with some known issues.

gif

Testing the MSIX package

Recently, we started publishing a MSIX package for Windows. This will eventually allow us to publish PowerShell 7 to the Windows Store. For now, if you wish to try out this package, you must be in Developer Mode and use Add-AppxPackage to install it. Double clicking it from the Windows Shell will not allow you to install the developer signed package.

Closing

Although this blog post focuses on new features, this release also contains many bug fixes as well as targeted performance improvements.

You can always get the latest version of PowerShell from https://aka.ms/get-powershell.

Expect more new features from the community and the PowerShell team in future Preview releases!

Steve Lee
PowerShell Team

The post PowerShell 7 Preview 4 appeared first on PowerShell.

Release of PowerShell Script Analyzer (PSScriptAnalyzer) 1.18.2

This post was originally published on this site

In keeping with the tradition of releasing improvements to PSScriptAnalyzer more often, we’re happy to announce that 1.18.12 is now available! As a dependency of PowerShell Editor Services (a module used by editor extensions like the PowerShell Visual Studio Code extension), this release is motivated by a desire to further stabilize our editor experience. At the moment, the Visual Studio Code PowerShell extension still ships with PSScriptAnalyzer 1.18.0. After fixing some undesirable edge cases between 1.18.1 and 1.18.2, we intend to ship an update to the Visual Studio Code extension that will include 1.18.2.

The blocking issue that it resolves is quite technical and should not concern end-users, but for those who are interested: starting with1.18.1, a performance optimization was added whereby we started to share and cache a PowerShell runspace pool instead of creating a new one for every command invocation. However, it turns out that there is an edge case where, when dealing with specific commands from thePackageManagementmodule, the runspace pool can get into a deadlock, which causes the execution of PSScriptAnalyzer to hang indefinitely. This is due to a bug inPackageManagementitself (a very unfortunate asynchronous API call that leads to the deadlock) but also PowerShell itself, which should be able to handle bad scenarios like this. Therefore, a workaround for this had to be implemented in PSScriptAnalyzer by blacklisting the PackageManagement commands.

Given that the other changes in this release are mainly fixes and small enhancements, we decided to not bump the minor version number. We ask that the community participate in testing and giving feedback on this update before it ships by default in the Visual Studio Code extension. You can make this new update with the Visual Studio Code extension start by executing the following command:

Install-Module -Name PSScriptAnalyzer -Repository PSGallery -Scope CurrentUser

Should you find that there are changes that you are not happy with, please report them here.

Optionally, you can roll back to the default included version of PSScriptAnalyzer by running Uninstall-Module -Name PSScriptAnalyzer.

In this release, we’ve made the following fixes

  • PipelineIndentation: More edge cases when using non-default values of this setting (NoIndentation in the Visual Studio Code extension) were fixed. This feature was only introduced in1.18.0and we hope the be closer to a state now where we could potentially change the default.
  • New compatibility rule profiles were added for non-Windows OSs on PowerShell 7 (preview). Additionally, fixes were made to profile generation to support macOS and Linux.
  • A fix was made to PSCloseBrace to correctly flag the closing brace of a one-line hashtable, correcting some broken formatting.

Enhancements were made in the following areas

  • When using settings files, error messages are now much more actionable.
PS> Invoke-ScriptAnalyzer -Settings /tmp/MySettings.psd1 -ScriptDefinition 'gci'

Invoke-ScriptAnalyzer : includerule is not a valid key in the settings hashtable.
Valid keys are CustomRulePath, ExcludeRules, IncludeRules, IncludeDefaultRules,
RecurseCustomRulePath, Rules and Severity. 
...

  • PSScriptAnalyzer has a logo now thanks to the community member @adilio
  • The formatter was enhanced to also take commented lines into account in multi-line commands
  • The formatter was enhanced to optionally allow correction of aliases as well. With this change, a setting in the Visual Studio Code extension will soon be made available to configure this. By default, this setting will not be on for the moment. We are open to feedback: while there are very likely a few people that would love for it to be enabled, it may upset others.
  • UseDeclaredVarsMoreThanAssignmentsnow also takes into account the usage of Get-Variable with an array of variables and usage of the named parameter -Name

We’ve also made some changes in our GitHub repository and changed the default branch from development to master to simplify the development workflow and be consistent with other repositories in the PowerShell organization. If you have a fork of the project, you will need to make this change in your fork as well or remember to use master as a base and open pull requests against master. This also means that the next version of the Visual Studio Code extension will point tomasterfor the documentation of PSScriptAnalyzer’s rules.

The Changelog has more details if you want to dig further.

Future Directions

We are thinking of following an approach similar to the Visual Studio Code extension where we make a version 2.0 at that drops support for PowerShell version 3 and 4. One of the next changes could be to improve how PowerShellEditorServices calls into PSScriptAnalyzer: currently, Editor Services uses the PSScriptAnalyzer PowerShell cmdlets which means that we have to create an entire instance of PowerShell for these invocations. Knowing that bothPowerShellEditorServicesandPSScriptAnalyzerare binary .NET modules, we could directly call into PSScriptAnalyzer’s .NET code by publishing a NuGet package of PSScriptAnalyzer with suitable public APIs. Given that PSScriptAnalzyer currently performs a conditional compilation for each PowerShell version (3, 4, 5, and 6+), dropping support for version 4 and 5 could help make the aforementioned move to an API model much easier to implement. Please give feedback if your use case ofPSScriptAnalyzerwould be impacted by this.

On behalf of the Script Analyzer team,

Christoph Bergmeister, Project Maintainer from the community, BJSS
Jim Truher, Senior Software Engineer, Microsoft

The post Release of PowerShell Script Analyzer (PSScriptAnalyzer) 1.18.2 appeared first on PowerShell.

PowerShell ForEach-Object Parallel Feature

This post was originally published on this site

PowerShell ForEach-Object Parallel Feature

PowerShell 7.0 Preview 3 is now available with a new ForEach-Object Parallel Experimental feature. This feature is a great new tool for parallelizing work, but like any tool, it has its uses and drawbacks.

This article describes this new feature, how it works, when to use it and when not to.

What is ForEach-Object -Parallel?

ForEach-Object -Parallel is a new parameter set added to the existing PowerShell ForEach cmdlet.

ForEach-Object -Parallel <scriptblock> [-InputObject <psobject>] [-ThrottleLimit <int>] [-TimeoutSeconds <int>] [-AsJob] [-WhatIf] [-Confirm] [<CommonParameters>]

 

Normally, when you use the ForEach-Object cmdlet, each object piped to the cmdlet is processed sequentially.

PS C:> 1..5 | ForEach-Object { "Hello $_"; sleep 1 } Hello 1 Hello 2 Hello 3 Hello 4 Hello 5 PS C:> (Measure-Command { 1..5 | ForEach-Object { "Hello $_"; sleep 1 } }).Seconds 5

But with the new ForEach-Object -Parallel parameter set, you can run all script in parallel for each piped input object.

PS C:> 1..5 | ForEach-Object -Parallel { "Hello $_"; sleep 1; } -ThrottleLimit 5
Hello 1
Hello 3
Hello 2
Hello 4
Hello 5

PS C:> (Measure-Command { 1..5 | ForEach-Object -Parallel { "Hello $_"; sleep 1; } -ThrottleLimit 5 }).Seconds
1

Because each script block in the ForEach-Object example above takes 1 second to run, running all five in parallel takes only one second instead of 5 seconds when run sequentially.

Since the script blocks are run in parallel for each of the 1-5 piped input integers, the order of execution is not guaranteed. The -ThrottleLimit parameter limits the number of script blocks running in parallel at a given time, and its default value is 5.

This new feature also supports jobs, where you can choose to have a job object returned instead of having results written to the console.

PS C:> $Job = 1..5 | ForEach-Object -Parallel { "Hello $_"; sleep 1; } -ThrottleLimit 5 -AsJob PS C:> $job | Wait-Job | Receive-Job Hello 1 Hello 2 Hello 3 Hello 5 Hello 4

ForEach-Object -Parallel is not the same as the foreach language keyword

Don’t confuse ForEach-Object cmdlet with PowerShell’s foreach keyword. The foreach keyword does not handle piped input but instead iterates over an enumerable object. There is currently no parallel support for the foreach keyword.

PS C:> foreach ($item in (1..5)) { "Hello $item" }
Hello 1
Hello 2
Hello 3
Hello 4
Hello 5

How does it work?

The new ForEach-Object -Parallel parameter set uses existing PowerShell APIs for running script blocks in parallel. These APIs have been around since PowerShell v2, but are cumbersome and difficult to use correctly. This new feature makes it much easier to run script blocks in parallel. But there is a fair amount of overhead involved and many times there is no gain in running scripts in parallel, and in fact it can end up being significantly slower than running ForEach-Object normally.

PowerShell currently supports parallelism in three main categories.

  1. PowerShell remoting. Here PowerShell sends script to external machines to run, using PowerShell’s remoting system.
  2. PowerShell jobs. This is the same as remoting except that script is run in separate processes on the local machine, rather than on external machines.
  3. PowerShell runspaces. Here script is run on the local machine within the same process but on separate threads.

This new feature uses the third method for running scripts in parallel. It has the least overhead of the other two methods and does not use the PowerShell remoting system. So it is generally much faster than the other two methods.

However, there is still quite a bit of overhead to run script blocks in parallel. Script blocks run in a context called a PowerShell runspace. The runspace context contains all of the defined variables, functions and loaded modules. So initializing a runspace for script to run in takes time and resources. When scripts are run in parallel they must be run within their own runspace. And each runspace must load whatever module is needed and have any variable be explicitly passed in from the calling script. The only variable that automatically appears in the parallel script block is the piped in object. Other variables are passed in using the $using: keyword.

$computers = 'computerA','computerB','computerC','computerD'
$logsToGet = 'LogA','LogB','LogC'

# Read specified logs on each machine, using custom module
$logs = $computers | ForEach-Object -ThrottleLimit 10 -Parallel {
    Import-Module MyLogsModule
    Get-Logs -ComputerName $_ -LogName $using:logsToGet
}

Given the overhead required to run scripts in parallel, the -ThrottleLimit becomes very useful to prevent the system from being overwhelmed. There are some cases where running a lot of script blocks in parallel makes sense, but also many cases where it does not.

When should it be used?

There are two primary reasons to run script blocks in parallel with the ForEach-Object -Parallel feature (keeping in mind that this feature runs the script on separate system threads).

  1. Highly compute intensive script. If your script is crunching a lot of data over a significant period of time and the scripts can be run independently, then it is worthwhile to run them in parallel. But only if the machine you are running on has multiple cores that can host the script block threads. In this case the -ThrottleLimit parameter should be set approximately to the number of available cores. If you are running on a VM with a single core, then it makes little sense to run high compute script blocks in parallel since the system must serialize them anyway to run on the single core.
  2. Script that must wait on something. If you have script that can run independently and performs long running work that requires waiting for somethings to complete, then it makes sense to run these tasks in parallel. If you have 5 scripts that take 5 minutes each to run but spend most of the time waiting, you can have them all run/wait at the same time, and complete all 5 tasks in 5 minutes instead of 25 minutes. Scripts that do a lot of file operations, or perform operations on external machines can benefit by running in parallel. Since the running script cannot use all of the machine cores, it makes sense to set the -ThrottleLimit parameter to something greater than the number of cores. If one script execution waits many minutes to complete, you may want to allow tens or hundreds of scripts to run in parallel.
$logNames.count
10

PS C:> Measure-Command { $logs = $logNames | ForEach-Object -Parallel { Get-WinEvent -LogName $_ -MaxEvents 5000 2>$null } -ThrottleLimit 10 }
TotalMilliseconds : 115994.3 (1 minute 56 seconds)
$logs.Count
50000

PS C:> Measure-Command { $logs = $logNames | ForEach-Object { Get-WinEvent -LogName $_ -MaxEvents 5000 2>$null } }
TotalMilliseconds : 229768.2364 (3 minutes 50 seconds)
$logs.Count
50000

The script above collects 50,000 log entries on the local machine from 10 system log names. Running this in parallel is almost twice as fast as running sequentially, because it involves some relatively slow disk access and can also take advantage of the machine multiple cores as it processes the log entries.

When should it be avoided?

ForEach-Object -Parallel should not be thought as something that will always speed up script execution. And in fact it can significantly slow down script execution if used heedlessly. For example, if your script block is executing trivial script then running in parallel adds a huge amount of overhead and will run much slower.

PS C:> (measure-command { 1..1000 | ForEach-Object -Parallel { "Hello: $_" } }).TotalMilliseconds
10457.962

PS C:> (measure-command { 1..1000 | ForEach-Object { "Hello: $_" } }).TotalMilliseconds
18.4473

The above example, a trivial script block is run 1000 times. The ThrottleLimit is 5 by default so only 5 runspace/threads are created at a time, but still a runspace and thread is created 1000 times to do a simple string evaluation. Consequently, it takes over 10 seconds to complete. But removing the -Parallel parameter and running the ForEach-Object cmdlet normally, results in completion in about 18 milliseconds.

So, it is important to use this feature wisely.

Implementation details

As previously mentioned, the new ForEach-Object -Parallel feature uses existing PowerShell functionality to run script blocks concurrently. The primary addition is the ability to limit the number of concurrent scripts running at a given time with the -ThrottleLimit parameter. Throttling is accomplished by a PSTaskPool class that holds running tasks (running scripts), and has a settable size limit which is set to the throttle limit value. An Add method allows tasks to be added to the pool, but if it is full then the method blocks until a new slot becomes available. Adding tasks to the task pool was initially performed on the ForEach-Object cmdlet piped input processing thread. But that turned out to be a performance bottleneck, and now a dedicated thread is used to add tasks to the pool.

PowerShell itself imposes conditions on how scripts run concurrently, based on its design and history. Scripts have to run in runspace contexts and only one script thread can run at a time within a runspace. So in order to run multiple scripts simultaneously multiple runspaces must be created. The current implementation of ForEach-Object -Parallel creates a new runspace for each script block execution instance. It may be possible to optimize this by re-using runspaces from a pool, but one concern in doing this is leaking state from one script execution to another.

Runspace contexts are an isolation unit for running scripts, and generally do not allow sharing state between themselves. However, variables can be passed at the beginning of script execution through the $using: keyword, from the calling script to the parallel script block. This was borrowed from the remoting layer which uses the keyword for the same purpose but over a remote connection. But there is a big difference when using the $using: keyword in ForEach-Object -Parallel. And that is for remoting, the variable being passed is a copy sent over the remoting connection. But with ForEach-Object -Parallel, the actual object reference is being passed from one script to another, violating normal isolation restrictions. So it is possible to have a non thread-safe variable used in two scripts running on different threads, which can lead to unpredictable behavior.

# This does not throw an error, but is not guaranteed to work since the dictionary object is not thread safe
$threadUnSafeDictionary = [System.Collections.Generic.Dictionary[string,object]]::new()
Get-Process | ForEach-Object -Parallel {
    $dict = $using:threadUnSafeDictionary
    $dict.TryAdd($_.ProcessName, $_)
}
# This *is* guaranteed to work because the passed in concurrent dictionary object is thread safe
$threadSafeDictionary = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()
Get-Process | ForEach-Object -Parallel {
    $dict = $using:threadSafeDictionary
    $dict.TryAdd($_.ProcessName, $_)
}

$threadSafeDictionary["pwsh"]

 NPM(K)    PM(M)      WS(M)     CPU(s)      Id  SI ProcessName
 ------    -----      -----     ------      --  -- -----------
    112   108.25     124.43      69.75   16272   1 pwsh

Conclusion

This feature can greatly improve your life for many work load scenarios. As long as you understand how it works and what its limitations are, you can experiment with parallelism and make real performance improvements with your scripts.

Paul Higinbotham
Senior Software Engineer
PowerShell Team

The post PowerShell ForEach-Object Parallel Feature appeared first on PowerShell.

Careers in Cybersecurity

This post was originally published on this site

Have you considered a career in Cybersecurity? It is a fast-paced, highly dynamic field with a huge number of specialties to choose from, including forensics, endpoint security, critical infrastructure, incident response, secure coding, and awareness and training. In addition, a career in cybersecurity allows you to work almost anywhere in the world, with amazing benefits and an opportunity to make a real difference. However, the most exciting thing is you do NOT need a technical background, anyone can get started.