Troubleshooting Tips: Orchestrator PowerShell Plug-in (SKKB1028)

It's only fair to share...Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+Digg thisPin on PinterestShare on TumblrPrint this pageEmail this to someone

In this post we will take a look at some at some common issues that one might experience when using the VMware vRealize Orchestrator (vRO) PowerShell Plug-In and especially when using HTTPS protocol or Kerberos authentication for the PowerShell host (PSHost).

 

Background and General Conciderations

Most use cases require that the PowerShell script be run with some kind of administrator level permissions in the target system with which vRO integrates. Here some of them: 

  • Add, modify, remove DNS records for Virtual Machines
  • Register IP address for a Virtual Machine in an IP management system.
  • Create, modify, remove mailbox for a user account.
  • Execute remote PowerShell commands against multiple Windows operating systems in the environment.
  • Run a PowerShell script (.ps1) file from within a PowerShell script file from vRO.
  • Access to mapped network drives from vRO.
  • Interact with Windows operating systems that have User Access Control (UAC) is enabled
  • Execute PowerCLI commands.
  • Integrate with Azure.

When you add a PowerShell Host you must specify a user account. That account will be used to execute all PowerShell scripts from vRO. In most of the use cases, like the once above, that account must be an administrator account in the corresponding target system with which the script interacts. That is in most cases some kind of domain level account.

In order to successfully add the PowerShell host with that account , and therefore use that account when executing scripts from vRO, there are some prerequisites that need to be met.

In addition the use cases mention require the PowerShell Host to be prepared for credential delegation (a.k.a Credential Security Service Provider (CredSSP) , double-hop authentication or multi-hop authentication) .

On a high level the requirements for adding a PowerShell Host in vRO that satisfies the above use cases are:

  • Port: 5986
  • Powershell remote host type: WinRM
  • Transport protocol: HTTPS (Recommended)
  • Authentication: Kerberos
  • User name: <Administrator_user_name>

On a low level the requirements for adding a PowerShell Host in vRO that satisfies the above use cases are:

  • PSHost: Configure WinRM and user token delegation.
  • PSHost: Configure Windows service principal names (SPN’s) for WinRM
  • PSHost: Import a CA signed server certificate containing Client Authentication and  Server authentication Exchange Key Usage Properties.
  • PSHost: Configure Windows Credential Delegation using the Credential Security Service Provider (CredSSP) module.
  • vRO: Edit the Kerberos Domain Realm (krb5.conf) on the vCO Appliance (Optional/Scenario specific)
  • vRO: Add the PS Host as HTTPS host with Kerberos authentication.
  • vRO: Use the Invoke-Command cmdlet in your PowerShell code.

Lab Environment

The following environment has been used in all of the examples below:

  • Active Directory Domain called vmware.com
  • Windows Server 2012 R2 (with FQDN lan1dc1.vmware.com)
    • Domain Controller
    • DNS Server.
    • PowerShell v4 Host.
    • .NET 4 Framework installed (4.0.30319)
  • Windows Server 2012 R2 (with FQDN lan2dc1.support.vmware.com)
    • Domain Controller
    • DNS Server.
    • PowerShell v4 Host.
    • .NET 4 Framework installed (4.0.30319)
  • Windows Server 2008 R2 SP1 (with FQDN lan1dm1.vmware.com)
    • Domain Member
    • PowerShell v3 Host.
    • .NET 4 Framework installed (4.0.30319)
  • vCO 5.5.1.0 Virtual Appliance (with FQDN vco-a-01.vmware.com)
    • vCO PowerShell Plugin installed (version 1.0.6.2283945)

The full lab logical design can be seen HERE.

 

Troubleshooting issues when adding a PSHost

To resolve most common issues when adding a PSHost for use with HTTPS transport protocol and Kerberos authentication, follow these steps:

  1. Prepare the Windows PSHost .
  2. For more information on all the configurations needed on the PSHost, visit Using CredSSP with the vCO PowerShell Plugin

  3. After preparing the PSHost test it to make sure it accepts execution or remove PowerShell commands.

    Start by testing simple commands . I like to use $env:computername PowerShell command that returns the hostname of the PSHost. You can use the winrs command in Windows for the test.

    Here an example of the syntax:

    winrs -r:https://lan1dc1.vmware.com:5986 -u:vmware\administrator -p:VMware1! powershell.exe $env:computername

  4. Continue by testing a command that requires credential delegation. I like to use a simple command,  like dir \\<Server_FQDN\<sharename>,  that accesses a share residing on a computer different then the PSHost itself.

    Here an example of the syntax:

    winrs -r:https://lan1dc1.vmware.com:5986 -ad -u:vmware\administrator -p:VMware1! powershell.exe dir \\lan1dm1.vmware.com\share

    Note: Make sure to specify the –ad command line switch.

  5. Prepare the vRO so that it can handle Kerberos authentication. You need this in order to use a domain level account when adding the PSHost.
  6. For more information about the Kerberos configuration on vRO for single domain, visit Using CredSSP with the vCO PowerShell Plugin.

    If you are planning to add multiple PSHosts and use domain level accounts for each PSHost that are from different domains (e.g. vmware.com and support.vmware.com) you need to take this into consideration when preparing vRO for Kerberos authentication.

    For more information about the Kerberos configuration on vRO for multiple domains, visit How to add PowerShell hosts from multiple domains with Kerberos authentication to the same vRO

    If you have done a mistake in the configuration you might see the following error then Adding the PSHost:

    Cannot locate default realm (Dynamic Script Module name : addPowerShellHost#12
    tem: ‘Add a PowerShell host/item8′, state: ‘failed’, business state: ‘Error’, exception: ‘InternalError: java.net.ConnectException: Connection refused (Workflow:Import a certificate from URL with certificate alias / Validate (item1)#5)’
    workflow: ‘Add a PowerShell

    If this is the case, go back and validate again all configurations.

  7. If the error continues, make sure the krb5.conf file is correctly formatted.
  8. For more information about the common formatting mistakes, visit Wrong encoding or formatting of Linux configuration files can cause problems in VMware Appliances

  9. Make sure to use the following parameters when adding the PSHost:
    • Port: 5986
    • Powershell remote host type: WinRM
    • Transport protocol: HTTPS (Recommended)
    • Authentication: Kerberos
    • User name: <Administrator_user_name>
  10. Note: The user must be at least a local administrator on the PSHost in order to be able to add the PSHost at all.

  11. If you still cannot add the host, make sure your VMware appliance can authenticate successfully using Kerberos against the domains you’ve configured. To do this you can use the ldapsearch command and test Kerberos connectivity to the domain.
  12. Here an example example of the syntax:

    vco-a-01:/opt/vmware/bin # ldapsearch -h lan1dc1.vmware.com -D "CN=Administrator,CN=Users,DC=vmware,DC=com" -w VMware1! -b "" -s base "objectclass=*"

  13. If your authentication problems continue , most likely there is a general authentication problem that might not be directly connected to the vRO Appliance, like :
    • Network related issue.
    • Blocked firewall ports
    • DNS resolution problems
    • Unresponsive domain controllers
    • ect..

 

Troubleshooting issues when executing scripts

Once you’ve successfully added the PSHost, it’s time to test PowerShell execution from the vRO.

To resolve most common issues when executing PowerShell scripts from vRO, follow these steps:

  1. White in vRO go to the Inventory tab and make sure you don’t see the the word “(unusable)” in front of the PSHost name. If this is the case, remove the PSHost and add it again to vRO.
  2. Use the Invoke a external script workflow that is shipped with vRO to test execution PowerShell commands. Again start with a simple command, like  $env:computername.
  3. Then process with a command that requires credential delegation. Again, as before, you can use a command, like dir \\<Server_FQDN\<sharename>.

    Note that doesn’t support credential delegation so you need to do a slight workaround to achieve this functionality. You need to wrap the command you want to execute around an Invoke-Command command.

    For more information how to achieve credential delegation from vRO, visit Using CredSSP with the vCO PowerShell Plugin.

    If you do not use the workaround and try to execute a command that requires credential delegation, you will receive error similar to the following.

    PowerShellInvocationError: Errors found while executing script <script>: Access is denied

  4. Use the SilentlyContinue PowerShell error action preference to suppress output from “noisy” commands. Such commands are those that generate some kind of non-standard output, like:
    • Progress par showing the progress of the command execution
    • Hashes and other similar content.
  5. Also don’t use code in your commands or scripts that generates popup messages or opens other windows or any other Graphical User Interface (GUI) 

 

Final Step

If all went well, go grab a beer.

It's only fair to share...Share on FacebookTweet about this on TwitterShare on LinkedInShare on Google+Digg thisPin on PinterestShare on TumblrPrint this pageEmail this to someone

Leave a Reply

Your email address will not be published. Required fields are marked *