In this post we will take a look at some at some common issues that one might experience when using the VMware vRealize Orchestrator (vRO) PowerShell Plug-In and especially when using HTTPS protocol or Kerberos authentication for the PowerShell host (PSHost).

• Introduction
• Lab Environment
• Background and General Conciderations
• Troubleshooting issues when adding a PSHost
• Troubleshooting issues when executing scripts
• Final Step

 

Background and General Conciderations

Most use cases require that the PowerShell script be run with some kind of administrator level permissions in the target system with which vRO integrates. Here some of them: 

• Add, modify, remove DNS records for Virtual Machines
• Register IP address for a Virtual Machine in an IP management system.
• Create, modify, remove mailbox for a user account.
• Execute remote PowerShell commands against multiple Windows operating systems in the environment.
• Run a PowerShell script (.ps1) file from within a PowerShell script file from vRO.
• Access to mapped network drives from vRO.
• Interact with Windows operating systems that have User Access Control (UAC) is enabled
• Execute PowerCLI commands.
• Integrate with Azure.

When you add a PowerShell Host you must specify a user account. That account will be used to execute all PowerShell scripts from vRO. In most of the use cases, like the once above, that account must be an administrator account in the corresponding target system with which the script interacts. That is in most cases some kind of domain level account.

In order to successfully add the PowerShell host with that account , and therefore use that account when executing scripts from vRO, there are some prerequisites that need to be met.

In addition the use cases mention require the PowerShell Host to be prepared for credential delegation (a.k.a Credential Security Service Provider (CredSSP) , double-hop authentication or multi-hop authentication) .

On a high level the requirements for adding a PowerShell Host in vRO that satisfies the above use cases are:

• Port: 5986
• Powershell remote host type: WinRM
• Transport protocol: HTTPS (Recommended)
• Authentication: Kerberos
• User name: <Administrator_user_name>

On a low level the requirements for adding a PowerShell Host in vRO that satisfies the above use cases are:

• PSHost: Configure WinRM and user token delegation.
• PSHost: Configure Windows service principal names (SPN’s) for WinRM
• PSHost: Import a CA signed server certificate containing Client Authentication and  Server authentication Exchange Key Usage Properties.
• PSHost: Configure Windows Credential Delegation using the Credential Security Service Provider (CredSSP) module.
• vRO: Edit the Kerberos Domain Realm (krb5.conf) on the vCO Appliance (Optional/Scenario specific)
• vRO: Add the PS Host as HTTPS host with Kerberos authentication.
• vRO: Use the Invoke-Command cmdlet in your PowerShell code.

Lab Environment

The following environment has been used in all of the examples below:

• Active Directory Domain called vmware.com
• Windows Server 2012 R2 (with FQDN lan1dc1.vmware.com)
  • Domain Controller
  • DNS Server.
  • PowerShell v4 Host.
  • .NET 4 Framework installed (4.0.30319)
• Windows Server 2012 R2 (with FQDN lan2dc1.support.vmware.com)
  • Domain Controller
  • DNS Server.
  • PowerShell v4 Host.
  • .NET 4 Framework installed (4.0.30319)
• Windows Server 2008 R2 SP1 (with FQDN lan1dm1.vmware.com)
  • Domain Member
  • PowerShell v3 Host.
  • .NET 4 Framework installed (4.0.30319)
• vCO 5.5.1.0 Virtual Appliance (with FQDN vco-a-01.vmware.com)
  • vCO PowerShell Plugin installed (version 1.0.6.2283945)

The full lab logical design can be seen HERE.

 

Troubleshooting issues when adding a PSHost

To resolve most common issues when adding a PSHost for use with HTTPS transport protocol and Kerberos authentication, follow these steps:

1. Prepare the Windows PSHost .

For more information on all the configurations needed on the PSHost, visit Using CredSSP with the vCO PowerShell Plugin

2. After preparing the PSHost test it to make sure it accepts execution or remove PowerShell commands.

   Start by testing simple commands . I like to use $env:computername PowerShell command that returns the hostname of the PSHost. You can use the winrs command in Windows for the test.

   Here an example of the syntax:

   winrs -r:https://lan1dc1.vmware.com:5986 -u:vmware\administrator -p:VMware1! powershell.exe $env:computername

Continue by testing a command that requires credential delegation. I like to use a simple command,  like dir \\<Server_FQDN\<sharename>,  that accesses a share residing on a computer different then the PSHost itself.

Here an example of the syntax:

winrs -r:https://lan1dc1.vmware.com:5986 -ad -u:vmware\administrator -p:VMware1! powershell.exe dir \\lan1dm1.vmware.com\share

Note: Make sure to specify the –ad command line switch.

3. Prepare the vRO so that it can handle Kerberos authentication. You need this in order to use a domain level account when adding the PSHost.

For more information about the Kerberos configuration on vRO for single domain, visit Using CredSSP with the vCO PowerShell Plugin.

If you are planning to add multiple PSHosts and use domain level accounts for each PSHost that are from different domains (e.g. vmware.com and support.vmware.com) you need to take this into consideration when preparing vRO for Kerberos authentication.

For more information about the Kerberos configuration on vRO for multiple domains, visit How to add PowerShell hosts from multiple domains with Kerberos authentication to the same vRO

If you have done a mistake in the configuration you might see the following error then Adding the PSHost:

Cannot locate default realm (Dynamic Script Module name : addPowerShellHost#12
tem: ‘Add a PowerShell host/item8′, state: ‘failed’, business state: ‘Error’, exception: ‘InternalError: java.net.ConnectException: Connection refused (Workflow:Import a certificate from URL with certificate alias / Validate (item1)#5)’
workflow: ‘Add a PowerShell

If this is the case, go back and validate again all configurations.

4. If the error continues, make sure the krb5.conf file is correctly formatted.

For more information about the common formatting mistakes, visit Wrong encoding or formatting of Linux configuration files can cause problems in VMware Appliances

5. Make sure to use the following parameters when adding the PSHost:
   • Port: 5986
   • Powershell remote host type: WinRM
   • Transport protocol: HTTPS (Recommended)
   • Authentication: Kerberos
   • User name: <Administrator_user_name>

Note: The user must be at least a local administrator on the PSHost in order to be able to add the PSHost at all.

6. If you still cannot add the host, make sure your VMware appliance can authenticate successfully using Kerberos against the domains you’ve configured. To do this you can use the ldapsearch command and test Kerberos connectivity to the domain.

Here an example example of the syntax:

vco-a-01:/opt/vmware/bin # ldapsearch -h lan1dc1.vmware.com -D "CN=Administrator,CN=Users,DC=vmware,DC=com" -w VMware1! -b "" -s base "objectclass=*"

7. If your authentication problems continue , most likely there is a general authentication problem that might not be directly connected to the vRO Appliance, like :
   • Network related issue.
   • Blocked firewall ports
   • DNS resolution problems
   • Unresponsive domain controllers
   • ect..

 

Troubleshooting issues when executing scripts

Once you’ve successfully added the PSHost, it’s time to test PowerShell execution from the vRO.

To resolve most common issues when executing PowerShell scripts from vRO, follow these steps:

1. White in vRO go to the Inventory tab and make sure you don’t see the the word “(unusable)” in front of the PSHost name. If this is the case, remove the PSHost and add it again to vRO.
2. Use the Invoke a external script workflow that is shipped with vRO to test execution PowerShell commands. Again start with a simple command, like  $env:computername.

Then process with a command that requires credential delegation. Again, as before, you can use a command, like dir \\<Server_FQDN\<sharename>.

Note that doesn’t support credential delegation so you need to do a slight workaround to achieve this functionality. You need to wrap the command you want to execute around an Invoke-Command command.

For more information how to achieve credential delegation from vRO, visit Using CredSSP with the vCO PowerShell Plugin.

If you do not use the workaround and try to execute a command that requires credential delegation, you will receive error similar to the following.

PowerShellInvocationError: Errors found while executing script <script>: Access is denied



3. Use the SilentlyContinue PowerShell error action preference to suppress output from “noisy” commands. Such commands are those that generate some kind of non-standard output, like:
   • Progress par showing the progress of the command execution
   • Hashes and other similar content.

Also don’t use code in your commands or scripts that generates popup messages or opens other windows or any other Graphical User Interface (GUI) 

 

Final Step

If all went well, go grab a beer.

DISCLAIMER; This is a personal blog. Any views or opinions represented in this blog are personal and belong solely to the blog owner and do not represent those of people, institutions or organizations that the owner may or may not be associated with in professional or personal capacity, unless explicitly stated. Any views or opinions are not intended to malign any religion, ethnic group, club, organization, company, or individual.
All content provided on this blog is for informational purposes only. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. The owner will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of this information.
Photos
Unless stated, all photos are the work of the blog owner and are licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. If used with watermark, no need to credit to the blog owner. For any edit to photos, including cropping, please contact me first.
Recipes
Unless stated, all recipes are the work of the blog owner and are licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. Please credit all recipes to the blog owner and link back to the original blog post.
Downloadable Files
Any downloadable file, including but not limited to pdfs, docs, jpegs, pngs, is provided at the user’s own risk. The owner will not be liable for any losses, injuries, or damages resulting from a corrupted or damaged file.
Comments
Comments are welcome. However, the blog owner reserves the right to edit or delete any comments submitted to this blog without notice due to
– Comments deemed to be spam or questionable spam
– Comments including profanity
– Comments containing language or concepts that could be deemed offensive
– Comments containing hate speech, credible threats, or direct attacks on an individual or group
The blog owner is not responsible for the content in comments.
This policy is subject to change at anytime.