Warning: call_user_func_array() expects parameter 1 to be a valid callback, no array or string given in /home/kalofero/public_html/blog/wp-includes/class-wp-hook.php on line 286
Using CredSSP with the vCO PowerShell Plugin (SKKB1002) | Spas Kaloferov's Blog

Using CredSSP with the vCO PowerShell Plugin (SKKB1002)

Posted on by Spas Kaloferov

Update Log:
11/01/2019 – added update for vRA 8.0
12/02/2016 – Updated “Edit the Kerberos Domain realm on the vCO Appliance” chapter.
12/01/2016 – Added “Security Hardening Chapter” chapter. Updated “Security Conciderations” chapter.
11/30/2016 – Added “Security Conciderations” chapter.

 

Lab Environment

The full lab logical design can be seen HERE.

Introduction

In this post we will take a look at common difficulties and challenges when using VMware vCloud Orchestrator Server (vCO) in combination with Microsoft Windows PowerShell to achieve remote script execution of PowerShell code.

Few common scenarios and roadblocks that can be seen when using vCO and the PowerShell Plugin are:

  • Execute remote PowerShell commands. In this scenario you are running a vCO workflow that executes PowerShell code on ServerA and the code itself needs to access ServerB.
  • Access to mapped network drives through vCO: In this scenario you are running a vCO workflow that executes PowerShell code which needs to access locally mapped network drive on the PowerShell host or remote network share on remote host.
  • Run a PowerShell script (.ps1) file from within a PowerShell script: In this scenario you are running a vCO workflow that executes PowerShell code which calls an external PowerShell script (.ps1) file and executers it. The file can be located locally on the PowerShell host, on a locally attached/mapped share on the PowerShell host or share on a remote host.
  • User Access Control (UAC) is enabled – Conceder a simple use case. After you provision a Windows VM , which has UAC activated, you want to add domain user/group to the VM’s local administrators group. This requires elevation of permissions.
  • Change permissions in a script – Conceder a simple use case. You have a script that many operations. Some require the use of administrator/elevated permissions , other do not. Changing permissions requires CredSSP.
  • Execute PowerCLI commands – WF executes a PS commands that connects to PowerCLI and then connects to vCenter/ESXi host. In this case you also need to supply additional credentials.
  • Azure Integration – Requires CredSSP, depending on what authentication and SSO is used.

All of these scenarios require the need of user token delegation. Windows Remote Management (WinRM) supports the delegation of user credentials across multiple remote computers. WinRM is Microsoft’s implementation of the WS-Management protocol, a standard Simple Object Access Protocol (SOAP)-based, firewall-friendly protocol that enables hardware and operating systems from different vendors to interoperate. The multi-hop support functionality can now use Credential Security Service Provider (CredSSP) for authentication. CredSSP enables an application to delegate the user’s credentials from the client computer to the target server.
CredSSP authentication is intended for environments where Kerberos delegation cannot be used. Support for CredSSP was added to allow a user to connect to a remote server and have the ability to access a second-hop machine, such as a file share.

Reference:
Multi-Hop Support in WinRM

Unfortunately this authentication is not supported by vCO PowerShell plugin at the time this article is being written. So, what can we do in order to be able to achieve the desired results in the two scenarios above.

The following things need to be set in order to achieve the use cases outlined above:

  • PSHost: Configure WinRM and user token delegation (CredSSP).
  • PSHost: Configure Windows service principal names (SPN’s) for WinRM
  • PSHost: Import a CA signed server certificate containing Client Authentication AND Server authentication Exchange Key Usage Properties.
  • PSHost: Configure Windows Credential Delegation using the Credential Security Service Provider (CredSSP) module.
  • vRO: Edit the Kerberos Domain Realm (krb5.conf) on the vCO Appliance (Optional/Scenario specific)
  • vRO: Add the PS Host as HTTPS host with Kerberos authentication.
  • vRO: Use the Invoke-Command cmdlet in your PowerShell code.

 

Security Considerations

There are some security considerations that you should be aware of before enabling CredSSP on Windows.

WS-Management encrypts all traffic by default and this is controlled by the AllowUnencrypted client and server WinRM configuration parameter. Even if you only work with HTTP (which is the default configuration) and not with HTTPS. Prior to Windows Server 2003 R2 this WInRM in an HTTP session is not encrypted.

By default, PowerShell remoting authenticates using a “Network Logon”.  Network Logons work by proving to the remote server that you have possession of the users credential without sending the credential to that server (see Kerberos and NTLM authentication). Because the remote server doesn’t have possession of your credential, when you try to make the second hop (from Server A to Server B) it fails because Server A doesn’t have a credential to authenticate to Server B with.

To get around this issue, PowerShell provides the CredSSP (Credential Security Support Provider) option. When using CredSSP, PowerShell will perform a “Network Clear-text Logon” instead of a “Network Logon”. Network Clear-text Logon works by sending the user’s clear-text password to the remote server. When using CredSSP, Server A will be sent the user’s clear-text password, and will therefore be able to authenticate to Server B. Double hop works!

Microsoft has made changes to Windows Server 2012R2 and Windows 8.1 to eliminate clear-text credentials from being stored in memory. This means that an attacker who runs Mimikatz will no longer see your clear-text credentials. An attacker will still see your NT password hash and your Kerberos TGT, both of which are password equivalent and can be used to authenticate as you over the network.

Additionally, even though your clear-text credential is not saved in memory, it is still sent to the remote server. An attacker can inject malicious code in the Local Security Authority Subsystem Service (LSASS.exe) and intercept your password in transit. So while you may not see your password with Mimikatz anymore, your password can still be recovered by an attacker.

An additional layer of protection can be added by using WinRM with HTTPS . HTTPS doesn’t just add another encryption layer; its main purpose is to verify the authenticity of the remote machine, thereby preventing man-in-the-middle attacks.

Note also that the WinRM configuration outlined in this article is for PoC purposes. Once you have everything up and running you should consider hardening the configuration.

References:
Accidental Sabotage: Beware of CredSSP
Compromising Yourself with WinRM’s “AllowUnencrypted = True”

 

Configure WinRM and user token delegation (CredSSP)

The first thing that need to be configured, if not already done when adding the PowerShell Host(s) in vCO is to configure the WinRM Service and Client on the PowerShell Host(s).

To check if WinRM has been enabled and configured go on the PowerShell host and run in [winrm e winrm/config/listener]. If it is not enabled you will receive message similar to this:

WSManFault
Message = The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: “winrm quickconfig”.

Error number: -2144108526 0x80338012
The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consultthe logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: “winrm quickconfig”.

 

If this is the first time you are configuring WinRM on the PowerShell hosts run the following commands to quickly configure the WinRM service and the WinRM HTTP and HTTPS listeners with default settings:

  • Winrm quickconfig.
  • Winrm quickconfig -transport:https (Be aware that you must open port 5986 for HTTPS transport to work.)

Note: If you are not running under the local computer Administrator account, you must either select Run as Administrator from the Start menu or use the Runas command at a command prompt.

The winrm quickconfig command (or the abbreviated version winrm qc) performs the following operations:

  • Starts the WinRM service, and sets the service startup type to auto-start.
  • Configures a listener for the ports that send and receive WS-Management protocol messages using either HTTP or HTTPS on any IP address.
  • Defines ICF exceptions for the WinRM service, and opens the ports for HTTP and HTTPS.

Note The winrm quickconfig command creates a firewall exception only for the current user profile. If the firewall profile is changed for any reason, winrm quickconfig should be run to enable the firewall exception for the new profile; otherwise, the exception might not be enabled.

When executing the [Winrm quickconfig -transport:https] command you may receive an error like this one:

C:\>winrm quickconfig -transport:https
WinRM service is already running on this machine.
WSManFault
Message
ProviderFault
WSManFault
Message = Cannot create a WinRM listener on HTTPS because this machine does not have an appropriate certificate. To be used for SSL, a certificate must have a CN matching the hostname, be appropriate for Server Authentication, and not be expired, revoked, or self-signed.

Error number: -2144108267 0x80338115
Cannot create a WinRM listener on HTTPS because this machine does not have an appropriate certificate. To be used for SSL, a certificate must have a CN matchingthe hostname, be appropriate for Server Authentication, and not be expired, revoked, or self-signed.

This is because no certificate matching the machine name is found on the local computer certificate store. Such certificate is needed in order to allow HTTPS communication. Make sure to get such certificate and install it on the local computer before you continue.

You might also receive this error even though you have issued a certificate and installed it into the computer certificate store. This happens when the certificate you are using has either only Client Authentication or Server Authentication Exchange Key Usage Properties. In order for your certificate to be valid WINRM certificate it must have both Exchange Key Usage Properties

To get the listener configuration, type [winrm enumerate winrm/config/listener] at a command prompt. Listeners are defined by a transport (HTTP or HTTPS)

Upon successful configuration you should see both HTTP and HTTPS listeners with default configuration similar to this one:

C:\>winrm e winrm/config/listener
Listener
Address = *
Transport = HTTP
Port = 5985
Hostname
Enabled = true
URLPrefix = wsman
CertificateThumbprint
ListeningOn = 127.0.0.1, 192.168.1.1, 192.168.2.1, 192.168.3.1, 192.168.4.1, 192.168.5.1, 192.168.6.1, 192.168.7.1, ::1, fe80::5efe:192.
168.1.1%20, fe80::5efe:192.168.2.1%30, fe80::5efe:192.168.3.1%31, fe80::5efe:192.168.4.1%21, fe80::5efe:192.168.5.1%22, fe80::5efe:192.168.6
.1%13, fe80::5efe:192.168.7.1%32, fe80::5c29:f88a:d53e:c393%33, fe80::5cd3:4401:93dd:5020%25, fe80::6868:73f5:f051:f859%36, fe80::6d8a:e62c:
58da:2c76%38, fe80::a15a:d339:4137:aa44%24, fe80::dda4:a92:430b:393e%35, fe80::f0c7:f6c9:a83c:194e%37

Listener
Address = *
Transport = HTTPS
Port = 5986
Hostname = LAN1DC1.vmware.com
Enabled = true
URLPrefix = wsman
CertificateThumbprint = 87 22 09 f1 e0 23 65 64 96 71 1b 17 fc 4e 43 1cf5 31 10 1e
ListeningOn = 127.0.0.1, 192.168.1.1, 192.168.2.1, 192.168.3.1, 192.168.4.1, 192.168.5.1, 192.168.6.1, 192.168.7.1, ::1, fe80::5efe:192.
168.1.1%20, fe80::5efe:192.168.2.1%30, fe80::5efe:192.168.3.1%31, fe80::5efe:192.168.4.1%21, fe80::5efe:192.168.5.1%22, fe80::5efe:192.168.6
.1%13, fe80::5efe:192.168.7.1%32, fe80::5c29:f88a:d53e:c393%33, fe80::5cd3:4401:93dd:5020%25, fe80::6868:73f5:f051:f859%36, fe80::6d8a:e62c:
58da:2c76%38, fe80::a15a:d339:4137:aa44%24, fe80::dda4:a92:430b:393e%35, fe80::f0c7:f6c9:a83c:194e%37

Key takeaways from this output are:

  • Both HTTP and HTTPS listeners are enabled.
  • Both HTTP and HTTPS listeners are listening for connections on all IPv4 and IPv6 addresses.
  • HTTP is listening on default port 5985 and HTTPS is listening on default port 5986

Now that we have checked the WinRM listeners let’s check the configuration of the WinRM Service and WinRM Client by running [winrm get winrm/config]

C:\>winrm get winrm/config
Config
MaxEnvelopeSizekb = 500
MaxTimeoutms = 60000
MaxBatchItems = 32000
MaxProviderRequests = 4294967295
Client
NetworkDelayms = 5000
URLPrefix = wsman
AllowUnencrypted = false
Auth
Basic = true
Digest = true
Kerberos = true
Negotiate = true
Certificate = true
CredSSP = false
DefaultPorts
HTTP = 5985
HTTPS = 5986
TrustedHosts
Service
RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;G
XGW;;;WD)
MaxConcurrentOperations = 4294967295
MaxConcurrentOperationsPerUser = 1500
EnumerationTimeoutms = 240000
MaxConnections = 300
MaxPacketRetrievalTimeSeconds = 120
AllowUnencrypted = false
Auth
Basic = false
Kerberos = true
Negotiate = true
Certificate = false
CredSSP = false
CbtHardeningLevel = Relaxed
DefaultPorts
HTTP = 5985
HTTPS = 5986
IPv4Filter = *
IPv6Filter = *
EnableCompatibilityHttpListener = false
EnableCompatibilityHttpsListener = false
CertificateThumbprint
AllowRemoteAccess = true
Winrs
AllowRemoteShellAccess = true
IdleTimeout = 7200000
MaxConcurrentUsers = 10
MaxShellRunTime = 2147483647
MaxProcessesPerShell = 25
MaxMemoryPerShellMB = 1024
MaxShellsPerUser = 30

Here are the things that need to be set in addition to the default configuration, if not already done so, for both the WinRM Client and WinRM Service:

  • Enable Kerberos authentication. (optional for both HTTP and HTTPS)
  • Enable Basic authentication. (for HTTP only)
  • Allow transfer of unencrypted data on the WinRM service (for HTTP only)
  • If WinRM host machine is in an external domain, specify the trusted hosts. (both HTTP and HTTPS)
  • Enable CredSSP authentication (for both HTTP and HTTPS)

Here are few of the commands to get you started:

winrm set winrm/config/service @{AllowUnencrypted=”true”}
winrm set winrm/config/service/auth @{Basic=”true”}
winrm set winrm/config/service/auth @{Kerberos=”true”}
winrm set winrm/config/service/auth @{CredSSP=”true”}
winrm set winrm/config/service/auth @{Certificate=”true”}
winrm set winrm/config/client @{AllowUnencrypted=”true”}
winrm set winrm/config/client/auth @{Basic=”true”}
winrm set winrm/config/client/auth @{Kerberos=”true”}
winrm set winrm/config/client/auth @{CredSSP=”true”}
winrm set winrm/config/client/auth @{Certificate=”true”}
winrm set winrm/config/client @{TrustedHosts=”host1, host2, host3″}

Here more information about each command:

c:\> winrm set winrm/config/service @{AllowUnencrypted=”true”}

Run the following command to enable basic authentication on the WinRM service.

c:\> winrm set winrm/config/service/auth @{Basic=”true”}

Run the following command to enable Kerberos authentication on the WinRM service.

c:\> winrm set winrm/config/service/auth @{Kerberos=”true”}

Run the following command to enable CredSSP on the WinRM service:

c:\> winrm set winrm/config/service/auth @{CredSSP=”true”}

Run the following command to allow transfer of unencrypted data on the WinRM client.

c:\> winrm set winrm/config/client @{AllowUnencrypted=”true”}

Run the following command to enable basic authentication on the WinRM client.

c:\> winrm set winrm/config/client/auth @{Basic=”true”}

Run the following command to enable Kerberos authentication on the WinRM client.

c:\> winrm set winrm/config/client/auth @{Kerberos=”true”}

Run the following command to enable CredSSP on the WinRM client:

c:\> winrm set winrm/config/client/auth @{CredSSP=”true”}

If the WinRM host machine is in an external domain, run the following command to specify the trusted hosts.

c:\> winrm set winrm/config/client @{TrustedHosts=”host1, host2, host3″}

Run the following commands to allow certificate/SSL:

c:\> winrm set winrm/config/service/auth @{Certificate=”true”}

c:\> winrm set winrm/config/client/auth @{Certificate=”true”}

Note: make sure to refer to the Security Hardening and Security Considerations chapters later on for more information about the security implications this WinRM configurations may pause.

All of the above changes and more can be made for a group of servers in the domain or for all servers in the entire domain via windows group policy object. The Windows WinRM GPO’s can be accessed as follows [start > run > gpmc.msc (find the computer OU of the servers and edit the existing or create new GPO) > Computer > Policies > Windows Components > Windows Remote Management (WinRM)]. This can streamline, standardize and automate the WinRM configuration throughout the domain or selected number of PowerShell hosts.

At this point the WinRM configuration in our lab looks like this:

// WinRM configuration LAN1DC1

C:\>winrm get winrm/config
Config
MaxEnvelopeSizekb = 500
MaxTimeoutms = 60000
MaxBatchItems = 32000
MaxProviderRequests = 4294967295
Client
NetworkDelayms = 5000
URLPrefix = wsman
AllowUnencrypted = true
Auth
Basic = true
Digest = true
Kerberos = true
Negotiate = true
Certificate = true
CredSSP = true
DefaultPorts
HTTP = 5985
HTTPS = 5986
TrustedHosts = vco-a-01, vco-a-01.vmware.com, lan1dc1, lan1dc1.vmware.com, lan1dm1, lan1dm1.vmware.com
Service
RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
MaxConcurrentOperations = 4294967295
MaxConcurrentOperationsPerUser = 1500
EnumerationTimeoutms = 240000
MaxConnections = 300
MaxPacketRetrievalTimeSeconds = 120
AllowUnencrypted = true
Auth
Basic = true
Kerberos = true
Negotiate = true
Certificate = true
CredSSP = true
CbtHardeningLevel = Relaxed
DefaultPorts
HTTP = 5985
HTTPS = 5986
IPv4Filter = *
IPv6Filter = *
EnableCompatibilityHttpListener = false
EnableCompatibilityHttpsListener = false
CertificateThumbprint = 87 22 09 f1 e0 23 65 64 96 71 1b 17 fc 4e 43 1cf5 31 10 1e
AllowRemoteAccess = true
Winrs
AllowRemoteShellAccess = true
IdleTimeout = 7200000
MaxConcurrentUsers = 10
MaxShellRunTime = 2147483647
MaxProcessesPerShell = 25
MaxMemoryPerShellMB = 1024
MaxShellsPerUser = 30

// WinRm configuration LAN1DM1

C:\>winrm get winrm/config
Config
MaxEnvelopeSizekb = 500
MaxTimeoutms = 60000
MaxBatchItems = 32000
MaxProviderRequests = 4294967295
Client
NetworkDelayms = 5000
URLPrefix = wsman
AllowUnencrypted = true
Auth
Basic = true
Digest = true
Kerberos = true
Negotiate = true
Certificate = true
CredSSP = true
DefaultPorts
HTTP = 5985
HTTPS = 5986
TrustedHosts = * [Source=”GPO”]
Service
RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;G
XGW;;;WD)
MaxConcurrentOperations = 4294967295
MaxConcurrentOperationsPerUser = 1500
EnumerationTimeoutms = 240000
MaxConnections = 300
MaxPacketRetrievalTimeSeconds = 120
AllowUnencrypted = true
Auth
Basic = true
Kerberos = true
Negotiate = true
Certificate = false
CredSSP = true
CbtHardeningLevel = Relaxed
DefaultPorts
HTTP = 5985
HTTPS = 5986
IPv4Filter = *
IPv6Filter = *
EnableCompatibilityHttpListener = false
EnableCompatibilityHttpsListener = false
CertificateThumbprint = 3e 20 4c 2f 23 82 a8 46 b5 c1 f2 73 2b b1 03 6a
7e 42 6a 3a
AllowRemoteAccess = true
Winrs
AllowRemoteShellAccess = true
IdleTimeout = 7200000
MaxConcurrentUsers = 10
MaxShellRunTime = 2147483647
MaxProcessesPerShell = 25
MaxMemoryPerShellMB = 1024
MaxShellsPerUser = 30

Key takeaways are:

  • Kerberos authentication is enabled which will allow to add the PowerShell host with Kerberos authentication in vCO
  • CredSSP is enabled which will allow to delegate user token(s)
  • Both listeners are enabled on default ports.
  • I’ve added all my PowerShell hosts which will take part of the script execution as TrustedHosts. Although this is not needed as both of my PowerShell hosts are members of the domain, this has been done for teaching purposes.

You may consider hardening the WinRM configuration after you have tested it with vCO and reached the desired results.

A quick test using WinRS utility tool can be done in order to check if CredSSP has been successfully configured. The tool relies on WinRM to execute remote commands, especially for headless servers.. You can think of WinRM as the server side and WinRS the client side of WS-Management. Making a call to WinRM with WinRS Is the closest match to the call that vCO does to WinRM when it calls for PowerShell execution.

To test a multi-hop authentication (CredSSP) and Kerberos authentication make a remote call to server lan1dc wither HTTP or HTTPS. The command calls LAN1DC1 and asks him to access remote share on LAN1DM1 (user token delegation).

PS C:\> winrs -r:http://lan1dc1.vmware.com:5985 -ad -u:vmware\administrator -p:VMware1! powershell.exe dir \\lan1dm1.vmware.com\share

PS C:\> winrs -r:https://lan1dc1.vmware.com:5986 -ad -u:vmware\administrator -p:VMware1! powershell.exe dir \\lan1dm1.vmware.com\share

If CredSSP is not activated on a server, you will receive error similar to the following:

PS C:\> winrs -r:http://lan1dc1.vmware.com:5985 -ad -u:vmware\administrator -p:VMware1! powershell.exe dir \\lan1dm1.vmware.com\share

winrs :
At line:1 char:1
+ winrs -r:http://lan1dc1.vmware.com:5985 -ad -u:vmware\administrator -p:VMware1! …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError
Winrs error:
The WinRM client cannot process the request. A computer policy does not allow the delegation of the user credentials to the target computer. Use gpedit.msc and look
at the following policy: Computer Configuration -> Administrative Templates -> System -> Credentials Delegation -> Allow Delegating Fresh Credentials. Verify that
it is enabled and configured with an SPN appropriate for the target computer. For example, for a target computer name “myserver.domain.com”, the SPN can be one of
the following: WSMAN/myserver.domain.com or WSMAN/*.domain.com.

The errors is normal at this point as we haven’t configure everything needed for CredSSP to work.

 

Configure Windows Credential Delegation using the Credential Security Service Provider (CredSSP) module

To allow windows to delegate credentials we will need to set a group policy object (GPO). This GPO’s can be found under [start > run > gpmc.msc (find the computer OU of the servers and edit the existing or create new GPO) > Computer Configuration > Administrative Templates > System > Credential Delegation]. For the purpose of this lab we will enable two policies:

  • Allow delegating fresh credentials
  • Allow delegating fresh credentials with NTLM-only server authentication

As this is our test lab we will enable these policies for the entire vmware.com domain, but normally this should be done only to the necessary servers.



After you have enabled the policies to force them on the PowerShell hosts, run [gpupdate /force]

After the command executes check the System event and search for Warning 10154:

Log Name: System
Source: Microsoft-Windows-WinRM
Date: 6/24/2014 4:28:24 PM
Event ID: 10154
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: LAN1DC1.vmware.com
Description:
The WinRM service failed to create the following SPNs: WSMAN/LAN1DC1.vmware.com; WSMAN/LAN1DC1.
Additional Data
The error received was 8344: %%8344.
User Action
The SPNs can be created by an administrator using setspn.exe utility.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
<System>
<Provider Name=”Microsoft-Windows-WinRM” Guid=”{A7975C8F-AC13-49F1-87DA-5A984A4AB417}” EventSourceName=”WinRM” />
<EventID Qualifiers=”7″>10154</EventID>
<Version>0</Version>
<Level>3</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime=”2014-06-24T13:28:24.000000000Z” />
<EventRecordID>13103</EventRecordID>
<Correlation />
<Execution ProcessID=”0″ ThreadID=”0″ />
<Channel>System</Channel>
<Computer>LAN1DC1.vmware.com</Computer>
<Security />
</System>
<EventData>
<Data Name=”spn1″>WSMAN/LAN1DC1.vmware.com</Data>
<Data Name=”spn2″>WSMAN/LAN1DC1</Data>
<Data Name=”error”>8344</Data>
</EventData>
</Event>

If this warning occurs it is again normal as SPN’s may not have been configured for the WinRM service at this point. We will do this later in this post.

In this use case i’ve aneblled CredSSP only on my Windows PS host. Depending on commands nad scripts you execute you might need to enable CredSSP on mulitple servers.

For more information on the different scenarious , visit
Introducing the Add CredSSP to a PowerShell script (vROCmdlet) workflow

 

Configure Windows Service Principal Names for WinRM

As you have seen previously, indicated by Warning 10154 in the System event logs of the PowerShell  hosts, SPN’s must be set for the servers taking part in the credential delegation. To set SPN’s for the WinRM service on LAN1DC1, run [setspn -S WSMAN/lan1dc1 lan1dc1] and [setspn -S WSMAN/lan1dc1.vmware.com lan1dc1] for both the hostname and FQDN name of the server.

To check if both SPN’s have been set run [setspn –L lan1dc1]

Make sure to do this on LAN1DM1 with corresponding hostname (lan1dm1) and FQDN (lan1dm1.vmware.com) also and any other server that might take part in the delegation.

 

Grant WinRM service access to the private key of valid certificate

Restart the WinRM service. If you do not receive the flowing error in the system event log on the PowerShell host, proceed to the next chapter:

Log Name: System
Source: Microsoft-Windows-WinRM
Date: 6/24/2014 4:29:56 PM
Event ID: 10156
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: LAN1DC1.vmware.com
Description:
The WinRM service failed to initialize CredSSP.
Additional Data
The error received was 0x8009030d.
User Action
Configure CertificateThumbprint setting under the WinRM configuration for the service. Use the thumbprint of a valid certificate and make sure that Network Service has access to the private key of the certificate.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
<System>
<Provider Name=”Microsoft-Windows-WinRM” Guid=”{A7975C8F-AC13-49F1-87DA-5A984A4AB417}” EventSourceName=”WinRM” />
<EventID Qualifiers=”7″>10156</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime=”2014-06-24T13:29:56.000000000Z” />
<EventRecordID>13105</EventRecordID>
<Correlation />
<Execution ProcessID=”0″ ThreadID=”0″ />
<Channel>System</Channel>
<Computer>LAN1DC1.vmware.com</Computer>
<Security />
</System>
<EventData>
<Data Name=”error”>0x8009030d</Data>
</EventData>
</Event>

Log Name: System
Source: Schannel
Date: 6/24/2014 4:29:56 PM
Event ID: 36870
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: LAN1DC1.vmware.com
Description:
A fatal error occurred when attempting to access the SSL server credential private key. The error code returned from the cryptographic module is 0x8009030D. The internal error state is 10001.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
<System>
<Provider Name=”Schannel” Guid=”{1F678132-5938-4686-9FDC-C8FF68F15C85}” />
<EventID>36870</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime=”2014-06-24T13:29:56.642249800Z” />
<EventRecordID>13106</EventRecordID>
<Correlation />
<Execution ProcessID=”612″ ThreadID=”900″ />
<Channel>System</Channel>
<Computer>LAN1DC1.vmware.com</Computer>
<Security UserID=”S-1-5-18″ />
</System>
<EventData>
<Data Name=”Type”>server</Data>
<Data Name=”ErrorCode”>0x8009030d</Data>
<Data Name=”ErrorStatus”>10001</Data>
</EventData>
</Event>

This happens because the user account under which the WinRM service is running doesn’t have access to the private key of the certificate that is being used for WinRM HTTPS communication. By default this should be the “NETWORK SERVICE” account.

To find a list of the certificates which are installed on the PowerShell host , run the folioing command and find the certificate issued for the local computer. Subject should be the FQDN name of the local PowerShell host. In our lab this is lan1dc1.vmware.com. After you have found the certificate note simple/unique containter name

PS E:\> certutil -store My
…
================ Certificate 2 ================
Serial Number: 18000000103c2d965e88d913d3000000000010
Issuer: CN=RootCA, DC=vmware, DC=com
NotBefore: 4/1/2014 5:29 PM
NotAfter: 4/1/2015 5:29 PM
Subject: CN=LAN1DC1.vmware.com
Certificate Template Name (Certificate Type): DomainController
Non-root Certificate
Template: DomainController, Domain Controller
Cert Hash(sha1): 87 22 09 f1 e0 23 65 64 96 71 1b 17 fc 4e 43 1c f5 31 10 1e
Key Container = d5dfb48df2cc6d7928e160b5a8f7cc8e_4ecd4f00-cade-48c7-9297-2140d7e6d7a8
Simple container name: le-DomainController-df6e74aa-331d-43b8-ad97-8002c54856ba
Provider = Microsoft RSA SChannel Cryptographic Provider
Private key is NOT exportable
Encryption test passed
CertUtil: -store command completed successfully.

At this point I would like you to notice that if you issue a [dir /as] in the [C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys] folder you will see a list of hidden files who’s names correspond to the [Unique container name] in the screenshot.

Next step is to grant the service under which the WinRM is running (in our case the NETWORK SERIVCE) read permission on this certificate using [icacls /grant “NETWORK SERVICE”:(R)]

To check the service under which the winrm is running, run the following command [sc qc winrm]

Check again the WinRM configuration and note the CertificateThumprint value. The CertificateThumprint value in WinRM must match the Key Hash value found for the certificate when running [certutil –store My].

PS C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys> winrm get winrm/config
Config
MaxEnvelopeSizekb = 500
MaxTimeoutms = 60000
MaxBatchItems = 32000
MaxProviderRequests = 4294967295
Client
NetworkDelayms = 5000
URLPrefix = wsman
AllowUnencrypted = true
Auth
Basic = true
Digest = true
Kerberos = true
Negotiate = true
Certificate = true
CredSSP = true
DefaultPorts
HTTP = 5985
HTTPS = 5986
TrustedHosts = vco-a-01, vco-a-01.vmware.com, lan1dc1, lan1dc1.vmware.com, lan1dm1, lan1dm1.vmware.com
Service
RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
MaxConcurrentOperations = 4294967295
MaxConcurrentOperationsPerUser = 1500
EnumerationTimeoutms = 240000
MaxConnections = 300
MaxPacketRetrievalTimeSeconds = 120
AllowUnencrypted = true
Auth
Basic = true
Kerberos = true
Negotiate = true
Certificate = true
CredSSP = true
CbtHardeningLevel = Relaxed
DefaultPorts
HTTP = 5985
HTTPS = 5986
IPv4Filter = *
IPv6Filter = *
EnableCompatibilityHttpListener = false
EnableCompatibilityHttpsListener = false
CertificateThumbprint = 87 22 09 f1 e0 23 65 64 96 71 1b 17 fc 4e 43 1cf5 31 10 1e
AllowRemoteAccess = true
Winrs
AllowRemoteShellAccess = true

Now to grant the WinRM service read access to the public key of the certificate, navigate to the MachineKeys folder and run  [icacls <Key Container> /grant “<User/Service Account>”:(R)]. In our case the command looks like this [icacls d5dfb48df2cc6d7928e160b5a8f7cc8e_4ecd4f00-cade-48c7-9297-2140d7e
6d7a8 /grant “Network Service”:(R)]

 

Edit the Kerberos Domain Realm on the vCO Appliance (Optional/Scenario specific)

In our lab we would like to use Kerberos authentication in vCO so we need to edit the krb5.conf file in the appliance to add the vmware.com Kerberos realm. Edit the krb5.conf file as follows:

vco-a-01:~ # vi /usr/java/jre-vmware/lib/security/krb5.conf
[libdefaults]
default_realm = VMWARE.COM
udp_preference_limit = 1
[realms]
VMWARE.COM = {
kdc = lan1dc1.vmware.com
admin_server = lan1dc1.vmware.com
default_domain = vmware.com
}
[domain_realm]
.vmware.com=VMWARE.COM
vmware.com=VMWARE.COM
[logging]
kdc = FILE:/var/log/krb5/krb5kdc.log
admin_server = FILE:/var/log/krb5/kadmind.log
default = SYSLOG:NOTICE:DAEMON

Note than an offten typo is domain_reams instead of domain_realm.

If you are working with the vRA 7.x Embedded vRO the file is located in /etc/krb5.conf.
If you are working with the vRA 8.x Embedded vRO the file is located in /data/vco/usr/lib/vco/app-server/conf/krb5.conf

If you are looking to add PowerShell hosts from multiple domains you must add them all to the Kerbeors file. To learn more, visit:
How to add PowerShell hosts from multiple domains with Kerberos authentication to the same vRO

For some common mistakes when editing files which can lead to malformation and wrong encoding, visit:
Wrong encoding or formatting of Linux configuration files can cause problems in VMware Appliances

Change file owner:

chown root:root /usr/java/jre-vmware/lib/security/krb5.conf

Assign permissions: (rw-r–r–) The owner may read and write a file, while all others may only read the file. A common setting for data files that everybody may read, but only the owner may change.

chmod 644 /usr/java/jre-vmware/lib/security/krb5.conf

Restart the vCO Appliance.

 

Add the PowerShell Host to vCO

You need to add a PS Host in vCO with the following properties:

  • Port: 5986
  • Powershell remote host type: WinRM
  • Transport protocol: HTTPS
  • Authentication: Kerberos
  • User name: <Administrator_user_name> (user must be at least local administrator on the PS Host in order to be able to add the host. Recommended is to user domain level account as this will satisfy most of the integration use cases that exist. )

If you want to use account other than the default administrator account to add the host with, visit
Adding vCO Powershell Host with account other than the default domain administrator account

Following are screenshots from an example PS host configuration.

If the adding the PowerShell host fails with error similar to the following, you probably have made a mistake when previously editing the Kerberos Domain Realm (krb5.conf) configuration file:

Cannot locate default realm (Dynamic Script Module name : addPowerShellHost#12
tem: ‘Add a PowerShell host/item8’, state: ‘failed’, business state: ‘Error’, exception: ‘InternalError: java.net.ConnectException: Connection refused (Workflow:Import a certificate from URL with certificate alias / Validate (item1)#5)’
workflow: ‘Add a PowerShell

If this is the case, go back and double-check the configuration.

Testing CredSSP

Now let’s test again CredSSP as we did before. To test a multi-hop authentication and Kerberos authentication make a remote call to server lan1dc1 over either HTTP or HTTPS. The command calls LAN1DC1 and asks it to list the contents of remote share on LAN1DM1 (user token delegation).

PS C:\> winrs -r:http://lan1dc1.vmware.com:5985 -ad -u:vmware\administrator -p:VMware1! powershell.exe dir \\lan1dm1.vmware.com\share

PS C:\> winrs -r:https://lan1dc1.vmware.com:5986 -ad -u:vmware\administrator -p:VMware1! powershell.exe dir \\lan1dm1.vmware.com\share

This time you should get results listed.

If you receive the following error go back to the Grant WinRM service access to the private key of valid certificate chapter and do the steps described there:

Winrs error:The WinRM client received an HTTP server error status (500), but the remote service did not include any othe
r information about the cause of the failure.PS C:\Users\Administrator>

Now let’s move the PowerShell code [dir \\lan1dm1.vmware.com\share] into a PowerShell script file called PS.ps1 and copy the file on the local E:\ drive on the LAN1DC1 PowerShell host. We are selecting E:\ drive only for testing purpose, but this can be any other local drive on the host.

Now test the CredSSP again, but this time specifying the PS.ps1 file. As expected this works again. The only change we have done is that we are not executing a PowerShell command directly, but have moved the PowerShell command into PowerShell script file and we are calling the PowerShell script file for execution.


Not lest call this same PowerShell script file from vCO.

In vCO I have already added both LAN1DC1 and LAN1DM1 as Powershell hosts and have used Kerberos for authentication. I have added these both as HTTP hosts and HTTPS hosts. We will use the [Invoke an external script] workflow to test the execution. This is the equivalent of running WinRS command above.


Although you would expect this command to compete successfully, this swill fail with the error below.

Q: Why ?
A: The –ad switch.

Take another look at the WinRS command we have executed previously

PS C:\> winrs -r:http://lan1dc1.vmware.com:5985 -ad -u:vmware\administrator -p:VMware1! powershell.exe E:\PS.ps1

Notice the –ad switch we have used. This specifies that the user’s credentials can be used to access a remote share, for example, found on a different machine than the target endpoint. Problem is that when vCO calls WinRM it doesn’t include this option, so user credentials are not used to access remote shares. At least not in the current vCO PowerShell Plugin. The workaround for this is explained later in this post.

 

Methods to add CredSSP to your script

Use Invoke-Command in the PowerShell code

As you have seen in this post vCO doesn’t allow user credentials to be used within a PowerShell call to access remote shares. In order to workaround this we must modify our current PS.ps1 code

We need to modify this as follows (this is only example code)

:
$username = “vmware\administrator”
$password = “VMware1!”
$psHost = “lan1dc1”
Write-Host “”
Write-Host “currently executing script on host:” $psHost
Write-Host “Connecting to remote host/share via credSSP”
$cred = New-Object System.Management.Automation.PSCredential -ArgumentList @($username,(ConvertTo-SecureString -String $password -AsPlainText -Force))
Invoke-Command -ComputerName $psHost -ScriptBlock { dir \\lan1dm1.vmware.com\share } -Authentication CredSSP -credential $cred

Key takeaways from this script:

  • For the sake of the example I have hardcoded the username and password that will be used to access remote shares.
  • I have used the ConvertTo-SecureString which converts encrypted standard strings to secure strings to pass the credentials to the PowerShell script that will be executed. It can also convert plain text to secure strings .
  • When running the Inveoke-Command cmdlet you need to specify the [–Authneticaiotn CredSSP] switch.

Now if you go back in vCO and execute again the [Invoke an external script] workflow as we did previously, this should succeed and you should see the credential delegation in action.

 

Security Hardening

Once you get everything up and running you should spend some time securing you scripts and WinRM.
Here are few advices

  • Store you passwords as SecureStrings in vRO.
  • Add the PS Host with HTTPS, and not HTTP to encrypt traffic b/n the vRO and the PS Host server.
  • Disable WinRM HTTP Listeners on all Windows servers including the PS Host. Use only HTTPS listener to encrypt traffic b/n servers.
  • Disable WinRM AllowUnencrypted on all Windows Servers so that WinRM traffic is encrypted even if you use HTTP.
  • Disable unused WInRM Authentication mentors.
  • Use the –UseSSL switch with Invoice-Command commands to force HTTPS.

Refer to the Security Considerations chapter for more information

.Also visit, Securing Your PowerShell Execution and Password in vRO (SKKB1035)

 

Use the Add CredSSP to a Powreshell Script vROC workflow

Instead of hardcoding the credentials in the PowerShell script you can use the Add CredSSP to a Powershell Script vROC workflow. This way your credentials will remain safe as SecureString in vRO and encrypted in the vRO databse, and not exposed. The workflow is part of the com.SpasKaloferov workflow library package.

If you want to learn more about the Add CredSSP to a PowerShell script (vROCmdlet) workflow , visit:
Introducing the Add CredSSP to a PowerShell script (vROCmdlet) workflow

If you want to download latest version of the vROC workflow package library, visit:
com.SpasKaloferov vCO (vRO) workflow library package

If you are new to what a vROC is, visit:
Let’s vROC with vRO!

 

Configure PowerShell execution policies and allow execution of files downloaded from the Internet.

Now that we have configured CredSSP and successfully used it in vCO, let’s take a look at few more challenges that may arise when dealing with remote PowerShell execution. Taking our scenario further lets say that we have downloaded a script file (pwoershell/bat/vbs/ect..) from internet or from any other location using a browser. It can also be that someone else downloaded this script file and we have copied it from them via USB/HDD/CD or any other means of transport of information. Key point here is that the script file has been downloaded originally from the internet.

Let’s say we have downloaded the PS.ps1 file from the previous example from internet. If you are running file, which is not digitally signed, which is local for the system/host change the PowerShell execution policy to Unrestricted , if file is remote (or on a locally mapped share) change it to bypass. If you do not set it to bypass you will get the following message each time you open file which is downloaded from the internet:

This is because windows threads this file as downloaded from internet.

To change PowerShell execution policies use the following commands:

To view the current execution policy

PS:\> get-ExecutionPolicy

To set the execution policy

PS:\> set-ExecutionPolicy –ExecutionPolicy [Unrestricted | Bypass | …]

Note: You can use Unblock-file (PowerShell v4 and above) to temporary allow access but in is not very usable in automation and scripts.

For lerning more about the command, visit:
Unblock-File

Q: How does Windows determine that the file has originated from the internet?
A: ADS data stream.

When file is downloaded from internet ADS stream is added and is used to store meta-information about the file. For example, the Zone identifier stores whether the file was downloaded from the internet. To check the ADS stream use the Get-Item PowerShell cmdlet as follows [ get-Item <Path_To_File> -Stream *] to list all streams for given file.

If the file was downloaded from the internet you will normally see something like [Zone.Identifier:$DATA 26]. If this is the case you can use the [remove-content ] PS command to remove the zone.identifier. The zone.identifier has 6 possible values

  1. NoZone = -1
  2. MyComputer = 0
  3. Intranet = 1
  4. Trusted = 2
  5. Internet = 3
  6. Untrusted = 4

Final Step

If all went well, go grab a beer.

<? include TEMPLATEPATH.”/../../../itBlogDisclaimer.php”; ?>

1 thought on “Using CredSSP with the vCO PowerShell Plugin (SKKB1002)”

  • When I went to run the commands
    Here are few of the commands to get you started:
    winrm set winrm/config/service @{AllowUnencrypted=”true”}
    I actually had to put the arguement in single quotes i.e.
    winrm set winrm/config/service ‘@{AllowUnencrypted=”true”}’
    otherwise I got invalid use of the command

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *