Wrong encoding or formatting of Linux configuration files can cause problems in VMware Appliances (SKKB1017)

Introduction

In this blog post we will look into some use cases that can lead to malformed or wrongly encoded files,  which when used on Linux/Unix/VMware Appliances, can break different functionality. In particular we will see how malformed krb5.conf can break Kerberos authentication on a VMware vRealize Orchestrator Appliance.

Lab Environment

The following environment has been used in all of the examples below:

  • Active Directory Domain called vmware.com
  • Windows Server 2012 R2 (with FQDN lan1dc1.vmware.com)
    • Domain Controller
    • DNS Server.
    • PowerShell v4 Host.
    • .NET 4 Framework installed (4.0.30319)
  • Windows Server 2012 R2 (with FQDN lan2dc1.support.vmware.com)
    • Domain Controller
    • DNS Server.
    • PowerShell v4 Host.
    • .NET 4 Framework installed (4.0.30319)
  • Windows Server 2008 R2 SP1 (with FQDN lan1dm1.vmware.com)
    • Domain Member
    • PowerShell v3 Host.
    • .NET 4 Framework installed (4.0.30319)
  • vCO 5.5.1.0 Virtual Appliance (with FQDN vco-a-01.vmware.com)
    • vCO PowerShell Plugin installed (version 1.0.6.2283945)

The full lab logical design can be seen HERE.

 

Use Case: Malformed Krb5.conf

Introduction

To illustrate how encoding or malformation can cause an issue, we will be using a krb5.conf file on a VMware vRealize Orchestrator Appliance as an example. We will replace it with one edited under Windows with different formatting and encoding and see how this can break Kerberos authentication on the appliance. We have taken the Kerberos configuration file as an example but this can happen with any other file on a Linux system.  Few common things that can cause a file to be unusable on a linux/VMware Appliance:

  • File doesn’t use ANSI as UTF-8 encoding. UTF-8 uses one to four bytes per code point and, being compact for Latin scripts and ASCII-compatible, provides the de facto standard encoding for interchange of Unicode text. It is used by FreeBSD and most recent Linux distributions, including VMware Appliances, as a direct replacement for legacy encodings in general text handling.
  • File is not formatted as UNIX. For example the file might be formatted as Dos\Windows. Problem is that line breaks are different between Windows and Unix. Unix uses only U+000A (LF) while Windows uses two characters for that: U+000D U+000A (CR+LF)
  • First bite is U+FFFE (the result of byte-swapping). This is again caused by incorrect formatting. For example the replacement character � (often a black diamond with a white question mark) is a symbol found in the Unicode standard at codepoint U+FFFD in the Specials table. It is used to indicate problems when a system is not able to render a stream of data to a correct symbol. It is most commonly seen when a font does not contain a character, but is also seen when the data is invalid and does not match any character. A poorly implemented text editor might save the replacement in UTF-8 form; the text file data will then look like this: 0x66 0xEF 0xBF 0xBD 0x72.  Since the replacement is the same for all errors this makes it impossible to recover what character was intended? This might be the case not only with the first bite character, but with any other character in the document that cannot be replaced during formatting.
    Reference: Specials (Unicode block)

 

Correctly encoded and formatted file

Let’s take the krb5.conf file as an example. Let’s say, for some reason, you have edited the file on a Windows system outside the Linux/VMware Appliance and you have copied it back afterwards. Editing such files on external systems or with external tools can sometimes break the encoding of the files. This will cause issues on the Linux/VMware appliance when the file is accessed and read.

Let’s see an original  krb5.conf file opened on Windows with Notepad++. Note that the file has not been edited and is direct copy from the Linux/VMware Appliance machine.

Note two things in the status bar :

  • Formatting is UNIX
  • Encoding is ANSI as UTF-8.

This is a file which in in correct formatting and encoding.

Malformed file

Lets take a look at a krb5.conf which was created entirely in Windows (wasn’t copied from a Linux appliance)

Note two things in the status bar

  • Formatting is Dos\Windows
  • Encoding is UCS-2 Little Endian. The encoding in your case might be different then UCS-2 Little Endian.

This file is “broken” and cannot be used on a Linux/VMware appliance.
For example, if the above file is viewed under Linux you will notice a character in the upper left corner which is causing a problem:

Same file viewed in a hex editor under Window shows the following:

Note the first bite U+FFFE (the result of byte-swapping). This is causing our issue. Note that this is just an example and the encoding in your case might be broken in another place in the file.
Important here is that two actions must take place in order for this, wrongly encoded and formatted file, to be usable again:

  • The encoding must be changed from UCS-2 Little Endian to ANSI as UTF-8
  • Formatting must be changed from DOS\Windows to UNIX

Fixing a malformed file

The first is easy to be accomplished. You can use Notepad++ and change the encoding from the Encoding menu to Encode in UTF-8 without BOM.

Note that the encoding in the status bar changes to ANSI as UTF-8.
So far so good.
This file is still unusable on a linux/VMware appliance.

The second part is the tricky part. Windows doesn’t offer native capability to format in UNIX, so you cannot change the Dos\Windows formatting so easily.

As mentioned in the beginning of the article the two most common malformations are:

  1. File is not formatted as UNIX.
  2. First bite is U+FFFE (the result of byte-swapping). This might be the case not only with the first bite character, but with any other character in the document that cannot be replaced during formatting.

In our case we have both:

  1. a character (U+FFFE ) at the very beginning of the document which needs to be removed.
  2. Formatting as Dos\Windows which introduces an extra character for line breaks.

To fix 1) you can always use an HEX editor and remove the malformed character. Theoretical you can use a HEX editor for fix 2) also by removing all the extra characters at the line break, but this can be a time consuming error and open to human errors.
To fix 2) you have two options:

      a) Use PowerShell
      b) Use Notepad++

To view these line breaks in notepad++ you must enable [View > Show Symbol > Show All Characters].

Unix formatted line breaks file:

Dos\Windows formatted line breaks file:

To use a) just save it as .ps1 script and run it. Note that krb5.conf below is the name of the file you want to edit. Note also that the file will be overridden upon execution of the script.

Get-item krb5.conf | ForEach-Object {
$contents = [IO.File]::ReadAllText($_) -replace "rn?", "`n"
$utf8 = New-Object System.Text.UTF8Encoding $false
[IO.File]::WriteAllText($_, $contents, $utf8)
}

To use b) and use Notepad++ select [Edit > EOL Conversation > UNIX/OSX Format]

Now you have fixed the “broken” file.

As  you can see from the status bar it is both:

  • Formatted is UNIX
  • Encoded in ANSI as UTF-8.

 

vRO (Add A PowerShell Host) Workflow fails with “cannot locate default realm” error

Let’s say we have the malformed file from earlier placed on a VMware vRealize Orchestrator Appliance and we are trying to perform one of the following actions:

  • Run a workflow that uses Kerberos authentication.
  • Add A Powershell Host with HTTS/Kerberos
  • Any other operation that might require Kerberos authentication.

In any of the above cases you might receive the following, or similar , error:

Cannot locate default realm

The error might vary depending on how and where the file is malformed.

The catalina.out log recorded the following:

2015-06-09 15:11:31.368+0300 [WorkflowExecutorPool-Thread-1] INFO  {administrator@vmware.com:Add a PowerShell host:EF8180808080808080808080808080803D80808001270557368849c62c352aa82:ff8080814dd83a8c014dd83b7c560005} [OverthereWinRMPowerShell] Connecting to cifs:WINRM_HTTPS://Administrator@support.vmware.com@lan2dc1.support.vmware.com:5986
73.145: [GC73.145: [ParNew: 14040K->856K(14464K), 0.0045480 secs] 158407K->145223K(228516K) icms_dc=4 , 0.0046340 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
Java config name: /usr/java/jre-vmware/lib/security/krb5.conf
>>> KdcAccessibility: reset
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 16 23 1 3.
>>> KrbAsReq creating message
getRealmFromDNS: trying vmware.com
2015-06-09 15:11:42.569+0300 [WorkflowExecutorPool-Thread-1] ERROR {administrator@vmware.com:Add a PowerShell host:EF8180808080808080808080808080803D80808001270557368849c62c352aa82:ff8080814dd83a8c014dd83b7c560005} [WinRmPowerShellClient] send message on https://lan2dc1.support.vmware.com:5986/wsman error , document in <?xml version="1.0" encoding="UTF-8"?>
<env:Envelope xmlns:env="http://www.w3.org/2003/05/soap-envelope">
<env:Header>
<a:To xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing">https://lan2dc1.support.vmware.com:5986/wsman</a:To>
<a:ReplyTo xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing">
<a:Address mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</a:Address>
</a:ReplyTo>
<w:MaxEnvelopeSize xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" mustUnderstand="true">153600</w:MaxEnvelopeSize>
<a:MessageID xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing">uuid:70C8CFB3-340A-4ED7-97E0-27020742BA78</a:MessageID>
<w:Locale xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" mustUnderstand="false" xml:lang="en-US"/>
<p:DataLocale xmlns:p="http://schemas.microsoft.com/wbem/wsman/1/wsman.xsd" mustUnderstand="false" xml:lang="en-US"/>
<w:OperationTimeout xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd">PT180.000S</w:OperationTimeout>
<a:Action xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/09/transfer/Create</a:Action>
<w:ResourceURI xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" mustUnderstand="true">http://schemas.microsoft.com/wbem/wsman/1/windows/shell/cmd</w:ResourceURI>
<w:OptionSet xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd">
<w:Option Name="WINRS_NOPROFILE">FALSE</w:Option>
<w:Option Name="WINRS_CODEPAGE">437</w:Option>
</w:OptionSet>
</env:Header>
<env:Body>
<rsp:Shell xmlns:rsp="http://schemas.microsoft.com/wbem/wsman/1/windows/shell">
<rsp:InputStreams>stdin</rsp:InputStreams>
<rsp:OutputStreams>stdout stderr</rsp:OutputStreams>
</rsp:Shell>
</env:Body>
</env:Envelope>
, document out [EMPTY],
2015-06-09 15:11:42.570+0300 [WorkflowExecutorPool-Thread-1] ERROR {administrator@vmware.com:Add a PowerShell host:EF8180808080808080808080808080803D80808001270557368849c62c352aa82:ff8080814dd83a8c014dd83b7c560005} [PowerShellHostManager] Cannot locate default realm
com.vmware.o11n.plugin.powershell.model.PowerShellException: Cannot locate default realm
at com.vmware.o11n.plugin.powershell.model.BaseSession.<init>(BaseSession.java:77)
at com.vmware.o11n.plugin.powershell.model.impl.winrm.WinRmSession.<init>(WinRmSession.java:31)
at com.vmware.o11n.plugin.powershell.model.impl.winrm.WinRmRemoteConnector.openSession(WinRmRemoteConnector.java:64)
at com.vmware.o11n.plugin.powershell.model.Host.openSession(Host.java:118)
at com.vmware.o11n.plugin.powershell.model.Host.invokeCommand(Host.java:259)
at com.vmware.o11n.plugin.powershell.scripting.PowerShellHostManager.validateConnectivityAndFunctionallity(PowerShellHostManager.java:145)
at com.vmware.o11n.plugin.powershell.scripting.PowerShellHostManager.validatePowerShellHost(PowerShellHostManager.java:96)
at com.vmware.o11n.plugin.powershell.scripting.PowerShellHostManager.update(PowerShellHostManager.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at ch.dunes.vso.sdk.WrappedJavaMethod.call(WrappedJavaMethod.java:142)
at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1473)
at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
at ch.dunes.scripting.server.script.DynamicFunction.call(DynamicFunction.java:179)
at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:1473)
at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:815)
at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:109)
at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394)
at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091)
at org.mozilla.javascript.InterpretedFunction.exec(InterpretedFunction.java:120)
at ch.dunes.scripting.server.script.MainScriptingObject.executeScript(MainScriptingObject.java:237)
at ch.dunes.scripting.server.script.MainScriptingObject.executeScript(MainScriptingObject.java:223)
at ch.dunes.workflow.engine.mbean.WorkflowScriptRunner.execute(WorkflowScriptRunner.java:192)
at ch.dunes.workflow.engine.mbean.runner.WorkflowItemTaskRunner.execute(WorkflowItemTaskRunner.java:43)
at ch.dunes.workflow.engine.mbean.runner.WorkflowItemTaskRunner.execute(WorkflowItemTaskRunner.java:24)
at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.executeItem(WorkflowHandler.java:923)
at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.requestElementExecution(WorkflowHandler.java:887)
at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.handleWorkflowTokenNextStep(WorkflowHandler.java:610)
at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.executeToken(WorkflowHandler.java:500)
at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.handleTokenExecution(WorkflowHandler.java:439)
at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.access$100(WorkflowHandler.java:101)
at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler$1.execute(WorkflowHandler.java:349)
at ch.dunes.model.ar.AccessRightsTemplate.executeWithAccessRights(AccessRightsTemplate.java:16)
at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.doExecute(WorkflowHandler.java:345)
at ch.dunes.workflow.engine.mbean.helper.WorkflowHandler.run(WorkflowHandler.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: com.vmware.o11n.plugin.powershell.remote.AuthenticationException: Cannot locate default realm
at com.vmware.o11n.plugin.powershell.remote.impl.winrm.KerberosTokenGenerator.generateToken(KerberosTokenGenerator.java:88)
at com.xebialabs.overthere.cifs.winrm.connector.JdkHttpConnector.sendMessage(JdkHttpConnector.java:69)
at com.vmware.o11n.plugin.powershell.remote.impl.winrm.WinRmPowerShellClient.sendMessage(WinRmPowerShellClient.java:292)
at com.vmware.o11n.plugin.powershell.remote.impl.winrm.WinRmPowerShellClient.openShell(WinRmPowerShellClient.java:284)
at com.vmware.o11n.plugin.powershell.remote.impl.winrm.WinRmPowerShellClient.open(WinRmPowerShellClient.java:81)
at com.vmware.o11n.plugin.powershell.remote.impl.winrm.CifsWinRmPowerShellConnection.open(CifsWinRmPowerShellConnection.java:132)
at com.vmware.o11n.plugin.powershell.remote.impl.WinRmPowerShellTerminal.startShellWithSerialization(WinRmPowerShellTerminal.java:72)
at com.vmware.o11n.plugin.powershell.model.BaseSession.<init>(BaseSession.java:71)
… 42 more
Caused by: javax.security.auth.login.LoginException: Cannot locate default realm
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)
at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at javax.security.auth.login.LoginContext.invoke(Unknown Source)
at javax.security.auth.login.LoginContext.access$000(Unknown Source)
at javax.security.auth.login.LoginContext$5.run(Unknown Source)
at javax.security.auth.login.LoginContext$5.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokeCreatorPriv(Unknown Source)
at javax.security.auth.login.LoginContext.login(Unknown Source)
at com.vmware.o11n.plugin.powershell.remote.impl.winrm.KerberosTokenGenerator.login(KerberosTokenGenerator.java:142)
at com.vmware.o11n.plugin.powershell.remote.impl.winrm.KerberosTokenGenerator.generateToken(KerberosTokenGenerator.java:82)
… 49 more
Caused by: KrbException: Cannot locate default realm
at sun.security.krb5.PrincipalName.<init>(Unknown Source)
at sun.security.krb5.KrbAsReq.<init>(Unknown Source)
at sun.security.krb5.KrbAsReqBuilder.build(Unknown Source)
at sun.security.krb5.KrbAsReqBuilder.send(Unknown Source)
at sun.security.krb5.KrbAsReqBuilder.action(Unknown Source)
… 64 more
Caused by: KrbException: Cannot locate default realm
at sun.security.krb5.Config.getDefaultRealm(Unknown Source)
… 69 more
Caused by: KrbException: Generic error (description in e-text) (60) – Unable to locate Kerberos realm
at sun.security.krb5.Config.getRealmFromDNS(Unknown Source)
… 70 more
2015-06-09 15:11:42.575+0300 [WorkflowExecutorPool-Thread-1] WARN  {administrator@vmware.com:Add a PowerShell host:EF8180808080808080808080808080803D80808001270557368849c62c352aa82:ff8080814dd83a8c014dd83b7c560005} [WorkflowItemTaskRunner] Script execution error on workflow : Add a PowerShell host / ‘addPowerShellHost'(item0) : Cannot locate default realm (Dynamic Script Module name : addPowerShellHost#16)
110.955: [GC110.955: [ParNew: 13707K->1600K(14464K), 0.0053760 secs] 158074K->146500K(228516K) icms_dc=4 , 0.0055080 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
110.987: [GC110.987: [ParNew: 14461K->1599K(14464K), 0.0102790 secs] 159361K->149682K(228516K) icms_dc=4 , 0.0103850 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
205.773: [GC205.773: [ParNew: 14463K->962K(14464K), 0.0169110 secs] 162546K->149241K(228516K) icms_dc=4 , 0.0170590 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
339.345: [CMS-concurrent-mark: 0.529/270.802 secs] [Times: user=9.18 sys=1.41, real=270.82 secs]
339.356: [CMS-concurrent-preclean-start]
339.364: [CMS-concurrent-preclean: 0.008/0.008 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
339.364: [CMS-concurrent-abortable-preclean-start]
363.008: [GC363.008: [ParNew: 13826K->670K(14464K), 0.0219980 secs] 162105K->149129K(228516K) icms_dc=4 , 0.0221870 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
CMS: abort preclean due to time 377.457: [CMS-concurrent-abortable-preclean: 0.094/38.093 secs] [Times: user=0.75 sys=0.13, real=38.10 secs]
377.462: [GC[YG occupancy: 6923 K (14464 K)]377.462: [Rescan (parallel) , 0.0186980 secs]377.481: [weak refs processing, 0.0077340 secs]377.489: [scrub string table, 0.0032630 secs] [1 CMS-remark: 148459K(214052K)] 155382K(228516K), 0.0299830 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
377.501: [CMS-concurrent-sweep-start]
377.661: [CMS-concurrent-sweep: 0.160/0.160 secs] [Times: user=0.17 sys=0.00, real=0.16 secs]
377.662: [CMS-concurrent-reset-start]
377.674: [CMS-concurrent-reset: 0.012/0.012 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
392.435: [GC392.435: [ParNew: 13534K->359K(14464K), 0.0120440 secs] 159014K->145840K(256932K) icms_dc=0 , 0.0121660 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
408.445: [GC408.446: [ParNew: 13222K->304K(14464K), 0.0110790 secs] 158702K->145784K(256932K) icms_dc=0 , 0.0112450 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
685.776: [GC685.776: [ParNew: 13168K->1116K(14464K), 0.0425550 secs] 158648K->146597K(256932K) icms_dc=0 , 0.0432600 secs] [Times: user=0.08 sys=0.00, real=0.04 secs]
708.946: [GC708.946: [ParNew: 13980K->811K(14464K), 0.0158050 secs] 159461K->146493K(256932K) icms_dc=0 , 0.0159650 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
858.814: [GC858.817: [ParNew: 13675K->1301K(14464K), 0.2605310 secs] 159357K->146983K(256932K) icms_dc=0 , 0.2642120 secs] [Times: user=0.34 sys=0.03, real=0.27 secs]
1133.996: [GC1134.002: [ParNew: 14165K->1337K(14464K), 0.3125040 secs] 159847K->147426K(256932K) icms_dc=0 , 0.3185550 secs] [Times: user=0.60 sys=0.00, real=0.32 secs]
2015-06-09 15:30:55.018+0300 [vcoSystemTaskScheduler-3] INFO  {} [PurgeSessionAdaptor] Started verification of 22 session(s) against the authentication provider.
2015-06-09 15:30:55.137+0300 [vcoSystemTaskScheduler-3] INFO  {} [PurgeSessionAdaptor] Verified 1 unique session(s) in 108ms.
1382.206: [GC1382.208: [ParNew: 14201K->1080K(14464K), 0.1752780 secs] 160290K->147197K(256932K) icms_dc=0 , 0.1780740 secs] [Times: user=0.34 sys=0.00, real=0.18 secs]

 

DISCLAIMER; This is a personal blog. Any views or opinions represented in this blog are personal and belong solely to the blog owner and do not represent those of people, institutions or organizations that the owner may or may not be associated with in professional or personal capacity, unless explicitly stated. Any views or opinions are not intended to malign any religion, ethnic group, club, organization, company, or individual.
All content provided on this blog is for informational purposes only. The owner of this blog makes no representations as to the accuracy or completeness of any information on this site or found by following any link on this site. The owner will not be liable for any errors or omissions in this information nor for the availability of this information. The owner will not be liable for any losses, injuries, or damages from the display or use of this information.
Photos
Unless stated, all photos are the work of the blog owner and are licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. If used with watermark, no need to credit to the blog owner. For any edit to photos, including cropping, please contact me first.
Recipes
Unless stated, all recipes are the work of the blog owner and are licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. Please credit all recipes to the blog owner and link back to the original blog post.
Downloadable Files
Any downloadable file, including but not limited to pdfs, docs, jpegs, pngs, is provided at the user’s own risk. The owner will not be liable for any losses, injuries, or damages resulting from a corrupted or damaged file.
Comments
Comments are welcome. However, the blog owner reserves the right to edit or delete any comments submitted to this blog without notice due to
– Comments deemed to be spam or questionable spam
– Comments including profanity
– Comments containing language or concepts that could be deemed offensive
– Comments containing hate speech, credible threats, or direct attacks on an individual or group
The blog owner is not responsible for the content in comments.
This policy is subject to change at anytime.

Leave a Reply

Your email address will not be published. Required fields are marked *