Automating .NET 3.5 offline installation on Azure roles

We have a .NET 3.5 based web application which runs on Azure. As a part of deployment we runs couple of start-up tasks. One of the start-up task install .NET 3.5 in the roles.

Recently we faced down time for this website. We did some analysis and found start-up tasks together takes around 30 minutes to finish. The reason for down time was azure Fabric Controller will wait 15 minutes for a role in an upgrade domain to start before proceeding to a role in the next upgrade domain. 15 minute limit is about how long the Fabric Controller will wait for a role instance to go to the Ready state during a role update. This cause both of our role instances to be in busy state and stop responding to HTTP requests for long time.

We checked the start-up scripts one by one and noticed that the start-up script that installs .NET 3.5 runs the below command:

PowerShell -Command "Add-WindowsFeature Net-Framework-Core" >> "%TEMP%\StartupLog.txt" 2>&1

This command connect to WSUS (Windows Server Update Service) to install .NET 3.5. Downloading .NET 3.5 files will take some time because NET 3.5 install files are around 250 MB. Note that our roles are running in 'Windows Server 2012 R2' VMs. Only way to automate .NET 3.5 installation in OS starting from 'Windows Server 2008 R2' is via 'Add-WindowsFeature' command.

If you have the .NET 3.5 install files available locally then we can run above command with -Source option:

Add-WindowsFeature Net-Framework-Core –Source "<Path-To-NET35-Install-Files>"

How to get NET35-Install-Files?

If you have Windows Installation media, you can find these files under "\sources\sxs".  You can compress contents of this directory and include it in the azure package or can upload to azure blob storage.

Now we need to modify the start-up task to get this compressed file, extract it and run 'Add-WindowsFeature Net-Framework-Core' with -Source option.

Note that we cannot extract these files to role's %TEMP% directory, because we need more than 250 MB disk space and if we use %TEMP% then we will get 'Not enough space on disk' error. So solution is to use Azure Role local store, allocate 350 MB space for the resource. Let's call this local resource as 'net35resource'

Since we have to un-compress the zip file, include 7z binaries in the azure package. Easy way to get these binaries is to install 7zip in your dev machine, go to 'ProgramFiles(x86)\7-Zip' and copy the files '7z.exe', '7z.dll' and '7-zip.dll'.

Now we will write a powershell script file, lets name it 'installnet35.ps1'.

# Method that returns path to the directory holding 'installnet35.ps1' script. 
function Get-ScriptDirectory
{
  $Invocation = (Get-Variable MyInvocation -Scope 1).Value
  Split-Path $Invocation.MyCommand.Path
}

# Gets path to the local resource we reserved for manipulating the zip file.
[void]([System.Reflection.Assembly]::LoadWithPartialName("Microsoft.WindowsAzure.ServiceRuntime"))
$localStoreRoot = ([Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment]::GetLocalResource("net35resource")).RootPath.TrimEnd('\\')

# .NET 3.5 source (in blob storage)
# Note that you can also include the .NET 3.5 source zip file in the package
$net35Source = "http://<storage-account-name>.blob.core.windows.net/<container>/net35.zip"

# Destination path for the zip file
$net35ZipDestination = Join-Path $localStoreRoot "net35.zip"

# Use WebClient to download the the zip file
$webClient = New-Object System.Net.WebClient
$webClient.DownloadFile($net35Source, $net35ZipDestination)

# Destination path to hold the extracted files
$net35ExtractDestination = Join-Path $localStoreRoot "net35"
$pathExists = Test-Path $net35ExtractDestination
if (!$pathExists)
{
  new-item $net35ExtractDestination -itemtype directory
}

# Build command to unzip
$zipTool = (Join-Path (Get-ScriptDirectory) "\7z.exe")
$unzipCommandArgs = "x " + $net35ZipDestination + " -o$net35ExtractDestination -y"

# Unzip the file
Start-Process $zipTool $unzipCommandArgs -NoNewWindow -Wait
$net35ExtractDestination = Join-Path $net35ExtractDestination "net35"

# Install .NET 3.5 using -Source option
Install-WindowsFeature NET-Framework-Core –Source Join-Path $net35ExtractDestination

Next we need a batch file to run the above powershell script in unrestricted mode. lets name it 'installnet35.cmd'.

PowerShell -ExecutionPolicy Unrestricted "%~dp0..\..\Startup\installnet35.ps1" >> "%TEMP%\StartupLog.txt" 2>&1

We need to include all these files in project's Startup directory as shown below.

Note that for all these files under 'Startup' you need to set 'Build Action' as 'Content' and 'Copy To output Directory' as 'Copy Always'

Last step is to register 'installnet35.cmd' as a startup task via 'ServiceDefinition.config' file.

<Startup priority="-2">
  <Task commandLine=".\Startup\installnet35.cmd" executionContext="elevated" taskType="simple">
    <Environment>
      <Variable name="EMULATED">
        <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
      </Variable>
    </Environment>
  </Task>
</Startup>

multipart/form-data: HTML file upload internal explained with C# implementation

In this post I will explain how browser encode an html form before sending to server and how to use a C# program to perform same type of encoding.

Below is a very basic html form with a file and two text html controls.

<!DOCTYPE html>
<html>
  <body>
    <form action="http://localhost:80/User/Register.php" method="post" 

          enctype="multipart/form-data">
      First name: <input type="text" name="fname"><br>
      Last name:  <input type="text" name="lname"><br>
      Photo:      <input type="file" name="photo"><br>
      <input type="submit" value="Submit">
    </form>
  </body>
</html>

This will render the form like:

When we click submit, browser encode the form data and send it to server. 

Note that we declare "multipart/form-data" (via form enctype attribute) as encoding to be used, this is the form encoding
that supports forms with 'file' control. Another encoding type (default) is "application/x-www-form-urlencoded"

If we look into the request send to the server using fiddler, it looks like:

The Content-Type "multipart/form-data" specifies how the form data set is encoded. Form data set is the sequence
of "name and value" of html controls defined inside 'form' node.

For example in the above form, data set is { { fname, "anu" }, { lname "chandy" }, { photo "<content-of-file>" } }

The encoding "multipart/form-data" says,  message contains a series of parts each part contain a name-value pair in
the 'form data set'. These parts are in the same order as the corresponding controls in the form. 

Value of the boundary attribute (in the above example "----WebKitFormBoundaryUOaE2JpZxwqbIieW") declare the boundary
that separates each parts.

Each part has following format:

// New line
// New line
--<value-of-boundary-attribute>
Content-Disposition: form-data; name="<control-name>" [filename="name-of-the-file-if-control-is-file"]
[Content-Type: <The-content-type-of-uploded-file-if-control-is-file>]
// New line
// New line
Value of the control or file content if control is 'file'

Each part starts with two dashes (--) followed by the boundary string defined by the Content-Type boundary attribute. 

After all parts, the message ends with a string that is formed by appending and prep-ending the boundary attribute by two dashed (--) 

Below is a C# program that shows how to use MultipartFormDataContent class to generate a message that conform to "multipart/form-data" protocol.

The usage is:

NameValueCollection formdata = new NameValueCollection();
myCol.Add( "fname", "anu" );
myCol.Add( "lname", "chandy" );

SubmitFormViaPOST("http://localhost:80/User/Register.php", // Server url
            "photo", // Name for file control
            "./images/MyProfilePic.jpg", // Path to the file to be uploaded
            "image/jpg" // The file Content Type,
            formdata, // Other form data
            CancellationToken.None).Wait();

Function definition is:

public async Task SubmitFormViaPOST(string serverUrl,
            string fileControlName,
            string filePath,
            string fileContentType,
            NameValueCollection formData,
            CancellationToken cancellationToken)
        {
            FileInfo fileInfo = new FileInfo(filePath);
            Uri RequestUri = new Uri(serverUrl);

            using (var multiPartContent = new MultipartFormDataContent("---------------------------" + DateTime.Now.Ticks.ToString("x")))
            {
                #region Build Request Content

                foreach (string key in formData)
                {
                    multiPartContent.Add(new StringContent(formData[key]), key);
                }

                FileStream fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read);
                StreamContent streamContent = new StreamContent(fileStream);
                multiPartContent.Add(streamContent, fileControlName, filePath);
                streamContent.Headers.ContentType = new MediaTypeHeaderValue(fileContentType);

                #endregion

                #region creates HttpRequestMessage object

                HttpRequestMessage httpRequest = new HttpRequestMessage();
                httpRequest.Method = HttpMethod.Post;
                httpRequest.RequestUri = RequestUri;
                httpRequest.Content = multiPartContent;

                #endregion

                #region Send the request and process response
                // Send Request

                HttpResponseMessage httpResponse = null;
                try
                {
                    cancellationToken.ThrowIfCancellationRequested();

                    HttpClient httpClient = new HttpClient();
                    httpClient.Timeout = TimeSpan.FromSeconds(300);
                    httpResponse = await httpClient.
                        SendAsync(httpRequest, cancellationToken).ConfigureAwait(false);

                    HttpStatusCode statusCode = httpResponse.StatusCode;
                    if (statusCode != HttpStatusCode.OK)
                    {
                        string errorResponse =
                            await httpResponse.Content.ReadAsStringAsync().ConfigureAwait(false);

                        throw new Exception(errorResponse);
                    }
                }
                finally
                {
                    if (httpResponse != null)
                    {
                        httpResponse.Dispose();
                    }
                }

                #endregion
            }
        }

The above function uses MultipartFormDataContent which abstract the logic associated with generating the message.  Take a look at below function which provide exactly same functionality as above but it takes care of the raw message generation that conform to "multipart/form-data" protocol.

public async Task SubmitFormViaPOST(string serverUrl,
            string fileControlName,
            string filePath, 
            string fileContentType, 
            NameValueCollection formData, 
            CancellationToken cancellationToken)
        {
            FileInfo fileInfo = new FileInfo(filePath);
            Uri RequestUri = new Uri(serverUrl);

            // Create HTTP transport objects
            HttpRequestMessage httpRequest = null;
            try
            {
                #region Build Request Content

                // 27 dashes then current ticks as hexa decimal
                string partBoundary = "---------------------------" + DateTime.Now.Ticks.ToString("x");
                byte[] partBoundaryBytes = System.Text.Encoding.ASCII.GetBytes("\r\n--" + partBoundary + "\r\n");
                string formContentType = "multipart/form-data; boundary=" + partBoundary;

                MemoryStream requestStream = new MemoryStream();

                // Write each form item one by one

                //
                // -----------------------------7d81b516112482
                //
                // Content-Disposition: form-data; name="fname"
                //
                //
                // Anu
                //
                // -----------------------------7d81b516112482
                //
                // Content-Disposition: form-data; name="lname"
                //
                //
                // Chandy
                //

                string formdataTemplate = "Content-Disposition: form-data; name=\"{0}\"\r\n\r\n{1}";
                foreach (string key in formData)
                {
                    requestStream.Write(partBoundaryBytes, 0, partBoundaryBytes.Length);
                    string formItem = string.Format(formdataTemplate, key, formData[key]);
                    byte[] formItemBytes = System.Text.Encoding.ASCII.GetBytes(formItem);
                    requestStream.Write(formItemBytes, 0, formItemBytes.Length);
                }

                requestStream.Write(partBoundaryBytes, 0, partBoundaryBytes.Length);

                // Write the file stream to upload

                // Content-Disposition: form-data; name="photo"; filename="myprofilephoto.jpeg"
                //
                // Content-Type: image/jpeg
                //
                //
                // file-content-goes-here......

                string headerTemplate = "Content-Disposition: form-data; name=\"{0}\"; filename=\"{1}\"\r\nContent-Type: {2}\r\n\r\n";
                string header = string.Format(headerTemplate, fileControlName, filePath, fileContentType);
                byte[] headerBytes = System.Text.Encoding.ASCII.GetBytes(header);

                requestStream.Write(headerBytes, 0, headerBytes.Length);

                FileStream fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read);
                byte[] buffer = new byte[4096];
                int bytesRead = 0;
                while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
                {
                    requestStream.Write(buffer, 0, bytesRead);
                }

                fileStream.Close();

                byte[] trailer = System.Text.Encoding.ASCII.GetBytes("\r\n--" + partBoundary + "--\r\n");
                requestStream.Write(trailer, 0, trailer.Length);

                #endregion

                #region creates HttpRequestMessage object

                httpRequest = new HttpRequestMessage();
                httpRequest.Method = HttpMethod.Post;
                httpRequest.RequestUri = RequestUri;
                requestStream.Position = 0;
                httpRequest.Content = new StreamContent(requestStream);
                httpRequest.Content.Headers.Remove("Content-Type");
                httpRequest.Content.Headers.TryAddWithoutValidation("Content-Type", formContentType);
                httpRequest.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");

                #endregion

                #region Send the request and process response
                // Send Request

                HttpResponseMessage httpResponse = null;
                try
                {
                    cancellationToken.ThrowIfCancellationRequested();

                    HttpClient httpClient = new HttpClient();
                    httpClient.Timeout = TimeSpan.FromSeconds(300);
                    httpResponse = await httpClient.
                        SendAsync(httpRequest, cancellationToken).ConfigureAwait(false);

                    HttpStatusCode statusCode = httpResponse.StatusCode;
                    if (statusCode != HttpStatusCode.OK)
                    {
                        string errorResponse = 
                            await httpResponse.Content.ReadAsStringAsync().ConfigureAwait(false);

                        throw new Exception(errorResponse);
                    }
                }
                finally
                {
                    if (httpResponse != null)
                    {
                        httpResponse.Dispose();
                    }
                }

                #endregion
            }
            finally
            {
                if (httpRequest != null)
                {
                    httpRequest.Dispose();
                }
            }
        }

ACS, WIF, ASP .NET and Microsoft Azure – Fixing the exception: System.InvalidOperationException: ID1073: A CryptographicException occurred when attempting to decrypt the cookie using the ProtectedData API (see inner exception for details). If you are using IIS 7.5, this could be due to the loadUserProfile setting on the Application Pool being set to false

This post is about fixing the exception mentioned in the title that I got in an ASP .NET application after deploying to Azure.

When we configure an ASP.NET application as claims aware application using Azure Access Control Service (ACS), we can easily enable login to our application using Windows Live, Google and Yahoo identity providers.

Windows Identity Foundation (WIF) is a set of classes which is a part of .NET framework enable us to easily integrate claim based identity in .NET application.

Authentication flow in nutshell

To debug this exception, we need a basic idea on WIF's role in authentication flow.

WIF::WSFederationAuthenticationModule redirect the user to ACS when user browse to a page which requires him to be authenticated. ACS shows list of enabled identity providers [IdP] (Live, Google, Yahoo), user selects one of the identity provider for login.

Suppose Google is the selected IdP, user will be redirected to Google login page. Once user enters his credentials Google verifies it, if valid, 'Google token issuer' creates a security token containing user's claims. Google IdP redirects browser to ACS with this security token. Once ACS receives this token it validate that it is issued by IdP (and optionally process it), now ACS returns this token (processed token) to user's browser and it again gets POST-ed to the web application.

The WIF::WSFederationAuthenticationModule [FAM] intercept this POST to the web application, parses the security token to extract the claims and prepare an IClaimPrincipal object from it, which is accessible to the application.

FAM calls into WIF::SessionAuthenticationModule [SAM] module, which creates a 'session token' from this IClaimPrincipal object. SAM also takes care of serializing 'session token' to cookie, write this cookie to HTTP response to client. On subsequent request when client present this cookie SAM takes care of deserializing this cookie to IClaimPrincipal object to be used by the application.

Now, lets understand the exception and fix it

WIF creates this browser session cookie so that user can continue to browse to other pages within the same application without having to re-authenticate with the identity provider for each page visit. For security WIF encrypt the cookie.

To read and write session/cookie tokens, WIF::SAM module  uses an instance of the SessionSecurityTokenHandler class, by default it uses Windows DPAPI (Data Protection API) for encryption and decryption of tokens.

For Encrypt:
byte [] encryptedCookie = ProtectedData.Protect( cookie, s_aditionalEntropy, DataProtectionScope.CurrentUser );

For Decrypt:
byte [] cookie = ProtectedData.Unprotect( encryptedCookie, s_ditionalEntropy, DataProtectionScope.CurrentUser );

Note the 'DataProtectionScope.CurrentUser' argument, this cause the protection APIs to use some key generated using current user's 'UserProfile' for encrypt-decrypt.

In order to be able to use DataProtectionScope.CurrentUser option within an application running in IIS, application pool associated with the application should have 'Load User Profile' flag set to 'true'. If this flag set to false, then application running under IIS will not be able to access the current user profile.

By default this 'Load User Profile' flag is 'true'.

All works fine if we host our application in a single machine with 'Load User Profile' on, which will always on by default.

Problem starts when you deploy your application to machine instances behind a load balancer. When user continue to browse the site (after login) the load balancer redirects some request to execute on machine-1 some on machine-2.

If machine-1 receives the first request after login then WIF uses machine-1's current user key to encrypt the cookie. When the next request goes to machine-2, WIF in machine-2 will fail to decrypt the cookie using machine-2's current user key.

You will get the error:

System.InvalidOperationException: ID1073: A CryptographicException occurred when attempting to decrypt the cookie using the ProtectedData API (see inner exception for details). If you are using IIS 7.5, this could be due to the loadUserProfile setting on the Application Pool being set to false.

This exception will be thrown from WIF method System.IdentityModel.Web.ProtectedDataCookieTransform.Decode (Byte[] encoded) 

Which uses DPAPI to decode the cookie, when 'DPAPI' fails to do so this method assumes it’s because of 'loadUserProfile' to set false and ask you to do it with above exception, but we know the reason for failure is not 'loadUserProfile' being set to false but because of using different keys for encryption and decryption.

I will explain two solutions to fix this exception:

Solution-1

One solution to this problem is configure WIF not to use the default 'SessionSecurityTokenHandler' (which uses DPAPI) instead use a custom one which uses a key (for encrypt and decrypt) that is shared across all machines. We can register our own 'SessionSecurityTokenHandler' instance within an event handler hooked to the event  'FederatedAuthentication.FederationConfigurationCreated'.

Add the following line in Application Start method in Global.asax.cs:

protected void Application_Start()
{
     // .. other configurations routes etc..
     FederatedAuthentication.FederationConfigurationCreated += OnFederationConfigurationCreated;
}

Now from 'OnFederationConfigurationCreated' register the 'SessionSecurityTokenHandler' as below:

void OnFederationConfigurationCreated(object sender, FederationConfigurationCreatedEventArgs e)
{
    List<CookieTransform> sessionTransforms =
        new List<CookieTransform>(
            new CookieTransform[] {
                new DeflateCookieTransform(),
                new RsaEncryptionCookieTransform(e.FederationConfiguration.ServiceCertificate),
                new RsaSignatureCookieTransform(e.FederationConfiguration.ServiceCertificate)
            }
        );


    SessionSecurityTokenHandler sessionHandler =
    new SessionSecurityTokenHandler(sessionTransforms.AsReadOnly());
    e.FederationConfiguration.
      IdentityConfiguration.
      SecurityTokenHandlers.AddOrReplace(sessionHandler);
}

You can see that for cookie transform, we use a certificate instance pointed by 'e.FederationConfiguration.ServiceCertificate', this will instruct WIF 

to use certificate for cookie encrypt and decrypt instead of DPAPIs.

Next step is to install a certificate in all machines: In Windows Azure you can install a certificate for machines (web roles) by uploading the certificate under the hosted service which holds the web roles 

(machines) that runs the web application.

You also need to configure your cloud project with the certificate, refer the section 'Certificates Page' in 

http://msdn.microsoft.com/en-us/library/ee405486.aspx , note that you need to set Current User\Personal (My) as the Store Name.

Note the thumbprint associated with the certificate and register this certificate as WIF ServiceCertificate via web.config. Add below line in web.config under: <configuration>/<system.identityModel.services>/<federationConfiguration> section:

<serviceCertificate>
   <certificateReference x509FindType="FindByThumbprint"
                              findValue="<cerrt-thumbnail>"
                              storeLocation="LocalMachine"
                              storeName="My"/>
 </serviceCertificate>

Solution - 2

In soultion-1 we registered an instance of 'RsaEncryptionCookieTransform' for performing encryption and decryption of cookies using 

shared certificate across all machines.

Another option is to use machine key for encrypt-decrypt. The idea is to use the machine key configured in web.config.

When you deploy your application to Azure, as a part of hosting the application, infrastructure will takes care of inserting a <machineKey> node 

in web.config.

<machineKey decryption="AES" decryptionKey="<decryptionKey>" validation="SHA1" validationKey="<validationKey>" />

Infrastructure will ensure 'decryptionKey' and 'validationKey' are same across all role instances (machines).

ASP.NET 4.5 provide APIs which uses these machineKey, for example MachineKey.Protect() protects the specified data by encrypting or signing

it using keys present in <machineKey>. Similarly MachineKey.Unprotect() the specified data that was protected by the Protect method.

Now we can write our own 'CookieTransform' using MachineKey APIs.

public class MachineKeyTransform : CookieTransform
{
    public override byte[] Encode(byte[] value)
    {
        if (value == null)
        {
           throw new ArgumentNullException("value");
        }

        return Encoding.UTF8.GetBytes(MachineKey.Protect(value, 
    "protectme"));
    }


    public override byte[] Decode(byte[] encoded)
    {
        if (encoded == null)
        {
            throw 
  new ArgumentNullException("encoded");
        }

        return MachineKey.Unprotect(Encoding.UTF8.GetString(encoded), 
    "protectme");
    }
}

We can now use instance of this class instead of RsaEncryptionCookieTransform in 'OnFederationConfigurationCreated'

Okay, that being said, in .NET 4.5 we don't have to do any of the above steps :), the latest WIF comes with Session Handler which uses machine key

as described above.

All we have to do is in web.config use built-in 'MachineKeySessionSecurityTokenHandler' instead of  'SessionSecurityTokenHandler'.

In this case we don't have to register for the event 'FederationConfigurationCreated' and register any 'Transform' instance.

<system.identityModel>
  <identityConfiguration>
    <securityTokenHandlers>
      <remove type="System.IdentityModel.Tokens.SessionSecurityTokenHandler,       

        System.IdentityModel, Version=4.0.0.0, 

        Culture=neutral, PublicKeyToken=b77a5c561934e089" />
      <add type="System.IdentityModel.Services.Tokens.MachineKeySessionSecurityTokenHandler, 

        System.IdentityModel.Services, Version=4.0.0.0, 

        Culture=neutral, PublicKeyToken=b77a5c561934e089" />
    </securityTokenHandlers>
  </identityConfiguration>

</system.identityModel>