OMS Log Analytics HTTP Data Collector API – Work notes

I believe that the Microsoft Operations Management Suite is yet another example of how cool Cloud based solutions can be. Instead of first having to spin up an entire backend infrastructure before you can actually start collecting data, with the Microsoft Operations Management Suite you can directly start focusing on the task of collecting and visualizing your data. This blog post is basically a summary of my notes and scripts that I created while exploring the OMS Log Analytics HTTP Data Collector API, that allows you to submit any type of data to an OMS Workspace.

Creating an OMS Workspace

First we create an OMS Workspace. Within the Azure Management portal. Select “New”  search for Log Analytics and then create a new OMS Workspace.

image

Once the provisioning process is completed, we have a new OMS Workspace.

image

Clock on the “OMS Portal” button to access the portal.

image

Managing OMS with PowerShell

The PowerShell Module “AzureRM.OperationallInsights” provides the cmdlets for managing OMS with PowerShell. More information on that can be found here:https://blogs.technet.microsoft.com/privatecloud/2016/04/05/using-the-oms-search-api-with-native-powershell-cmdlets/

When running the command:

Get-AzureRmOperationalInsightsWorkspace

We get the the following output:

image

Submitting Data to Log Analytics

The Microsoft Operations Management suite provides a HTTP Data Collector API to send data to Log Analytics. A detailed overview including code examples is documented here:
https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-data-collector-api

In the following example, we are going to submit custom data to the OMS workspace we just created. Let’s start with setting some variables:

#OMS workspace Name
$OMSWorkspacename = "APIDemo"

# identify ResourceGroup Name
$resourcegroupname = (Get-AzureRmOperationalInsightsWorkspace | Where-Object {$_.Name -eq "$OMSWorkspacename"}).ResourceGroupName

# Workspace ID
$customerId = (Get-AzureRmOperationalInsightsWorkspace | Where-Object {$_.Name -eq "$OMSWorkspaceName"}).CustomerId.guid

# Primary Shared Key
$sharedKey = (Get-AzureRmOperationalInsightsWorkspaceSharedKeys -ResourceGroupName $resourcegroupname -Name $OMSWorkspacename).PrimarySharedKey

As a next step we define the LogType and the TimeStampField reference. The LogType is the name of our custom log that will be created. The TimeStampField is a reference to the Field that contains the date-time stamp. (For more details look at the Request Headers section).

# Specify the name of the record type that you'll be creating
$LogType = "MyComputers"

# Specify a field with the created time for the records
$TimeStampField = "DateValue"

Next we define the data that we want to submit:

$json = @"
[{
    "MyComputerName": "Computer10",
    "MyModel": "T460",
    "MyManufacturer": "Lenovo",
    "MyLocation": "Zurich",
    "DateValue": "2017-02-08T12:13:35.576Z"
},
{
    "MyComputerName": "Computer11",
    "MyModel": "T450",
    "MyManufacturer": "Lenovo",
    "MyLocation": "Amsterdam",
    "DateValue": "2017-02-08T12:13:35.576Z"
},
{
    "MyComputerName": "Computer12",
    "MyModel": "T470",
    "MyManufacturer": "Lenovo",
    "MyLocation": "London",
    "DateValue": "2017-02-08T12:13:35.576Z"
}]
"@

And finally the code to submit the data to OMS, The Build-Signature and Post-OMSData functions originate from example provided by Microsoft.

# Create the function to create the authorization signature
Function Build-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)
{
    $xHeaders = "x-ms-date:" + $date
    $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource

    $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)
    $keyBytes = [Convert]::FromBase64String($sharedKey)

    $sha256 = New-Object System.Security.Cryptography.HMACSHA256
    $sha256.Key = $keyBytes
    $calculatedHash = $sha256.ComputeHash($bytesToHash)
    $encodedHash = [Convert]::ToBase64String($calculatedHash)
    $authorization = 'SharedKey {0}:{1}' -f $customerId,$encodedHash
    return $authorization
}


# Create the function to create and post the request
Function Post-OMSData($customerId, $sharedKey, $body, $logType)
{
    $method = "POST"
    $contentType = "application/json"
    $resource = "/api/logs"
    $rfc1123date = [DateTime]::UtcNow.ToString("r")
    $contentLength = $body.Length
    $signature = Build-Signature `
        -customerId $customerId `
        -sharedKey $sharedKey `
        -date $rfc1123date `
        -contentLength $contentLength `
        -fileName $fileName `
        -method $method `
        -contentType $contentType `
        -resource $resource
    $uri = "https://" + $customerId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01"

    $headers = @{
        "Authorization" = $signature;
        "Log-Type" = $logType;
        "x-ms-date" = $rfc1123date;
        "time-generated-field" = $TimeStampField;
    }

    $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing
    return $response.StatusCode

}

# Submit the data to the API endpoint
Post-OMSData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($json)) -logType $logType

If all went well we get a return code of “200” All possible return codes are documented here in the Return Codes section. Let’s have a look if the data was successfully submitted, for this we can run the following command:

$dynamicQuery = "* Type=MyComputers_CL"
$result = Get-AzureRmOperationalInsightsSearchResults -ResourceGroupName $ResourceGroupName -WorkspaceName $OMSWorkspacename -Query $dynamicQuery -Top 100
$result.Value | ConvertFrom-Json

image

OMS Schema and custom fields

When submitting data for the first time to OMS, the log analytics schema is extended with the custom fields defined within the json object. I wrote a little helper function that retrieves the Log Analytics schema information. The Get-OMSSchemaInfo cmdlet can be downloaded from here.

Get-OMSSchemaInfo -ResourceGroupName $ResourceGroupName -WorkSpaceName $OMSWorkspacename | Where-Object {$_.OwnerType -like "MyComputers*"}

image

Okay, so far so good, we have submitted data, the schema has been extended and we can see the data we submitted to OMS. Now let’s head over to the OMS portal.

The OMS Portal

Open the OMS Portal and select “Log Search”

image

Next select “Count of all data collected grouped by Type

image

image

image

Note: Should you not see the full data but only the TimeGenerated information close all browser windows and open the OMS portal again. (Thanks to Cameron Fuller for the tip).

You can also see the new created custom fields within the OMS Portal via Settings, Data, Custom Fields.

image

In a next blog post, I’ll walk us through the creation of an OMS Dashboard.

Hope you found this useful.

Other useful resources that helped me throughout my OMS exploration process:

Leave a Reply

Your email address will not be published. Required fields are marked *