Running Datadog on Azure App Service

Last week covered sending .NET logs straight to Datadog with no Agent. That's the quickest way to get logs flowing, but logs are only part of the picture. When a request is slow, you want to see where the time went: the database call, the partner API, the Service Bus send. That needs APM tracing, and on a VM it would come from the Datadog Agent running next to your app.

App Service doesn't let you install an Agent the usual way. Instead, Datadog provides two App Service-specific setups: a site extension on Windows and a sidecar container on Linux.

#What you get

• Distributed APM traces from automatic instrumentation of ASP.NET Core, HttpClient, SQL clients, the Azure SDKs and more, with no code changes
• Custom metrics through DogStatsD
• Trace IDs injected into your logs, so you can go from a slow trace to the exact log entries it produced
• Traces enriched with App Service metadata, such as the instance and plan

#Windows: the site extension

On a Windows App Service, Datadog ships as an App Service site extension:

1. Stop the app. Datadog's documentation is firm about this: the extension won't install or update correctly while the app is running.
2. In the portal, open the app's Extensions blade, add the Datadog .NET extension (Datadog.AzureAppServices.DotNet), then start the app again.
3. Add the app settings:

DD_API_KEY   = @Microsoft.KeyVault(VaultName=my-vault;SecretName=DatadogApiKey)
DD_SITE      = datadoghq.com
DD_SERVICE   = orders-api
DD_ENV       = production
DD_VERSION   = 1.4.0
DD_LOGS_INJECTION = true

The extension is available on Basic, Standard and Premium plans, not on Free or Shared.

Logs on Windows are still agentless. The extension handles traces and metrics. For logs, use one of last week's options: turn on the tracer's direct submission with DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS=ILogger, or keep the Serilog sink. With DD_LOGS_INJECTION on, those logs carry the trace IDs.

#Linux: the sidecar container

Linux App Service supports sidecar containers that run next to your app. Datadog's sidecar image, datadog/serverless-init, plays the part of the Agent: it receives traces on port 8126 and forwards them, and it can also collect log files.

The setup has three parts.

1. Add the tracer to your app with the Datadog.Trace.Bundle NuGet package. It ships the tracer inside your deployment under a datadog folder.

2. Add the sidecar. In the portal, add a sidecar container to the app using the image index.docker.io/datadog/serverless-init:latest, listening on port 8126.

3. Add the app settings:

DD_API_KEY   = @Microsoft.KeyVault(VaultName=my-vault;SecretName=DatadogApiKey)
DD_SITE      = datadoghq.com
DD_SERVICE   = orders-api
DD_ENV       = production
DD_VERSION   = 1.4.0
DD_LOGS_INJECTION = true
DD_TRACE_ENABLED  = true
WEBSITES_ENABLE_APP_SERVICE_STORAGE = true

CORECLR_ENABLE_PROFILING = 1
CORECLR_PROFILER         = {846F5F1C-F9AE-4B07-969E-05C26BC060D8}
CORECLR_PROFILER_PATH    = /home/site/wwwroot/datadog/linux-x64/Datadog.Trace.ClrProfiler.Native.so
DD_DOTNET_TRACER_HOME    = /home/site/wwwroot/datadog

The CORECLR_* settings turn on the .NET profiling API, which is how the tracer instruments your code automatically. The paths point into the folder the NuGet package added to your deployment.

Logs on Linux can come through the sidecar. Write your logs as JSON files under /home/LogFiles/, for example with Serilog's file sink and a JSON formatter, and the sidecar picks up files matching /home/LogFiles/*.log. Set DD_SERVERLESS_LOG_PATH if you write them somewhere else. Shared storage (WEBSITES_ENABLE_APP_SERVICE_STORAGE=true) is what lets the app and the sidecar see the same files. If you'd rather not manage log files, agentless logging still works here too.

#Script it, don't click it

Setting a dozen app settings by hand on every app, in every environment, is how one of them ends up with the wrong DD_ENV. Datadog supports the datadog-ci aas instrument command, Terraform, Bicep and ARM templates for both setups. Put it in the same infrastructure code as the App Service itself.

#Don't forget the platform side

The extension and sidecar see inside your app. For App Service's own metrics, such as CPU, memory, HTTP queue length and instance count, plus Azure resource logs, set up Datadog's Azure integration as well. Seeing your traces next to the platform metrics is often what explains a slowdown.

#Things that catch people out

• Deployment slots. Mark DD_ENV as a slot setting so staging reports as staging, and make sure the extension or sidecar is set up in every slot you swap.
• Version tags. Set DD_VERSION from your pipeline's build number, not by hand, so Datadog can compare releases.
• Extension updates on Windows need the app stopped again. Plan them like a deployment.
• Cost. APM is billed per host, and on App Service that roughly means per instance. Check what scaling out does to your bill.

#Takeaway

Agentless logging is the quick start. For full APM on App Service, use Datadog's site extension on Windows or the serverless-init sidecar on Linux, add the tracer and the DD_* settings, keep sending logs agentlessly or through the sidecar, and script the whole setup so every app and slot is configured the same way.