Skip to navigation

Microsoft Azure

With Kubernetes

Deploying Deepgram on Microsoft Azure requires some preparation. In this guide, you’ll provision an Azure Kubernetes Service (AKS) cluster with an NVIDIA GPU node pool, create shared storage for Deepgram model files, and install Deepgram with the official Helm chart. You’ll run every step in your local terminal with the Azure CLI.

Prerequisites

Make sure you’ve completed the requirements in the Self-Hosted Introduction.

Choose a GPU VM size

Deepgram Engine requires a dedicated NVIDIA GPU, and the GPU you choose determines which models you can run. Pick the VM size before you create the cluster.

This table maps the Azure VM series that ship NVIDIA CUDA drivers to those compatibility rules.

Azure VM seriesExample sizeGPUNova-3, Nova-2Aura-2Flux STTFlux TTS
NCasT4_v3Standard_NC8as_T4_v31 × NVIDIA T4 (16 GB)✅✅❌❌
NVadsA10_v5Standard_NV36ads_A10_v51 × NVIDIA A10 (24 GB)✅✅✅❌
NC A100 v4Standard_NC24ads_A100_v41 × NVIDIA A100 (80 GB)✅✅✅✅
NCads H100 v5Standard_NC40ads_H100_v51 × NVIDIA H100 NVL (94 GB)✅✅✅✅

VM size names and GPU counts change as Azure adds series. To confirm a size is available in your region, run az vm list-skus --location <your-region> --size Standard_NC --output table, and check Microsoft’s GPU-optimized VM sizes for current details.

Request GPU quota

New Azure subscriptions often have no quota for GPU VM families, and GPU capacity is limited in many regions. Request a vCPU quota increase for your chosen VM family in your target region before you create the cluster. Follow the steps in Requesting Quotas for NVIDIA GPUs.

Azure CLI

The Azure CLI provides programmatic access to manage your Azure services. Many of the same actions are available in the Azure portal.

  1. Follow the installation guide to install the CLI locally. You need version 2.72.2 or later, which adds the --gpu-driver option this guide uses.

  2. Sign in and select the subscription to bill for this cluster.

    Shell
    az login
    az account set --subscription "YOUR_SUBSCRIPTION_ID_OR_NAME"

Choosing a region

This guide provisions resources in the Azure westus2 region. To deploy to a different region, change the LOCATION variable and confirm that your chosen GPU VM family is available there.

kubectl

The Kubernetes command-line tool, kubectl, runs commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs.

Install it with the official Kubernetes guides, or run az aks install-cli.

Kubernetes packages with helm

Helm is the package manager for Kubernetes. A Helm chart defines a Kubernetes application so you can install and upgrade it. This guide uses Helm to install Deepgram and the NVIDIA GPU Operator. See the installation guide for details on how to install it locally.

Create a cluster

Azure Kubernetes Service (AKS) is a managed Kubernetes service. AKS manages the availability and scalability of the Kubernetes control plane that schedules containers, manages application availability, and stores cluster data.

  1. Set the variables this guide reuses, then create a resource group and an AKS cluster. The cluster starts with a small system node pool. You’ll add node pools for each Deepgram component next.

    Shell
    RESOURCE_GROUP=deepgram-self-hosted
    CLUSTER_NAME=deepgram-self-hosted
    LOCATION=westus2
    ENGINE_VM_SIZE=Standard_NC24ads_A100_v4 # Replace with the GPU VM size you chose
    az group create \
    --name $RESOURCE_GROUP \
    --location $LOCATION
    az aks create \
    --resource-group $RESOURCE_GROUP \
    --name $CLUSTER_NAME \
    --node-count 1 \
    --node-vm-size Standard_D4s_v5 \
    --enable-cluster-autoscaler \
    --min-count 1 \
    --max-count 3 \
    --generate-ssh-keys
    az aks get-credentials \
    --resource-group $RESOURCE_GROUP \
    --name $CLUSTER_NAME
  2. Create a separate node pool for each Deepgram component (API, License Proxy, Engine). Adjust the VM sizes and node counts to your needs. You may wish to consult your Deepgram Account Representative when you plan your cluster’s capacity.

    Shell
    az aks nodepool add \
    --resource-group $RESOURCE_GROUP \
    --cluster-name $CLUSTER_NAME \
    --name apipool \
    --node-vm-size Standard_D8s_v5 \
    --node-count 1 \
    --enable-cluster-autoscaler \
    --min-count 1 \
    --max-count 3 \
    --labels k8s.deepgram.com/node-type=api
    az aks nodepool add \
    --resource-group $RESOURCE_GROUP \
    --cluster-name $CLUSTER_NAME \
    --name lpool \
    --node-vm-size Standard_D4s_v5 \
    --node-count 1 \
    --enable-cluster-autoscaler \
    --min-count 1 \
    --max-count 2 \
    --labels k8s.deepgram.com/node-type=license-proxy
    az aks nodepool add \
    --resource-group $RESOURCE_GROUP \
    --cluster-name $CLUSTER_NAME \
    --name enginepool \
    --node-vm-size $ENGINE_VM_SIZE \
    --node-count 1 \
    --enable-cluster-autoscaler \
    --min-count 1 \
    --max-count 8 \
    --gpu-driver none \
    --labels k8s.deepgram.com/node-type=engine

    Skip the AKS GPU driver install

    AKS can install an NVIDIA driver for you, but AKS selects the driver version. Deepgram requires driver >=580 with the open kernel modules, so create the Engine node pool with --gpu-driver none. The deepgram-self-hosted Helm chart then installs a compliant driver with the NVIDIA GPU Operator. Don’t use --enable-managed-gpu on this node pool.

    You can only set --gpu-driver when you create a node pool. To change it later, create a new node pool.

  3. Create a dedicated namespace for Deepgram resources.

    Shell
    kubectl create namespace dg-self-hosted
    kubectl config set-context --current --namespace=dg-self-hosted

Configure persistent storage

Deepgram Engine pods read model files from a shared volume. The Helm chart can download models automatically only for Amazon EFS, so on AKS you create an Azure Files NFS share, populate it from inside the cluster with a one-shot Kubernetes Job, and point the chart at the resulting PersistentVolumeClaim.

Azure Files NFS shares need a Premium (SSD) storage account and network access from your cluster’s subnet. Review the NFS prerequisites before you continue. If the AKS identity lacks the required roles, volume creation or mounting fails.

  1. Create a StorageClass for NFS shares. The Retain reclaim policy keeps your model files if you delete the claim later.

    Shell
    cat <<'EOF' | kubectl apply -f -
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
    name: dg-models-nfs
    provisioner: file.csi.azure.com
    reclaimPolicy: Retain
    allowVolumeExpansion: true
    parameters:
    protocol: nfs
    skuName: Premium_LRS
    mountOptions:
    - nconnect=4
    - noresvport
    - actimeo=30
    EOF
  2. Create a PersistentVolumeClaim for the models. Premium file shares have a 100 GiB minimum size.

    Shell
    cat <<'EOF' | kubectl apply -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
    name: dg-model-storage
    namespace: dg-self-hosted
    spec:
    accessModes:
    - ReadWriteMany
    storageClassName: dg-models-nfs
    resources:
    requests:
    storage: 100Gi
    EOF
    kubectl get pvc dg-model-storage

    The claim should reach the Bound status within a few minutes.

  3. Run a one-shot Job that mounts the claim and downloads the model files provided by your Deepgram Account Representative.

    Replace the wget lines below with the model URLs supplied by your Deepgram Account Representative.

    Shell
    cat <<'EOF' | kubectl apply -f -
    apiVersion: batch/v1
    kind: Job
    metadata:
    name: dg-model-downloader
    namespace: dg-self-hosted
    spec:
    backoffLimit: 3
    template:
    spec:
    restartPolicy: Never
    containers:
    - name: downloader
    image: alpine:3
    command: ["sh", "-c"]
    args:
    - |
    set -eu
    apk add --no-cache wget ca-certificates
    cd /mnt/models
    # Replace these with the model URLs from your Deepgram Account Representative.
    wget -c https://link-to-model-1.dg
    wget -c https://link-to-model-2.dg
    # ... continue for all model files
    volumeMounts:
    - name: models
    mountPath: /mnt/models
    volumes:
    - name: models
    persistentVolumeClaim:
    claimName: dg-model-storage
    EOF
  4. Wait for the Job to complete, then delete it. The claim and the model files remain.

    Shell
    kubectl wait --namespace dg-self-hosted --for=condition=complete job/dg-model-downloader --timeout=30m
    kubectl logs --namespace dg-self-hosted job/dg-model-downloader
    kubectl delete --namespace dg-self-hosted job dg-model-downloader

Configure Kubernetes Secrets

Deepgram strongly recommends following best practices for configuring Kubernetes Secrets. Refer to Securing Your Cluster for more details.

The deepgram-self-hosted Helm chart takes two Secret references. One is a set of distribution credentials that allow the cluster to pull images from Deepgram’s container image repository. The other is your self-hosted API key that licenses each Deepgram container that’s created.

  1. Complete the Self Service Licensing & Credentials guide to generate distribution credentials and a self-hosted API key.

  2. If you use an external Secret store provider, such as Azure Key Vault, configure cluster access to these two Secrets, and name them dg-regcred (distribution credentials) and dg-self-hosted-api-key.

  3. If you don’t use an external Secret store provider, create the Secrets manually in your cluster.

    1. Using the distribution credentials username and password generated in the Deepgram Console, create a Kubernetes Secret named dg-regcred.

      Shell
      kubectl create secret docker-registry dg-regcred \
      --docker-server=quay.io \
      --docker-username='QUAY_DG_USER' \
      --docker-password='QUAY_DG_PASSWORD'

      Replace the placeholders QUAY_DG_USER and QUAY_DG_PASSWORD with the distribution credentials you generated in the Self Service Licensing & Credentials guide.

    2. Create a Kubernetes Secret named dg-self-hosted-api-key to store your self-hosted API key.

      Shell
      kubectl create secret generic dg-self-hosted-api-key \
      --from-literal=DEEPGRAM_API_KEY='YOUR_API_KEY_HERE'

      Replace the placeholder YOUR_API_KEY_HERE with the Deepgram API key you generated in the Self Service Licensing & Credentials guide.

Deploy Deepgram

Deepgram maintains the official deepgram-self-hosted Helm chart. You can reference the source and Artifact Hub listing for more details. We’ll use this chart to deploy Deepgram services in your cluster.

  1. Fetch the repository info.

    Shell
    helm repo add deepgram https://deepgram.github.io/self-hosted-resources
    helm repo update
  2. Save the following as my-values.yaml. It pins each component to the node pool you created, mounts the Azure Files claim as the model volume, and lets the chart install the NVIDIA GPU Operator. See the chart README for every available option.

    my-values.yaml
    global:
    pullSecretRef: "dg-regcred"
    deepgramSecretRef: "dg-self-hosted-api-key"
    scaling:
    replicas:
    api: 1
    engine: 1
    auto:
    enabled: false
    api:
    affinity:
    nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: k8s.deepgram.com/node-type
    operator: In
    values:
    - api
    resources:
    requests:
    memory: "4Gi"
    cpu: "2000m"
    limits:
    memory: "12Gi"
    cpu: "4000m"
    engine:
    affinity:
    nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: k8s.deepgram.com/node-type
    operator: In
    values:
    - engine
    resources:
    requests:
    memory: "24Gi"
    cpu: "5000m"
    gpu: 1
    limits:
    memory: "40Gi"
    cpu: "8000m"
    gpu: 1
    modelManager:
    volumes:
    customVolumeClaim:
    enabled: true
    name: dg-model-storage
    modelsDirectory: "/"
    licenseProxy:
    enabled: true
    affinity:
    nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: k8s.deepgram.com/node-type
    operator: In
    values:
    - license-proxy
    resources:
    requests:
    memory: "4Gi"
    cpu: "1000m"
    limits:
    memory: "8Gi"
    cpu: "2000m"
    # The Engine node pool was created with `--gpu-driver none`, so the GPU Operator
    # installs the NVIDIA driver (>=580, open kernel modules) and container toolkit.
    # The chart's default container toolkit (v1.15.0) doesn't register the `nvidia`
    # runtime with the containerd 2.x that current AKS nodes use, so pin a newer one.
    gpu-operator:
    enabled: true
    toolkit:
    version: v1.17.8-ubi8
  3. Adjust the values to match your cluster.

    • Set scaling.replicas.api and scaling.replicas.engine to the initial number of replicas. Keep the Engine replica count at or below the number of nodes in enginepool, because each Engine pod requests one GPU.
    • If you chose a VM with less memory than the example, lower the requests and limits so the pods fit on the node.
    • For Aura text-to-speech, each Engine needs two GPUs. Set engine.resources.requests.gpu and engine.resources.limits.gpu to 2, and start from the Deploy TTS Services guide.

    Node pool autoscaling, which you configured with the Azure CLI, is separate from pod autoscaling in the chart. To enable pod autoscaling, reach out to your Deepgram Account Representative to discuss which limits and values suit your traffic. See Autoscaling Best Practices.

  4. Install the Helm chart with your values file.

    Shell
    helm install deepgram deepgram/deepgram-self-hosted \
    -f my-values.yaml \
    --namespace dg-self-hosted \
    --atomic \
    --timeout 1h
    # Monitor the installation in a separate shell
    watch kubectl get all

    Allow time for the first install

    The GPU Operator installs the NVIDIA driver on each GPU node, and AKS may need to provision a new node first. This can take several minutes. If a pod isn’t scheduled as you expect, kubectl describe pod <pod-name> lists the events that explain why.

Verify the GPU setup

Confirm that the cluster advertises your GPUs and that the installed driver meets Deepgram’s requirements.

  1. List the GPUs Kubernetes can schedule on each Engine node.

    Shell
    kubectl get nodes \
    --selector k8s.deepgram.com/node-type=engine \
    --output custom-columns='NAME:.metadata.name,GPUS:.status.allocatable.nvidia\.com/gpu'
  2. Check the driver version that the GPU Operator installed.

    Shell
    DRIVER_POD=$(
    kubectl get pods \
    --namespace dg-self-hosted \
    --selector app=nvidia-driver-daemonset \
    --output jsonpath='{.items[0].metadata.name}'
    )
    kubectl exec --namespace dg-self-hosted $DRIVER_POD -c nvidia-driver-ctr -- nvidia-smi
    kubectl exec --namespace dg-self-hosted $DRIVER_POD -c nvidia-driver-ctr -- cat /proc/driver/nvidia/version

    The nvidia-smi output should list your GPU and a driver version of 580 or later. The version string should read NVIDIA UNIX Open Kernel Module.

Test your Deepgram setup with a sample request

Test your Deepgram deployment on AKS with an audio file.

  1. Launch an ephemeral pod to send your test request from.

    Shell
    kubectl run --namespace dg-self-hosted api-client \
    --stdin --tty --rm \
    --image=curlimages/curl \
    -- /bin/sh
  2. Inside the ephemeral pod, download a sample file from Deepgram, or supply your own file.

    Shell
    wget https://dpgr.am/bueller.wav
  3. Send your audio file to your Deepgram setup for transcription.

    Shell
    curl \
    -X POST \
    --data-binary @bueller.wav \
    "http://deepgram-api-external.dg-self-hosted.svc.cluster.local:8080/v1/listen?model=nova-3&smart_format=true"

    If needed, adjust pieces of the above command:

    • the query parameters to match the directions from your Deepgram Account Representative
    • the service name deepgram-api-external
    • the namespace dg-self-hosted

You should receive a JSON response with the transcript and associated metadata. You’re all set: your self-hosted setup is working.

Troubleshoot common AKS issues

SymptomLikely cause and fix
enginepool creation fails with a quota or OperationNotAllowed errorYour subscription has no quota for that VM family in the region. Request quota, or choose another region or size.
unrecognized arguments: --gpu-driverYour Azure CLI is older than 2.72.2. Upgrade it with az upgrade.
Engine pod stays Pending with Insufficient nvidia.com/gpuThe GPU Operator hasn’t finished installing the driver and device plugin on the node. Check kubectl get pods --namespace dg-self-hosted, and wait for the nvidia-* pods to reach Running.
nvidia-operator-validator, nvidia-device-plugin-daemonset and other GPU Operator pods stay in Init, and pod events show no runtime for "nvidia" is configuredThe NVIDIA container toolkit is too old for the containerd 2.x on your nodes. Set gpu-operator.toolkit.version to v1.17.8-ubi8, as in the values file above, and run helm upgrade.
nvidia-driver-daemonset pod fails or restarts repeatedlyThe node OS or kernel isn’t supported by the GPU Operator driver image. See NVIDIA’s supported platforms, and use the default Ubuntu node image.
dg-model-storage stays Pending, or pods can’t mount itThe AKS identity can’t create or reach the storage account. Review the NFS prerequisites in Microsoft’s Azure Files guide.
Engine pod exits at startup with a missing model errorThe model files aren’t in the root of the share. Check the downloader Job logs, and confirm the file names match what your Deepgram Account Representative provided.

For general Kubernetes issues, see Troubleshooting.

Next steps

Your Deepgram services are accessible within your cluster through the deepgram-api-external Service that the Helm chart created.

To accept requests from outside the cluster, expose the API with an Azure load balancer. In your my-values.yaml, set api.service.type to LoadBalancer. Add the service.beta.kubernetes.io/azure-load-balancer-internal: "true" annotation under api.service.annotations to keep the load balancer private to your virtual network. See Microsoft’s AKS load balancer documentation for details. Your installation automatically balances received requests across the cluster, so the load balancer serves as the ingress endpoint into the cluster.

To stop charges when you’re done testing, delete the resource group. This removes the cluster, node pools, and model storage.

Shell
az group delete --name $RESOURCE_GROUP --yes --no-wait

What’s next

Now that you have a basic Deepgram setup working, take some time to learn about building up to a production-level environment, as well as helpful Deepgram add-on services.