Free API Key

License Proxy

Suggest Edits

For customers deploying Deepgram’s on-premises solution in production, Deepgram recommends the License Proxy, which is a caching proxy that communicates with the Deepgram-hosted license server to ensure uptime and simplify network security.

📘

The Deepgram License Proxy is not necessary to test or deploy Deepgram services onprem. It is typically only deployed in highly available production environments.

Use Cases

  • For production deployments, the License Proxy allows your deployed on-premises services to continue to run even if your deployment loses connectivity to the Deepgram license server.

  • If network security requirements dictate that traffic over the web is allowed from only certain hosts, the License Proxy can be deployed statically while ASR services scale elastically.

  • If your customers will deploy Deepgram software as part of your on-premises solution and their traffic must flow back to your environment, you may deploy the License Proxy to relay traffic to the Deepgram license server.

  • If you have strict network traffic requirements, you can designate License Proxy to be the only container with access to the public internet. See Single Point Network Traffic Control for more details.

Architecture Overview

An on-premises Deepgram implementation of the License Proxy functions in the following way:

  1. The on-premises Deepgram services are configured to make their licensing requests against a hostname associated with a load balancer.

  2. The load balancer passes a licensing request to one of multiple instances of the License Proxy.

  3. In normal operation, the proxies will then pass on requests to the Deepgram-hosted license server. If the license server is unreachable for any reason, then the Deepgram License Proxy will allow the other Deepgram components to continue to run for a pre-configured amount of time.

📘

Deployment Recommendation

Deepgram recommends deploying onprem-license-proxy as a singleton for a given on-premises system.

System Requirements

For the License Proxy container, we recommend at least 1 GB RAM. When monitoring, if usage is consistently approaching the 1 GB limit, then increase to 2 GB RAM.

Deploying the License Proxy

Prerequisites

If you aren't certain which products your contract includes or if you are interested in adding the License Proxy to your on-premises deployment, please consult a Deepgram Account Representative. To reach one, contact us!

Installing

Deepgram makes all of its products available through Quay, a container image repository service.

  1. Follow the Self Service Licensing & Credentials guide to get credentials for Quay and login.

  2. Download the License Proxy container image:

    docker pull quay.io/deepgram/onprem-license-proxy:IMAGE_OR_RELEASE_TAG

🖥️

Be sure to replace the IMAGE_OR_RELEASE_TAG placeholder value with the appropriate tag, as described in the Deploy Deepgram Services guide.

Deploying

Before you can run your on-prem deployment with the License Proxy, you must configure the License Proxy. To do this, you will need to update your Docker Compose and container configuration files, then restart any running containers.

Update Your Docker Compose File

Add the following service to your docker-compose.yml, nested under the services header:

license-proxy:
  image: quay.io/deepgram/onprem-license-proxy:IMAGE_OR_RELEASE_TAG

  ports:
  - "8443:8443"
  - "8089:8089"

  # Invoke the License Proxy service
  # Make sure you `export` your on-prem API key secret in your local environment
  command: -v serve --license-key "${DEEPGRAM_API_KEY}" --host 0.0.0.0 --port 8443 --status-port 8089

🖥️

Be sure to replace the IMAGE_OR_RELEASE_TAG placeholder value with the tag of the images you've downloaded.

Also remember to export the Deepgram On-Premises API key secret you generated in the Deepgram Console as DEEPGRAM_API_KEY.

You will also want to make sure the Engine and API containers are relying on the License Proxy. Add the following line to the service definition for the existing api and the engine services in your docker-compose.yml file.

depends_on:
  - "license-proxy"

Update Your Container Configuration Files

If you are using custom configuration files for your Deepgram services, you will need to update your api.toml and engine.toml files. Add a line that specifies the URL to your deployed proxy:

[license]
server_url = "https://license-proxy:8443"

Update Your Services

Restart existing containers to begin directing licensing requests through the proxy, as well as starting the License Proxy itself:

docker compose up -d

Afterwards, inspect the logs for errors, and test that a request is processed as expected.

Monitoring

The License Proxy provides a status endpoint that indicates whether the proxy succeeded in relaying the most recent licensing request to the license server.

Querying the Status Endpoint

You can reach the status endpoint via port 8080 at the default /v1/status route. You can test this with a simple curl command (we use the command line utility jq to format the output):

curl http://localhost:8080/v1/status | jq

You should receive a response similar to:

{
	"state": "Connected",
	"last_successful_checkin": "2022-01-01T22:10:48.744573703Z",
	"trust_expiration": "2022-01-08T22:10:48.744573703Z"
}

Interpreting the Status Endpoint Response

Response fields include:

  • state: The proxy’s behavior based on the most recent licensing request. Possible values include:
    • Ready: Indicates that the proxy has started but has not yet relayed a licensing request.
    • Connected: Indicates that the most recent relay of a licensing request succeeded.
    • TrustBased: Indicates that the most recent relay of a licensing request failed, and the proxy returned a cached response.
    • Failed: The most recent relay of a licensing request failed, and the proxy is outside of the trust window.
  • last_successful_checkin: The timestamp of the last successful relay of a licensing request.
  • trust_expiration: The timestamp past which the proxy will no longer return a cached response for licensing requests that fail.

Setting Up Automated Alerts

If you use an automated metrics and alerting system, you should query the proxy’s status endpoint regularly (for example, once each minute) and set a warning when the state is TrustBased for a significant time period (for example, one hour). If you receive a warning, you can contact Deepgram support to troubleshoot the connection to the license server while knowing that deployed services will continue to run due to trust caching.

Single Point Network Traffic Control

No inference data is sent to Deepgram from an onprem deployment. For example, if you submit an ASR request in your onprem deployment, Deepgram services will not send the audio data or corresponding transcription outside the onprem deployment.

Additionally, no incoming network traffic will ever be initiated by Deepgram's servers into an onprem environment. All ingress traffic to Deepgram services can be blocked in your onprem environment.

Minimal outgoing network traffic originates from Deepgram services to communicate with the Deepgram License Server. These are small, regular messages to verify that your account has a valid license; these messages never include any inference data. With that said, some customers like an extra layer of confidence that no inference data is leaked from the environment.

If the License Proxy is deployed in an onprem environment, Deepgram services can be configured so all license verification traffic will flow through it. API, Engine, and other containers will not make any outgoing network connections, meaning you can block all ingress and egress network traffic from the machines hosting these containers. Only the machine hosting the License Proxy container will need to allow egress network traffic.

Notably, the License Proxy never handles inference data directly, so many customers' security requirements are satisfied by having License Proxy be the only container with egress access to the public internet.

Whitelist for egress traffic

To maintain high availability, Deepgram does not publish a static list of IP addresses for our License Server. If you wish to block egress traffic and explicitly whitelist communication to the Deepgram License Server, you may whitelist the hostname https://license.deepgram.com.

If you wish to direct traffic through a firewall or other proxy, our onprem containers communicate over HTTPS and respect the Unix standard HTTPS_PROXY and ALL_PROXY environment variables.

Updated 2 months ago