Prometheus Integration

Prometheus is a powerful monitoring and alerting toolkit, often used for collecting and querying various system metrics. This guide outlines the steps required to integrate Prometheus with an self-hosted instance.

Kubernetes

Deepgram supports self-hosted deployments on Kubernetes via the deepgram-self-hosted Helm chart. Prometheus will be installed and automatically configured if either scaling.auto.enabled or kube-prometheus-stack.includeDependency is set to true.

If you would like to add additional configuration, see the kube-prometheus-stack Helm chart for available values. See the Overriding Values from a Parent Chart documentation for details on setting these values in the kube-prometheus-stack subchart from the deepgram-self-hosted parent chart.

Docker/Podman

Self-hosted deployments utilizing Docker or Podman will need to manually install and manage Prometheus.

Prerequisites

Before proceeding, ensure that you have the following:

Access to the self-hosted instance with the metrics endpoint exposed. See the Metrics Guide for details. Note that this guide assumes you chose port 9991 for the HOST_PORT.
Prometheus installed on your machine or server. Visit Download Prometheus for installation instructions.
Local system permissions to modify the Prometheus configuration file (usually named prometheus.yml)

Integrate with Deepgram Engine

Step 1: Add Prometheus Target

Add the target to the Prometheus configuration file in the prometheus.yml file. Locate the scrape_configs section, and add a new job with the Engine container instance as a target:

scrape_configs:
  - job_name: "dg_self_hosted_instance"
    static_configs:
      - targets: ["<ENGINE_INSTANCE_URI>:9991"]

Replace <ENGINE_INSTANCE_URI> with the IP address or hostname of the Engine container. Save the configuration file.

Step 2: Reload Prometheus Configuration

To apply the changes made to the prometheus.yml file, you need to reload the Prometheus configuration. You can do this by restarting the Prometheus service:

sudo systemctl restart prometheus

Step 3: Verify the Integration

Open the Prometheus web interface, usually accessible at http://localhost:9090, and navigate to the "Targets" page (under the "Status" menu). You should see the dg_self_hosted_instance job listed with a status of "UP", indicating that Prometheus is successfully scraping metrics from the Engine container.

Querying Metrics

With the integration complete, you can now query the collected metrics using the Prometheus web interface or API. You may consider using a tool like Grafana for handling visualization and alerting.