Deploy the Kafka connector for real-time reporting (RTR)

This page covers steps to deploy the Kafka sink connector that consumes RTR topics and writes transformed data into reporting tables.

On this page

  1. Installing the Kafka connector
  2. Troubleshooting

These steps require a Unix-compatible shell. On Windows, use Git Bash, WSL, or an equivalent terminal emulator.

Installing the Kafka connector

Follow these steps to configure and deploy the Kafka connector Helm chart for RTR.

Verify that you are connected to the correct Kubernetes cluster before proceeding. To confirm, run kubectl config current-context.

  1. Locate the Kafka connector Helm chart in the NEDSS-Helm repository.

  2. Configure values.yaml. Replace all placeholder values before installation.

    To retrieve your Kafka bootstrap server endpoints, see Get bootstrap brokers in the AWS MSK documentation.

    image:
  # Kafka Connect image
  repository: confluentinc/cp-kafka-connect
  # Replace with the target release version tag, e.g. v1.0.1
  tag: <release-version-tag>

sqlServerConnector:
  config:
    # [SME REVIEW: confirm databaseName should be rdb_modern, not rdb]
    connection.url: "jdbc:sqlserver://nbs-db.EXAMPLE_FIXME.nbspreview.com:1433;databaseName=rdb;encrypt=true;trustServerCertificate=true"
    # SQL Server username for reporting database
    connection.user: "EXAMPLE_FIXME"
    connection.password: "EXAMPLE_FIXME"

kafka:
  # Kafka bootstrap server endpoint from AWS MSK
  bootstrapServers: "EXAMPLE_FIXME"

  3. Install the pod:

    helm install -f ./kafka-connect-sink/values.yaml cp-kafka-connect-server ./kafka-connect-sink/

  4. Verify the pod is running:

    kubectl get pods

  5. Validate the service.

    The Kafka connector is an internal service with no ingress. Validate it as part of RTR pipeline validation.

After Debezium deploys successfully, continue to Deploy Java services.

Troubleshooting

If issues persist after initial troubleshooting, contact support at nbs@cdc.gov.

Database connection errors

If the service has trouble connecting to the database, run the following command to reset the ConfigMap:

kubectl delete configmap cp-kafka-connect-sqlserver-connect