Deploy Data Availability

This section provides instructions for on-premises deployment of the Data Availability service, which extracts data from the modernized NBS cloud implementation so you can keep copies of selected tables from RDB, ODSE, and SRTE. Complete Validate API endpoints before starting this page.

This page is part of the optional NND Service (Data Sync) section. CDC is evaluating long-term support for this service. If your STLT has a use case, contact nbs@cdc.gov.

On this page

  1. Prerequisites
  2. Components for Data Availability service
  3. Set up the Data Availability service
  4. Appendix: Data Sync service
  5. Next steps

Prerequisites

  • Complete Validate API endpoints.
  • Keycloak client ID and client secret: Retrieve these from your Keycloak instance. In the NBS realm, navigate to Clients > nnd-keycloak-client > Credentials > Client Secret.
  • Data service URL: Retrieve this from your NBS environment.

Components for Data Availability service

The Data Availability service includes:

  • data-sync-service.jar
  • .cmd files (only for Windows environments)
  • .sql files (for configuration table)

The service supports multiple options to download the data:

  • Direct database sync
  • Drop incremental .JSON files into an AWS S3 bucket
  • Drop incremental .JSON files into a local directory

Set up the Data Availability service

Download the Data Availability service files (.jar, .cmd, and .sql) from the NEDSS-NNDSS v7.13.0 release page. Under Assets, download the v7.13.0.NEDSS.NBS.Modernized.Documentation.zip file and locate the files in the data-sync/NND_SERVICE/ directory. Save the files to a secure directory with executable permissions to run the services.

Before you proceed, choose where to send the data:

  • SQL database
  • Amazon S3 bucket
  • Local directory

The SQL database option uses a different setup (see Appendix). For AWS S3 bucket and local directory options, pass the appropriate arguments in the .cmd file based on the selected option. Templates are provided in the release documents and README.md.

Step 0: Choose the option for Data Availability service

Use one of these three download options:

  1. Direct insert into the database:
    • Default setup: Syncs all tables into one database
    • Custom setup: Syncs tables into separate databases as preferred (e.g., target databases match source databases)
  2. Drop JSON files in S3 bucket
  3. Drop JSON files in a local file system

Step 1: Create the data config table

This step is important because this config table ensures that tables sync correctly.

Default setup for Data Availability

  • Use this if downloading data into S3 bucket or file location, or if having all tables in one database is acceptable.
  • Use the .sql file (create_data_config.sql) provided in the release package to create a new database (optional) and the required config table.
  • You can name your local database as preferred.
  • Ensure the poll_data_sync_config table is created.
  • Run the script poll_config_insert.sql to insert the static/lookup data into the config table.

Custom setup for Data Availability

  • Use this if you prefer multiple databases and want to redirect table inserts per your preference.
  • Create any sync database with your chosen name.
  • The service requires the poll_data_sync_config table (create_data_config.sql).
  • For each new database, create this configuration table and update its records as needed (poll_config_insert.sql), depending on sync requirements.

Example: By default, Data Availability syncs all tables in the RDB database, without considering their original source.

To direct SRTE table data into a separate SRTE database:

  1. Create the SRTE database in the on-premises environment.
  2. Create the poll_data_sync_config table in that database.
  3. Insert only SRTE-specific configurations into the table.
    • These can be identified by the source_db field (e.g., source_db = SRTE).

Config table scripts

Step 2: Configure the .cmd files

  • Find the configurable .cmd script file in the release materials.
  • Replace the argument values with your own.
    • Important: Do not include spaces between argument name and value. The proper format is: arg_name=arg_value.
  • For customizations, refer to the README.md file.

Appendix: Data Sync service

Setup for downloading data into SQL database (on-premises)

  • Use this setup if the SQL sync option is chosen.
  • Ensure relevant tables exist in the designated database before sync.

  • Execute the following .sql scripts to create necessary tables required to download the data.

  • Database creation scripts (includes RDB, SRTE, and RDB Modern) create_test_db SQL scripts

Next steps

Continue to Case notifications.