Connecting to an external data source

When integrating an external data source (such as a database, file system, etc.) into your application, whether using Streamlit or FastAPI, it’s important to manage your credentials securely and establish a reliable connection. Below are the steps to connect to an external data source in a secure and efficient manner.

1. Set up Credentials for the External Data Source:

To securely store and access credentials (such as API keys, database passwords, or other sensitive information), use the Secrets Management feature on the Health Universe website.

Important: Secrets can only contain letters, numbers, or the - and _ characters. They must start with a letter or number. For example, DB_PASSWORD, API_KEY, or DATA_SOURCE_CREDENTIALS.

2. Access your secret in your code:

Once you've created the secret, you can retrieve it securely using os.environ.get() in your Python code. Make sure to use uppercase for the secret name when accessing it:

import os

DB_PASSWORD = os.environ.get("DB_PASSWORD")
API_KEY = os.environ.get("API_KEY")

3. Connect to the External Data Source from your Code:

Once you have your credentials, you can establish a connection to the external data source. The process for connecting depends on the type of data source. Below are examples for connecting to a database and an API.

• Connecting to a Database: To connect to a database (e.g., PostgreSQL, MySQL, etc.), you can use a library like psycopg2 or sqlalchemy.

• For PostgreSQL: pip install psycopg2

import psycopg2
import os

# Retrieve credentials from secrets management
DB_HOST = os.environ.get("DB_HOST")
DB_NAME = os.environ.get("DB_NAME")
DB_USER = os.environ.get("DB_USER")
DB_PASSWORD = os.environ.get("DB_PASSWORD")

# Connect to the database
connection = psycopg2.connect(
    host=DB_HOST,
    dbname=DB_NAME,
    user=DB_USER,
    password=DB_PASSWORD
)

# Example function to query the database
def query_database(query):
    with connection.cursor() as cursor:
        cursor.execute(query)
        result = cursor.fetchall()
    return result

• Connecting to an External API: To connect to an external API, you can use the requests library.

• For requests: pip install requests

import requests
import os

# Retrieve credentials from secrets management
API_KEY = os.environ.get("API_KEY")

# Example function to call the external API
def query_api(endpoint, params=None):
    headers = {
        "Authorization": f"Bearer {API_KEY}"
    }
    response = requests.get(endpoint, headers=headers, params=params)
    if response.status_code == 200:
        return response.json()
    else:
        return {"error": "Failed to fetch data"}

4. Use the External Data Source in Your Streamlit or FastAPI App:

Once the connection to the external data source is established, you can easily use it within your Streamlit or FastAPI app.

• Streamlit Example:

import streamlit as st

st.title("External Data Source Example")

query = st.text_input("Enter SQL query to execute:")
if query:
    result = query_database(query)
    st.write(result)

• FastAPI Example:

from fastapi import FastAPI

app = FastAPI()

@app.get("/query_database")
async def query_database_endpoint(query: str):
    result = query_database(query)
    return {"result": result}

@app.get("/query_api")
async def query_api_endpoint(endpoint: str, params: dict = None):
    result = query_api(endpoint, params)
    return {"result": result}

Additional Notes:

• Security Considerations: Always retrieve your credentials securely using the Secrets Management process. Never hardcode sensitive data directly into your codebase.

• Connection Pooling: For databases, consider using connection pooling to optimize performance and resource usage. Libraries like SQLAlchemy or psycopg2 support pooling.

• Error Handling: Always include error handling when working with external data sources, as network issues, incorrect credentials, or unavailable resources may cause failures.

try:
    result = query_database(query)
except Exception as e:
    st.error(f"Error querying the database: {e}")

By following these steps, you can securely and efficiently connect to an external data source from your Streamlit or FastAPI application.