Building an Azure SQL to Dataverse Migration Solution with the New Dataverse SDK for Python

Microsoft's new Dataverse SDK for Python (Public Preview) opens exciting opportunities for developers, data engineers, and Power Platform professionals to integrate enterprise data with Dataverse using Python. The SDK supports core capabilities such as CRUD (DML) operations, schema-related operations (DDL), file uploads, and seamless integration with the Python ecosystem including Pandas and Jupyter Notebooks.

In this article, we'll build a practical use case: migrating customer data from Azure SQL Database to Microsoft Dataverse using Python .

Solution Architecture

Components

• Azure SQL Database

• Python Runtime (Local, Azure Function, Azure VM, GitHub Actions, etc.)

• Dataverse SDK for Python

• Microsoft Dataverse Environment

• Microsoft Entra ID Authentication

Business Scenario

A company maintains customer master records in Azure SQL.

Azure SQL Column	Dataverse Column
CustomerID	Customer Number
CustomerName	Name
Email	Email Address
Phone	Telephone
Country	Country

Goal

1. Read customer records from Azure SQL.

2. Check whether customer exists in Dataverse.

3. Update existing records.

4. Create new records if not found.

5. Generate migration statistics.
   Prerequisites

1. Dataverse Environment

Create a Dataverse table:

Customer Master

Columns:

• Customer Number

• Name

• Email Address

• Phone

• Country

2. Install Python SDK

Microsoft has made the SDK available through PyPI and GitHub as an open-source project.

pip install PowerPlatform-Dataverse-Client
  

Additional packages:

1. pip install pandas

2. pip install pyodbc

3. pip install sqlalchemy

Step 1: Connect to Azure SQL

  
    from sqlalchemy import create_engine
import pandas as pd

server = 'myserver.database.windows.net'
database = 'CustomerDB'
username = 'sqladmin'
password = 'Password'

connection_string = (
    f"mssql+pyodbc://{username}:{password}@{server}/{database}"
    "?driver=ODBC+Driver+18+for+SQL+Server"
)

engine = create_engine(connection_string)

query = """
SELECT
    CustomerID,
    CustomerName,
    Email,
    Phone,
    Country
FROM dbo.Customers
"""

df = pd.read_sql(query, engine)
  

Step 2: Authenticate to Dataverse

  
    from dataverse_client import DataverseClient

client = DataverseClient(
    tenant_id="tenant-id",
    client_id="app-id",
    client_secret="client-secret",
    environment_url="https://org.crm.dynamics.com"
)

Recommended Authentication

Use:

• App Registration

• Client Secret

• Managed Identity (Azure-hosted workloads)

This approach avoids interactive user authentication.

Step 3: Read Existing Dataverse Records

To prevent duplicates:

existing_customers = {}

records = client.retrieve_multiple(
    table_name="new_customermasters"
)

for row in records:
    existing_customers[
        row["new_customernumber"]
    ] = row["new_customermasterid"]

Step 4: Create or Update Records

for _, customer in df.iterrows():

    customer_no = customer["CustomerID"]

    payload = {
        "new_customernumber": customer_no,
        "new_name": customer["CustomerName"],
        "new_email": customer["Email"],
        "new_phone": customer["Phone"],
        "new_country": customer["Country"]
    }

    if customer_no in existing_customers:

        client.update(
            table_name="new_customermasters",
            record_id=existing_customers[customer_no],
            data=payload
        )

        print(f"Updated: {customer_no}")

    else:

        client.create(
            table_name="new_customermasters",
            data=payload
        )

        print(f"Created: {customer_no}")

Step 5: Logging and Error Handling

import logging

logging.basicConfig(
    filename='migration.log',
    level=logging.INFO
)

try:
    # migration code
    pass

except Exception as ex:
    logging.error(str(ex))

Step 6: Generate Migration Summary

total_records = len(df)

print("Migration Completed")
print(f"Processed Records : {total_records}")
print(f"Created Records   : {created_count}")
print(f"Updated Records   : {updated_count}")
print(f"Failed Records    : {failed_count}")

Performance Optimization

For large migrations (100K+ records):

Use Batch Processing

batch_size = 1000

Process records in chunks.

Parallel Processing

Use

concurrent.futures

for multiple migration threads.

Incremental Loads

Store

LastModifiedDate

and migrate only changed records.

Automation Options

Once tested, the Python solution can be hosted in:

Azure Function

Trigger:

• Daily

• Hourly

• On-demand

Azure Container Apps

For large-scale enterprise migrations.

Azure Automation Account

For scheduled synchronization.

GitHub Actions

CI/CD driven migration deployment.

Conclusion

The new Dataverse SDK for Python bridges the gap between enterprise data engineering and Power Platform. With direct Python access to Dataverse, organizations can leverage familiar tools such as Pandas, Jupyter, AI/ML libraries, and Azure services while maintaining Dataverse security and governance.