Building an Azure SQL to Dataverse Migration Solution with the New Dataverse SDK for Python

Microsoft's new Dataverse SDK for Python (Public Preview) opens exciting opportunities for developers, data engineers, and Power Platform professionals to integrate enterprise data with Dataverse using Python. The SDK supports core capabilities such as CRUD (DML) operations, schema-related operations (DDL), file uploads, and seamless integration with the Python ecosystem including Pandas and Jupyter Notebooks.

In this article, we'll build a practical use case: migrating customer data from Azure SQL Database to Microsoft Dataverse using Python .

Solution Architecture

Components

Azure SQL Database
Python Runtime (Local, Azure Function, Azure VM, GitHub Actions, etc.)
Dataverse SDK for Python
Microsoft Dataverse Environment
Microsoft Entra ID Authentication

Business Scenario

A company maintains customer master records in Azure SQL.

Azure SQL Column	Dataverse Column
CustomerID	Customer Number
CustomerName	Name
Email	Email Address
Phone	Telephone
Country	Country

Goal

Read customer records from Azure SQL.
Check whether customer exists in Dataverse.
Update existing records.
Create new records if not found.
Generate migration statistics.
Prerequisites

1. Dataverse Environment

Create a Dataverse table:

Customer Master

Columns:

Customer Number
Name
Email Address
Phone
Country

2. Install Python SDK

Microsoft has made the SDK available through PyPI and GitHub as an open-source project.

pip install PowerPlatform-Dataverse-Client

Additional packages:

pip install pandas
pip install pyodbc
pip install sqlalchemy

Step 1: Connect to Azure SQL

  
    from sqlalchemy import create_engine
import pandas as pd

server = 'myserver.database.windows.net'
database = 'CustomerDB'
username = 'sqladmin'
password = 'Password'

connection_string = (
    f"mssql+pyodbc://{username}:{password}@{server}/{database}"
    "?driver=ODBC+Driver+18+for+SQL+Server"
)

engine = create_engine(connection_string)

query = """
SELECT
    CustomerID,
    CustomerName,
    Email,
    Phone,
    Country
FROM dbo.Customers
"""

df = pd.read_sql(query, engine)

Step 2: Authenticate to Dataverse

  
    from dataverse_client import DataverseClient

client = DataverseClient(
    tenant_id="tenant-id",
    client_id="app-id",
    client_secret="client-secret",
    environment_url="https://org.crm.dynamics.com"
)

Recommended Authentication

Use:

App Registration
Client Secret
Managed Identity (Azure-hosted workloads)

This approach avoids interactive user authentication.

Step 3: Read Existing Dataverse Records

To prevent duplicates:

existing_customers = {}

records = client.retrieve_multiple(
    table_name="new_customermasters"
)

for row in records:
    existing_customers[
        row["new_customernumber"]
    ] = row["new_customermasterid"]

Step 4: Create or Update Records

for _, customer in df.iterrows():

    customer_no = customer["CustomerID"]

    payload = {
        "new_customernumber": customer_no,
        "new_name": customer["CustomerName"],
        "new_email": customer["Email"],
        "new_phone": customer["Phone"],
        "new_country": customer["Country"]
    }

    if customer_no in existing_customers:

        client.update(
            table_name="new_customermasters",
            record_id=existing_customers[customer_no],
            data=payload
        )

        print(f"Updated: {customer_no}")

    else:

        client.create(
            table_name="new_customermasters",
            data=payload
        )

        print(f"Created: {customer_no}")

Step 5: Logging and Error Handling

import logging

logging.basicConfig(
    filename='migration.log',
    level=logging.INFO
)

try:
    # migration code
    pass

except Exception as ex:
    logging.error(str(ex))

Step 6: Generate Migration Summary

total_records = len(df)

print("Migration Completed")
print(f"Processed Records : {total_records}")
print(f"Created Records   : {created_count}")
print(f"Updated Records   : {updated_count}")
print(f"Failed Records    : {failed_count}")

Performance Optimization

For large migrations (100K+ records):

Use Batch Processing

batch_size = 1000

Process records in chunks.

Parallel Processing

Use

concurrent.futures

for multiple migration threads.

Incremental Loads

Store

LastModifiedDate

and migrate only changed records.

Automation Options

Once tested, the Python solution can be hosted in:

Azure Function

Trigger:

Daily
Hourly
On-demand

Azure Container Apps

For large-scale enterprise migrations.

Azure Automation Account

For scheduled synchronization.

GitHub Actions

CI/CD driven migration deployment.

Conclusion

The new Dataverse SDK for Python bridges the gap between enterprise data engineering and Power Platform. With direct Python access to Dataverse, organizations can leverage familiar tools such as Pandas, Jupyter, AI/ML libraries, and Azure services while maintaining Dataverse security and governance.