Can anyone assist with performing BulkInsert and BulkUpdate operations on Snowflake tables using Dapper and C#. I am encountering issues where the values are not being inserted into the Snowflake tables. Any help would be greatly appreciated. Thanks in advance!
public async Task BulkInsertAsync(string tableName, IEnumerable data, int batchSize = 1000)
{
int totalRowsAffected = 0;
var batches = data.Batch(batchSize);
foreach (var batch in batches)
{
string connstring = "connection string";
using SnowflakeDbConnection conn = new SnowflakeDbConnection();
conn.ConnectionString = connstring;
conn.Open();
using var transaction = conn.BeginTransaction();
try
{
var dataTable = batch.ToDataTable();
var sql = GenerateBulkInsertSql(tableName, dataTable);
var rowsAffected = await conn.ExecuteAsync(sql, dataTable, transaction: transaction);
transaction.Commit();
totalRowsAffected += rowsAffected;
}
catch
{
transaction.Rollback();
throw;
}
}
return totalRowsAffected;
}
private string GenerateBulkInsertSql(string tableName, DataTable dataTable)
{
var columns = string.Join(", ", dataTable.Columns.Cast().Select(c => c.ColumnName));
var values = string.Join(", ", dataTable.Columns.Cast().Select(c => ":" + c.ColumnName));
return $"INSERT INTO {tableName} ({columns}) VALUES ({values})";
}
public static DataTable ToDataTable(this IEnumerable data)
{
var dataTable = new DataTable();
var properties = typeof(T).GetProperties();
foreach (var prop in properties)
{
dataTable.Columns.Add(prop.Name, Nullable.GetUnderlyingType(prop.PropertyType) ?? prop.PropertyType);
}
foreach (var item in data)
{
var row = dataTable.NewRow();
foreach (var prop in properties)
{
row[prop.Name] = prop.GetValue(item) ?? DBNull.Value;
}
dataTable.Rows.Add(row);
}
return dataTable;
}
Jayraj ChhayaPosted Oct 22, 2024, 6:46 AM
To perform a successful BulkInsert into Snowflake tables using Dapper, ensure that your connection string is correctly configured and that the SnowflakeDbConnection is properly instantiated. The provided code structure is generally sound, but there are a few key areas to verify:
Connection String: Ensure that the connection string is valid and includes necessary parameters such as user, password, account, and warehouse.
DataTable Conversion: The
ToDataTablemethod should accurately reflect the structure of the Snowflake table. Ensure that the data types in your C# model match those in the Snowflake table.SQL Command: The
GenerateBulkInsertSqlmethod constructs the SQL command. Ensure that the generated SQL is valid and that the table name is correctly specified.Error Handling: Implement logging within the catch block to capture any exceptions that may provide insight into why the insert is failing.
Here’s a refined version of your
BulkInsertAsyncmethod with added logging: