Integrating all Discrete Heterogeneous Application of an Enterprise Leveraging Biztalk Server

INTRODUCTION

This white paper is an attempt to address the enterprise application integration initiative in an enterprise, for integration of all heterogeneous discrete applications, step by step, leveraging BizTalk Server.

Usually, I had observed that, in an enterprise, there are many successful applications deployed on the application and web servers, accomplishing the task of managing the business processes mainly revolving around the business domain.. While I have found each and every application to be great in their own, but the big picture revealed many caveats and limitations, the principal one being the inability of the business to update the mainframe, SAP etc. online and read data from the same online; and the secondary ones being that all the applications are like silos achieving some part of business functionality in their own best possible way, but not being able to share common business functionalities because of lack of or absence of any integration facility. This struck my mind, when SOA came into the market.

Well, the basic idea is to integrate all the heterogeneous discrete applications using BizTalk so as to facilitate software reuse, getting a holistic picture of Business Process Management and finally moving towards higher levels of SMM (SOA Maturity Model). The first and foremost benefit is software reuse (code reuse) without any slightest ambiguity, and the second most important one is that during migrations to higher technologies in future, one needs to migrate only the service internals (within a service boundary) and need not rewrite the entire application, otherwise wasting lot of dollars in turnkey / fixed bid / T & M projects. And then last but not the least, it would also facilitate 'data reuse' which would be clear as you proceed through this white paper step by step. For the sake of heads up, I have coined the term 'data reuse' as something which would allow one application to share a set of tables / schemas of some other application's database seamlessly. Well, if you don't like the term you can rename it to something else but still, our objectives would remain basically the same.

Please don't fall asleep after reading this much, as the real thing would follow soon. Hence without much ado about anything let's jump on to the first step.

Steps To Migrate To EAI And SOA in an Enterprise

STEP - I

Right now we have the following applications running in the enterprise:

ASP.NET web applications
Desktop WinForms applications
Desktop WPF applications
WPF Browser applications
ASP.NET MVC 2 applications
Silverlight 4.0 applications on top of ASP.NET Web applications
Silverlight 4.0 applications on top of ASP.NET MVC 2 web apps
Console applications
Windows Service applications
Other applications

Almost all the applications are presumably or more or less following the CBLA (component based layered architecture) which looks like the following:

Now let's turn our attention to the data access layer which is nothing but a collection of entity classes and access classes having methods declared therein, finally being compiled into a dll. Please don't be afraid. You need not change anything much.

The objective is that the BizTalk middleware has to communicate with these dlls (the data access layers). And in production environment first two BizTalk servers would belong to NLB cluster, whilst the last two would belong to Active-Passive cluster. The first two would facilitate the communication using HTTP, SOAP, WCF through BizTalk Isolated Host (the IIS process) and the last two would facilitate communication through POP3, MSMQ, FTP, FILE through BizTalk in-process host (BTSNTSvc.exe). Now, in such a scenario, the Microsoft DTC kicks in, which we enable during installation (please refer to the installation guide sent by me earlier in your inbox).

If such a scenario needs to be supported by the data access layer dlls, then each class in data access layer need to inherit from System.EnterpriseServices.ServicedComponent namespace. Please don't be afraid, it's not much work. Just inherit all the classes from ServicedComponent and rebuild your data access layer. This is to be done such that BizTalk can access the data access layer in a multibox installation scenario, since now the dll can take part in MSDTC transactions as well. Well, not yet so. After that you need to register the dll in each box (or just one box in case of development server) using regsvcs utility provided by .NET framework, and even better if you strong name the dlls and place the same in GAC too (for each machine in case of production environment and single machine in case of development environment). To substantiate my argument I am providing two links below for you to refer:

http://social.msdn.microsoft.com/Forums/en-US/biztalkgeneral/thread/9b25aae4-ceff-4782-a299-120b753319e0

http://www.biztalkgurus.com/blogs/biztalk/archive/2009/10/28/got-a-great-biztalk-tip-or-trick-and-you-could-win-richard-seroter-s-soa-patterns-book.aspx

In the first link you would see that my blog post in MSDN pertaining to the above argument has been voted as helpful post and in the second one I have won a prize for presenting the best BizTalk tip / trick in year 2009 from Stephen W Thomas (a Microsoft MVP and BizTalk guru). Also in numerous other posts I have advised others to follow the above principle and got applauded for the same eventually. Now a skeptic might ask that why not use the WCF SQL adapter or the Oracle adapter provided by BizTalk in such cases. The answer is that doing so would soon lead the production environment into deadlock situation under heavy message traffic, the prime reason being that the ambient transactions started by the aforementioned adapters is having an isolation level of Serializable (mandated by default by BizTalk), whilst the default transaction isolation level of both SQL Server and Oracle server is Read Committed. Hence I do not recommend using database adapters while communication from BizTalk to databases. The data access layer methods could be called from a serializable .NET component, which is in turn called from within an Expression shape in BizTalk orchestration. In order to substantiate my arguments I am furnishing the following links in MSDN forums:

http://social.msdn.microsoft.com/Forums/en-US/biztalkgeneral/thread/2f4e1b87-63ab-496f-9949-30196497ea44/#8ad3f3e4-5c67-4d83-b0e6-dd2b16869deb

http://social.msdn.microsoft.com/Forums/en-US/biztalkgeneral/thread/86a29e83-9de7-4133-b236-79f432bf689b/#08e129ad-6966-4e00-a2cc-47ea0d9146ee

In one of the links, Andrew Zhu, the moderator from Microsoft, has marked my post as answer and in the other link I got a vote for posting a helpful blog.

STEP - II

Now, let's turn our attention to the business logic layer. Please do not be afraid, as here too, we need not change anything. We only need to add something. But before that I need to explain what tight coupling means and how loose coupling is better than that. Hope, until now, you have not found my explanation to be rather insipid. So, if you are exhilarated, please read on. Let's say you have a method in business logic layer like the following:

List<PolicyHolders> GetPolicyHolderDetails (int fromDate, int toDate, int userId)

{

...

}

Now, this method must have been called because of some event raised due to some user interaction like selecting a row in a gridView control or selecting an item in some dropDownList control. So we see that here the business logic layer method is tightly coupled to the UI layer, which means that the context (the English term context, not the technical term) of the operation (the method call) is provided by the UI layer. Had this method existed independently or autonomously, then it would have been quite insignificant as far as business process realization is concerned. So, now what we need to do. We need not touch our respective business logic layers anyway. All that we need to do is that we need to add an extra layer on top of it such that the method call takes the said 'context' as parameter. Welcome to the term 'contract' in SOA, nothing but a DataContract (a class decorated with DataContract attribute) declared in WCF which becomes a parameter to the method. Actually in BizTalk, we would have corresponding schemas declared (the .xsd files) and we shall pass an instance of the corresponding message to the method either as System.Xml.XmlDocument or Microsoft.BizTalk.XLANGs.Message parameter. No wonder that SOA design is often called as "contract first approach". For this, we need to clearly demarcate our business logic methods into various subject areas of our overall business functionality. Now for each such subject area, we need to define a canonical schema (also called document normalization in BizTalk parlance). Such methods would internally extract the relevant fields from the message corresponding to the aforesaid schema and then call our existing business logic methods. This would introduce loose coupling.

In a nutshell, now the SOAP method call depends upon the message passed and not upon some UI control selection. Please do not worry, if I am not able to explain you properly then I am going to discuss in detail about the same with examples in a meeting room (you need to book Terabyte or Gigabyte in advance for that). For each subject area, we are going to have an orchestration in BizTalk and obviously a canonical schema to work with. The UI layer would just send the SOAP message to the BizTalk middleware and then the BizTalk orchestration, after receiving the message, would fire up an instance of itself, which would then use an expression shape and some C# code to extract the requisite parameters from the message and call our existing BLL method as shown at the beginning of this subsection. The method in turn would call the DAL layer methods.

Finally, all such orchestrations would be exposed as WCF services. So now we have autonomous, coarse-grained loosely coupled services having explicit boundaries (one per subject area of the application). We are slowly getting used to the SOA jargons like, 'autonomous', 'coarse-grained', 'explicit boundaries', 'schemas and contracts' and 'loose coupling'; is it not so. Once such orchestrations are exposed as WCF services, just any application UI could call these services, provided it can send the requisite message as parameter during such SOAP calls. Thus we are not only fulfilling the promise of "code reuse" but also of "data reuse". Come on, we have now moved to SOA Maturity Model Level 2 already, wherein enterprise-wide, all applications are sharing their respective business functionalities with others with the help of autonomous, coarse-grained loosely coupled services.

For the receive or send ports corresponding to the BizTalk orchestrations, we might have to use various adapters like FILE adapter, Host Adapter for Files (VSAM files), Host Adapter for Applications (CICS transactions), WCF-BasicHttp adapter, WCF-WSHttp adapter etc. The host adapters would facilitate communication with mainframes, whilst the WCF adapters would facilitate communication with other discrete applications. So uptil now, our Service Oriented Modeling and Architecture (SOMA), looks like the following:

STEP - III

Now that we have integrated all our applications, moved to SOA Maturity Model Level 2, so what next? We still have some inherent weaknesses in our enterprise architecture, that is, the client must explicitly know which WCF service to invoke, which means that the client must know the exact endpoint of the WCF service to invoke. Had it not been better if client could have sent a SOAP message to a bus and let the message finds its own WCF service with the help of routing and transformation features of the aforementioned bus. Welcome to ESB Toolkit 2.1.

Typical ESB Toolkit 2.1 Itinerary for WNL

Here we use two-way itinerary on-ramp (OnRamp.Itinerary.Response) receive port with WCF-BasicHttp receive location. Then we use a messaging extender for performing some transformation using a static resolver (map) or we apply some business rules using a BRE resolver or we do both. Then we use an orchestration extender to hook up the orchestration we have created in step II for a given subject area. Then we return the processed message back to the itinerary on-ramp as response message.

By using ESB Toolkit 2.1 we would be moving to SMM Level 3. But this endeavor would happen after sometime, that is, after we fully achieve level 2. And in my opinion, it would take at least another year. We shall hook all our orchestrations to our itineraries using Orchestration extenders, all our maps using transformation extenders, all our receive ports using itinerary on-ramp and for send port we would have only one dynamic send port corresponding to off-ramp. For knowing what these jargons are, please read through ESB Toolkit 2.1 documentation in MSDN. The enterprise service bus would have message routing and transformation capabilities and it would take care of routing and transforming external messages coming from clients. In that case our architecture would look like:

Now, this completes our SOMA architecture as per IBM standards. SCOM 2007 provides monitoring and management of services, health check of services, alerts, events etc.

CONCLUSION

Well, that's pretty cool isn't it? And it is also not involving huge amount of work as has been ostensibly felt hitherto. Let's strive to make all companies as agile enterprise. Thanks and Bye!