Introduction
Our Business Analyst went abroad last week to meet a new customer who selected us to develop his Content Management System. The Analyst had met and extensively discussed details about the business need with the customer. Our hard working Business Analyst had returned after two weeks with on hand detail Business Model document, where he together with a coworker prepared the formal requirement specification in a hurry, during this process they conference with the customer to clarify some requirements. Formal requirement document was sent to the customer for initial approval, where it was returned with minor adjustments. The requested minor adjustments were made by the business analyst himself to get the final approval from the custom end. It was a good effort; we got the fixed set of requirements after three weeks, since we first met the customer. Business Analyst directed the business model to system architect. Soon, the Non-Functional Requirement Document was finished and was approved by the customer with no changes. As the next immediate process, the user cases were defined. System Architect finalizes the design of the system together with the System Analyst. The Database Admin was followed by identifying/ defining the entities and their relations together with DDL (Data Definition Language) of the system. The UI (User Interface) designer designed the system user interface to finalize the initial design effort. At last, the deployment model was defined and starts implementing the system steadily, after getting all required approval from the customer end. What I just mentioned, was an imaginary system design process, which has limited real world applicability. In real world practice, there are many variables for one to feel that system design methodology needs to be adjusted, from customer to customer, as well as from project to project. This article is a vigorous attempt to introduce a concept (model) that stands against all extreme cases of modern day system development life cycle.
Designing a system needs thorough study of the problem domain. This implicitly means time, followed by money, which threaten the customer, who doesn't have an adequate visibility over the advantages of a properly designed system. It just makes things worst, when the current day diversity of requirements argues the value of one's experience. If one soon says that "Oh, This is just the same solution we delivered last year" he soon be ended up loosing the customer by delivering the wrong product.
There are good design methodologies out there, but often due to time limitations (where client push for quick releases, while developer struggle with extremely tight deadlines) solution providers aren't confident to apply them. Today, Solution Providers tend to implement systems without designing them properly. More often, the development starts after loosely evaluating the depth of the project. The initial System Requirement Specification is often changed at the middle part of the project, due to insufficient understanding, the customer had, before starting the project. The customer often starts to understand the system requirements as the solution provider starts to release it. The initial releases of the system allows customer to get a feel of the system, which encourage (unfortunately) him/ her to refine the system at the middle of the development. However, some of these changes lead to shake the whole foundation of the system, forcing designers to rethink their initial design, even though for customer's none technical eye, it could see as a minor modification. A fight, between customer's money versus developer's 24 hours a day.
The customer estimates the correct time to enter the market at the initial stage of the project. The massive competition at the software market place makes it important to take decision to pin-point accuracy. The most important is to hit the market on time.
What Solution Providers Do Today?
You don't need civil engineering expertise to understand the instability of a building foundation put up, not knowing the number of stories of the finished building. The same philosophy applies to software architecting as well, expanding a system, which is developed for a limited set of features is just adding more problems than features. Higher percentages of solution providers push for add-hoc development approaches, with extreme programming to achieve tight deadlines. The customers also encourage them, with having that daydream of re-factoring the system, once it is running at a profit earning stage. Another reason for customers to treat system designing a low priority is the time the designers/ analyst take to investigate/ design a system. The Customer, according to some terminology is the GOD, is regularly expecting tangible outputs from the solution provider, but a proper system designing process may makes him wait for several months. In order to support the customer need, solution provider often plan for by weekly or monthly releases from the starting day of the project. This extreme need of release the system regularly, while developing it without a proper design, burnout resources at the middle stage of the project.
The developers have used to start the system development process, with a scale down version of the system and gradually patch rest of the system around it, which result an unstable product with lots of repetitive, badly grouped codes. Additionally, it creates programmer dependant code modules, since each module is coded by individuals with different skill set. The inconsistent code makes debugging of the system a nightmare. This cost time as well as money, in many direct/ indirect ways, while making it impossible to expand.
Why We Should Design System Properly? Do I have to answer this?
Handle regular changes, upgrades/ enhancement easily and dealing with issues related to resource bandwidth, are some of the few advantages, you gain with a properly designed system. Consistent code allows managers to pull/ push resource from/ to different section of the project easily.
In order to drive the system development effort smoothly, the designer should introduce a steady design/ coding pattern at the early stage of the process, allowing every developer to gain mastery over it. This opens the opportunity to remove the architect's dependency, at later stage of the project.
How you Approach to Design your System
Let me highlight first, that I do not draw the famous user case diagram and sequence diagram, when design systems. In-fact I have another approach that let you understand the system better and directly draw the class diagram. But before drawing the class diagram, I encourage you to draw an activity diagram and/ or a system overview diagram and/ or a module interaction diagram and/ or any other type of diagram, which help you to fully understand (feel) the system.
As I reiterate many times before, designing a system mainly depend on designer's understanding of the problem domain, so better you understand, easier the system design would be. However in modern days, it is less probable to assume that designers get a sufficient time frame to fully investigate the system, before starting the design.
As the first step of designing the system, let's try to maximize the understanding, we have about the system. If you are given a System Requirement Specification (SRS), read it over and over, till you feel comfortable. Then, refer to some online resources, such as articles, open source and other similar types of projects. The main goal of this effort is to correctly gauge the scope of the project. This understanding help designer to better predict the future of the system and keep adequate space for later expansions. (Note: New comers always need help, from experiences, to correctly identify the resource pools, since the complex business requirements are hardly seen by new technical eyes.) In addition to this, new comers have to be attentive not to underestimate important business requirement by evaluating them from technical angles. This is one of those crucial areas, where client could be extremely rigid and even would decide to reject the whole system, complaining that solution provider delivered a wrong product.
Discover the system by asking questions about it. I have a technique that I used, since the very first system, I designed. Here, I am presenting it for you, as the second step of approaching to design a new system. Open a Note pad or get a piece of paper, and then start asking question about the system. In this effort you are free to ask any question, but in order to get the best result, I encourage you to start with three main questions (about input, process, and output of the system) as listed bellow.
1. What are the Input(s) of the system?
2. What are the Processes of the system?
3. What are the Output(s) of the system?
In order to optimize the throughput, start this process with a fresh mind by forgetting all the information you gathered by reading and referring to various resources. This is a rethink of the system with a fresh mind. This approach will help you to understand and define the product margins accurately. In this effort, interestingly, the questioner as well as the answerer will be the same and it will be you. A better question would be a one that creates several derivative questions by the answer given to it. So an answer to a question may produce another question (one or more) or a leaf level operation or function of the system (use case of the system). The questions that are not answerable may need to be directed to the client, and those, which are not answered by the client has to be creatively handled by the system designer in a way, so that it doesn't affect the stability of the system. The questions, which are not answerable as yet needed to be creatively grouped into separate module(s) for later implementations.
As an example, let's try to apply this methodology to discover the feature list of a simple Document Management System (DMS). The questioning will start at a very abstract level with the three main questions.
What are the inputs going to be like?
Files such as word, txt, media files etc (I would say any type of files)
What are the processes of the system?
It is a document management system and it will help my client to manage his document properly.
What are the outputs?
Allow user to view file(s) online as well as allow them to checkout/ remove file(s).
If you get fairly good answers to all three questions above, I would say that you have understood the system. The above three questions, virtually can be used to discover any software system. The answers, you write to these three questions will create many more derived questions to continue. This process will help us to identify all external/ internal (system) actors of the system as well as all use cases or rather in simple term, leaf level functions of the system (A leaf level function is an independent, granular level operation that describes a specific function of the system). The process of asking question and writing answer will continues until we get to leaf level functions, where we cannot ask any more questions about the answer.
Let's go and analyze the second question little more to see, what it will discover for us..
What is the input going to be like?
Files such as word, txt, media files etc (I would say any type of files).
Are we going to have any restriction over document types?
Probably NO, but I think we need to check for virus before any upload.
Note: This introduces a new module to the system named Input Validation Module.
How we going to check for virus?
We need to have a module that can read a data of a file and check whether it contain virus or not. This module will take in a file path and return us a Boolean to inform virus yes/ no state.
Note: This can be treated as a leaf level function of the system.
Who is going to upload documents?
Employees of each department of the organization will upload.
Who is an employee, how we identify him?
All employees will be registered with the DMS system at first, and will issue a user name and a password.
How we going to register users?
System will have an interface to register users.
Who is a user?
A user of this system can be from four different division named Administrative, Account, Marketing and Development. Each user will belong to one of these four roles. A user from each division will have different privileges. A user from Marketing or Development division will have default privileges where they can upload and mange their documents but users from account division can over look marketing division and administrative division can over look any other division apart from doing default functions of the system. (Administrative>Account>Marketing = Development).
Note: Above answer introduce us two modules called user management, Security and Privileges. Let's not dig in to that section right now.
What is a default privilege of a user?
This includes upload documents, view documents online, check out documents for editing and delete a document. Again any document uploaded will be shared among users of the same division automatically. A document belong to a one division can be shared with another division only by explicitly giving permission. A private document will not be supported by this system (confirmed by the client software requirement specification).
What is over looking a user?
This is the privileges of accessing documents uploaded by others user without needing any permission from the document owner.
What data user needs to insert to register with the system?
This should list the user metadata.....
How a documents flow through our system?
Select and upload the document by the user > check for virus (Is there any other validation needed here? I need to talk to the client) > Files added for approval > Administrator Approve and publish the file > File is available.
Do we need to notify users, when a new document is successfully uploaded or received?
The user who uploads the document will get an email noting the reject/ accept state. In addition to that we have to send another notification to users who have subscribed to be notified as this operation happens.
Note: This introduces a new module to the system named Notification Module.
Are we going to have versioning?
Yes we need to have versioning.
Can a user undo an update?
I need to talk to the client about this.
Can multiple user checkouts a one document?
Let's stop this now.....
Look at the way it expands, it just keeps on expanding as you ask questions. I omit lots of the questions, since the article is getting longer, but I believe that this sample question and answer set will provide enough guidance to understand the concept. This process need to be repeated (go through the questioning and answering process again and again) several times to fully discover the system. Firstly, you have to recognize the main modules of the system, I have already identified couple of modules named input validating module, user management module, security module, notification module and not last but least the main document management module. If you do not have enough practice, then you may find it harder to identify the modules of the system at once, but as you practice this method several times, you can identify the modules of the system easily (This comes with experience, and to get this experience quicker, you can do this process over and over on the same system, until you feel comfortable with the module break down). When I first started with this approach, I never think of the module break down, but just tried to recognize the full list of leaf level function of the system (granular level use cases of the system), then I started to group them in to classes (classes will group same types of functions/ operations) and then group the classes to form modules. I recommend new comers to take that route. Once you gain mastery over the concept, you can directly break the system into modules without asking any question at all. Then you may further break each module by asking question (as explained above) about each module separately. Once you designed couple of systems, the module breakdown will become easier and concept will become friendlier. This experience will also reveal you that some modules are repeatedly coming in every other system you design.
You may follow this process to identify all the leaf level functions and main/ sub modules of the DMS system.
Breaking your system with an easy concepts
In this application design approach, we have to treat the software system as a real world organization, which operate on a pure manual environment. I have found that any software system can be easily mapped to a real world organization. This mapping helps you to easily control the designing effort of the system and allows you to handle updates, changes and enhancement, very easily. When you have that luxury of visualizing the software system via a pure manual real world organization (more friendly), you can quickly and accurately find the most appropriate place to apply changes, without hurting the expandability of the system. This approach will better suite for complex systems, but will obviously suite for simple systems too. In order to successfully map a software system to a real world organization, you need to correctly analyze the software system in a way, so that it explains, how it would process in a pure manual environment. The next step is to automate that manual process with a software model. It is highly recommended to keep similar naming convention in between manual system and software system for better visualization.
Let's have a look at a typical organization as bellow and try to see, how it would respond to a typical request.
Figure-1 Showing a typical organization hierarchy
CEO - Chief Operation Officer of this system is acting as the front end (User Interface of the software system) of the organization. The main responsibility of the CEO is to interact with out side and identify/ guide managers in the correct sequence to complete a task. The CEO will directly map to the front interface of our sample Document Management System (DMS).
Super senior, senior and junior Managers - The manager's duty is to utilize resources (workers or sub managers) in the correct sequence to complete a particular task. A complex organization may have several levels of managers (super senior managers, senior managers and junior managers). Higher level managers will use one or many lower (immediately) level managers to complete a task. According to our diagram, we have three senior level managers named Accountant, Manager Operation, and Manager Delivery, where each manger is given set of junior managers to perform their duties. As you can see, all junior managers are equipped with group of workers.
Workers - Workers do all granularly level operations and more often, workers are given pure independent tasks, which they can perform without making any dependency over any other task or worker.
Rules of the Organization -
- Higher level managers have comprised with the knowledge of the capacity of junior level worker groups. Hence, they know what worker to pick, in order to complete a task.
- Same level entities are not allowed to communicate with each other, but if a task is needed two or more same level managers to complete, then that task will be handled by the super manager, who is senior to the managers, who needed to be involved to complete that task.
- Any entity is only responsive to the immediately higher entity, and only utilizes the immediately lower entities.
In order to understand the organization well, let's see how this organization will respond to a typical request, made by the CEO.
Figure-2 A request is been processed utilizing different types of workers of the organization
In this sample the CEO of the organization has issued a request named "request 1" to "Manager Operation" (This operation assumes to be an independent one from Accountant and Manager Delivery). The CEO knows the manager, who he needs to issue the command to process this particular request successfully. In this case "Manager Operation" has received the request from the CEO and he has proceeded by making the request named "request 1.1" to "Manager 1". The "Manager Operation" and CEO are at waiting mode now while "Manager 1" is processing the first part of the request. The "Manager 1" has triggered a request named "request 1.1.1" to "Worker 3" to complete the "Task 1". As the "Manager 1" received the result of the "Task 1" he has issued the next request to complete the "Task 3" to the same worker. After getting the result, the "Manager 1" has used another worker named "Worker 1" to complete the "Task 2", which also complete the "request 1.1.2" and "request 1.1". The "Manager 1" has responded to "Manager Operation" with result of "request 1.1". The "Manager Operation" has picked the correct junior manager to handle the second part of the main request i.e. "request 1.2". As a result, "Manager 3" has received the "request 1.2" from the "Manager Operation", where he has responded to it by making the "request 1.2.1" to "Worker 4" to complete the "Task 3". Finally, "Manager 3" has responded to "Manager Operation" with the result of "request 1.2" which complete the whole processing of the "request 1".
In our system design approach, we will also use the same concepts to break and design the software systems. You will first identify the granular/ leaf level functions of the system (Tasks of the system) and will group them in to classes, where each class is responsible for similar types of operations. These classes are pretty much similar to workers of the above diagram. As the number of workers of the organization increases, you add managers to manage the worker group and same concept will be used to control (manage) managers. When it comes to the software system, we will add managers to manage worker classes, however during this process, you need carefully group classes of the system to form modules as well (The modules can be formed by grouping similar types of classes). The number of managers and the depth of the manager pool inside a module can be decided by the complexity of the module (or the amount of worker class loaded or grouped in to the module). These modules will be treated as separate divisions of an organization, where each module will be controlled by one or many mangers, positioned considering the complexity of the system. These module controlling managers will help modules to interact with each other.
Start designing the Document Management System
Initial analysis of the Document Management System opens up several leaf level functions as well as couple of modules. As explained before, it is important to break the system in to modules before designing it, better you modularize the system, easier the system maintenance would be. So let's modularize our DMS system horizontally as well as vertically. This will help you to design each module separately by treating each of it as a separate division of an organization. Even though we didn't fully analyze the DMS system, I guess that, what we have discovered is detail enough to explain the concept with an example. As the third step of designing the system, let's draw a system architecture diagram, which help us to understand the key modules of the DMS system and their interaction as well as positioning.
Figure-3 System Architecture Diagram of the Document Management System
Manager Operation - Manager Operation is acting as the head for all divisions, where divisions are the Document Management Division, Email/ Notification Division, and User Manager/ Security Division. The Manager Operation will coordinate all functions of the system while helping each division to interact with one another as and when it is appropriate.
E.g.:- Once a document is published, several emails needed to be sent, so then manager operation will first request the manger of the DMS division to publish the document and depending on the state of the publication the manager operation will request the email/ Notification division to send the correct email to the document owner and other subscribers. In this attempt the system has used two divisions to complete the task and these two divisions have being controlled by the Manager Operation.
DMS, Notification, User Manager/ Security - These are similar to three divisions of a generic organization. I have separated the leaf level workers in to three groups named Data Access Layer, Template Handlers and Other Operations. The DAL (Data Access Layer) is dealing with all database related operation such as get, add, update, and delete data, where as the section named "Other Operations" is responsive to any other leaf level operations as required.
Common Operation - This module does the common operations. A correctly designed common operation module can be easily reused in any system. This division is shared among all component or modules of the system. In real world organizations also you find common divisions, such as company library, company canteen, reception etc. In most cases may use this module to group classes that log exception/ transactions, store shared objects, handle errors etc.
Let's further analyze the User Management module of our Document Management System to see, how it support functioning three basic features of the user management module named add, edit and remove user(s) (Please refer to the class diagram bellow). According to the diagram, firstly, you have the system user interacting with the DMS interface. In this DMS system, we only have one (to make the sample a simple one) senior manager named "Manager Operation". There is a one junior level manager to control the user management module, which handles user management/ security related operations. The manager named "User Manager" directly communicating with the junior manager of the sub module named DAL (Data Access Layer for the User Manager module). The data access layer is responding to the corresponding immediate manager only via the abstract class named "HandlerData". Inside the DAL you can see there are three classes to handle three basic types of data related operations named Editing, Adding and Removing of data (In this sample all handlers are same as workers of the above described organization).
Figure 4 - Class Diagram of Scale down Version of a UserManager Module
E.g.:- Adding a User: The user is requested to fill the registration form via the user interface. As the user click on the submit button, the UI side is expected to do all required client side validation to verify the input. Then the user profile is stored inside the model class named "User" (where it can store user profile with correct entity relations) pass it to the main manager of the DMS system named "ManagerOperation" by invoking the method named "AddUser". The main operation manager correctly identifies the module to talk to complete the request. So it invokes the method named "AddUser" of the manager named "UserManager" (of the module named "UserManager") where that talk to the DAL (Data Access Layer) via junior manager class named "HandlerData" to store data in the database. Once this process is completed the "ManagerOperation" evaluate the status of the operation. Then, depending on the status, it will invoke the main manager of the notification module to send a welcome email to the newly registered user.
The analysis of this system shows that generic DMS system can be easily mapped to a real world organization, by continuing this way you may build the whole DMS to form a organization that manage documents. This approach will produce a consistent system that has distributed its functionality across several modules, allowing easy maintenance. This method can be used to design any system, including a web site, web service, other types of services (windows service etc), any form based application (windows form etc) or library, where as if you follow any other architecture, design/ coding pattern you may advice to use a one method for web site and another to form based application and may be some thing totally new when it comes to any other types.
Object Orient Concepts
The designers are welcome to use any object oriented concept as and when they are appropriate. Please refer to online resources.
Design Patterns
Please refer to online resource to find details about design patterns.
Modular Based Development and Reuse of Modules
The modules of the system have to be treated as divisions of an organization. This also means that each division has to have one or many managers, depending on the complexity of the module. The divisions also can be broken into several sub modules, where each module has to be treated as sub divisions of the division. It is recommended to have separate sub modules inside each module to handle leaf level operations such as accessing a database, accessing a file server, or any other leaf level operation. The modules can be designed using the expertise you have, on various design patterns and object oriented programming concepts.
Properly design modules can be reused in other systems too. The most famous modules that can be used across systems are Logging, Notifications, Exception and File Directory IO. As you start with this approach you will find it harder to reuse the modules (This will happen until you correctly gauge the depth of the module or until you learn to design truly object oriented modules), which you have already developed, but it is recommended to expand the function of the module until they are rich enough, to be used in the next system. This approach will gradually create very powerful, more complete, and functionality rich modules, after they are being reused in couple of projects. Here, other than delivering a smart product to your customer, you can open up a new market, if you can develop extensible framework for all commonly use modules.
Quickly Develop the Data Access Layer (DAL)
It is very important to separate the data access layer. The Data Access Layer (DAL) consists of classes that directly operate with database. This separation will not only better modularize the system but also helps developers to edit data access layer (which cause lots of changes at the early stage of the project) without hurting any other part of the system. For more information about this section, you may refer to following two articles.
Group same types of operation in to one class
When you identified similar types of functions or operation group them together. In above example I have created three separate classes for Add, Edit, Delete operations (refers to Figure 4). This way you can keep consistence coding across the methods of the class, since all the methods of the class are doing similar types of operation. This way you can handle exception/ logging etc the same way for all the functions of the class.
Keep Front Layer Free from any Application code
User Interface or the Front Interface of your application has to be kept free from application logic related coding. The whole idea behind this approach is to change the front layer (User Interface) without hurting other part of the system.
Figure 5 - Explain how the front end can be easily replaced by different types of interfaces
E.g.:- As an example think of a web site you developed using .NET/ ASPX pages. In that case front interface of the application is the set of ASPX pages and their code behind files. If you have followed the concept above as you were implementing the system probably you should have set of ASPX pages that are free from application logic and also another set of modules which contains the core application logic. This simple breaking allows you to introduce a web service interface (ASMX file) to distribute the system across two machines to gain performance. This can be done just by replacing the ASPX files with a web service interface. This will separate the heart of the application to a separate application server and ASPX files to a web server as the following diagram explains.
Visualize the System
This allows designers to visually model the system to capture the structure and behavior of architectures and components. Visual abstractions help you understands the bigger view, while opening hidden areas of the system. This include identifying component of the system correctly; understanding how each component of the system fit together; understanding how each component communicate with each other; and finally make each component design consistently. In my practice I draw an activity diagram and a System Architecture Diagram (as drawn here) if the project is very complex and only the second when the system is not very complex. So I encourage you to visually display the system before starting the design.
Naming Convention
- Use lengthy meaningful/ readable names when naming variables, methods, classes, modules and any other.
- Follow the naming convention of the technology owner, if your application is developed using Microsoft .NET then follows the Microsoft standard and if it is any thing else then follow their naming convention, by doing this you will have the luxury of directly using their sample code in your application with out going through that naming convention adjustment process.
Think Fresh and Approach like a Newcomer
Today the technology changes at a rapid rate, allowing new things to evolve every day. When we designing system, it is very important to know what we have around us, today. This let you design the system with most up-to-date technologies. But by going to do that, never let famous standard control the system designing effort. I have heard some people say that "we design our system according to X model, so it is not recommended to do that" never ever make bottlenecks inside your system just because you specifically followed some famous standard or model or just because everybody else is doing it that way. There are no magical formulas that suite for everything, so simple theory is if it does not suite you, change it or adjust it, without making faulty decisions. Think what is needed? Identify, what suite you most? Then take your decisions and let what is available or what that designing concept/ article says stand aside. Concepts are there to help you and better guide you but not to control you. Try to be creative, but by doing so, do not reinvent the wheel. Good luck for your fresh approach..
Join the conversation! Your thoughts help the community grow.