Software Engineering Matters

To have a successful domain model, we need to separate the different concerns from the model. A domain model is easily cluttered with code from the UI, database and other supporting code. When this clutter happens, it becomes harder to sieve through the code and determine what's the actual domain. The current trend to mitigate this problem is by separating the system into different layers and minimize the interactions between those layers.

This chapter presents a general layered architecture that comprises:

• User interface layer
• Application Layer
• Domain Layer
• Infrastructure layer

The user interface does not communicate directly with the domain layer. Instead it does so through the application layer. This separation is based on the Model-View-Controller compound pattern and makes use of the heuristic that the view (the user interface layer) is more likely to change compared to the model. Therefore, instead of putting code that talks to the model directly in the view, we have a middle-man: the controller (application layer). The infrastructure layer provides the technologies for persisting the domain objects and presentation.

Two patterns from the Patterns of Enterprise Application Architecture (P of EAA) were briefly mentioned:

• Unit of Work
• Transaction Script

I found it useful to skim through those patterns from the P of EAA book while reading this chapter.

This chapter also gives some warning about architectural frameworks that might seem to be useful but actually stymie the development of a domain model because the framework demands that files fit certain constraints. This is especially true if you are following the framework blindly. Steve McConnell talks about this in Code Complete 2 when he mentions programming into a language instead of programming in a language. McConell was using Visual Basic to program the UI and Visual Basic makes it really tempting to place all the business code into the forms (.frm files). In fact, Visual Basic advocated keeping everything in the form file and did not make it easy to delegate functions from the .frm file to the associated .bas file. But that did not stop McConnell. He came up with his own convention to minimize the business logic in the .frm files. The moral of the story: do not just program in the framework, program into it.

The "Smart UI" anti-pattern is a very common pattern for novice UI designers. When using a visual editor that allows drag and drop of widgets and auto-generation of code, most novices just dump all the code into the auto-generated placeholders. For instance, if you are using Visual Studio's Form editor, it is easy to just double click on the button and be transported to the callback code that is to be executed when that button is clicked. Without analyzing further, a user might be tempted to put all the code inside that callback handler instead of delegating the function to some controller.

Often, auto-generated code leads to code that is hard to maintain and hard to understand. Auto-generated code does not try to be "smart"; it tries to be safe. When creating new functions, the safest thing would be to call them function1, function2 and so on. This helps eliminate the chances of a name clash. But these names make no sense to a developer.

What the "Smart UI" anti-pattern is trying to show, is that you can still have a cluttered code base if you are not careful with the tools you use. A visual editor is suppose to make it easy to generate the user interface, but it is up to the developer to be skeptical about how to organize the code. Do not just blindly follow the suggested auto-generated placeholders.

In conclusion, this chapter focuses on the need for separation of concerns between the different layers. Leaving the model clean from all extraneous code helps developers focus their effort on creating a richer model instead of wasting time trying to figure out what actually belongs in the model and what does not.

This chapter emphasizes the importance of the model and the implementation. It is useless to use a different model when designing and a different one when actually implementing the system. Not only does this mean that whenever something change you are going to have to modify two models (a prime candidate for introducing inconsistencies and bugs) but you would also have deal with the mapping between the two models. It is always easier to deal with just one model. And from the previous chapter, that one model will help promote the use of an ubiquitous language.

The process of binding the model and its implementation is called model-driven design. And together with the ubiquitous language, forms the basis for domain-driven design. Model-driven design is more easily implemented in a language that supports modeling. Object-oriented languages do this well and thus is more suited for this task. Procedural languages hide the model by operating in terms of passing data around between functions.

A good article always gives concrete examples. And this book has done exactly that in the example for the PCB board. I have had some experience using CAD tools (HDL Designer) for designing microprocessors in one of my computer engineering class so the terms he used are similar to the ones that we use when designing. Instead of nets we called them signals and in addition to buses, we also had bundles which could contain signals and buses. The CAD tool was helpful in letting us connect nets and buses using the mouse, but that was just a visual representation of it. Underneath all that GUI, the design was still embodied in VHDL (or Verilog) which is stored as a text file. And sometimes, when using buses, you have to manually edit the names of the individual nets in a tabular manner.

The mechanistic manner would surely work since everything was basically in VHDL which is very editable by text manipulating scripts. But the model-driven design manner was much cleaner to a programmer. The ability to unit test each the methods were also an added advantage.

One thing that was probably enforced but not mentioned explicitly in the text was that each net and bus name were unique. You cannot have buses with the same name in the same layout because that is not legal syntax and would only confuse the simulator. Nets and buses with the same names are going to be treated the same.

The example of Internet Explorer's favorites is a good example of the importance of a model that matches the user's mindset. A matching model has a lower learning curve for the user and lets the user make use of the features of the application easily.

In conclusion, this chapter shows the importance of having a model that is tied closely to the implementation. Without this tight relationship, the implementations and the underlying abstractions become harder to reconcile as the project requirements evolve leaving the system in a state of code rot.

If you have ever played the game where a group of people sit in a row and you try to to pass a message verbally from the first person to the last person, you know how fun this activity can be. Not only is the sentence deformed in every way imaginable, but the original meaning is usually completely lost.

This chapter emphasizes the importance of communication. All major software projects involved more than one developer, so it is important that every developer communicates clearly to one another. One way to do this is to use well-established terminologies. A queue to a data programmer might mean a data structure but to a non-programmer, it can validly refer to "a braid of hair worn at the back". So it best to make sure that everyone is on the same wavelength before using a term. As the book claims, a project faces serious problems when its language is fragmented across the team members.

And thus the concept of a ubiquitous language arises. With this ubiquitous language, the same terms can be applied in verbal and written communication. You use the same terms in code and in your documentation and in your daily communications. It's unlikely to have this ubiquitous language from day one of the project. But as your project matures, you will find yourself building the vocabulary of the language and refining it to make it more precise.

One very interesting theory in anthropology is the Sapir Whorf theory. It says that you can only understand the world if you can describe it using some language (not necessarily written since some languages have no written form). So the canonical example is that the Hopi people are not able to come to grasp with the notion of future and past since their language does not support it. They have trouble understanding what "yesterday" or "tomorrow" means. The same thing can be applied for domain modeling. Without a suitably rich language and vocabulary, developers will never be able to see what they are actually missing. Enhancing your language helps enhance your model as well.

There were some terms in the diagrams in the chapter that I did not really understand. For instance, cargold and hazmat. I found out that hazmat referred to HAZardous MATerials but I still have no idea what cargold is. We will probably revisit this diagram soon but it would be good to know what cargold is.

Evans dedicates a section for modeling out loud. After reading that section, it felt like a CRC: Classes, Responsibilities, Collaboration session to me. In a CRC session, you have people talk about the problem and try to create suitable classes as models for the problems. This seems like a good way to come up with some models as well. CRC sessions are highly interactive and everyone gets a chance to say something. We can use the general idea of CRC cards without getting too much into the details.

UML is good for getting ideas across quickly in a diagrammatic manner. However, Evans is quick to realize that a diagram is just a diagram and nothing more. You can definitely represent a lot of things in a diagram but there are certain things that are easier to describe using words. In effective communication, the key is to convey the intent as clearly and quickly as possible. When you have to scrutinize the picture for "hidden" meanings, then you might as well use text. Over-advocating one tool is always a bad idea. All tools have their limitations and knowing those limitations enables you to make judicious use of them.

All in all, by using an ubiquitous language in as many parts of the development as possible, we can come up with a consistent way of describing the system that makes sense to the stakeholders. As the book says: "[O]ne model should underlie implementatoi, design and team communication. Having separate models for these separate purposes poses a hazard."

The class has finished the SAIP book and is now moving over to the a new book: Domain-Driven Design by Eric Evans. The reviews have been good so far and I expect this book not only to be a good read but also to be educational as well without all the dry formalisms of a textbook. All posts on readings from this book will be prepended with the acronym DDD.

While reading the introduction to this book, I was curious as to how much the model referred to in this book is similar to the metaphor in XP. I was not able to find anything in the index about XP or metaphor so I assume that this book does not dwell on that issue. However, this issue of a model and metaphor is pretty interesting (to me at least).

In Extreme Programming Explained, Kent Beck defines the system metaphor as:

A story that everyone - customers, programmers, and managers - can tell about how the system works.

A model for the domain seems to be fulfilling some of that task. With a suitable domain model, one can more easily implement and design the system to correspond to the model. The model itself also serves as the basis for communicating amongst the developers when they refer to parts of the system. Instead of calling the method funcA, they will call it applyFilter for instance. Moreover, the model is an abstraction of the most important features of the system; it allows one to describe the internal workings of the system in terms of these abstractions. With these abstractions, the developers can communicate better with the people who are using the software. You do not talk to Photoshop users about the convolutions involved mathematically to change the image; you talk to them in terms of filters and masks.

Unfortunately, since less than 2 pages were allocated to discuss the system metaphor in XP, most people do not actually know the real meaning of the metaphor and it is hard to compare it with the concept of a domain model.

Anyway, Chapter 1: Crunching knowledge presents a process of effective modeling. It's called knowledge crunching because like number crunching, you start of with globules of unrelated data and try to synthesize something. And the best knowledge crunchers do not work alone; they talk to domain experts. Want to create software that deals with gene structures? Talk to a cellular biologist. The best knowledge crunchers also see "real" pervasive links (not fragile links) between the clumps of knowledge. They can formulate new relationships and remove obsolete ones. Their knowledge of the system grows as they become more experienced.

Evans has outlined the methods for effective modeling:

1. Binding the model and the implementation
   While abstractions are nice for communicating and getting the initial idea across, building concrete instances are also important. Building something forces you to think about the parts that are still missing from your domain model.
2. Cultivating a language based on the model
3. Developing knowledge-rich model
4. Distilling the model
   Do not be afraid to throw something out or formulate a new model if the old models do not scale well. Your previous models serve as guides and pointers for you. Your new one model will build upon those lessons that you have learned.
5. Brainstorming and experimenting
   New paradigms come and go. You need to know when to embrace them.

In the example about shipping books, Evans shows how important it is to capture important domain knowledge and not let it be hidden in the code. In his example, the concept of overbooking was being implemented using a simple if... then clause in the code. While this certainly works, it does not immediately convey the message of overbooking to the code reader. By abstracting that overbooking if... then clause into a different class (a strategy), it makes it clearer what the if... then clause actually serves. Clear code is not only easy to read and understand, but it also conveys important underlying features across clearly.

I am reading Martin Fowler's Patterns of Enterprise Application Architecture. I have been flipping between patterns because the project that I am documenting, Typo, makes used of most of them. Actually Typo is built on top of the Rails framework, so it is more correct to say that Rails makes use of them and Typo builds on them.

To get a better feel for Rails, I have been perusing Agile Web Development with Rails. It's a nice book. More tutorial-like in contents than a real book on the architecture of Rails. Between reading that book and the source code, I think I have a pretty good idea how Rails is structured. The source code is not as easy to read as I had hoped. Some files are over 2000 lines long because they were comments that were lengthy examples of how some concept works. Also, there is heavy use of meta-programming around so it is easy to miss something since you cannot just do a search for that term that you expect.

Anyway, there are three patterns in Fowler's book that deal with how to handle inheritance within databases. I read them all and was really astonish that I have a sense of deja vu for these patterns. Seems like we had already covered these patterns in my database course albeit with different names for them.

So here is how those patterns correspond to the picture above:

• Single Table Inheritance - The Null Value Approach
  Create a super-node kind of table that can accommodate all children of the current class. Fields that are not used will be left null.
• Concrete Table Inheritance - The OO approach
  Create a table for each concrete class. Fields common to classes will be duplicated across the tables.
• Class Table Inheritance - The E/R approach
  Class Table Inheritance will try to eliminate duplication between elements by specifying a parent class and then using foreign keys to link them together

The Class Table Inheritance scheme is potentially going to be able to save space but it is hard to actually retrieve stuff since you have to do multiple joins on the tables.

The OO approach is going to be really explosive! For every possible combination of child-parent relationship, you are going to have a table for it. Each kind of relationship will thus only fit in table by default. I can imagine that if you ever need to change you schema, you will probably be in for a lot of trouble.

And now, as if that sense of deja vu was not enough, Rails also uses one of the methods above fpr handling complex relationship between tables. Rails has built-in functions for associations and inheritance. It uses the simplest of the three methods above: Single Table Inheritance.

This is the last chapter of the book. Somehow, I feel that most of the contents here should have been placed either in the prologue or in the preceding chapters. For example, the evolution of software from subroutines, modules, objects and frameworks should have been mentioned earlier to give the reader an idea of how software has evolved.

This chapter mentions the shortcomings of the some of the techniques introduced in the previous chapters and I am glad that the authors acknowledge those shortcomings. For instance, in Chapter 5: Achieving qualities, readers were presented with a very unclear description on how to proceed once the development team has a plethora of tactics that they want to employ. Chapter 5 suggested that from those tactics, one could easily pick a right type of architectural style to apply. I am doubtful that it is that simple. If it were that simple, then there would not be a need for all this discussion in the first place.

The authors also discuss the existence of documentation within a tool environment. Again, I feel that this really should have been mentioned in the chapter about documenting the architecture. I agree that the authors might be wary about mentioning tools that they have neither used nor analyzed closely. But a brief mention of those tools might actually be helpful. After all, those tools are used in practice so they must be beneficial in some way. And more readers will be familiar with it compared to the Dali tool that was discussed extensively.

I appreciate the book devoting a sidebar to software architecture in education. However, regardless of how many course you take in the university, software architecture cannot really be taught. You can be exposed to a series of techniques like what was done in the book but then again, you only really learn those techniques if you apply them in real life scenarios. And that's why I am a fan of classes that have lots of projects. There are many things that have been left out from this book (and judiciously so) for the sake of keeping it simple. As the title of the book hints, software architecture is something that exists right now as a form of practice, and there has not been any significant advances in trying to codify the theory of it.

This book is an attempt to describe software architecture. It's not easy task, and while the book could be improved, it does a pretty good job at it.

There was some discussion in class today about why Java Applets did not take off as much as Sun Microsystems had hoped. While I really do not know the answers, here are my thoughts.

• No broadband. 10 years ago broadband Internet was not as pervasive as it is today. So there wasn't really much need for highly interactive web pages. Imagine trying to download a few megabytes worth of data over a 28.8 kbps modem. Now imagine if every web page out there embedded an animating Java Applet.
• Heavy JVM. Even today, most people don't really like web pages embedded with Java applets. Windows does not really ship with the latest version of the Java Virtual Machine. Our university uses a classroom system called Compass based on Java. Almost nobody likes it. Enough said.
• Browser compatibility. Not every browser in the 1990s supported Java Applets. And most people did not bother installing a new one.
• Java was not a buzzword (yet!). Most consumers did not know or care about Java. And pointy-haired bosses did not know enough to jump on the bandwagon.
• No web designer knows the language. This is very important. The first few Java Applets were really hideous compared to what was out there. I suspect that it was because web designers did not get on board yet. To most web designers. Java is a heavy language to learn just to make animated applets. You need to know about threads, classes, methods and objects. Even today, most web designers prefer the simplicity of HTML, CSS and some javascript.
• It just looks weird. The UI from the AWT looks weird on the browser. And that makes the entire web site feel clunky.

I am interested in hearing what other people think. Java Applets were something introduced in the wrong time period. Had it been introduced later when it was more mature, it might have appealed to more people. I guess it is a compromise between how early to enter the market and when to wait for maturity of your technology.

This is one of the more interesting chapters in the book. I am glad that the authors acknowledge that most start-ups do not build everything from scratch; it handicaps their market penetration.

How to be a programmer:

    
   intellisense 
        ||
        \/
       code >>> compile >>>>> run >>>> success ;-)
        /\         ||          ||         
        ^^         \/          \/

        ^^      errors       errors 
        ^^          \\       //
        ^^           \\     //

        ^^             google
        ^^              ||
        \\              \/
         \<<<<<<<  cut N paste

Figure 1: What actually happens during development. Or why no one codes alone anymore.

Using off-the-shelf components sounds really convenient. After all, why rewrite what you can get for free from open source projects or buy for a small fee from other companies? There are various very real reasons why you do not want to use off-the-shelf components all the time as the book points out.

Here are some real risks that have to be addressed before committing to any off-the-shelf component:

• Maintenance: Whose job is it to maintain that component that you have just borrowed? Will you have to hire someone to maintain that component full-time? What if the company that owns that component goes bankrupt?
• Cost: Will there hidden fees in the long run? For instance, annual license fees? Upgrade fees?
• Quality: How good is that component? Is it safe, reliable and secure? How would you find out?
• Compatibility: Does the component fit with the rest of your system? Or would you have to bend over backwards just to accommodate it?
• Long term goals: Can I scale using the current component? In other words, is there a component that I can upgrade to when I need more features?

As the chapter proposes, the best way to mitigate some of those risk is to isolate the system from directly accessing the off-the-self component. Never access the component directly from code. Instead, use a proxy and provide more general APIs for accessing that component. That way, should you need to change that component (maybe to a different vendor) you will minimize the changes to your system.

All in all, though there are clear risks involved with using COTS product, it is unlikely that every company will go build their software from scratch. Even if they intend to establish their own product line architecture, they will seldom do it from scratch. It just takes too long. The time gained can be used to do more testing, or add new features, or come up with the next version of the software. In other words, the time saved is better spend focusing on other parts of the system.

I think this form of reuse is most apparent in the web. There is a lot to be gained from sharing your data with people. For instance, by making part your data accessible to other web companies, you can charge them a small fee and also increase traffic to your site. One example would be Flickr.com, a photo sharing website. By giving stable APIs that enable people to query their database (not private data) a lot of website can integrate Flickr's services. Not only does this save the website from having to implement their own photo sharing service (who knows how long that will take. And who would actually use it?) but it also helps Flickr get more publicity. So it could be a win-win situation for both parties.

Chapters 16 and 17 are case studies on systems that utilize Java and its technologies.

Chapter 16 was a description of the architecture of J2EE and how it achieved the quality attributes of its stakeholders for distributed computing. However, it was too detailed for me to appreciate since I have not used J2EE or EJB before. It did point out the robustness of J2EE and how Sun Microsystems invested a lot of time and money to make it handle most of the situations. One interesting aspect of this architecture was the design to support the older standard for distributed computing using CORBA. I found this to be a wise step on the part of Sun Microsystems; no one is likely to switch over everything to just use your company's new technology. It has to be a gradual shift that helps build confidence in the new product. It was because of neglecting this fact that the Air Traffic Control system we saw back in Chapter 6 did not take off.

Chapter 17 focused on using J2EE for wearable computing devices. These computing devices had unique requirements that had to be catered to. For instance, some of them had small screens, limited input abilities, low power consumption and limited processing power. Sometimes, these devices were so varied that entirely new input/output devices had to be designed from the ground up. While the company, Inmedius Corporation, decided to use J2EE, I am rather surprised that they did not turn to J2ME, the Java standard for mobile devices. J2ME seemed like a better choice for micro devices that had limited processing power since it offers a simpler API and smaller memory footprint.

Though I am no in favor of this, I suspect that an entire CS undergraduate degree can be taught entirely using Java and its technologies. Java has grown into a behemoth with components for distributed computing (J2EE and EJB), embedded systems (J2ME), web architecture (JSP) and it even has enough of the best practices of software engineering (object oriented design, aspect oriented design) thrown in. If this topic is your cup of tea, you might want to read The Perils of Java Schools.

As the books mentions, reuse is great for any business out there. It is much too costly to invest in a "one-trick-pony" kind of architecture that only fits the profile for one kind of product or serve. Reuse justifies the return on investment for a large project that can easily take years to complete and requires thousands of employees working on it. Reuse empowers commercial companies to roll out a new product line quickly by sharing similarities between previous products. And more importantly, reuse reduces the need to reinvent the wheel.

So it seems that in this chapter, we have introduced the notion of reusability as a desired quality attribute. The SEI defined reusability as: "the degree to which a software module or other work product can be used in more than one computing program or software system". If reusability is a desired quality attribute, it must also be incorporated as part of the ATAM when evaluating the software architecture. However this is always easy. In fact planning for reusability is hard.

Making things reusable takes a bit of prediction (which comes not only from experience but from observing the trend in the market and the technologies that might emerge in the future). I was reading the Pentium Chronicles this summer and the chief architect, Bob Colwell, described the creation of the P6 architecture that would be the product line architecture for the Pentium 3, Xeon and even the Pentium 4. The product line architecture for a microprocessor has to be almost perfect to accommodate any future changes. It is prohibitively expensive to change the fabrication equipment so any chief architect worth his salt must design an architecture that can last a few years down the road. Not only that, this same architecture must work with several generations of chips. To achieve that, Colwell had to be smart on where he made his assumptions when he designed the architecture. Realizing that fabrication technology would continue to improve, Colwell made it a point that his architecture would be able to support many more transistors in the future and that would improve the performance of the chip. My point here: if you want reusability, you have to plan (and make predictions) for it early on.

If you are designing the architecture for the first time, there is very little you can do to help you deal with reuse. Knowing the future plans for the company might help so that you can anticipate on the components that are likely to change. So you can make those components more modular. But how would you know if people are actually going to be using your architecture? The best way I can think of is to "eat your own dog food". Use your own architecture as you develop it. From there you can find the missing abstractions that need to be in for reusability. Or you can release your framework as an open source project and get other people to comment and contribute as well. This is exactly how Ruby on Rails was created. It was initially developed as an internal framework for 37signals who later released it for public use.

Also, when do you actually know if you are going to need it again? There is a principle in XP called YAGNI: You ain't gonna need it. So there is a conflicting case for making things reusable and also for keeping the design simple. If you have read my previous post on reuse, you would have also read the article I linked to. In it, the author mentions how you should just code as normal and get things working first. Only then do you try to eliminate code duplication by refactoring. The more times you have to eliminate that duplication, the more likely that the code you are dealing with is a candidate for a framework or library so that you can make it more general for reuse in the future.

But another problem with reuse is getting people to actually use the same architecture. There are many architecture out there that are overly complex that most people do not need all the features. For instance, many developers are leaving the world of J2EE and EJB for more lightweight architectures that can get the job done. J2EE can definitely accomplish everything you need for your enterprise but because it is so general, there are so many components that you need to know first that it becomes a greater hurdle. What I am trying to say is that there is a limit to what people will be using your framework for. There is a threshold of "enough, that works for me". Going beyond that leads to complexity that defeats the purpose of reuse.

It is very hard to make an architecture programming language agnostic and still be deployable in a short amount of time. But if you realize early that you need your architecture to work with different programming languages, then you need to design another layer for it. This is exactly what the .NET platform has done. All programs compile to CLR so you can code in C#, C++, Ruby or Python and still be able to run that program using the .NET framework.

So, reuse has not pervaded the society of computer science because it is just hard to make it work right. There are various other reasons for this, but sometimes one has to ask: is reuse really that important? Not every system out there needs to be reusable. And not every part of a system needs to be reusable. By cutting down on reusability, we can actually create less complex systems.

---

The subtitle for this chapter reads: "re-using architectural assets". It suddenly dawned to be that in this class, we have looked at the terms architecture, design and implementation. However we have not looked at frameworks. I have not come across a concrete definition for frameworks but from the diagram on p. 478 of the book it would appear that frameworks are a form of abstraction that falls below architecture.

Diff b/w Framework and Architecture:

"Architecture: an idea about how to structure your app, possibly shared through natural language documentation and structured methods, like UML.

Framework: an implementation of an architecture that someone can use as the basis for a working application. "

So loosely speaking, I have seen various frameworks being applied. In fact, in my Software Engineering II class we read a paper on that entitled: Components, Frameworks, Patterns that discusses frameworks are reuse techniques for object-oriented systems. However, I have not really seen the idea of a software architecture actually being reused.