Software Engineering Matters

In my previous post I mentioned how there aren't any reliable refactoring tools for AOP yet. Seems like this problem is slightly more involved than the refactoring that most of us are used to. I have been reading a bunch of papers on this subject (mostly the top hits from the search for Aspect-Oriented Refactoring in Google Scholar.). The paper that best summarizes the problems and challenges is Aspect-Oriented Refactoring: Classification and Challenges by Jan Hannemann.

There are three main issues with refactoring aspects:

• Aspect aware refactoring tools
  If you are using AOP with your OO program, it is no longer safe to just call a normal refactoring like rename method. Calling rename method might actually not respect the aspects that are tied to that particular join point in the code. Thus, you affect the aspects that are advising the code. In all but the most trivial cases, this can have undesirable effects.
• Aspect-oriented programming refactoring
  AOP itself is a rich language. As with any language, there is a potential for refactoring to make the code easier to read and maintain.
• OO to AOP refactorings
  Most programs are going to be a combination of OO and AOP programming styles. To get the best of AOP, we have to gradually address the points of crosscutting concerns in our OO program and then refactor them to use aspects. There is research being done on aspect mining - automatically detecting crosscutting concerns and presenting the choice for refactoring those concerns into aspects.

A fourth issue, I would like to add is being able to refactor from aspects to the normal OO constructs. Sometimes this might be desirable. For instance, an easier and more efficient refactoring to the code might be possible by first pushing the aspects back into normal code, perform some organization on that source code and then refactor into a new aspect.

Another paper - Refactoring of Aspect-Oriented Software - claims to have created a prototype to addresses the first bullet point above. However, I am not able to get a hold of their implementation which is a plug-in for Eclipse. The paper has identified the steps for each refactoring to be performed by hand (and also to be verified so that it behaves correctly). It is a good start and it is written along the lines of the first thesis on refactoring by William Opdyke.

Aspect-Oriented programming brings with it the interesting issue that it is no longer sufficient to treat the original source code as the only thing that matters. As programs evolve, they will be written with many different languages that are intertwined together. Each language will be complicated enough that it can be refactored to be made better but refactoring would also cause repercussions in other parts of the program. Solving (or at least getting a better understanding of the problem) from AOP refactoring will be useful when we have to deal with the same issue in domain-specific languages.

This post is based on the paper Design Pattern Implementation in Java and AspectJ by Jan Hannemann and Gregor Kiczales. There is also a video related to AOP and it might be beneficial to watch.

I have been reading quite a few papers on Aspect-Oriented Programming recently. Mostly I believe that there is a potential use for aspects. For instance, in the previous post I mentioned how it could be used to extend otherwise closed classes or libraries.

This paper is a discussion of implementing the original 23 GOF design patterns in AspectJ. Undeniably, anyone should already know that it is possible to implement the GOF design patterns in any programming language. The more important issue is how easy it is to implement in AspectJ compared to plain old Java? I read the paper and did not feel that it was comparably simpler. My criteria for simplicity is based on how easy it is to understand the code and how many lines of code is needed.

The authors argue that one of the drawback of using design patterns in Java is how mangled the code becomes. Furthermore, they argue that additional code needed to implement the design pattern interferes with the domain model underneath. This gets progressively worse as patterns are composed together. They claim that it becomes harder to reason about the code and the interactions because of this mangled structure. And thus, the propose a solution in AspectJ.

True, with AspectJ the authors managed to achieve their goal of modularity and reuse. But modularity and reuse do not necessarily make the code simpler to reason about. In fact, artificially imposed modularity makes the code very hard to read and understand. Design patterns avoid this form of modularity. Instead, they present a theme for understanding the code. For instance, in the Observer pattern the most important idea is understanding the roles played by the Subject and the Observer. Once you have identified them, reasoning becomes simpler.

In the video (25th minute), Kiczales tries to convince people how aspects can in fact make it easier to reason about the programs once we think at a higher level: what we want to accomplish rather than how we want to do that with objects and other OO design patterns. There were the usual question of whether aspects will create spaghetti code since you can weave things into any class. Whether his argument is convincing or not, it is for you to decide.

I downloaded the source code for the examples in this paper and loaded it inside Eclipse with the AspectJ Developement Tools installed. While it does provide some useful features for notifying the programmer which aspects affect the current line of code it may not really be intuitive at first (maybe there is no known paradigm for tools in AOP yet). Refactoring support for AspectJ has not been implemented so it hard to actually make changes. This makes it hard for developers to use AspectJ in a piecemeal growth manner.

One issue that the authors failed to address properly is how to compose several patterns together. Composing aspects together is a tricky process because undesirable interactions may occur based on the order of weaving and the parameters that are used to identify the pointcuts. Since patterns seldom exist in isolation, the authors should have addressed this issue more carefully in their paper. The source code for the examples do not demonstrate patterns interacting together.

My conclusion is that I am more interested in finding out any new patterns that would emerge with the use of AOP. The authors themselves acknowledge that the implementation for some of original 23 design patterns were transparent in more dynamic languages such as Lisp or Scheme. The authors did cite some papers and research that was in progress to address those issues and I will probably be taking a look at them.

Aspect-Oriented Programming (AOP) is now a popular buzz word. Well since it has survived for nearly 10 years, I figured that it was worth taking a look at it.

So far I have read two papers on AOP: A Disciplined Approach to Aspect Composition and a draft of a paper to be submitted. I will refrain from detailing the draft in this post. Suffice to say it introduces 4 broad patterns that give a better perspective on what AOP can do and how to do it.

In general, there is nothing in AOP that OO cannot do. Well, in general, there is nothing in OO that you cannot do with procedural code or functional programming. So, while AOP is nothing that great, it could be something that could be useful for certain programs. If you have heard a bit about AOP you might remember the examples about monitoring and logging. Rest assured that AOP is a lot more than just plain monitoring and logging.

For instance, AOP is an interesting way to do program transformation without actually modifying the source code. For instance, you cannot really modify the String class in Java. For instance, you might want to add the method camelCase or snakeCase. Trust me these methods can be handy if you are trying to do some metaprogramming. But since class String is close to modification, you cannot add those methods directly. Instead what you would usually do is subclass the String class or add some String utility class. With AOP you could use introductions to simulate adding that method to the class String. Everywhere you do "some string".camelCase() it actually modifies the code to call a camelCase method somewhere else (not in class String).

Programmers in Ruby and Smalltalk will not consider the example above relevant since they are free to change the internals of the classes because its open for modifications. However, consider the potential AOP has for proprietary systems. You will never be able to get the source code for those systems to hack on. But at least now with AOP you can do dynamic woven and get the behavior that you want do. This seems like some ersatz compared to actually modifying the code but at least it provides itself as an alternative.

AOP still lacks the tools that make it useful though. For instance for Java and Smalltalk we are used to IDEs that implement "Call hierarchy", "Type hierarchy", "Show all implementors", etc. As far as I know, there are no such tools for AOP yet. Maybe AOP is not mature enough that we know what kinds of tools that we need. For instance, it is not likely to be hard to come up with some modeling language like UML to describe AOP.

At the moment, one of the most frustrating things about AOP is the inability to see how your code works. Aspects can be woven in statically or at run-time. And depending on run-time conditions, different aspects might be woven in potentially overwriting previous behavior.

An interesting approach to AOP might be design your program in the normal OO way first. And then refactor to aspects. For instance, design your program with all the superfluous monitoring and logging baked in first. And then as you begin to see the underlying domain to your program, you can refactor those that do not really belong into aspects. Therefore, you are not trying to fit your program into the AOP mental model but rather you are trying to get to your domain model by moving things out into aspects. In a nutshell, AOP complements OO design.

AOP is not a panacea for all programming problems. However, it might be useful for making certain mundane programming tasks simpler. Personally for me AOP borders on the capabilities of metaprogramming. AOP seems to be an external fix for languages with weaker reflection and metaprogramming capabilities. Instead of baking a heavyweight reflection system into the language, you can add it on with an aspect compiler. So people who do not use AOP will not have to pay for the cost of having that ability built-in.

InfoQ: Domain Driven Design Quickly:

"The most complicated aspect of large software projects is not the implementation, it is the real world domain that the software serves. Domain Driven Design is a vision and approach for dealing with highly complex domains that is based on making the domain itself the main focus of the project, and maintaining a software model that reflects a deep understanding of the domain"

In this class, we read the excellent book Domain-Driven Design by Eric Evans. The book itself was interesting and useful, but it was too thick and heavy to bring around. And as far as I know, there isn't a legal .pdf copy of it. Well thankfully, InfoQ has been kind enough to condense the tome into a 106 page document in .pdf. This will certainly be useful. I have not read through the entire 106 pages but from the Table of Contents it seems to cover a good chunk of the book. This .pdf will definitely serve as a convenient review and summary of the book for me.

Some caveats. The text layout is a bit funky for the book. It definitely has not gone through a good typesetting process (it looks like it was written in Word with left-right text justification turned on) but it is readable. Some of the images are also blurry and that shows how much effort was put into this .pdf. But it is free and it a good effort to getting more people interested in Domain-Driven Design.

[Python-3000] Refactoring tool available (work in progress):

"In the sandbox I've been working on a refactoring tool, which could form the basis for a Python 2.x -> 3.0 conversion tool. I'd like to invite folks here to give it a try and give me a hand. It certainly needs more work, but I think that the basic infrastructure is sound. Check out sandbox/2to3/: http://svn.python.org/view/sandbox/trunk/2to3/."

(Via Guido Announces Refactoring Tool.)

I am not a regular Python user but this tool seems pretty interesting. I have not seen any other tool that could potentially refactor (or transform) a language into a new version of its own syntax yet. Most just involve changing the structure of code while preserving the behavior. However the syntax is limited to the current syntax. Using the refactoring tool to help migrate old syntax to new is while preserving behavior is certainly a good idea.

This post is based on the paper ArchJava: Connecting Software Architecture to Implementation by Jonathan Aldrich, Craig Chambers, and David Notkin [ACN]. We heard of ArchJava in my previous post on Software Architectural Primitives.

This paper stresses the importance of tying the software architecture to its implementation. Consistency between the architecture and implementation is important for analysis and reasoning. The current state of software development usually decouples the software architecture from its implementation. [ACN] argue that this is undesirable since it becomes harder to keep the architecture and implementation in sync. To emphasize their point, they introduce the term communication integrity:

A system has communication integrity if implementation components only communicate directly with the components they are connected to in the architecture.

A system with communication integrity, the authors argue, enable architectural reasoning about its implementation. Thus, they propose ArchJava, an extension to the Java programming language as a way to maintain communication integrity. ArchJava adds new language construct to support components, connections and ports.

What really surprise me is the resemblance this extension has with VHDL. VHDL is a hardware description language that uses ports, components and connections to describe electrical components in code. VHDL is meant to mimic the electrical components closely. It was hard to program in VHDL because you need to keep track of the in and out ports that each component provides. However, because of those ports, it was actually easier to ensure that our components had the right specifications.

In VHDL IDEs. there are usually tools that help you make the connection between components. You can drag a line to show that one port was connected to the other; it would then generate the code that actually did that connection for you. Apparently there is no such tool for ArchJava and the developer has to manually connect the components using code.

One of the problems I have with ArchJava is the fact that if you decide to use it somewhere in your program, you would need to use it everywhere. It is not possible to just describe one part of the system (maybe the more complicated parts) and leave the simpler parts alone. Therefore, existing programs that want to take advantage of ArchJava have to be rewritten. The paper has an example of rewriting. And while rewriting did not take a lot of time, it is not a trivial task either. This could discourage exiting projects from adapting ArchJava.

I also suspect that ArchJava makes programming more restrictive. For developers using dynamic languages, developing in static-type languages already feels very restrictive. I suspect that it would feel even more restrictive once ArchJava comes into the picture. It would require discipline to follow the those rigid requirements. This might be a good thing but it is probably hard to write a quick prototype using ArchJava.

The main advantage I see with ArchJava is that connections between modules is now explicit. This can be useful for developers when they need to do a quick review if they are actually following the specification.

In my opinion, ArchJava is not meant for everyone. If your development project requires stringent adherence to specifications, then ArchJava might be a solution. But for most projects, it is probably too restrictive.

For the class project in CS527, I am documenting the Typo content management system. Typo is the back-end that powers this weblog.

We are supposed to follow the documentation style from the Documenting Software Architecture Views and Beyond book. There is some good advice there but following it blindly will make the documentation too cryptic for even the most enthusiastic readers.

I have tried to follow the different views that the book suggests. The end product will have module view, component-and-connector view and allocation view types. Whenever possible, I have also included the code excerpts from Typo as well as links to external sites that describe the rationale behind certain design decisions. Each view is explained in the text and related views and tied together.

I do not intend to produce something that mimics the example at the back of the book. The example at the back of the book is meant to cater to various stakeholders so there is an obscene number of views. Moreover, each view follows a template that needs to be described as well.

The stakeholders for Typo are mainly Rails developers and web designers. Therefore they are more familiar with some of the terms that we will be using. I have written it in such a way that it is possible to read the entire documentation from cover to cover without it being too boring. After all, what is the use of documentation that no one reads. This documentation also serve as a guide on where to obtain more information on Typo. There is no authoritative source of documentation for Typo at the moment, even though there are various tutorials on the web on how to modify certain extension points for it.

The advantage of an open source project is the availability of the source code. Thus, this documentation should serve as an overview of the relevant modules in the source code. Moreover, Typo is an evolving project so producing a detailed documentation that will only last for one version is not going to be useful. Instead, by documenting the major modules that are unlikely to change, that documentation remains useful even in the next release.

Incidentally, this is the first time I am using LaTeX for writing so there could be some weird artifacts with how the text looks. I describe my process of learning LaTeX on my personal weblog.

An auto-generated draft is available here. It will be updated frequently but I want to get a version out first since Prof. Johnson requested that we post a copy by the end of the week --- Saturday is the end of the week. The sections on Module views and Allocation Views are pretty complete but the rest are probably going to undergo more changes. Right now the figures have explanations in the text but they do not have a key associated with them. The book suggests that each figure be accompanied by a key. I am still deciding if this is really necessary.

Update: The link to the documentation now contains a complete version of the documentation. By complete, I mean that it has all the views that I had planned to address. I have also just read through it and corrected some minor mistakes. Unless I decide to include a new view or receive comments to change certain aspects of the documentation, the final version will be similar to the current one. The first draft is still available from here for comparison.

The final version of the pattern that I wrote for this class is now available. The html version can be found here and the pdf version can be found here.

The intent of the pattern:

Design a framework so that it enforces standard naming conventions for mapping classes to resources or events. A programmer only needs to write the mapping configurations when the naming convention fails.

I admit that the name is not coined by me. Instead it has beeb used actively in the Rails community.

As I mentioned during the writer's workshop session for this pattern, I was more interested in trying to give a definition to the phrase Convention Over Configuration. The term has been used loosely by various developers and I hope my description will offer developers a better understanding of the underlying concepts and the consequences for using Convention over Configuration.

One form of convention that I have seen - but not mentioned in the paper because it might make things too complicated - is the fact that code and resource organization can also be a form convention. For instance Rails expects to find all the ActiveRecord classes under the app/models folder. By having a convention for placing code and other resources, Rails does not need to traverse all the directories when it needs to find those files. Also, it does not need a configuration to tell it where to look for those files.

This is my first attempt at a pattern and it has been an interesting experience. It is very tempting to assume that the reader knows too much or too little. In the former case, I would make the mistake of writing too little and use too many terms. In the latter case, I tend to write too much and the underlying meaning to the pattern might be lost. In this final version, I feel that I have struck a balance between the two and have included sufficient references in case the reader wants to find out more.

The most important part of writing the pattern is actually getting feedback from other people. I was privileged to have the opinions of four members of the class and that really helped me identify the parts of the writing that need to be clarified.

Hopefully somebody finds this pattern and its description useful.

This blog post is primarily based on the paper Worse is Better by Richard Gabriel. I have taken the liberty of reading most of the articles that are linked on that page to get a better feel of this topic. Another article that is worth reading is Worse is Worse by Jim Waldo.

One of the problems (I hear) that most new comers to Lisp have is the fact that the community itself is pretty divided. There is no "one" implementation of Lisp¹ (the standard is supposed to be Common Lisp). And there is no one "right" way to do things. In fact, having more than one way to do things is probably one of the appealing features of Lisp. Nonetheless to a new comer who is learning a functional language for the first time, the arduous task of learning a new programming paradigm and trying to get things to work in the simplest way possible might be too much. And the worse scenario is when new comers are chagrined for mentioning that which-must-not-be-mentioned: Lisp and its hideous parentheses².

But I digress. The main point of this article is not really about Lisp. It's about the Worse is Better approach to software design.

Worse is better - Wikipedia, the free encyclopedia:

"Worse is better, also called the New Jersey style, is the name of a computer software design approach (or design philosophy) in which simplicity of both interface and implementation is more important than any other system attribute (including correctness, consistency, and completeness). [Contrast this to the MIT style of software development]"

So why did Gabriel choose Lisp as an example to his statement? It's probably because Lisp is purported to be a very powerful programming language. It has correctness, consistency and completeness. Every new programming paradigm can be expressed in Lisp: functional programming, procedural programming, object-oriented programming, etc. However, it's interface and implementation leaves much to be desired. With this great power comes a greater responsibility on the programmer to not shoot his own foot.

There has been criticism that Gabriel's essay is not substantiated in any way. His opponents argue that Gabriel's examples with Lisp vs. C and ITS vs. Unix are not real examples that can support his claim. The truth is that, Gabriel himself is not certain of his claim. He has written other essays under the pseudonym Nickieben Bourbaki that rebuts his Worse is Better essays. What Gabriel has done is merely stating the obvious and offering a possible explanation for it. There are a lot of things out there in the world today that are far from perfect and yet they are being used more than their perfect counterparts. It could be because of marketing or price but the truth is that better is not always better.

Personally, I agree that Lisp is really powerful. You can make use of macros and meta-programming and define new domain-specific languages inside Lisp: you have the power to define another programming language within this programming language. Why would you need another programming language if you can do it all in Lisp? However, I am more of a fan of using various languages for different tasks. Want to quickly do some string manipulation? A quickly thrown together script with some regular expressions would get the job done. Need to do some logic programming? Using Prolog and the built-in tools would make it simpler.

That is the philosophy that the Pragmatic Programmers take. They believe in learning a new language each year. After all, you never know what you might discover. This is the power that you have with variety. The little pieces themselves might not be complete but by combining different pieces together you end up with something simple yet powerful. So standalone products might seem less-than-perfect on their own but when combined with other less-than-perfect components, something good might come about.

The Pragmatic Programmers Language of the Year:

"Learn at least one new [programming] language every year. Different languages solve the same problems in different ways. By learning several different approaches, you can help broaden your thinking and avoid getting stuck in a rut."

Worse is Better is not an excuse for creating sub-standard software as Waldo pointed out in Worse is Worse. It's a guideline that when faced with the conflicting decision between simplicity and power, always choose simplicity. Simplicity allows more people to understand it and with more people using the software, the development team will get better feedback. This feedback allows the developer to improve the areas that really need to be improved. Simplicity also opens the software to a wider range of users and developers. Why limit software to the aristocratic few?

Then again, having a provably correct implementation that takes care of all the cases has a certain appeal as well. This is the struggle between being pragmatic and being perfect. But who gets to decide what is worse or better? In what sense is C worse than Lisp? In what sense is Microsoft worse than Linux? After all, one man’s meat is another man’s poison³.

¹ Another interesting read (as is the case with his other posts) is Lisp is Not an Acceptable Lisp by Steve Yegge

² In fact in some communities, any questions on the syntax or justification for a particular feature is viewed as a personal attack on the community. The community then would lynch the defector in cyberspace usually through the means of nasty posts and comments. See Software Development: It's a religion

³ It appears that after a product gets pass the tipping point, regardless of how bad it is, it will survive for many years to come. This scenario will continue regardless of how good a competitor's product is. Finding out the way to break this tipping point will make anyone in marketing very very rich.

The following post is based on the chapter The Architecture of Complexity: Hierarchic Systems from the book The Sciences of Artificial by Herbert Simon.

The chapter excerpt that I was reading from is a scanned from the book. That made it really hard to read the words and it could have slightly affected my evaluation of the chapter which was based on a paper of a similar title. Anyway, I felt that this paper was a bit too slow moving and verbose. I do not enjoy interdisciplinary approaches when explaining things especially if the examples themselves are presented with too much detail. Being a book, it caters to a broader audience so including various examples might be more helpful to the reader. But, sometimes these examples masquerade the intentions of the author.

Nonetheless, the paper does have some interesting points. The gist of it is that complex systems are usually composed hierarchical components. These hierarchical components can stand alone (and indeed the author calls these stable intermediate forms) but they are more useful when combined into systems that interact. For instance, in Computer Science, we might combine the java.swing packages with the java.io packages when designing our program. By combining those hierarchical components, we actually create even more complex systems albeit in a piece-wise manner so that we can understand the evolution of the system.

This decomposition into hierarchical standalone components is a way to tackle the complexity. Once we have a hierarchical structure, it's easier to describe and analyze the underlying complexity. We can take a component and analyze it in isolation first. After that, we could combine components together in different configurations and reason about how they interact. We can continue doing so until we have achieve our objective. Having a way to describe a complex system in simpler terms makes reasoning about the system itself simpler.

In design, we strive to decompose components so that they have high cohesion internally and low coupling externally. Complex systems that exhibit this decomposition can be more easily understood by other developers.

Indeed, most of the architecture or design that we see in software today is based on some form of hierarchy. Inheritance in object-oriented languages is a form of hierarchy. Classes themselves are a form of components that can stand alone or interact with other classes. In conclusion, decomposing a system into hierarchical components is a useful approach to not only manage the evolving complexity but it is also an useful way to describe the system to someone else.

It is also important to realize that hierarchical decomposition is not the only way to handle complexity. It certainly is one of the more well-known methods but there could be other methods yet to be discovered.