CHAPTER 1

Background

Over 40 years ago Parnas (1972) suggested that software be decomposed into modular systems. He recommended modular systems as being more comprehensible and maintainable. OOP has, and is, being used to satisfy some of the objectives as proposed by Parnas. Even so, many software concerns cannot be decomposed into independent units. Kiczales, et al, (1997) points out that some aspects of software appear scattered and tangled throughout the target code and cannot be encapsulated in a single OOP class or classes. These concerns cannot be decomposed from the rest of the system using functional, procedural or object oriented programming languages. Concerns may be tightly coupled to other aspect of the code. When the code is tightly coupled with other concerns it is considered tangled with those concerns. Other concerns appear throughout the target program. They cannot be localized to a particular class. These concerns are considered scattered.

LEGEND: Code A Code C

Code Concern B Code D

Figure 1 – Abstract Pictoral of Code Scattering. If this represents four software concerns coded in a program then Code Concern B is scattered.

. Software aspects that are intertwined with other concerns (are tangled) and cannot be localized to a class (are scattered) are defined to be cross cutting concerns. Thus, cross cutting concerns are both scattered and tangled. Some commonly cited cross cutting concern examples are logging, error detection, security, and as we will show in this proposal, unit testing. Many more cross cutting concerns have been identified.

Concern A

Concern B

b

Figure 2 – Abstract Pictoral of Code tangling.

As has been stated by Parnas, and for a variety of reasons (maintainability cited most often), OOP tries to encapsulate similar concerns in a single class. As an example, when a requirements change is demanded of a particular programmed feature, it would be easier if the changes were localized to one particular class rather than scattered throughout a program (thus this feature would be considered more maintainable). Therefore, as an adjunct to OOP, AOP has been proposed. AOP, or Aspect Oriented Programming, localizes or encapsulates cross cutting concerns to a particular class or classes. The AOP class or classes are called advice. The advice is joined, or weaved, into the target program using a set of join points by the compiler. Typical join points are the entry and exit points of methods or get and put of class instance properties (although new join points have been, and are being, proposed.) The programmer specifies both the advice and the join points. The source code has localized advice and this is easier to maintain and comprehend. If one were to look at the object code, it would have the same tangling and scattering as the original code.

AOP is of interest to the Software Engineer. AOP can reduce the cost of software, speed development, and at the same time increase software quality. This is an elusive and rare combination. In project management it is often said “FAST, GOOD or CHEAP, pick any two of the three.” AOP achieves all three of these qualities simultaneously by simplifying the coding, increasing cohesion and decreasing coupling – indeed a rare combination features. But what is the cost? Certainly, the complexity of the compiler increases. We need language features to specify the join points, and the compiler needs to weave the advice into the target program. Additionally the programmer must understand, embrace, and learn the AOP programming paradigm. This entails learning new ways of expressing his or her code and learning a few new language constructs. The effort invested up front is rewarded with AOP as it allows the previously tangled concerns to be decomposed into encapsulated advice. This has been demonstrated to simplify the design for a number of classic and well defined cross-cutting problems. Thus the primary benefits of AOP are quality, development time, and cost. Blocks to AOP programming are new and more complex tools, and new language constructs to learn.

Another problem for the software engineer (other than maintainability) is the testing of software. Software testing directly affects software quality. A specific type of testing is unit testing. Unit testing is a type of testing where the smallest units of software are independently tested. A unit test typically entails testing a particular class instance. Unit testing is often augmented by integration testing. With integration testing the tested units are brought together and tested for correctness. The Agile programming paradigms Scrum and Extreme Programming (XP) both rely heavily on unit testing. In XP, unit tests are created before, during and after code creation. XP programmers have a saying, “code a little, test a little, code a little, test a little.” Before code is evem written, the unit tests are created. After writing the unit test the simplest code is written that satisfies the unit test. In the latter stages of the software cycle after the code has been written, refactoring is common. The refactored code is unit tested with the same unit test it was created with to confirm correctness. Unit testing is a strong ally to refactoring in these latter stages. It serves both regression testing and validation testing. Code refactoring becomes easier and less likely to introduce new bugs. Unit testing is an integral part of writing code to the XP programmer. XP programmers are dogmatic unit testers.

Unit testing is important in all lifecycles of the code to the XP programmer. If we look at the waterfall model in particular, then they would use unit testing in the requirements phase, design phase, implementation phase, verification phase and maintenance phases. Because of its high importance the unit test code is maintained and saved in the code repository along with the target code. Unit testing often serves as a documentation of the program under test.

Programmers often talk about Black Box testing and White Box testing. Unit testing can be categorized as one of these. In Black Box testing the Class under test is considered to be a ‘black box’. Only the external interface is visible and tests are done through there. The internal structure of the class (or program) is not visible or may not be understood by the testing person. Only visible may the public methods, their parameters, and properties of the class. In White Box testing however, the unit test framework would have access to internal workings of the Class. Not just its public interface. These would include private methods and properties as well as any other resources; code, threads and libraries. Often White Box testing provides more opportunity to test the class fully because of this privileged access to and/or knowledge of the internal structure. In some cases, the only way (or a way) to achieve White Box testing without modifying the code under test, and to gain access to the internal states, properties and methods is to break encapsulation.

The Black Box testing pattern is often only a simple test pattern. Figure 4 illustrates that a simple test pattern has a number of inputs and a number of expected outputs. Outputs that do not fall within the expected range are failures. When White box testing is employed then a number of alternative test pattern’s open up. Figure 5 illustrates a code path test pattern. With this pattern, the objective is to exercise paths in the target code. Of course, to use this pattern we must know the internal workings of the program under test. Many other documented test patterns exist for White box testing.

CLASS

INPUTS

OUTPUTS

Black Box Testing

OUTPUTSS

INPUTS

White Box Testing

Figure 3 – Black box testing does not test inside the class, only it’s interface. The term “White box Testing” implies knowledge and/or access to the internal structure of the program or class.

CODE

Condition A

Condition B

EXPECTED RESULT

EXPECTED FAILURE

Figure 4 – A Test Pattern that is Simple.

CODE

Code Path A

Code Path B

PATH RESULT

PATH RESULT

Figure 5 - A Test Pattern for Code Path testing.

The Agile methods Extreme Programing and Scrum methods require that unit testing be automated. This is to encourage frequent unit testing as code is written since testing is a labor-intensive process. Automated Testing Frameworks provide the set of tools needed for Automating Unit testing. Thus, automated testing can be repeated quickly and easily. The automated testing framework is used to automatically execute and evaluate the test cases. The automated testing framework will also provide a means to remove the test code when a production build of the software is desired.

Unit testing is a cross cutting concern. This is because the unit testing is typically performed on every method of every class in the target program. Thus, it is scattered. Secondly, the methods and classes under test require interfacing to both their entry and exit. Often, as in the White Box code path testing case, we need access to select, internal code path points. On entry, we provide the method with a set of test data. On exit we evaluate the outputs and/or side effects of the class under test. Thus, the unit test is tangled with the class under test. Finally, Unit testing is a separate concern then the program under test (perhaps the lone exception would be if the target program were a unit test program or framework itself!) Thus Unit testing satisfies the criterion of being a separate concern that is both tangled with the code it tests, and, it appears throughout the program under test (it is scattered).

Problem Statement(s)

One of the key tenants of unit testing is that the unit test should isolate the particular class or method under test. This is so each unit test clearly tests a particular feature. If there is an error it is clear where that error originates. But, it is difficult to isolate unit testing to a particular class in some instances. In some instances unit testing becomes integration testing when the class is tightly coupled to another class or other resource such as a database. If a test fails, you do not know where the failure originated

For an example in the literature, Mackinnon, Freeman and Craig (2001) point out that an essential aspect of unit testing is that we test one feature at a time to know exactly what you are testing and where the issues are. They point out a difficulty occurs when the test has to set up domain state or the domain code causes side effects.

Research Goal

In the proposed research, we will present a framework for unit testing of software enhanced by Aspect Oriented Programming. The first part of the research will be a critical review or survey of the current tools for unit testing. We will present a critical survey of popular unit test framework: NUNIT, and nMock, Test, TypeMock, Visual Studio, Moles, Pex, for the .NET languages may be included. The purpose is to generate a view of common functionality required of unit test frameworks. The research will categorize this functionality. The intent of this critical review is to provide a basis, or measuring stick, with which to judge the generated framework.

The research will generate the range of unit test patterns common for C# language implementations for unit testing of code. We will demonstrate how AOP can implement these test patterns. This phase will also present a library of Mock objects and how they can implemented with an AOP solution. A framework will be developed that will automate the unit testing of C#. The C# framework developed will demonstrate \an interface to automate unit testing. The framework will use OOP design patterns to implement the functionality. It will be based on UML models. The interface will preset several mock objects strategic to an aspect oriented .net unit test framework. Finally, our AOP Unit Test Framework will support the automation of testing. We will demonstrate how AOP’s ability to enhance modularity can speed frequent changes to the program under test by breaking the inheritance link between program under test and unit test classes.

In a final stage the resulting AOP framework (or AOP enhanced framework) will be compared to the frameworks in the 1^st phase critical review. The framework will be judged successful if it can be demonstrated that an AOP solution can span the range of test problems non-AOP based solutions provide AND it solves the coupling problem as described in the problem statement. The successful completion will be verified as the AOP unit test framework is applied to a sample set of unit test problems.

Relevance and Significance

Unit testing is a very important issue to the software engineer and the managers of programmers. Testing can be very labor intensive and a significant part of the software cycle. One of the first significant characteristics is that unit testing keeps the programmer out of the debugger. In “The Mythical Man-Month”, Fredrick Brooks Jr (1975), postulates that 50% of ‘conventional’ software project time is spent debugging! It is safe to say that debugging is a significant part of any software project.

The cost of debugging is not solely the programming team time either. The later in the development cycle the bug is found the more expensive it is to fix. Post release fixes can incur upgrade and recall costs. Unit testing focuses the time normally spent debugging in later stages of development, up front in the earlier stages of development to create the unit tests.

An AOP solution to unit testing promises to accelerate the development cycle. It will provide ways for the programmer to mock objects that previously required significant resources to mock. It will enable white box testing of unmodified production code thus introducing less new errors that are tied to the test framework. It will enable the automation of unit testing and speed development by ‘untying’ the program under test class and the unit test class.

Many software engineering paradigms practice the labor-intensive unit testing process. Just the Agile paradigms alone include: GSD, Agile Modeling, Agile Unified Process (AUP), Crystal Clear, Scrum,Extreme Programming (XP), Open Unified Process (OpenUP), Feature Driven Development (FDD), Kanban (development), and Velocity tracking to name a few.

It is safe to say that unit testing is a pervasive and its’ effective practice is important to the software engineer.

Barriers and Issues

A research question that requires further exploration is how to gage the effectiveness of this research. In the goal section of this proposal success is demonstrated by a solution that spans the range of unit test problems. The real goal should be to reduce the burden on the programmer(s) or speed up the development (unit test) cycle, This is a difficult measurement but probably not impossible. A brute force method to compare an AOP solution to the non-AOP solutions would be do set up some double blind studies. Perhaps this could be done as part of a Master Course in Extreme programming. One group of student programmers would use a conventional unit test framework, and a second group of student programmers could run the project with the AOP unit test framework. The group members would be chosen very precisely to minimize any variations in student programming ability. This could occur later in the semester after their abilities are measured or judged. Perhaps the computer they use for development could be instrumented to record the development cycle. What data would you instrument for and collect? Would each team need to follow a set and precise development program?

Although this would not be a very precise blind experiment, it may lead to some insight as to the effectiveness of the AOP unit test framework Williams, et al, (2004) develop metrics that may be helpful in determining the effectiveness of an AOP framework.

Another way to test the effectiveness of this framework is possible. Work on benchmarking unit test tools has been proposed (Cyrille, 2007). This work uses AOP based tools to measure the code coverage of a test suite.

This could be a future proposal as follow on research based on the results of this research.

Definition of Terms

AOPD: Aspect Oriented Programming Development. This software practice attempts to put cross cutting concerns in separate classes. These classes are then woven into the target program to a series of join points with an Aspect Compiler.

Cross Cutting Concerns: Aspect or bit of code that represent abstract ideas that are defined and contained. Cross cutting concerns are both scattered in the target program AND the code is tangled with other concerns.

Aspect: An aspect is a software design decision (Kiczales et al., 1997). AOPD is concerned with those software aspects that are difficult to cleanly capture in code.

Tangled: In the context of AOPD it means two or mode aspects whose code cannot be decomposed from each other. A class with tangled code has a low cohesion.

Scattered: In the context of AOPD it means a code aspect that appears in many places (and classes) in the target program. Classes with scattered aspects have a high coupling between classes.

Unit test: In this paper it is defined as used in the software programming paradigm Extreme Programming. In this software method the smallest pieces of code are independently tested.

Extreme Programming: A pragmatic and methodical method of software development that depends on meticulous testing to build software systematically.

Chapter 2: Review of the Literature

Test driven development is not a new software design process. There is abundant commercial products and scholarly literature. The IEEE has published a standard for test driven development.("IEEE Standard for Software Unit Testing," 1986). This specification should be the start of any framework for unit testing. Another IEEE standard that adds an overall context is ANSI/IEEE Std 829-1983, ("IEEE Standard for Software Test Documentation," 1983). It describes the basic information needs and results of software testing.

Commercial and open source products exist for unit testing. These include jMock, nMock. These are frameworks for Mocking Java and .Net classes respectively.

A very interesting White Box testing Framework is Pex for the .Net framework. Pex automatically generates the test suite. It finds interesting input and output pairs to include for testing. From the Pex website “Pex seeks to generate a test suite with high code coverage, where each test case invokes the method-under-test with particular values.”

Also for .Net are the Moles and Stubs to Mock the ..Net library and resources. Moles and Stubs are being replaced by the “Fakes Framework”. . An equivalent for Java programmers is jMock. Open Source Unit Testing frameworks include nUnit and jUnit. These frameworks provide the classes and features to automate the tests.

Research into “The Fragile Pointcut” as applied to unit test cases will be of significant importance to this research. A new point cut has been proposed (Sakurai & Masuhara, 2008) that lessens or solves this problem. They call this the test based point-cut. The test based point-cut matches a section of a program’s execution history. The author gives a concrete example of a login authentication failure. The match would be on the variable in test cases that holds the invalid password. The point cut would match login method calls whose failed execution path matches those of the test case. In this way the point cut remains faithful even if the target program changes as long as the test case is maintained!

GATE is an AOP based automated testing framework. (Feng, Liu, Kerridge, 2007). Gate tests the non-functional concerns of the program under test. This includes reliability, dependability, system performance and programming standards enforcement. Using software product line techniques this framework creates automatic and reusable test cases. Gates attempts to extend the concept of the unit test from functional to non-functional concerns.

Test Oracles have been written in JUnit and JML and found to be laborious. (Cheon & Leavens, 2006). Chen and Leavens combine the Modeling language, such ans JML and testing framework, such as JUnit, and instead of focusing on input and output values, they determine correctness by monitoring the behavior of the method being tested to determine pass or fail.

One attempt to write an aspect oriented framework for unit testing is FLEXTEST (Sokenou & Vösgen, 2005), In Flextest they develop a framework using AspectJ that is similar to JUnit. The privileged class of AspectJ is used in addition to test classes to adapt to the object oriented program under test. One popular definition of AOP they present is that AOP is “quantification and oblivions.” The authors demonstrate how the quantification of AOP can help in writing test cases that return similar results. Flextest does not attempt to solve the mock and stub problems addressed in this proposed research.

In “Endo-Testing: Unit Testing with Mock Objects” (Mackinnon, T., Freeman, S., & Craig, P. 2001) the authors discuss the Mock Object. They point out the difference between a Mock object and a code stub. They show how the Mock object can be used to test the class from the inside. Thus the term “Endo-Testing”. Although they DO NOT show how AOP can be applied to this problem, research in this area should build upon their work. Here they define the Mock object “A Mock Object is a substitute implementation to emulate or instrument other domain code. It should be simpler than the real code, not duplicate its implementation, and allow you to set up private state to aid in testing. The emphasis in mock implementations is on absolute simplicity, rather than completeness.”

In “Testing Object-Oriented Programs using Dynamic Aspects and Non-Determinism” Michael Achenbach & Klaus Ostermann (2010) create an AOP framework to model Mocks and stubs. In their work they develop a framework for “increasing the expressiveness” of mock objects in Ruby. The framework is called ExploR. They develop a library called TwistR that uses dynamic AOP to inject mock objects and test local design adaptations. Finally they combine both and show how this can be applied to testing of multi-threaded programs. They claim to turn non-deterministic programs into deterministic tests to create reproducible results.

“Abstract Test Aspect: Testing with AOP” (Wenner,2004) shows how to use Aspects to check contracts in a hierarchy of classes a la the Liskov Substitution Principal. Here an abstract test case pattern is used to create the test which the child classes must pass. In this paper they show how to use AspectJ to create an AOP solution to the abstract test case.

Chapter 3: Methodology

Approach

This research will be conducted in three phases. The first phase is a critical review of the commercial tools available. We will construct a vector of unit test patterns. In this first phase a survey of Mock objects, libraries, and stubs will also be presented. Mocks and stubs will be categorized by unit testing problem. A matrix of white, grey and black box test patterns will be created. A vector of approaches to the typical unit testing problems will be created.

In the second phase of the research an Aspect enhanced Unit Testing framework, or alternative frameworks, will be developed that can be constructed to implement the mock objects, stubs, test classes and white box testing frameworks that address the issues outlined in the problem statement of this idea paper. This phase will start with a set of software requirements derived from the first phase of the research. UML models will be created and appropriate OOP patterns selected (singleton, etc…).

In the last phase of this research, we will measure the effectiveness of the AOP enhanced Unit Testing framework by comparing to those tools that were critically reviewed in the first phase of the research. The effectiveness of the AOP enhanced unit test framework will be judged by how well it spans and encompasses the range of solutions to unit testing problems as compared to the original range determined in the phase 1 critical review. Secondly, unit test problems will be selected from Open Source Code projects. The AOP unit test framework will be applied to these problems to confirm successful completion.

Resource requirements

· This project will take 12 months to complete after approval of proposal.

· This project will require 2500 hrs of research time.

· 20 hrs of secretarial time will be required for publishing and binding.

· Standard library resources (computer, printer, internet, research database).

Deliverables & Milestones

· Dissertation – will explain the groundwork, research, relevant work, applied methodology, criterion for choosing tools used in the critical review.

· Critical review of available unit testing tools and solutions. A matrix of current solution to mocks, stubs, white, grey and black box test patterns.

· Aspect Oriented Enhanced Unit Test Framework – Code framework In C#

Figure 6 – Deliverable Schedule.

Summary

This research will address some common problems with unit testing that are not otherwise easily solved using standard OOPs tools. This research will apply AOP to develop a testing framework to solve common problems programmers have with unit testing and white box testing. AOP will increase modularity when applied by the software engineer to those software aspects whose modularity is otherwise elusive. AOP can be used to increase the modularity of the unit test suite as well. This will be a productive enhancement to the software team.

References

Williams, L., Krebs, W., Layman, L., Antón, A., (2004) Toward a Framework for Evaluating Extreme Programming, Empirical Assessment in Software Engineering (EASE), pp. 11-20.

Binder, R. V. (1996), Testing object-oriented software: a survey. Softw. Test. Verif. Reliab., 6: 125–252.

Juristo, N.; Moreno, A.M.; Vegas, S.; Solari, M.; , (2006) "In Search of What We Experimentally Know about Unit Testing," Software, IEEE , vol.23, no.6, pp.72-80, Nov.-Dec.

F. P. Brooks, (1975) Mythical Man Month: Essays on Software Engineering. Reading, MA:Addison-Wesley.

C. Artho, Z. Chen, S. Honiden, (2007) “AOP-Based Automated Unit Test Classification of Large Benchmarks”, AoAsia.

D.L. Parnas (1972). On the Criterion to be used in decomposing systems into modules. Communications of the ACM. 15(12):1053-1058. December 1972.

Dijkstra, E.W. (1976) A Discipline of Programming. Prentice Hall, Englewood Cliffs, NJ.

Yankui Feng, Xiaodong Liu, Jon Kerridge, (2007) "A product line based aspect-oriented generative unit testing approach to building quality components," compsac, vol. 2, pp.403-408, 2007 31st Annual International Computer Software and Applications Conference.

Mackinnon, T., Freeman, S., and Craig, P. (2001). Endotesting: unit testing with mock objects. In Extreme Programming Examined, G. Succi and M. Marchesi, Eds. The XP Series. Addison-Wesley Longman Publishing Co., Boston, MA, 287-301.

Wenner, R. (2004). Abstract Test Aspect: Testing with AOP, Extreme Programming and Agile Processes in Software Engineering. J. Eckstein and H. Baumeister, Springer Berlin / Heidelberg. 3092: 237-241.

Achenbach, M. and K. Ostermann (2010). Testing object-oriented programs using dynamic aspects and non-determinism. Proceedings of the 1st Workshop on Testing Object-Oriented Systems. Maribor, Slovenia, ACM: 1-6.

Artho, C.; Zhongwei Chen; Honiden, S.(2007) "AOP-based automated unit test classification of large benchmarks," Computer Software and Applications Conference, COMPSAC 2007. 31st Annual International , vol.2, no., pp.17-22, 24-27 July 2007 doi: 10.1109/COMPSAC.2007.75

Yoonsik Cheon and Gary T. Leavens. (2002) A simple and practical approach to unit testing: The JML and JUnit way. In Boris Magnusson, editor, ECOOP 2002, volume 2374 of LNCS, pages 231–255. Springer June 2002.

Feng, Y., Liu, X., and Kerridge, J., A product line based aspect oriented generative unit testing approach to building quality components. Proceedings of the 1st IEEE International Workshop on Quality-Oriented Reuse of Software, Beijing, China, pp., 2007.

Cheon, Y., & Leavens, G. (2006). A Simple and Practical Approach to Unit Testing: The JML and JUnit Way

ECOOP 2002 — Object-Oriented Programming. In B. Magnusson (Ed.), (Vol. 2374, pp. 1789-1901): Springer Berlin / Heidelberg.

Cyrille, A. (2007). AOP-based automated unit test classification of large benchmarks.

IEEE Standard for Software Test Documentation. (1983). IEEE Std 829-1983, 1-48. doi: 10.1109/ieeestd.1983.81615

IEEE Standard for Software Unit Testing. (1986). ANSI/IEEE Std 1008-1987, 0_1. doi: 10.1109/ieeestd.1986.81001

Kiczales, G., Lamping, J., Mendhekar, A., Maeda, C., Lopes, C., Loingtier, J.-M., & Irwin, J. (1997). Aspect-oriented programming ECOOP'97 — Object-Oriented Programming. In M. Aksit & S. Matsuoka (Eds.), (Vol. 1241, pp. 220-242): Springer Berlin / Heidelberg.

Sokenou, D., & Vösgen, M. (2005). FlexTest: An Aspect-Oriented Framework for Unit Testing Quality of Software Architectures and Software Quality. In R. Reussner, J. Mayer, J. Stafford, S. Overhage, S. Becker & P. Schroeder (Eds.), (Vol. 3712, pp. 257-270): Springer Berlin / Heidelberg.