My Photo

My Online Status

SW Process

Best Books on Design and Development

Changing Jobs

It's been a long time since I blogged last time. Well now there is something interesting happening in my life and that gives me a reason to write something here again.

I am changing to a new job. My new employment starts at the end of this month (April) and I'm pretty excited about that. In fact I haven't been so excited in a long time. The reasons I accepted this new opportunity include following:

  • This is a new startup company and it will be interesting to see how things get going.
  • We will be building something new. In my current job I've been mostly involved with legacy code and it's nice to implement something completely new for a change.
  • I get to use my Flash video skills in addition to Java (I've implemented this nice little Flash video player called FlowPlayer)
  • I get to learn new technology.
  • I have a change to work with talented people.
  • The Web site that we will build will kick ass

I plan to start writing about software development and about this new project here in this blog.

Blogged with Flock

Agitator Developer Testing Tool

Recently I participated in an “evaluation” of the Agitator testing tool. We wanted to see if Agitator would help us in our mission to write bug-free code using Test Driven Development TDD).

This evaluation was done during a two day workshop where we tried the tool with the code base our team is currently working on. We also did an exercise in which we used the tool when developing a toy application from scratch. In this exercise we used Agitator together with Junit and the programming was done using Test Driven Development(TDD).

I have some experience in TDD and I believe that it helps in producing high quality code. I also think that it requires quite a lot of effort to write truly comprehensive unit test suites. With this in mind I was optimistic when we started the evaluation – a tool that makes unit testing (developer testing) easier would be valuable.

Agitator is a tool aimed to help developers in writing unit tests.The value proposition of Agitator is to make the effort of writing comprehensive unit tests suites much smaller by automating the creation of unit tests.

How it works

In brief, Agitator works as follows:

  1. The classes in the agitated project are instrumented using byte code manipulation.

  2. Agitation is done for the class that is under development. Agitator instantiates the class hundreds of times using different constructor parameter values. The values are (more or less) random. Similarly public methods are invoked with random values.

  3. The results of the agitation are shown to the developer as observations. These observations include the range of possible values of a class field, the range of possible values of a method parameter, and the exceptions that were thrown and in which situations.

  4. The developers inspects the observations. The observations can highlight bugs for example when illegal field values were detected or when unwanted exceptions were thrown. In this case the developer should fix the bug and promote the observation to an assertion. All these assertions form a test suite that can be used in regression testing. Any observation can be promoted to an assertions, assertions can be also added manually and they can be modified.

The whole set of assertions form a test suite. The test suite can also include all hand written Junit tests and the Agitator can also run those.

The test suite (assertions) is stored in an XML file that can be committed to version control and be shared by all developers. The test suite can be run using Ant and be included in continuous integration builds (e.g. with CruiseControl).

In addition to what is described here there are more supporting features in the product. See the Agitar Software Web site for more info.

Dashboard

The results of agitation are presented in a Dashboard. Dashboard is a Web page showing the test results, test coverage and McCabe complexity metrics. The combination of code complexity metrics,number of classes' clients, and unit test coverage is used to highlight high risk classes. Overall the dashboard provides quite good overview to the health of the code base. The Dashboard can be generated as part of a nightly build.

Benefits

  • Eases unit test creation. Agitator focuses on verifying object state and method parameters and makes test creation for those easy. It does not seem to provide much help in testing business logic (at least we did not learn how to do that during our two day workshop).

  • Finds bugs even when using Test Driven Development (TDD)

  • The Dashboard enables constant monitoring of quality levels, and the setting of target values that should be met before code is released from the team.

These are in fact quite significant benefits.

Drawbacks

  • Requires some effort to be effectively applied in legacy code bases.

  • Might confuse developers if they try to learn TDD at the same time.

  • Takes some time for developers to learn.

  • Difficult to estimate if the tool provides the value promised. The ROI is highly dependent on how actively the tool is used.

  • Eclipse is the only supported IDE.

These are not too bad to tackle.

Conclusion

The tool is not cheap. A license for one developer costs thousands of euros per year. Still I believe that  the tool provides enough value to justify the investment provided that the developers start actively using it. In new development I would definitely use it. But I hope that there would be a smaller entry fee for new licenses.

In existing large code bases the adoption is not that straightforward. First there is some up-front effort to configure and develop the required factory classes etc. so that the tool can be effectively used with the existing code base. Then there are other things that affect the decision: Project schedules and how they are affected when developers start learning a new tool, IDEs used by the team, what kind of new features are being implemented and are there good opportunities for Agitator based testing (are there business objects with state, for example), and so on.

I hope Agitator was free. Then it would be easy to start experimenting with it. Throwing a lot of money into something that has a high probability to fail is not that attractive.

technorati tags:,

Good Management and Bad Management

Hitler I recently finished reading Behind Closed Doors, Secrets of Great Management. The book really got me thinking about management and how I've been doing it in my short career as a SW development manager. In this post I'll explain what it was that got me thinking.

There is quite a big difference between the management described in the book and in the management that I've seen in practice. Here are some of the things that great managers do according to the book. This is not an exhaustive list of things discussed in the book, the ones described here felt important to me.

Provide feedback. Great managers provide timely, both positive and negative feedback. Timely means that it is almost immediate. Feedback should be given not only when really major things happen but also based on more or less normal work related things.

Great managers coach people. They identify the goals and needs of the people they are managing and they people achieving these goals. Great managers also coach people so that they concentrate on things that support the the goals of the team and the organization. Coaching is an ongoing activity, you cannot provide coaching with in meetings that occur once every half a year.

Great managers involve the team in decision making. In the book there is a nice story in which the book's Great manager facilitates a new forum involving a group that will meet regularly. The first thing that the group does is that they brainstorm and decide how the forum will do their work, what tools and practices they will use, and so on. This is superior to the way this would be done in a typical Finnish company. In my experience it is usually the manager who decides how things will be done and as a result the people find it hard relate to the results. They might not be motivated to follow the manager's approach regardless of how good that might be. Team involvement generates motivation and commitment.

Avoid "micromanagement". I have discussed this in some other entries in this blog. This book also has the view that managers should not focus on giving people detailed instructions on how to do a task. There are three excellent articles related to this in Joel on Software about different management styles. Joel calls his preferred style Identity Management.

According to the authors the most important tool that their book presents are the one-to-one meetings between the manager and the individual people that belong to his team. One-to-ones are used to facilitate feedback and coaching for example. These meetings are held weekly and therefore they address things in timely fashion: Feedback is given on time and coaching for upcoming goals and events can be given on time.

After reading this book I was convinced that what was described was indeed Great management. Management is about people and the manager should concentrate on people. My sin has been that I have been concentrating too much on technical stuff and ignoring management. I have a technical background and it's hard to completely stop coding and other technical work. But technical work has a tendency to eat all time, at least it has that effect on me. When that happens, the management suffers and the team goes on without a manager.

Reality

The sad thing is that in my career I have never experienced great management as described in this book. Here are some observations from the reality I have seen it in the five different Finnish companies I have worked in.

The feedback comes usually late if ever. It is common to receive feedback only after some major mistakes have been made. Situation gets even worse if there's no coaching - Negative feedback is completely absurd if there was no advice and the things went wrong. In Finland we have these personnel appraisal systems where development discussions between a manager and an employee are conducted once every half a year. The idea is for the employee and the manager to sit down and do away with goal setting, feedback and coaching all in one shot. This is quite different from having weekly sessions. But maybe those weekly one-one-ones are only appropriate for the happy extrovert people living in USA, we Finns are completely different creatures and trying this here would be a disaster... or i don't know.

There is no coaching. Managers are too busy working on "more important stuff" and ignoring people. Because there is no coaching and support the new managers have to learn things the hard way. New managers are forced to make mistakes and learn from them. It would better to provide coaching where the more experienced manager shows how to avoid at least the most oblivious traps.

Managers make most of the decisions themselves. The managers may discuss with the different stakeholders before they make the decision but truly involving the people who do the work in decision making is rare. Obviously it is good if the manager discusses with the workers before he makes the decision but it is still not as good as having the group members in a joint session where they brainstorm and then decide. In software development the workers have the best knowledge and ignoring that fact is a mistake.

I'm not claiming that I do things any better than described here. I have been guilty of committing all the sins described here.

Many technical managers have technical backgrounds. Because of that they usually haven't received too much training on how to manage people. Additionally it is hard to switch from a purely technical job to a management position. For me some of the management stuff is just boring! I used to think that management is just paper shuffling that anyone is capable of doing. I guess I'm still thinking like that: Certainly you don't need a university degree to do management. University degrees are needed in more difficult jobs, software development being one of those!

technorati tags:

Team seating

Our team has a new office space and a new seating arrangement. As you can see from the image below we have two groups of tables to be used by our two project teams. There are four developers sitting closely together in both groups.

Seating_4

We had the opportunity to select new tables and we settled with simple rectangular shaped ones without any extra widgets. (Well there are those adjustable monitor stands in four tables, but otherwise these are as simple as they get.) The rectangular shape is good as we want to work in pairs from time to time. Pairing was impossible with our old tables shown here:

Oldtable

As you can see the one who is sitting in front of the keyboard is located in the corner of the table and his unfortunate pairing partner is forced to peeking around his shoulders if he wishes to see the monitor and even that way the monitor is too far for any practical work to happen. This kind of table is not really meant for software development.

Our new tables don't have any corners where one could hide. Additionally two developers are always sitting besides each other so that they can easily share information, ask questions, and do pair programming. We haven't settled into using pair programming seriously yet but the idea of sitting besides each other will probably have similar effects. Communication is the key here: The four guys sitting together should be in close communication with each other.

This side-by-side programming concept is described in Crystal Clear. The whole idea of having the teams sitting around a common table is from that book.

The only downside we have encountered with our new room is that it is noisy. The room does not have any soft materials that would eat out the sounds and therefore they keep echoing around. That could be fixed for example with a big soft sofa...

Design Maps

James Shore has posted design maps on his blog. Ours is also featured. In general I really like the stuff James writes on his blog, there are interesting writings for example about Fit.

Amplify Learning!

I just finished reading Mary & Tom Poppendieck's Lean Software Development. It is a good source of knowledge about the principles and ideas behind the practices used in Lean and Agile development. I found the chapters about amplifying organization's learning and about empowering the team especially interesting.

I once worked in an organization where the basic principle was to spread all kinds of knowledge to all people in the organization. For example, developers were encouraged to learn most of the different tools and programming languages used in the company. We did not have designated DBA's for example. To compensate the organization valued learning and developers were expected to know how to do schema design and code in PL/SQL. Some developers who found databases close to their heart wanted to study them more closely and as a result they had the skills of a DBA. But even those developers did not stick to databases only. Similarly it was normal that developers wanted to learn how to communicate effectively with customers. This principle of spreading knowledge lead to following practices: Testing sessions where the whole company participated: This was a great way to find bugs and to spread knowledge about company's products. Design sessions with participants from different teams: Again a great way to learn the domain and the products. All products had a similar dead-simple architecture that made it possible for anyone to work on any of the products with ease: This is known as common code ownership nowadays.

This arrangement provided a lot of flexibility. Project staffing was easy because there was less areas where the knowledge is concentrated into a few busy specialists. The project's required skill-set could be gathered from regular "staff engineers". It is also motivating for the personnel, at least for those who want to learn and grow in their profession. It can be quite frustrating if the organization wants people to concentrate in small areas and does not allow them to expand. Personally I panic if I am not able to learn in a certain job (I have switched jobs once just because of this).

For some developers it is natural to learn more than just one specialty. Scott W. Ambler calls these people "generalizing specialists" describing them as follows: "A generalizing specialist is someone with one or more technical specialties who actively seeks to gain new skills in both their existing specialties as well as in other areas, including both technical and domain areas.". I agree with Scott that the best developers can be described as generalizing specialists. These developers are typically passionate about their profession. They are capable of seeing the "big picture" both in business and in the technical area. They are able to communicate with customers, do requirements analysis if needed and they are also good programmers. There is no need to distinguish between analysts, DBAs, testers, programmers - We can use the term "developer" for anyone who participate in the construction (development) of software.

Poppendiecks' book talks about amplifying learning and the principle described above is one way of doing it although it is somewhat different from what is described in the book. The chapter "Empowering the Team" talks about building communities of experience within the organization. In this approach the knowledge about different key areas is developed within communities. As an example an organization might have DBA and product manager communities. The DBA community develops expertise on databases and the product manager community develops domain knowledge. The project teams are then composed of people gathered from these communities. This arrangement is similar to the the traditional matrix organization.

The matrix organization has the advantage that people can relate to other experts within their communities of expertise. It is easier to facilitate learning through common discussions, meetings, courses and other activities within the community. Therefore it might be easier to build really "deep" expertise in these communities. The challenge is then how to effectively use this experience in the projects and how to ensure that people don't turn into too narrowly focused specialists. I think the challenge can be tackled with following in place:

  • Communities of experience are fairly loose organizations that mainly provide opportunities for learning and for exchanging experiences.
  • Main work happens in agile project teams where people from different communities work in close co-operation and learn from each other.
  • Organization values learning and therefore encourages employees to become multi-skilled
  • Organization has people who want to learn

I am thinking the result of this could be quite an optimal organization that both encourages people to become multi-skilled and also provides communities where key expertise areas are built and maintained effectively.

In this interview Scott W. Ambler talks about generalizing specialists. It is well worth a listen.

Lean Software Development

I used to work at TietoEnator in a smallish business unit called Dialogos Team that developed (and still dovelops) an ERP system called Lean System. I left TietoEnator in 1998 after working there for 5 years. The Lean System ERP was somewhat based on the Lean principles. I think Lean manufacturing was hip during that time. But more interesting in all this is that this shop was applying Lean principles also to software development and they started doing that in the early nineties.

Lean principles in SW context have only recently been brought to public attention by the Agile movement. There is a book called Lean Software Development, An Agile Toolkit  that describes how to apply Lean thinking to software development. I just started reading that book, and therefore will not comment on that more yet. This book was published in 2004.

Here are some things that we did in Dialogos back then:

Spreading the knowledge and skills to everyone. The principle was that every developer should know about everything and every developer should be able to work on any module (common code ownership). We tried to avoid the situation where each individual specializes to some specific tasks. This provided good scalability making it possible to take in and serve new customers in fast pace.

Common design meetings. We used to call these "specification meetings" ("speksauspalaveri" in Finnish). People from both from sales/marketing and from development participated in these sessions. There were no architects responsible of creating the specifications. In fact nobody was talking about architects during that time; might be better if it was like that still now!

Simple design. (Actually I think the design was too simple, somewhat smelly.) Simple design made it possible to share the code base with everyone.

Team focus. The organization gave teams a lot of power making them motivated and effective. Teams were collocated in open space offices.

I have recently realized that many things were really well in Dialogos. The product was also a success that attracted a lot of customers. Not sure if things are so well nowadays: If I remember correctly we started formalizing the process into a heavier and more bureaucratic direction already before I left.

The Java Posse Podcast Rocks

I just found about the Java Posse podcasts. Now I have listened to their episodes during two days while driving to the office and back. I have to say I like it! I used to listen to the JavaCast when it was still alive. Now that seems to be dead and buried and the Java Posse continues in the same tradition. It is a good way to stay informed about new releases, JSRs etc. The hosts seem to be experienced and knowledgeable and therefore their opinions are also worth listening to.

I also found out how to turn off the RDS function in my car stereo so that the frequency stays locked into the frequency my iPod's radio transmitter uses. Listening podcasts and audio books in my car is the most important use case I have for my ipod. Suddenly it does not matter if the traffic is stuck because it only provides some more time for listening!

Is there some other podcasts available I should chedk out?

What is good in incremental development?

Scrum Overview @ Mountain Goat Software

Probably the most straightforward way to bring some agility to your process is to introduce iterations to your development. It was the most obvious starting point for me and our team anyway. An iteration, as I see it in this post, has following properties:

* It has a fixed length. Typically the length is from one week to one month.

* A product increment is implemented within an iteration. The product increment should have some value to the customer or at least to some user. It should be also shippable meaning that it can be deployed into production if necessary.

* Iterations don't mean rework: Certain number of stories are completed within an iteration and after that they are not normally reworked in the following iterations.

I started writing down some of the advantages this kind of iterations bring along and here is the result. I have been reading Crystal Clear by Alistair Cockburn and some of the things presented here are from that book. Some of them are actual experience that we have gained when using iterations in our projects.

Project planning and control

Iterations are a tool to do project management. Traditionally, in the non-agile era, the project manager was the ultimate authority doing most of the planning, estimation, work allocation and so forth. This fuhrer-style management is not very motivating for the team members. Additionally, if the fuhrer does all this in isolation the resulting plans are likely to be suboptimal. Nowadays, in the Agile era, the whole team is involved in these tasks. The project manager (or the technical lead) is responsible for coordinating the activities. This management version is more effective than the traditional version in many ways and I will discuss some of those next.

Before the iteration is started you have a planning session where the whole team (plus other stakeholders) sit together to check priorities and do estimates. Usually some design is also needed when doing this: Stories are split into smaller stories/tasks that can be estimated more accurately. Involving the whole team in this planning activity is a good thing. It promotes motivation and team spirit. The team will be more willing to commit to the plans they have themselves created together with the customer. Our experience with this approach has been positive. The team involvement is one of the most valuable things that our agile-like-process has provided us so far.

Iterations provide a way to track progress. Only running tested software is counted as progress - Specification work, for example, does not count and neither do features that are "80 % completed". The remaining 20% may contain surprises, and it usually does, when we are talking about SW development. If you follow this rule of counting only running tested software, it is clear what was achieved in an iteration. At the end of the iteration you know the team's velocity and that gives you a good starting point to estimate how many more iterations are needed to complete the project.

Gantt charts?

My opinion is that Gantt charts don't work well as a project planning and tracking tool. In the projects I have worked in during my career, the "project plan" has almost always been in the form of Gantt charts. Perhaps the most important reason for this is that MS Project uses them by default. Software development is so unpredictable that the chart will be out of sync with the reality soon after the project has been started. The inaccuracy of typical estimations is another reason why the bar lengths in the chart either lose their meaning or cause unnecessary delay. The delay comes when the developer tries to change the reality in order to stick with the plan. A typical developer will use all the time that was allocated for him to complete a task either doing unnecessary polishing or by simply slacking off.

The iteration approach is much simpler and works in practice. It also helps in squeezing maximum productivity out of the team! There is no room for unnecessary polishing or slacking off. You will not be able to get as good results if you base your planning and tracking on Gantt charts.

Quality

Iterations are a way to get early feedback from the real users of the system. This requires that you either deploy the product increment to the customer's server (or users' workstations) or alternatively organize a demo for the users.

Iterations also provide early wins for developers. The team should produce working software starting from the beginning. To make this clearer, here is an example plan that does NOT use iterations correctly:

  • 1st iteration: Functional specification
  • 2nd iteration: Architecture specification
  • 3rd iteration: Database schema
  • 4th iteration: Infrastructure
  • etc...

This approach will not produce early feedback or early wins. The developers have to wait until iteration 5 to see the first complete features. It is better to organize the development in such a manner that working features are created starting from the 1st or 2nd iteration. You need only enough planning, database tables and architecture to support the first set of features. More tables, more design, and more infrastructure is created as more features are implemented. The architecture also evolves when the project goes on. The first complete feature that uses all the application layers proves that the design decisions are correct and that the chosen technology can be used to create a functional system. You and your developers need this information as early as possible.

Process improvement

After the iteration you can have a session where you reflect what was good and what could be improved. You pick two or three things or practices that you think will improve the way the team works and develops software. The team will try these in the next iteration. And after the next iteration you reflect again to check if those practices were good or not and act accordingly. If the iterations are short enough the improvement will be fast!

I like this approach of trying things out. What I don't like is that there is some authority, like a project manager, or an Architect group, that comes and says that "from now on you will do this/that and, by the way, you will also use this framework because our architects decided so". Again I find that it is much better to have the team plan together how to improve their own way of working. The result is that the team will be more motivated to follow through.

Team communication

As I already mentioned above, iterations provide two points where a lot of teamwork is involved: The start and the end of an iteration. In the start there is the planning session and at the end there is the retrospective. Both of these provide an opportunity for the team to get together. This is of course more important if the team is distributed, like it has been in our case. If the team is collocated and sitting together (like it really should), the value of this is not so big because the team is already communicating a lot in the course of normal day-to-day work.

My Agile reading list

Recently I've been reading books about software development methodologies. So far it has been mostly about Agile processes. You can find the books on the sidebar on my "Currently reading" and "SW process" book lists. I am happy to receive book recommendations, if you have some, my dear readers (there are 1 or 2 readers, I know).

I have some 10 years of experience in working with more "traditional" processes (waterfall style) and I know what kind of problems they have. Now I have been turning more to the agile processes (see my previous entry about Agile Development).

After some real-world experience and theory studying I'm now convinced that this is the way to go. To make this realization somehow official, I just signed the Manifesto for Agile Software Development. Below are the values of the manifesto. They are well worth of repeating here...

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

Recent Posts

Recent Comments

Blog powered by TypePad

Currently Reading

  • Jakob Nielsen: Prioritizing Web Usability

    Jakob Nielsen: Prioritizing Web Usability
    Packed with lots of usability recommendations for Web sites. I have already picked lots of tips that I will use in improving my site. Rating coming later, the beginning of the book is promising.

Management