Skip to content

Posts tagged ‘Iterative Software Development’

Explaining the business value of iterative development

A few people wonder how changing the software development approach can impact business results, namely costs and ROI.

We often use a diagram similar to the one presented below to explain the relations between various decisions and outcomes.

Iterative Software Development Applied to Business Intelligence

Iterative Software Development Applied to Business Intelligence

Starting with the decision of using an iterative development approach as opposed to a traditional waterfall approach for the development of a BI solution results in 4 immediate outcomes:

  • Improvement in requirements gathering by quickly developing on the initial requirements to deliver a prototype. After a few hours of discussion with a business user, there is usually enough information to start developing a first key performance indicator (KPI). At this point, the development team will select based on perceived value (see below) and low complexity a first KPI that they will develop and present to the business user. The business user will then re-assess his needs and initiate a new iteration to either add more KPIs or develop the first one further.
  • Reduce long term planning by avoiding a long term perspective. It is obvious that focusing on the long term forces the team to plan, architect, and model accordingly since it is unlikely they would have many attempts at doing this right. Reducing the long term perspective to the current iteration forces the development team to plan, architect, and model for the context of the current iteration. This does not mean the team should not think about future impacts. On the contrary, while the team needs to keep in mind the future overall BI application they should remained focus on the current iteration – focus being the key word.
  • Determine ROI of additional features by assigning business value to each additional request. Since the development team has a small time frame to deliver (current iteration), they cannot attempt to deliver everything and as a consequence the business users need to select which KPI should be handled during the iteration. Using a “business value factor” helps the business users and the development team understand and agree on the components that need to be build to deliver value quickly. Although simple, the prioritization exercise forces business users to make decisions and realize that it is not possible to deliver everything. Using long term iteration typically delays those hard decision and leads people to believe they can include everything and that everything will indeed be delivered.
  • Quickly responding to market changes by not committing the development team to long development cycles. In a traditional approach where the project is planned at the beginning and rarely re-assessed, using small iteration cycles allows a change in priority when the business context needs it. Knowing the development team is committed to a 3 to 4 weeks development cycle allows business users to change the content of the next iteration if the business landscape forces them to do so.

These 4 consequences of an iterative approach to a BI development then bring 2 important benefits to the organization.

  • The overall development time to deliver on the requirements is greatly reduced since the team worked with the business users to better understand and define their requirements through the use of a prototype. As such, development efforts were invested in well-known and understood requirements as opposed to speculations based on a limited understanding.
  • In addition, by focusing on the KPI that delivered the highest value – and not those who have low business value – while allowing for changes in the development priorities when the market conditions dictate such changes allow the development team to deliver value to the business users much faster than in a traditional approach where the users would only have visibility to the results at the end of the development project.

These 2 intermediate outcomes lead to 2 critical business value: reduced costs and improved ROI.

In light of the fact that a staggering 64% of systems functionalities are rarely or never used (Standish Group Study Reported at XP2002 by Jim Johnson), focusing on the requirements that were derived through the use of prototyping reduces waisted development efforts. As such, using a traditional development approach would have led to spend almost 3 times more time, efforts, and money on the real requirements.

In addition by having the development team select the real requirements with the highest business value ensures that efforts are spent on the tasks that will help the organization and possibly end the BI project once the cost of the remaining KPI exceed their estimated value.

Experience has showed us that changing the development approach to an iterative one does bring value to the organization. Hopefully, using such a diagram will make it easier to have such a conversation with the business community as opposed to within the IT department.

An iterative and incremental approach for BI projects

The un-impressive track record in the development of BI applications led us to conceive an improved approach to reduce the initial investment required to launch a BI project in order to deliver key performance indicators (KPIs) much more quickly and at lower cost.

The incremental development approach we came up with is aligned with the business priorities while allowing business users to further develop the knowledge of their business and better define their expectations.

As opposed to the traditional waterfall approach to building a BI application which is: plan the project, define the requirements, prepare the architecture, develop, test, move to production in a sequential manner, we propose to use an iterative approach during which we build layers of the entire data warehouse through iterations.

For example, instead of spending time in defining in a high level requirements, work with the end users to understand their business and the indicators they use to successfully manage it. Using an iterative approach, the objective of our process is to work backward starting with the required performance indicators back to the source systems.

An Iterative and Incremental Approach for BI Projects

An Iterative and Incremental Approach for BI Projects

As can be seen in the diagram, the intent is not to divide the project into phases where data is extracted from the source systems first, then move to a centralized repository and so forth. Instead of the traditional approach, we use a layer approach where each indicator is developed from source system to the presentation layer.

Although our approach requires a substantial paradigm shift for most people having experienced a traditional waterfall approach for BI
projects, the notions we propose are simple, logical and straightforward. This approach relies on AGILE principles and
more specifically on SCRUM practices that have been demonstrated to be successful in other types of initiatives. Our objective is to present them in a very concise manner in this post.

Our proposed approach to BI development can be summarized by the following principles:

  • Segment the development of components into small iterations of 3 to 4 weeks and frequently deliver key performance indicators at the end of each iterations;
  • Repeat the iterative process and use increments to add components to the BI architecture and infrastructure;
  • Prioritize the development of the key performance indicators that will deliver the highest value to the organization first;
  • Using a top-down approach going from aggregation to granularity, develop each KPI from end-to-end or from source systems to presentation layer;
  • Avoid “analysis-paralysis” forces by using a “good enough” approach in documentation, architecture, modeling, project management
    and other overhead activities;
  • Learn from each of the iteration and adapt the process appropriately to increase efficiency.

Our approach promotes a process that encourages frequent inspection and adaptation, a set of engineering best practices that allow for
rapid delivery of high-quality software, and a business approach that aligns development with customer needs and company goals. With this
approach, the project delivers in small increments with minimal planning, rather than long-term planning.