Can you use Agile development for BI projects?
I wanted to summarize Tuesday morning’s session at the TDWI BI Executive Summit.
Larissa Moss (President of Method Focus) presented “Agile Development: What is it and will it work for BI?” where she explained that after failing at a DW project she was working on using the waterfall approach, she stumbled into Agile in the mid ’90s. After explaining what Agile actually is and how it differs from a traditional waterfall approach, she asked the “$64,000 question“, “Can Agile work for BI?” to which she answered “It depends what you call BI“.
Larissa argues that Agile BI works well for companies that don’t have to deal with the complexity of the data (i.e. don’t deal with building an underlying data warehouse). As such, she explained that using an Agile BI approach to do a data warehouse project where the development team has to deal with end-to-end solution including data standardization, data modeling and ETL cannot work due to the complexity and the un-known associated with the underlying data.
Larissa finished her presentation by proposing a solution to address this situation – Extreme ProgrammingTM which is a 16 steps Agile methodology she developed to address the pitfalls of the traditional waterfall approach. Her book called Extreme ProgrammingTM is scheduled to be released shortly.
Jim Hill (Director of Data Management at 1-800-contacts) then presented “Applying Agile Techniques to BI” and how his organization has been successful for the past 4 years in using Agile Development “for all software development projects and BI activities“.
Jim explained the process used by his business users to log their requests (user stories) which are then prioritized weekly and included in weekly iterations. He gave an example of 1-800-contacts’ Call Center BI which is deployed to over 400 agents’ desktops and for which the data is refreshed every 15 minutes.
Following Jim was Wyatt Weeks (Group Manager Business Intelligence at Sport Authority) who presented “Agile Data Warehousing at Sports Authority”. Wyatt told the audience that his team had chosen to use Agile Development for the first time on a “Large, high risk project with short time lines” and following their success, the team has continued to use Agile to deal with changing priorities.
Wyatt’s Scrum team consists of 7 to 9 members and all phases of their BI projects (architecturee, data modeling, ETL, and report development) are delivered through monthly sprints. He highlighted the benefits of Scrum: team collaboration, enforces communication and visibility, and clear definition of priorities.
The 3 speakers gave good presentations and agreed that indeed Agile Development can be used for BI projects as long as it is adapted to the reality and constraints of the organization.