The following is a transcript of a live online chat hosted by Scott Ambler (Practice Leader, Agile Development, IBM Rational Software) and Per Kroll (Development Manager, Rational Methods, IBM Rational Software) held on April 13, 2007. Scott and Per dispel the misconceptions around Agile Development and explain how you can incrementally adopt Agile practices. christinelijia: My team has been reorganized under Rational recently. During the area meeting last week, agile processes have been mentioned as a key innovation. How to adapt to lean processes is put on my team's business goal as well. I am just wondering where there is any resources to help my team to use the right approach and adapt agile in a correct way? ScottAmbler: We're going to start in about 5 minutes ScottAmbler: There are both IBMers and customers invited to this chat tomyoung: Hi everyone, we'll get started in just a minute. tomyoung: I'd like to go ahead and get this officially kicked off. On behalf of IBM developerWorks, I want to thank you all for joining Agilistas Scott Ambler and Per Kroll for this chat. As you may know, IBM Rational just announced its Agile Development solution this week. To enable rapid global software delivery, IBM Rational provides technology, best practices, and thought leadership to make your Agile projects succeed with end-to-end automation, distributed teams, painless governance, a real-time project view, and faster software iterations. To begin, I'd like to have Scott and Per kick us off with a few words about the topic of our discussion, and then it will be open for questions. ScottAmbler: Just thought I’d start off with a brief prepared statement. I believe that agile software development (ASD) has crossed Moore’s “technology adoption chasm”. While Agile was once considered viable only for small co-located teams we’re seeing Agile approaches taken in much more complex situations. The improvements in product quality, team efficiency, and on-time delivery are motivating companies to apply agile development to a broader set of projects. Agile hence needs to adapt to deal with the many business, organization, and technical complexities today’s software development organizations are facing such as team size, distribution, compliance requirements, shared infrastructure, and so on. The inherent complexities require us to find ways to scale agile effectively. pkroll: To add what Scott said… so people have moved from “write all requirements on index cards and all models on whiteboards” to ‘you want to be lean and agile, but you also need to deal with all the business complexities’ that Scott talked about… This maturing of agile is necessary to make it mainstream, and also puts Rational solutions in focus. It requires looking at how to scale agile practices, and we have a lot of experiences helping customers with dealing with these complexities… tomyoung: Thanks for those introductions. I'd like to go ahead and open it up for questions now. ScottAmbler: Christine, a good starting point is www.ibm.com/rational/agile/ and for IBMers there is an all-in-one page on XL eskoviak: could u describe how u see the recent rational announcement in terms of agile suchko: This Agile stuff all sounds great to developers in the trenches but how do we "sell" it to the organization as a whole? christinelijia: do you have a more specific ASD that was used in some teams? ScottAmbler: Suchko, I think that the secret is to speak in terms that management wants to hear. In the June issue of Dr. Dobb's Journal I have a column on exactly this issue. Mgmt wants to hear about reduced time to market, improved ROI, ... business stuff, not techy stuff pkroll: eskoviak - with the announcement, we wanted to make sure that people understand what Rational has to offer in the agile space. we see that many companies are struggling with how to adopt 'agility at a scale' and we think we have a lot of good to offer.... ScottAmbler: Christine, do you mean methodology wise? christinelijia: yes. e.g. XP, RAD... ScottAmbler: Suchko, the URL for DDJ is www.ddj.com. The June issue should be online the second week of May Anjaribm: Members of Agile teams should be familiar with each other; practice makes perfect, how does this fit with global efforts / projects ? wsteger: How does the new Agile offering relate to the work being done on the OpenUp process? eskoviak: Our architecture group is attempting to "sell" to senior management the concept of agile--IT management gets it, it is the business that does not seem to. They still like mega projects ScottAmbler: Christine, many teams are using RUP in an agile manner and the OpenUP (www.eclipse.org/epf/) as well ScottAmbler: Christine, we're also seeing adoption of Scrum, particularly with RUP, as well as some practices from XP, Agile Modeling, Agile Data, ... pkroll: wsteger - OpenUP is one piece in the solution we offer. OpenUP is one of several processes offered through Eclipse Process Framework (EPF). We have also captured Scrum, XP, etc. We are also offering a commercial process product (RMC) and it extends and add value to EPF.... So, through this EPF/RMC platform, we provide guidance on a lot of practices and processes... As we move forward, we are e.g. refactoring RUP to be built as a set of extensions to OpenUP, so you can start with OpenUP, and then add the aspects of RUP you need... ScottAmbler: Anjar, the global environment clearly adds some communication challenges. Improved tooling around requirements management, modeling, defect tracking, CM, ... is definitely required. Earlier Per mentioned that index cards and whiteboard sketches won't cut it. He was definitely right CraigSuter: Scott - I understand about speaking businesses to the execs ... but the issue is tying a new development methodology to those concepts. I think the data comes anecdotally after the first project. That is my diffic tomyoung: For anyone just joining the chat, we're talking with Scott Ambler and Per Kroll about Agility at Scale. Please jump in with any questions you might have. ScottAmbler: eskoviak, a good approach is to ask management how well the mega projects have been going. Are they getting good value for the money? Are these projects on time? Are they truly implementing software which meets the needs of stakeholders, or only delivering software that reflects the specification written months or years earlier? If things aren't going well then perhaps they need to rethink their approach ScottAmbler: CraigSuter, there is some data out there. The agile conferences, and to a lesser extent the academic community in general, has some good research results to share. I've been running surveys with DDJ, see www.ambysoft.com/surveys/ which show that agile is being adopted widely. A recent survey, the data isn't available yet, seems to show that agile works really well in practice eskoviak: we're slowly getting there because the mediocre to poor value of the large projects is being realized. We feel we have a golden opportunity--but we need the first couple of iterations to succeed! bobbiea: does this approach also work for smaller projects that are more "enhancements to an existing system" not brand new. pkroll: Craig, Scott and I have also done a fair amount of work around 'agile governance'. I have found that the key things to win executives is to make sure that they feel that agile is disciplined (which it is) and that you can do more effective governance.... christinelijia: Hi, Scott, my team was (internal) audited for ISO 9001 from time to time. How should we pass ISO audit and at the same time adopt those light weight methodologies? ScottAmbler: Craig, other surveys by other organizations are showing similar results. I think my ddj survey was a bit better than most because I had a wide range of responses from both agile and non-agile folks pkroll: Craig, Scott and I are soon publishing a paper on 17 practices on agile governance... This type of messaging has really me when talking to various organizations / execs... ScottAmbler: Bobbiea, you can definitely use this with legacy systems. Michael Feathers has written a really good book on this subject, and in Agile Database Techniques I explore things from working with legacy data sources bobbiea: great thank you wsteger: Will the agile governance paper address best practices for managing multiple agile teams working on subsystems of a large scale system? CraigSuter: Per - the bottom line often comes down to proving in a business case that your alternative - agile or otherwise - has a quantifiable cost benefit over what is being done now. Most of the metrics I have seen in the literature are for small projects. CraigSuter: This governance paper sounds great ... when is it being published? ScottAmbler: Christine, I highly suggest that you pass the ISO audit with flying colors. Seriously though, there's nothing stopping you from being compliant with standards such as ISO and still being agile. The standards obviously constrain you, but nowhere in the standards do they tell you to be ineffective mcchesne: If you organization is used to making money on a large SW releases that come every 2 to 3 years ( new licenses and upgrades ) how do they sell the small Agile code releases that could come much more frequently? pkroll: wsteger - yes, it talks about some project level practices, but primarily focuses on organizational practices to aid in governance, compliance, etc. without introducing the overhead often associated with these things... So we discuss how to think about more effective governance bodies, how to coordinate a series of projects, and how to leverage iterative development for 'fact-based' governance... ScottAmbler: CraigSuter, if you go poking around the Agile Modeling site there's an article entitled "where's the proof" which might prove illuminating. If your organization waits for a mountain of proof then they'll end up so far behind their competition that they're only playing catch up by that point. If that's what they want to do then great, but they need to realize the implications of what they're doing ScottAmbler: Christine, what issues are you dealing with when it comes to ISO audits and agile? pkroll: CraigSuter - we will be done in roughly a month... we have not yet decided where / how we publish it, but expect it in May... hmaupin: Mcchesne, you needn't release every iteration except to QA. The multiple iterations avoid the end of cycle log jam, and make your 1-2 year timetables practical. ScottAmbler: Mcchesne, there is significant benefit to be had by short iterations because they reduce the feedback cycle and improve your ability to detect and address problems faster ScottAmbler: hmaupin, exactly ScottAmbler: hmaupin, you should also show your stakeholders your work on a regular basis to get feedback tomyoung: For anyone just joining the chat, we're talking with Scott Ambler and Per Kroll about Agility at Scale. Please jump in with any questions you might have. ScottAmbler: What other issues are people currently dealing with? ScottAmbler: Just thought I'd plug www.ibm.com/rational/agile/ once again. pkroll: Craig - good point, You do not adopt agile because you want to adopt agile. You adopt it because you want to gain certain benefits... I think there is overwhelming evidence for agile and iterative development. Check out Ch 5+6 in Craig Larman book "Agile and Iterative Development for Manager's". I am also working on a survey of effectiveness of agile practices with NC State, USC, and a few groups in IBM. All data looks really promising... 93% productivity increase for products moving towards daily integration and regressions testing, as an example... SMBurk1: What are the three biggest challenges you face when you try to get an large organization to adopt an agile approach? CraigSuter: Thanks Per ... I will get that book directly. ScottAmbler: SMBurk1, i think the biggest challenge is convincing middle mgmt to do it. Senior management is usually desperate to improve productivity and the people in the trenches want to get better too BAAndrew: Have you seen any examples where the social engineering aspects of agile have been successfully brought into a non-iterative environment? ScottAmbler: SMBurk1, another issue is convincing non-programmers to become more agile. You can really hobble an agile team by forcing them to follow the traditional data processes, the traditional QA processes, .... The data, QA, ... folks need to interact with the agile teams in a reasonable agile manner wfreds: Scott, most of what I've read about documentation for agile projects focuses on minimizing design and specs--but nothing about creating help for the end-user. how do we provide end-user assistance when there are no specs and the product keeps changing? jmorriso: Scott, if an organization has a lot of difficulty in adopting iterative development, does that mean that Agile is out of the question for them? Or, do you think that Agile somehow makes iterative easier to adopt? pkroll: SMBurk1, in the end, adopting agile and any other major process improvement is a organizational change effort. There is a lot we can learn from other industries on that topic... I love Kotter's 8 steps personally... But yes, I agree with Scott, middle mgmt is the biggest hurdle... ScottAmbler: SMBurk1, a third issue is giving people the support (training, mentoring, ...) that they need to learn the new approach. The rules for agile are definitely different. It requires a pretty big cultural change CraigSuter: I think the organization has to understand that adopting agile is more than a development team effort. The entire organization has to buy in and change their practices too. Otherwise, you have the same ole stuff being called agile ScottAmbler: wfreds, treat user docs like another requirement for the project. They should be estimated, prioritized, and created as appropriate. I highly suggest waiting until later in the project to do the docs once the system has stabilized. You can write user docs in parallel, particularly for stable aspects of the system, but you're at risk if you do it too early SMBurk1: middle mgmt FUD (fear, uncertainty, doubt) is definitely a big hurdle. But that means that senior mgmt must make it "safe" for middle mgmt to change :-) pkroll: BAAndrew - I do not have any great examples, but Alistair Cockburn has a lot of good writing on the topic... A lot of that stuff is basic stuff that makes sense, but we do not always do it, but agile has brought a lot of attention to the importance of social engineering... Treating people with respects has however not anything specifically to do with agile... ScottAmbler: wfreds, wrt support, how does Amazon, eBay, ... do support with their constantly changing product? They have a easy to use UI vcherw: What levels of testing does agile recommend within an iteration before you present components to a stakeholder? bobbiea: We have been using RUP for our projects for the last 2 years. I don't think it will be difficult to move from that to Agile but we also have been following the BTMS process. Does Agile replace BTMS or do we have to fit it in somewhere? wfreds: Scott, yes, I'm trying to focus on making the UI usable, including getting embedded assistance into it rather than writing more doc. but we always seem to be scrambling at the end to catch up ScottAmbler: vcherw, agilists do a lot of testing, far more testing that what we typically see in traditional projects. We do developer-level TDD and customer-level TDD if possible. This is confirmatory testing. I highly suggest having a small, independent test team doing exploratory testing as well ScottAmbler: vcherw, my January column in www.ddj.com covered agile testing strategies in a fair bit of detail vcherw: thanks BAAndrew: I wanted to agree with Scott about a couple of points. He's right in stressing the importance of seeing any kind of process change as changing your whole project team. We're implementing use cases and if you only focus on the business analysts and/or developers you're not going to be successful. You have to look at those who not only write but also those who use the use cases. This is even more important with something as large at a move to agile. pkroll: jmorriso - iterative is a core fundamental practice for agile development, no questions about it. However, you need to move to agile development incrementally, so maybe there are a number of other practices to start with. Some practices that will make it much easier to move towards iterative include developer / unit testing, good configuration management practices, ensuring a good dialog between customer and project, and so on... WNM: hi scott this is bill m - if you can make it fit in this dialog, i'd appreciate a few words on what IBM is doing in the agile business intelligence space christinelijia: hi, Scott, you've mentioned that often a few agile methodologies are used together. Do you see some drawback of using part of some practice? for example, with fast iteration and without documentation in place, the code could be very hard to read for any one new ScottAmbler: Bobbiea, agile doesn't replace BTMS but I bet agile could be applied to improve it. RUP can definitely be used in an agile manner and many people have done so. So, if you're already doing RUP then you might just need to tweak some stuff to be agile ScottAmbler: WNM, not enough yet ScottAmbler: WNM, BI is definitely ripe for agilizing. There seems to be an understanding within that community that they need to take an iterative approach to development, which is a start. They need to go beyond that and start adopting usage centered approach to requirements (e.g. use cases ), database refactoring, and database testing. BAAndrew: The other thing I wanted to agree with Scott about is the need for support other than just training. So much of learning a new process in learning the "art" of the process which cannot be successfully taught in a classroom. Mentoring and other support structures really help make such an initiative successful. I'll shamelessly include a plug here for my talk at the upcoming RSDC that will be dealing with one such method for this kind of support. It will be RA03. pkroll: Christine - in reality, few people adopt e.g. all practices in XP, but rather take a few XP practices and maybe combine with Scrum as a mgmt approach... That is natural... You do however need to make sure that the combination you put together make sense.... Iterative development without automated regression testing is a bit dangerous, as an example.... dlippert: We're doing user stories with ReqPro as a repository - our customer and development team are in two different locations. Any plans for Rational tools to better support this? ScottAmbler: WNM, I think that the adoption of use cases should be pretty easy for the BI community to adopt if they were to recognize the need to look beyond data. Refactoring will be hard to adopt because it goes against the false assumption within the database community that database schemas are hard to evolve. Database testing also seems to be a huge blind spot that needs to be addressed. We have our work cut out for us pkroll: Christine - another example is refactoring without high degrees of unit testing / developer testing.... hmaupin: christinelijia, good point. Fast does not mean un-commented. The code should be self documenting, meaning you take the time to comment incrementally, instead of a mad rush at the end to re-figure it out. The team must agree on acceptable practice in commenting their code, just for the reason you cite. Extraction tools such as javadoc can convert the code to more user friendly form, but the code is the core document. tomyoung: Once again, for anyone who just joined recently, we're talking with Scott Ambler and Per Kroll about Agility at Scale. If you have any questions, please don't hesitate to ask. pkroll: dlippert - we have done a number of things in the past, such as web clients which we have incrementally improved. I cannot talk about futures, but we are in general committed to supporting distributed development... Are you working with an IBM team on resolving your issues? They are in a better position to talk about futures, assuming NDAs, etc ScottAmbler: hmaupin, you're definitely correct. Agilists have a tendency to write significantly higher quality code, often because we're following common coding guidelines but also because we refactor our code whenever we detect quality problems. As a result the code is much more readable and easier to maintain ScottAmbler: BAAndrew, let's talk offline about an RSDC opportunity that I have for you. pkroll: We need more questions... Anybody has questions regarding agile adoption of RUP??? christinelijia: hmaupin, you are right. I think commitment from the development team is very important for any process. I agree with Scott that having a common coding guideline is also very important CraigSuter: Not so much a question ... but a comment! I have a hard time believing RUP can work with agile ScottAmbler: One of the things that you'll discover in the agile adoption effort, particularly at the beginning, is that it will be difficult for people to imagine how it works. Agile changes a lot of the rules by accomplishing the goals of development in different manners. CraigSuter: I know you sell the product SMBurk1: Per and Scott - could you share your thoughts on tools you would like to see for the future to support agile efforts, things which would make working with tools more like whiteboarding? dlippert: Per, we don't have any specific contact that is getting our input on what would be helpful tooling-wise. Not sure how to hook into the folks who gather Rational tooling requirements. We've had a Rational mktg person ask us to use our customer as a case study -- but our customer isn't overly happy with ReqPro (it is not meant for the WAN). ScottAmbler: CraigSuter, long before I joined IBM I wrote the book Agile Modeling (which you can never have too many copies of, BTW). In that book I wrote a significant amount on the topic of making RUP agile. To be fair, many organizations choose to implement RUP in a rather dysfunctional manner, but that's a choice that they make. Other orgs choose to implement RUP in an agile manner too. I suggest going agile. dlippert: We started with RUP and keep pushing for more agility. Our iterations are getting shorter. We switched from use cases to user stories, although I'm finding user stories are harder to organize. Any tips would be helpful. pkroll: Craig, well, I disagree... But let's start with the problem... RUP is a process framework, , and can be applied in an agile and non-agile fashion. I have written for years about issues we have with small teams taking the largest possible version of RUP and trying to adopt it. Then they say “RUP isn’t agile”. Dough?! We need people to choose lighter versions of RUP… Over the last years, we have produced RUP for Small Projects and RUP for Maintenance, and many other processes. See if those makes more sense for your project. And only apply what makes sense for your project, do not mindlessly follow RUP, apply what adds value to your project… PeteDaGuru: I worry that with more geographically-dispersed development going on, fewer Agile techniques will be able to be used. Not just because of tooling issues (network latency), but human latency of not having everyone sitting next to each other physically. Any thoughts about how this might be addressed ? ScottAmbler has left the room. Bill Hudacek: oops. BAAndrew: In an agile adoption of RUP in a large organization that is highly resistant to change (suffering from a certain level of change burnout), what is a generally considered best strategy: small-group adoption of the whole program, or best-practice-at-a-time of the whole organization? Or a combination of the two? I just can't imagine how to begin anything agile at my company. tomyoung: Scott inadvertently dropped off, but he's getting back on as fast as possible... sorry for the problem. pkroll: dlippert - right, so user stories needs to be organized... What about identifying use cases and using them as containers for user stories? In my latest book "Agility and Discipline Made Easy - Practices from OpenUP and RUP" (with Bruce Macisaac) I wrote a little on that topic, I think on practice 9... So, either organize user stories with UCs as containers, or do UCs and break them down into pieces that can be implemented in a few days... hmaupin: PeteDaGuru, this is an issue. The individuals on the team need to aggressively pursue communication. Pick up the bloody phone and call, use IM to clarify, conference in anyone that is needed. Tools like WebEx and other sharing techniques are useful for code reviews and debating specific topics. It really gets down to each individual initiating communication as if they are simply wandering over to a cube or desk to seek advice or ask a question. pkroll: PeteDaGuru - You may want to check out the future Jazz technology that we demoed at EclipseCon, JavaOne, and will demo at Rational User Conference. This has been developed by IBM Rational and IBM Research... When you log in, you see e.g. exactly who is logged in and working on your project, you see exactly what they do, if you are working with somebody, you can instant message them to ask them questions, and they can see exactly what you are working on, so you can collaborate in context of the work you are doing... There is a lot of other cool things they are doing... ScottAmbler has joined the room. hmaupin: pkroll, is this an Eclipse plugin, or is it dependent on the Rational suite or products? ScottAmbler: SMBurk1, a lot of people are using some of the existing Rational tools in agile environments right now. We've got some really cool stuff coming out soon, in particular Jazz is something to keep your eye on. pkroll: BAAndrew - Sounds like you need to showcase some early victory! Choose a team of people that are open to change, provide the right coaching, make sure the team is successful, and broadcast the success. Make sure that the project is important enough so people care, and make sure that you have people on the project that have the organizations trust... suchko: Is Jazz available anywhere yet to check out? tomyoung: I hate to say it, but we're pretty much out of time. We're going to close out these last few questions, then shut down. I'd like to thank everyone for joining, and thanks to Scott and Per for hosting the conversation today! dlippert: Per, but if the idea of user stories is that your customer can write them, then using use cases makes it harder for the customer to write. We are using business use cases for the business process and plan to trace user stories to them. PeteDaGuru: External site for Jazz community is http://jazz.net. It is still forming pkroll: hmaupin - it is based on eclipse, and focuses on team collaboration.... Developed by many of the key people that worked on eclipse platform.... hmaupin: Kewl! pkroll: dlippert - it is fine to organize by business use case...... You can also create a UC model in let's 2 h with your customer, and then use those UCs as containers... This does not prevent customers from writing user stories.... Anjaribm: Thanks. ScottAmbler: dlippert, user stories are very, very slim. You still need to do analysis, and that requires skill. The XP folks say that a user story is "a reminder to have a conversation with your customers" but what they're saying is do some analysis. So, user stories at best are the tip of the iceberg ScottAmbler: Looks like we're out of time. Just thought I'd yet again plug www.ibm.com/rational/agile/ . pkroll: Thanks everybody for all the great questions! ScottAmbler: Thanks for the questions. See you around on DeveloperWorks in the forums! tomyoung: Thanks all! For future reference, the transcript for this chat will be posted at www.ibm.com/developerworks/rational/chat/ambler.html and at www.ibm.com/rational/agile/ suchko: Thanks for the information!