How Agilent’s Global Software Team Doubled Its Productivity

Insights

In 2015, we undertook a lean/agile transformation at Agilent Technologies that roughly doubled our productivity over three years.

Along the way, we learned…

  • Leadership habits that enable success

  • The critical importance of releasing quality software on a repeatable cadence

  • That keeping commitments to customers is a competitive advantage

  • How to do coordinated agile development on a global scale

  • Effective ways to coordinate with internal stakeholders while focusing on our customers

  • Effective ways to work with outsource developers

We struggled with repairing field quality, delivering on a cadence, and building credibility and trust with customers and internal partners.

(See Insights for more detail.)

This was a team effort. Many people contributed to the success recounted here. For brevity, I’ll use “we” to acknowledge those contributions.

Five foundation steps to scaned and distributed agile development

Background

Agilent Technologies delivers Instruments, software, consumables, and services for chemical analysis, diagnostics and pathology, life sciences research, and food, water, and environmental testing. Headquartered in California, the company employs people and serves markets worldwide. Sales are B2B, and the company serves regulated markets including Pharmaceuticals and Healthcare.

Agilent’s Software and Informatics Division (SID) is exclusively responsible for enterprise software products. Its lab data systems support a range of Agilent and third-party instruments and are designed to be deployed in various configurations. When I joined the company in late 2014, these were all on-premise.

Competitors Waters and Thermo Fisher had newer lab software that was a better fit for enterprise customers – most of the installed base. It offered broader instrument coverage, was easier to use, and was localized in China, where there was rapid growth in analytical test labs. Dominance in enterprise software for labs meant they could charge a toll to connect competitive instruments like ours. The toll forced discounts on the prices of our instruments. Low-end competitors eroded instrument prices from below.

As a result, there was very little pricing power at the low and middle markets, and discounts reached levels that would embarrass a consumer audio retailer. It was clear to our CEO and sales leaders that winning in software was vital to owning our destiny. There was strong C-level sponsorship for the company to reinvent its software offerings and its organizational capability to deliver software effectively.

Assessing the situation

In the winter of 2014, the employees of Agilent’s Software and Informatics Division were stressed. They were struggling to deliver a modern replacement for their venerable but aging Chemstation and EZChrom software, and they had slipped their delivery date several times. They had persistent field quality problems with their legacy products, and the sales and support teams were skeptical of the SID team’s empathy for customers. Customers were annoyed and did not want to discuss new software: they saw quality issues and a history of inaction on past requests. Why spend more time talking if nothing will change?

I took the SID General Manager role in winter of 2014. My predecessor and his senior leaders intended to retire but were open to staying on to help me transition and hire their successors. The previous team had fought successfully to establish an independent software division in a hardware company and get its portfolio and budgeting consolidated. The next step was to build credibility and trust.

SID was a global team. We had two main sites in California and Germany, smaller ones elsewhere in Europe, and a significant outsourcing partner in Pune, India. At that time, about twenty scrum teams were working on new and legacy software – eight products in all. The division had its support team, but the lab software sales specialists reported to the company’s global sales leaders. I had been told to move the deployment team from within the division to the company’s services business unit.

The division’s people were intelligent, experienced, and motivated but stumbling in execution. The team’s ideas about what mattered in the market were at odds with the views held by sales and support, especially concerning the importance of serving the large installed base. We also heard frustration from long-time customers, especially those who had bet their future on products that were no longer the focus of investment with the push to deliver new software.

We served several markets. The largest were pharmaceuticals and the chemical and energy industries. The first was highly regulated, and required software that resisted tampering and supported regulatory audits. Chem/Energy needed software that was very easy to use and focused on fast cycle times and highly specialized analyses, sometimes at remote locations, conducted by technicians with minimal training.

We had underinvested in both of these constraints. As a company, we had a fragmented software portfolio with as many as fifty titles covering instruments in the lab rather than one (as our competitors did) for most of their routine tools, plus those of competitors. There was a great deal of catching up to do.

I resolved to take ninety days to observe, meet, and get to know people, begin to learn the business, visit customers, develop a strategy and plan, hire my leadership team, and write up our observations and a set of goals in cooperation with my boss.

There had been a great deal of focus on silo financial performance in the prior administration. Mike McMullen, the new CEO when I joined, asked us all to focus on our customers and to act like one company rather than a collection of silos. In a private meeting, I asked Mike what I should do if taking care of a customer meant a P&L hit for my business. He told me to do right by the customer, track the expense, and that he would find a way to make us whole if necessary. As it happened, I never had to make a sacrifice large enough to call in that promise, but it was encouraging to know that our CEO was willing to back his talk with cash.

Problems to Solve

We found several issues holding the team back, including fear of failure, execution issues, and a lack of customer contact and empathy.

  • As a software business in a hardware company, we found that software was an afterthought in accounting models and other sources of internal evidence that worked well for hardware sales. Leads in our CRM system were tied to hardware SKUs rather than software unless the deal was software only, so we had no win/loss information other than anecdotes from sales reps. We could not know what software was attached to what hardware or where our installed base was. We could see sales by quarter, sales region, and product or product line, but that was about all. We needed data to understand our sales and our market independently

  • Persistent field quality issues and lack of responsiveness eroded trust with the sales and support teams. As a result, sales people tended not to want our product marketers to go on field visits with them.

  • We had multiple systems for tracking work, so there was no easy way to see all the work requests and commitments in one place. The system that tracked support tickets was a rich source for defect reports and other customer information, but it did not connect to the systems of record used by the developers. Nor did the company’s strategy. Developers used at least two systems, and our outsource vendor used another for tracking work.

  • Some dev teams used Kanban boards, while others operated on Scrum or another agile framework. Each team had its own idea of what “agile” meant and how to do it. Estimates were not comparable from one team to another, so it was hard to understand how much work capacity was available or what could fit into a release. In small-scale agile development, the canonical unit of estimation is a “story point,” and industry dogma is that a story point is an arbitrary unit. This makes it impossible to estimate how much effort will be required to deliver a capability that spans agile teams or sprints. Portfolio-level thinking is a grind under those circumstances.

  • Builds required multiple transatlantic hops and took a week if they succeeded. Builds failed at least as often as they grew, and with the long cycle time, it could take even longer to track down the problem and fix it since developers had moved on to other work by the time the build failed. Tests were mainly manual, and because of the integration issues, they tended to happen late and, therefore, had a high surprise factor. The overall result was that the system wasn’t always integrated. It was hard to do demos to get stakeholder feedback for future iterations, and there were plenty of late-breaking surprises to contend with since integration tests happened late and there were fewer demos than desired.

  • Long Manual test cycles dominated the end stage of any release. Exacerbating long and uncertain build cycles was a lack of a quick and definitive smoke test to help developers find and fix their own issues immediately rather than wasting precious test time.

  • Persistent field quality problems with legacy products meant the product development team members were constantly interrupted with fires to fight. Still, because of schedule pressure, the team made relatively cursory efforts to deal with them. Some serious field issues had aged two or three years without resolution. The result was frequent interruptions from negative customer feedback, an angry support team, and frustrated sales reps. This made it even harder for developers to meet other commitments. Nobody was happy.

  • The team feared bringing bad news to leaders, so they downplayed risk and uncertainty, so bad news often exposed too late for leaders to marshal help.

    There was a tendency to expect the Quality team to “get out of the way” rather than hold R&D accountable to high standards and drive improvement based on evidence. This helped drive persistent field quality issues.

    We had a major schedule slip a month or two before the promised release date of the new software. This seemed to be a surprise even to the program manager.

  • The team overused email, and this caused decisions to eddy even at the developer level. This seemed to be a habit in Agilent, which had previously rewarded silo efficiency and “accountability” (loosely translated as punishment for failure). It seemed that people felt their job was done when they passed an email request on to someone else rather than when the problem was solved.

  • We lacked clarity on the scope of the work. Stories were often confusing or bigger than a team could complete in a sprint. There was a perception that features and scope were more important than time to market. Quality and reliability tended to go begging. Our Product Owners had not coalesced into a mature practice, and story quality was not what it needed to be.

    Some of our Product Managers and hardware business partners felt anxiety about replacing trusted legacy applications. I heard "if we don't have everything, we have nothing" - they feared nobody would buy our new software if it didn't do everything the old software did.

    We would go on to prove this untrue, but it caused people to overemphasize scope early on. What was true: there is a huge amount of work required to duplicate all the good parts of a legacy product. There are faster and lower risk strategies for modernizing legacy code than a complete rewrite.

  • There was a great deal of reliance on expert opinions and not nearly enough evidence available to evaluate them. Many employees had decades of experience, and had earned PhDs in related fields. They had grown up as chemical analysis or life sciences specialists. Our customers were increasingly not specialists, and purchasing power at major accounts had shifted from scientists to Procurement and IT departments. The organization reflected the market of two or three decades ago, and were slow to change thinking to reflect these new conditions.

  • We had underinvested in developing sources of evidence of our own, too. Our ancient licensing system was obsolete, coarse, and poorly instrumented. It required manual administration. There was little ability to control feature mix in the software based on the license. We also discovered that there were license leaks when we delivered upgrades. The choices the team had made twenty years ago in the license system were still locked in, and hampered the marketing team's ability to deliver or even think about new business models. We couldn't see how our software was attached to other orders, or vice versa

The Road to Better

Build Trust and Credibility

It was clear that we needed to build trust and credibility, so our first step was to improve our predictability – to deliver what we promised and get field quality issues under control. Knowing that we had to delay the launch of our new product, we decided to take the time to restructure and transform the way we worked with the expectation that this delay would pay off later.

Five Key Values

In our shared conference room I hung a handwritten poster emphasizing five key points for us to bear in mind as we went about changing the way we worked. Predictability, Teamwork, Evidence, Simplicity, and Customer focus became the principles that drove our transformation. In our operating plan we defined and expanded on these ideas. 

Predictability

Predictability comprised three aspects. First, doing as we said we would do, or giving timely warning if we were in trouble. Second, a clear way for stakeholders to request work from us, with a transparent decision process. Finally, meeting our customer commitments. That also meant being careful about what promises we made. It's OK to say "no" as long as there is a good reason. Note: many people have interpreted “predictability” to mean being dead accurate on cost and time estimates, even in a scope driven environment, and therefore a death march. I’m looking for a less freighted term, but meanwhile am using Predictability to mean the three characteristics above in the context of frequent, high quality releases on a repeatable cadence, where scope is treated as a dependent variable.

Poster with "Predictability, Teamwork, Evidence, Simplify, Customer Focus"

Five key values for improvement

Payload improvement over 6 releases

Teamwork: Less Email, More Collaboration

Observing Agilent’s tendency to overuse email, we asked people to converse directly in person or via audio or video calls whenever possible, to speed decisions and results. It looked like part of the silo culture led people to think in terms of checking off their own to-do list rather than solving problems. We needed to get our people to own problems and solve them together. That began with modeling those behaviors ourselves, and supporting the effort required to investigate customer needs and address issues.

Becoming Evidence-based

We were a company of very bright people who had grown up in labs like those of our customers, so there were many strong opinions, but often those opinions were out of date or not informed by a broad perspective on the market. We had to get people to develop and rely on evidence, recognize conflicting opinions, and ask for evidence to make well-informed decisions. This helped take egos out of discussions, and gave more agency to people who had not had as much customer contact. Relying on evidence wherever possible also created the psychological safety the team needed to emerge from its conflict-averse prior culture. We leaders also did our best to model candor and show the team that speaking up was encouraged, that good ideas could come from anywhere, and that bad news delivered in a timely way was a good thing.

Putting Customers First

The team tended to tolerate or even embrace complexity rather than look for ways to make things simpler for ourselves and our customers. There was a definite tendency in the company for divisions and departments to act independently to optimize their perceived interests even when it inconvenienced customers. This spilled over into product design: it was a major win for the company to adopt a common industrial design for the power switch across all instruments. Our software reflected this – it was highly flexible and capable for specialists, but not easy to use for enterprise customers who increasingly staffed their labs with less expensive techs and needed a simple and common way to run most of their lab workflows. The market was moving rapidly toward labs staffed this way, with supervisory roles reserved for specialists. We needed to catch up. Customer focus means putting customer interests ahead of parochial ones, and getting out of the building to understand more deeply who our customers were and how they worked with products like ours. Customer collaborations eventually became a source of growth and learning for us. 

Taking Action

One Place to Track Work, One Way to Work

After setting out these principles, we decided to have one place to track all our work. We had too many systems of record and multiple approaches to development in our team. From a combination of kanban and “agile in name” approaches we decided to converge on doing Scrum one way across all teams, and engaged Agile and Product consultants to help us develop a common practice. We included our outsource teams in this training, and required these teams to use our tools, systems, and processes - which meant partnering with our IT department to create the necessary connections and authorizations. We chose two-week sprints and a rather long nine month release cycle initially, and trained all of our teams the same way. We migrated all of our work into Jira and Confluence, and began to discuss connecting our support desk system so that tickets could be linked to defects.

This is a short description of a heroic effort on the part of the team at SID as well as our partners in IT. Having one approach and one tracking system meant one dashboard for all that work. It enabled a consistent review format from program to program.

From Project to Product

Culturally and procedurally, our team and company operated on a project model. A new release effort started with the Product Marketing team defining what they considered to be a minimum scope necessary to make a return that the company needed to justify the investment. Because releases were frequently delayed – in part because of the insistence on fixed scope – marketers often were unsure when they would get another chance to get a capability to market. As a result, they tended to ask for everything when they had the chance. This helped perpetuate a vicious cycle of underestimation and slips. One of the hardest things I had to do as GM was to drive the team toward putting Quality and Cadence ahead of Scope. It was worth it.

It was difficult for the team, used to the old way of doing things, to come to terms with the idea that we would sacrifice scope in order to hit a release cadence, or even more improbably ask the developers to stop planned work and help customers resolve issues with our software. As it happened, exposing developers and our leaders to hot customer situations, sometimes by flying teams to customer sites, was a revelation for many. They came back with enthusiasm and a greatly improved first-hand understanding of why our products mattered and how they were used in real life. Spreading customer insight into the development team made story writing simpler and led to both better design decisions and more give and take between developers and product owners. 

My senior leaders and I had to work hard to convince the team that Scope must come after Quality and Cadence in the canonical priority list of development. In hindsight, this was one of the biggest struggles in making the change succeed, and it was likely that teams would easily revert to prior practice without ongoing leadership commitment. 

If you knew scope might get ejected late in a release, you would do some things differently. A first change would be to attempt to complete as much as possible as early as possible in the release. The more scope is complete, the less risk there is of having partially finished work at the end. So rather than doing fifty things in parallel for the whole release cycle, a good adaptation might be to focus and finish ten things at a time, five times over. Knowing that some scope might have to be ejected, it’s also a good idea to have work in progress that is always in a usable and tested state, so that partial work can be released. This is a core idea of agile development, but teams may fail to do it unless they have very good habits. One way of enforcing these habits is to have a very frequent and regular demo cadence, and to require that stories pass testing before they are considered done. This is part of good “Definition of Done” hygiene.

Program Office

We created a program office tasked with improving our business processes and leading interactions with our many stakeholders in hardware divisions – the beginning of portfolio management. Our program director led monthly Portfolio Management meetings to create a transparent way of requesting work from us, negotiating, and communicating decisions. We encouraged the senior leaders of other divisions to participate and help develop clear priorities. It wasn't easy, but we settled into a rhythm as our credibility developed. 

The program team also led our large Sprint Zero meetings and the planning sessions that led up to each Sprint Zero, kicking off the feature release cycle. One of the critical roles of the program office was to work with stakeholders to get a clear ranked feature list far enough ahead of each Sprint Zero that there was time to negotiate a scope that would fit in our available capacity. This involved a great deal of back and forth as initial estimates often exceeded our capacity by a factor of three! Over time, the team got better at planning and estimating – only up to the “fill line” – to avoid wasting time planning things that had no chance of fitting in the release. 

Customer Commitment Focus

One immediately helpful change was to create a marketing role responsible for meeting customer commitments, which included resolving defects in a timely way. I asked the leadership team to model commitment to quality by attending a brief weekly conference to direct action on critical issues. Our incumbent quality leader took the role and flourished in it, and I had the opportunity to promote another quality leader who had not been immersed in the “get out of the way” culture but instead was passionate about building a forceful quality team and driving improvement. We set a goal to release a patch for 80% of critical or serious field defects within six weeks as a way of formalizing our commitment and measuring progress. As it turned out, we were so successful that we had to change the goal. There were so few outstanding field defects after the first couple of releases that it was hard to meet our target.

Regulations and Validation

Our industry was just beginning the transition to cloud computing when I joined Agilent. Lab informatics software was mission-critical for production in most of our customers’ labs, and they were very cautious about making changes that might disrupt work. Lab and IT managers had to do a great deal of testing and documentation when they deployed an upgrade, but still wanted bugs fixed quickly with no extra baggage to test. We took care to distinguish between releases containing only bug fixes, and releases that contained new features and capabilities (Feature Releases, which were numbered from FR-1 onward). Bug fix releases could be tested at much lower effort as long as customers trusted us to have good release discipline. We chose a nine month feature release cycle initially as much because of our own build and test cycle times as the market necessities of on-premise software. Later, we found ways to shorten those cycles dramatically, but the market still needed what it needed.Paying Down Quality Debt

We asked one of our engineering directors to lead a program to work down our quality debt of aging field defects in legacy products. We dedicated a team of developers to this and asked them to visit customer sites to gather the information they needed to identify root causes. We had developers and managers returning from field visits as heroes for our customers, with new insight to bring to their work. It felt like we won in more ways than working down the quality debt. Having developers with a deeper customer understanding made the work of Product Owners simpler too. The net result of our efforts to pay down quality debt was renewed credibility with key accounts, and with our sales and support people. This led directly to growth in upgrades, and reduced the amount of time our team was spending fighting fires, increasing productivity.

Product Lifecycle

Our Quality director made changes to our Product Lifecycle that specified how our agile development process fit into the traditional gated process that was standard at the company, and which was necessary for us to meet our regulatory commitments to our customers. We found a relatively simple way to achieve this that gave us the freedom to do periodic releases without excessive documentation. 

Investing in Architecture

We found that our legacy products had outgrown their designs and toolchains. In many cases we had products stuck on old versions of compilers or key libraries because of underinvestment in architecture and devops. We saw this issue beginning to develop in our new product as well. There were ad hoc decisions intended to get the product to market that had led to performance issues or that made it more difficult to adapt our learning about the market once we had launched. To correct this we decided to create an architecture group. Our intent was to signal our commitment to being deliberate about architecture in our products, to develop a culture around architecture, and to establish a clear architectural vision for our products that was tied clearly to the direction of our markets. We wrote architectural reviews into our Definition of Done -- effectively giving the architects the power to shape releases if they needed it. Over time the architects developed what we came to call our target architecture and a roadmap to achieve it. We allocated a significant fraction of our development capacity in each release to implementing capabilities that made progress toward our target architecture.

Unbreaking the Build

Initially we were lucky to get one build done per week. Our buildmaster installed traffic lights over each scrum team's work area to show where the build had broken. If your build passed successfully, your light would be green. If you broke the build, everyone could see that red light. It was amazing how quickly broken builds got resolved after that. We still had a long build cycle, but overall we moved to several builds per week. 

Over time we set up a devops team and asked them to cut build cycle time with the goal of having a build plus smoke test take no longer than a coffee break. We also invested in improving our automated test coverage at multiple levels – unit, functional, and integration testing. The QA team used low-code test development tools for UI and integration level testing, and our developers did the lower level work. The devops team came up with ways to automate the creation and loading of virtual machines to perform load testing. They brought test coverage up and cycle time down to about thirty minutes over the course of a couple of years.

Product Owners and Story Quality

Product Owner is the most challenging job title in agile development. A PO must be insightful about the end use of the product and its domain, while also being able to translate requirements into clear, concise stories that developers understand and estimate. They must work well with both developers and marketers. They must be good negotiators and have enough grasp of business to be able to identify value and work with evidence to justify their backlog rank decisions. We did not have a strong PO community initially. Story quality and customer connection both needed work. To address this, we organized PO’s under a single leader globally and worked to create a culture around product ownership practices and continuous improvement.

Developing a Capacity Model

In our first Feature Release (FR-1) we saw that the amount of work requested of us was at least three times what we had capacity to complete in the release, but it took a great deal of estimation effort from our most senior technical people and POs to realize that. Part of the problem was that we did not have a way to estimate how much capacity was actually available to assign to new requests. As a result, we wasted precious effort estimating work that had no chance of fitting in the release, and set incorrect expectations about what could be accomplished. I asked our Program Director to lead the development of a capacity model so that we could do a better job estimating those features that would actually fit next time. 

We had a single system for tracking work and work requests. We decided to buck the current teaching in agile development and ask all of our teams to use a reproducible unit of work so that we could compare estimates and add them up. We defined a story point as the work a developer could accomplish in a day. From this foundation we were able to create a relatively simple model that gave us a good estimate of the capacity to deliver stories to market, as well as an estimate of where the rest of our capacity was going. Initially we found we had only about 30% of our capacity available for discretionary work. The remainder was to address quality issues, both field defects and defects found in manual testing, to buffer inaccurate estimates, to compensate for time lost to attrition and personal emergencies, to maintain legacy products (quality issues again mostly). This was alarming, but also an indication that there was an opportunity to improve our productivity greatly for no extra cost by addressing some of the issues. The capacity model not only made these opportunities visible, it quantified improvement as we addressed them. Over the course of the next four releases we got our “payload” capacity up to about 65% by addressing many of these issues, starting with field quality

Faster Cycle Time

As we improved we wanted to reduce the planning burden and emotional toll of long release cycles. The emotional toll came from the understanding that if some request was not included in the next release, it would be a whole release cycle before we made another decision, and two cycles at best before that capability might become available. We thought that if we could get a release cycle down to three months (an eternity by SaaS standards), we could relieve this burden and greatly reduce the planning overhead as well. To do this efficiently we had to reduce overhead in our releases. This included the number of sprints reserved for final testing once the code was frozen. We divided our objective into two sub-goals: reduce the time required to run a build and smoke test cycle, and increase test automation to improve our confidence that code that passed the smoke test would integrate well with the rest of the system. 

Over time, the DevOps team were able to reduce the cycle to forty minutes – about a hundredfold improvement over the initial state. They achieved this mainly by profiling the build and relieving bottlenecks, using containers to automate the deployment of virtual test configurations to remove manual effort from test configuration, and making good choices about what work to do in the quick build and what could be done less frequently.  Our test and dev teams grew smoke test coverage continually during this time as well, resulting in a much lower defect burden on manual test.

Results

By FR-6 we achieved 65% payload thanks to improvements in field quality, reduced outsource turnover, and improved developer productivity from more completed work and less planning waste. We continued to scale the team globally and where we had started with two large employee sites and one large outsource vendor, we entered FR-6 with a third large employee site and two more outsource vendors to add scale. 

Credibility and Trust with Internal Partners

Credibility and trust were critical to our ability to gain cooperation from our internal partners. Because of the changes we had made, we became the clock that our partner divisions used to time their own releases. Because we were open to good ideas from anywhere and seemed to know what we were doing, we were able to build an outstanding team in Australia to partner with the division there, and the GM of that division made financial sacrifices to fund it.

Credibility and Trust with Customers

Credibility and trust became a competitive advantage as well. Our customers trusted our ability to deliver what we said we would, and mistrusted our competitors on the same scale, that they often treated a feature planned for our next release as if it already existed for the purpose of making purchase decisions. This was a very powerful competitive weapon, a direct result of our emphasis on a predictable cadence of high quality releases. 

Rapid Business Growth

We realized direct business benefits from our transformative efforts. OpenLab CDS achieved 65% compound annual growth over the five years from initial release to FR6. It was one of the most successful product launches in company history.

These achievements happened because of the dedication and effort of many people at Agilent, especially in SID. It was an honor to have had the opportunity to lead during this time, and a source of lasting satisfaction to me that we achieved great business results by doing well for our customers.

Copyright © 2023-2025 John W Sadler Jr - All Rights Reserved