Origins of the Product Manager vs Product Owner Dilemma
Introduction
The product management domain and the Product Manager role in the software industry were thrown into flux with the emergence of Agile/Scrum software development practices and the accompanying Scrum Product Owner role.
The upheaval in software companies that committed themselves to Agile/Scrum was initially relegated to confusion on how the traditional Product Manager and the new Scrum Product Owner roles should be reconciled. Are these two roles essentially the same? Are these two roles fundamentally different? Should or can these two roles be integrated into one capacity? Should one role be canceled in favor of the other? Can one person assume and successfully fulfill both roles?
At many software companies, the Product Manager vs Product Owner dilemma was ultimately resolved by officially abolishing the Product Manager title and having only Scrum product owners, or by redefining and expanding the Product Manager role to also include the responsibilities of the Scrum Product Owner role.
This review presents the events and circumstances which elevated Agile/Scrum to its high level of popularity, provides an alternative critiquing viewpoint on the merit of Agile/Scrum practices, and explains and analyzes some of the impact Agile/Scrum had on software companies and on the product management domain and the Product Manager role.
Business Motivation for Faster Delivery
There is a business backdrop that helped Agile/Scrum software development practices become so popular. Business executives are always looking for better ways to improve their businesses. It could be the employment of new technology, the application of more efficient internal processes, the acquisition of cheaper labor or reduction of labor, or just about anything that can legitimately help a business succeed in achieving its business objectives.
The motivation to change a company's internal processes and use a new method is due to the perception that the new method promises to deliver a combination of gains in time, cost, and quality. In business, that promise which is made explicitly or implicitly allures the business stakeholder with a pledge to ultimately decrease costs and/or increase revenue.
Affecting the decision making process at companies is the fact that the world we live in is quickly changing and becoming more dynamic. Customers expect value, innovation, and technological advancements, which they take for granted and want fast. Consequently, rapid delivery has become a perceived necessity for the survival of companies and this market dynamic has put more pressure on company executives to find ways to more rapidly deliver products to market.
The release of the document known as the Agile Manifesto was timely and it also tapped directly into the increased business motivation of accelerated delivery.
Timing is Everything, The Agile Manifesto
The Agile Manifesto, a proclamation document listing preferred values and principles related to custom software development, was published in 2001. The context of the Agile Manifesto was custom software development and its objective was to provide a better way of building custom software for customers in a manner that would be productive and would reduce undue negotiations with the customer.
The Agile Manifesto is an attempt at creating a Framework, a basic conceptional structure - a collection of ideas. Frameworks are overwhelmingly theoretical and primarily comprised of organized recommendations which pertain to a particular aspect of doing business.
The Agile Manifesto is a simple, perfunctory framework document that does not provide any revelation or any innovation whatsoever. There are thousands of similar book chapters, articles, and blog posts to be found that list ways, tips, rules, guidelines, etc. to better deliver software. The Agile Manifesto's espoused values of communication, collaboration, discovery, and experimentation are all worthy values which one will find being applied in many industries and institutions throughout the centuries.
The Agile Manifesto's background is rooted in lightweight software development. The concept of lightweight software development has been around under different names since the 1960s. It is based on iterative and incremental development principles, and promotes continuous planning, development, and testing work. Modern lightweight software development began to re-emerge in the late 1990s with method variations such as Extreme Programming (XP), Crystal Clear, Scrum, and Feature-driven Development (FDD).
Two of the co-authors of the Agile Manifesto, Ken Schwaber and Jeff Sutherland, had collaborated during the late 1990s to create Scrum which is a lightweight software development method. In 2001, Ken Schwaber and Mike Beedle (another Agile Manifesto co-author) had published a book titled Agile Software Development with Scrum. These two publications and several other coinciding events had caused the concept of Lightweight Software Development to become synonymous with and recognized as Agile Software Development, and also to be heavily associated with Scrum.
The timing of the Agile Manifesto's publication and the release of Schwaber and Beedle's Scrum book during the dotcom meltdown was perfect. Everybody was reeling from the effects of the late 1990s dotcom implosion and the ensuing early 2000s high-tech sector recession. There was a void in the market and an ongoing search for a faster delivery method that would help to quickly bring products to market and do so faster and with lower costs than the software development methods that were used during the dotcom era. Lightweight software development methods were viewed as possible candidates.
The Agile Manifesto, the Scrum book, and the Scrum method itself were perfectly positioned time wise to offer an alternative and supposedly better way of developing software. The success of the Agile Manifesto and popularity of Scrum were overwhelmingly due to happenstance and circumstance.
The Rise of Scrum
For many software companies, the search for a rapid delivery technique culminated in Scrum. Capitalizing on the aforementioned supporting events and relations, Scrum became the most popular Agile-based software development method with over seventy percent of Agile adoptions being Scrum. For many people the terms Scrum, Agile, and Agility all became synonymous, intertwined, and interchangeable.
Selecting Scrum as the company's new software development method was easy for many company executives because of the wholly unsubstantiated promises, such as "With Scrum you build a prototype in three months whereas with Waterfall it will take a year", that were being circulated and propagated by people and companies with a vested interest. There was also the bandwagon or herd effect, a situation where newcomers blindly adopt trends because others have merely done so before them.
Yet Scrum is remarkably crude with obvious undertones of generalization, relabeling, and sports culture.
For example, Scrum generalizes the most fundamental components of a development method such as documentation by having one product backlog document instead of several requirements and project documents, and by deconstructing specialized roles and replacing them with homogeneous developers and one generalized project manager titled product owner. The Scrum Product Owner, a key role in Scrum, is just a renamed and generalized title for a Software Development Project Manager (with some added responsibilities under Scrum).
Scrum also relabels the most fundamental components of the development method. For example, documents are renamed to be artifacts, meetings renamed to be ceremonies, and postmortems renamed to be retrospectives. Nothing is really new or even improved.
Furthermore, Scrum is seemingly patterned after American popular sports culture. The concepts of Scrum are very similar to American football with downs (sprints), quarterback (product owner), coach (scrum master), team huddle and locker sessions (ceremonies), etc.
All this generalization, relabeling, and sports game patterning makes Scrum conceptually very simple. Indeed, rapid delivery requires simplification and more peer collaboration. Processes must become more simplified and adaptable. However, the increased fluidity, simplification, and speed in delivery nearly always mean less precision in the product development process and in the resulting product feature set.
Scrum is simple and easy to grasp, which is an advantage and part of the reason for its popularity. Scrum is so simple that any novice only needs to undergo a two-day course to be trained and certified as a Scrum Master.
But Scrum does not provide any clarity on some extremely cardinal topics which would indicate how to govern Scrum and how Scrum should fit in and interface with other methodologies being practiced at various company departments such as marketing and product management. For example, does Scrum attempt to encroach into the problem space, reside solely in the solution space, or outright own both spaces? There are so many voids in Scrum at every conceivable level, conceptual to operational, which render Scrum as markedly incomplete.
Exacerbating the already flawed structures and missing information in Scrum is the fact that the authors of Scrum occasionally used colloquial or amorphous language in their books and writings. Consequently, the situation of Scrum becoming popular but simultaneously being unclear and incomplete had overnight given rise to an army of Agile/Scrum consultants, advisors, and experts whose job was to interpret Scrum for the masses, just like clergymen explaining scripture to their flocks, and help software companies implement Scrum.
Scrum, by its own design, was successful only in proving itself as a flawed and incomplete method for software product development. Although it may seem counterintuitive, Scrum's practices are actually restrictive and are inappropriate for managing complex business and technical undertakings.
Ultimately, Scrum did not yield the promised results with about fifty to eighty percent of Scrum adoptions failing.
When a food recipe consistently fails in numerous restaurants around the world, people eventually realize that changing the geography, kitchenware, chef, and kitchen staff will not help since the problem is the recipe. When it came to Scrum, many people, particularly the Scrum consultants, were for obvious reasons reluctant to blame the method for its own failure to deliver and instead blamed the practitioners for their supposed lack of commitment, lack of discipline, and an unaccommodating company culture. The misguided explanation often repeated was "Processes do not cause failures. People cause failures."
In the brilliant Zero-defects Code memo, Chris Mason, the author and a Microsoft developer, states "The point of enumerating our problems is to realize that our current methods, not our people, cause their own failure." So while clearly individual personalities and business cultures always have a bearing on a project's outcome, so does the process. A bad process can corrupt anything and anyone.
The practitioners themselves did not like getting blamed for Scrum's pitfalls and the executives were disappointed that Scrum had failed to deliver the promised improvements. But at this point in time the damage had already been done.
The Impact on Product Management
Scrum fundamentally harms the product management domain and the Product Manager role in the software industry. For technical people, technical knowledge and technology are highly valued and considered the company's core competency. The authors of the Agile Manifesto and Scrum are very technical people and that is reflected in their notions in which product development is the core and other corporate functions, such as product management, are overwhelmingly considered to be a part of or subservient to product development.
Based on this mindset, it did not seem outlandish to the authors of Scrum and its supporters to opine that product management is required to conform to and apply any product development framework presently being used by the development team, be it Waterfall, Agile/Scrum, or anything else.
Furthermore, in their Scrum Guide book (2012), Schwaber and Sutherland explicitly advocate that product managers be retrained to become Scrum product owners. This guideline alone had an immediate detrimental effect on product managers and the product management domain being practiced at software companies. At the individual level, many product managers, and other people assuming other critical roles which were folded into the Scrum product owner role, lost their jobs as their roles were not officially designated in Scrum. At the company level, crucial product management practices which are so critical to a company's commercial success disappeared because there was nobody at the company to practice them. There were additional negative ripple effects that touched product marketing and user experience.
Regardless of whether their original job title was retained or modified, many product managers were converted into product owners. Yet again, the blurring effect had taken its course and now Agile/Scrum, a lightweight software development method, was considered to be a form of product management. This introduced and caused the phrase Agile Product Management, the epitome of an utter oxymoron, to become prevalent. The road to coining the new job title of Agile Product Manager, a euphemism for Scrum Product Owner, was short.
Scrum and all the other Agile software development methods are not product management methodologies since the product management domain resides in the problem space (identifying and articulating market problems from a product planning perspective). Anybody practicing Scrum or any other Agile software development method is invariably in the solution space. Accordingly, a person assuming the capacity of a Scrum Product Owner must be a member of the technology/development/engineering team.
The converted product manager now found himself constantly serving the development team and acting as a Scrum Product Owner, a managerial role that according to Scrum is responsible for defining, prioritizing, and maintaining Product Backlog Items (PBI) which are contained in the Product Backlog. The problem is that by Scrum's own definition the Scrum Product Backlog contains everything related to the product, and therefore by inference the Scrum Product Owner becomes responsible for everything related to the product. This is a formula for an unavoidable failure because the product owner role cannot intrinsically scale to accommodate complex or large software development projects.
The Impact on Software Companies
Scrum negatively impacts software companies and their employees. Especially notable is the rudimentary nature of Scrum which impedes software companies who are engaged in developing intricate and complex software products or serving sophisticated markets.
In the short term and because of the adoption of Scrum, people were fired, retrained, and relocated. In the long term, the move to Scrum fundamentally altered the software company's processes, culture, and mindset, often without betterment. But there are alternatives.
Software companies that are adversely affected by Scrum go through a process of realization, recognition, and then stages of phasing out Scrum and phasing in a different method. Scrum's pitfalls have created a market dynamic to replace Scrum with better thought-out alternatives such as flexible implementations of Waterfall, Scaled Agile Framework (SAFe), Disciplined Agile Delivery (DAD), and also adaptations of Rational Unified Process (RUP).
Needless to say, the new Agile methods have all capitalized on the experience drawn from Scrum to bring back very clear delineations between product management and product development, reinstate and recognize the Product Manager role as being wholly distinct from any product development role, introduce leveled documentation, promote role specialization, and more. All of which are contrary to Scrum's principles.
Summary
Henry Louis Mencken, a shrewd American writer and journalist, said that "For every complex problem there is an answer that is clear, simple, and wrong." Scrum is that clear, simple, and wrong answer to a strong need in the software industry for a rapid delivery method. After more than a decade of experience, even the staunchest Agile consultants have come to terms with reality and moved on to focus on other forms of Agile methods and principles instead of Scrum.
The Agile/Scrum experiment hurt many in the software industry. It had a profound negative impact on many people's careers, completely skewed the product management domain, disrupted the relationships and interfaces between different corporate departments, and failed to deliver the promised benefits.
There are many very important lessons to draw from the Agile/Scrum experiment. Among them are that successful companies rely on professional business functions that have efficient interfaces between them and that there is absolutely no substitute for a solid process that is supported by logical thinking.