The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win

While driving to work, Bill Palmer, Director of Midrange Technology Operations at Parts Unlimited, gets a call from Laura Beck, Vice President of Human Resources, who wants to meet. Though Palmer initially worries he is being fired, Beck reveals at the meeting that the company’s CIO and VP of IT have left the company, and Palmer is being promoted to VP of IT. Palmer immediately rejects the role because most of the company’s VPs in IT last only two years, but he agrees to meet with Masters, the CEO. Beck brings Palmer to Masters’s spartan office, where Palmer again rejects the VP role. Masters compliments Palmer’s work on a prior acquisition, impressing Palmer, and he explains that the company needs the Phoenix Project to succeed to help them catch up to their competition.

Palmer ultimately accepts the role. Masters emphasizes that the temporary CIO, Chris Allers, who is also VP of Application Development, needs to focus on the Phoenix Project, so Palmer should address anything that might draw focus from Phoenix. Masters compares IT Operations to plumbing, saying he expects technology to work without worry. Palmer’s first task is to address a problem with the payroll system. A payroll delay could lead to work stoppage and employee hardships.

Palmer calls CFO Dick Landry, who found the payroll error. Landry says the payroll issue is an IT problem, and he brings Palmer to meet Ann, his head of operations. She explains that the partially automated system returned all hourly employees’ data with zeros for hours worked and pay due. The group has until 5 pm to resolve the issue, and Palmer recommends establishing a backup plan to estimate pay and correct the data later. Landry notes that auditors will be upset, and Palmer complains about regulations, noting that he hopes to avoid contacting the payroll developers.

Palmer heads to the Network Operations Center (NOC) to meet with Wes Davis, Director of Distributed Technology Operations, who is in charge of the networks Palmer needs to monitor, and Patty McKee, the Director of IT Service Support. Lamenting that Masters has not announced his promotion, Palmer explains his new role, which upsets Davis but not McKee. They explain that they think a SAN upgrade caused several process outages, including payroll. An engineer called Brent was pulled from Phoenix Project work to help, meaning he will not complete another task for Sarah Moulton, the SVP of Retail Operations. Palmer notes that Moulton usually blames IT Operations for anything that goes wrong, but she is rumored to be in line to replace Masters as CEO. Palmer says to keep Brent on the SAN issue, while Palmer handles Moulton.

Palmer, McKee, and Davis visit Brent, who explains that they have been meaning to upgrade the SAN for a while. During the upgrade, things did not go as planned, causing payroll and other databases to fail when the upgrade was completed. When IT Operations rolled back the upgrade, the SAN crashed altogether. Ultimately, Palmer comes to suspect that the problem is the result of a series of events involving CISO John Pesche and a developer trying to go on vacation, leaving IT Operations to fix the problem. Palmer laments the struggles of working with multiple teams.

Palmer and McKee call Pesche, who eventually admits to having one of his developers implement a tokenization program to protect personal identifiable information (PII). When asked why he did not go through the change process McKee designed, Pesche says it takes too long, noting that the company’s systems audit is less than a week away. After the call, McKee explains that people often ignore processes and procedures, making changes on their own, and Palmer says he will enforce the Change Advisory Board (CAB). McKee and Palmer go through all the changes made recently, eventually determining that the tokenization program is at fault for the payroll failure, but the SAN still needs to be fixed. At 3 pm, Palmer tells Landry and Ann to go ahead with the estimated payments, and they fix the timekeeping application by 7 pm. At 11 pm, they resolve the SAN issue. At home, Palmer receives an email detailing that Parts Unlimited’s failure to accurately pay their employees will be front-page news the next day, and he plans to apologize to Landry in person.

In the morning, Palmer has hundreds of emails and too many voicemails to cover before starting his day. He asks Ellen, his new assistant, to listen to voicemails and reach out with any information from Masters or Landry. In an email, Sarah Moulton accuses Palmer of delaying the Phoenix Project and arranges a meeting with Palmer and Masters for 10 am Davis assures Palmer that the payroll issue took priority, adding that the developers keep telling IT Operations that they are not providing the correct test environment for Phoenix. Palmer demands that Development and IT Operations meet to resolve the specifications of the Phoenix test environment.

In response to complaints about the delays to Phoenix, Palmer explains the payroll issue. Moulton blames Palmer personally for the delays, accusing him of laziness and incompetence. After Allers promises that development will be done soon and Phoenix complete in a week and a half, Davis explains that Development has not provided the specifications needed to proceed with Phoenix, and Palmer backs him up. Allers rails that IT Operations never comes to Development meetings, and Moulton repeats her accusation that Palmer is at fault. Masters agrees with Moulton, and they plan to release Phoenix in just over a week.

After the meeting, Davis reveals that Brent is the only person capable of attending meetings with developers, and Palmer says to send him to the developers, combining the Development and IT Operations teams, and fix network outages without Brent. In his office, Palmer’s computer dies, and Ellen arranges for desktop support. McKee and Palmer are the only people who attend the CAB meeting, and Palmer plans a mandatory CAB meeting for Friday. Palmer’s replacement laptop is old, and Ellen explains that Pesche’s Tuesday patches broke over 200 laptops. Palmer notifies everyone of the mandatory CAB meeting, and Davis calls to complain. He explains that the previous CIO tried to enforce CAB meetings, and everyone revolted, but he agrees to attend and try to help.

Palmer wakes early and finds an email from Nancy Mailer, the Chief Audit Executive, calling him to an early meeting to discuss internal audit findings. At the meeting, Tim, an auditor, hands out the audit findings. Pesche tries to claim credit for the tokenization project, which Nancy says will not impact the SOX-404 audit in three weeks. Davis yells at Pesche and Tim for not understanding IT. Tim explains that the Phoenix Project is taking up most of their time, limiting their ability to make changes. Mailer understands, and they tentatively agree to submit a plan of action in just over a week.

Pesche blames Palmer and Davis for security breaches, and Palmer tells him to address security concerns with Phoenix. Palmer tells Davis that they need a comprehensive list of what everyone is working on so he can ask Masters for more resources. According to Davis, almost everything requires Brent. Palmer responds that they need to hire more people like Brent. They call McKee and make a plan to contact everyone, compiling a list of all IT commitments in the company. From there, Palmer will be able to figure out what additional resources they need and provide an informed request to Masters.

The developers’ update reveals that the Phoenix Project is farther behind than Palmer anticipated, leaving much of the testing to later releases. Palmer, McKee, and Davis go over the IT workloads, discovering that they have an average of one person per project, meaning that even just the audit findings will take a full year to complete. Palmer complains to Davis that they need someone to watch the SAN and track outages in order to reduce distractions from Phoenix. Davis suggests they ask Masters for seven new people, but adds that it would take a year to bring new hires to full productivity.

At the CAB meeting, Palmer asks why people do not use McKee’s change system and learns that people find it too complicated and time-consuming. Palmer instructs everyone to write their changes on index cards, describing the change, the system it affects, and why it is needed. Pesche encourages the group, and they discuss what a change is, and which changes are important. Everyone leaves the meeting satisfied, except Palmer and McKee. Palmer says they will reconvene on Monday, and McKee complains that a lot of work went into the old process being discarded. Palmer comforts her, and she agrees to write new change submission instructions.

Later, McKee reveals that they have received over 200 more requests for changes for the next week, in addition to the 50 changes she expected. This makes them both nervous, but Palmer is glad people are using the new system. McKee wants to halt requests, but Palmer says they need to keep their momentum and enthusiasm. McKee agrees to bring in additional people to review and organize changes.

Palmer is called to Masters’s conference room to meet Erik Reid, a technology genius and prospective board member. Palmer initially mistakes Reid for a delivery man before introducing himself properly. Palmer tells Reid he is trying to instill discipline and consistency in IT Operations, and Reid responds that Palmer cannot improve IT until he knows what work is. There are four types of work, he says. Palmer correctly states that projects, like Phoenix, are a type of work, but Reid says Palmer needs to figure out the other three types.

Reid brings Palmer to a production plant, where he describes the Theory of Constraints, which appears in Eliyahu Goldratt’s book The Goal. According to Goldratt, bottlenecks in a manufacturing process are the only relevant place to make improvements, since improvements before and after the bottleneck do not affect the overall workflow. Palmer says that IT is nothing like manufacturing, but Reid disagrees, adding that Palmer can never improve the IT Operations situation at Parts Unlimited without learning the Three Ways, which he outlines. The First Way organizes DevOps, the flow from Development to IT Operations. The Second Way enhances feedback loops to reduce quality concerns. The Third Way fosters a culture of continual improvement through failure and practice. However, Reid says, before he can learn the Three Ways, Palmer must figure out the four types of work. Reid encourages Palmer to reach out when he figures them out. After the meeting, Palmer is awestruck and puzzled about the meaning of work. Palmer’s replacement laptop dies.

Palmer goes to Masters’s office for a meeting first thing in the morning, but Moulton is already there. Palmer resents that Moulton is taking time from his 30 minutes with Masters, waiting 11 minutes to get Masters alone. Finally, he presents his case: IT Operations is understaffed, and critical resources are needed for Phoenix and compliance with the audit findings. Palmer doubts they can accomplish both in the given timeline. Masters dismisses Palmer’s argument, telling him that IT Operations must accomplish both tasks and there is no room for additional budget or hiring. Palmer needs to work with what he has or ask other departments to share their budget. Palmer is crushed, but Masters does not seem to care about getting work done well. Moulton interrupts their last three minutes, and Palmer throws away his presentation.

Palmer meets with McKee and Davis to review the 437 submitted changes, which all three find overwhelming. First, they identify the riskiest changes, agreeing that they will need to review and approve them individually. Next, they identify low-risk changes, agreeing to preapprove them while still requiring a change request submission. They agree that middle-risk changes should be left to the discretion of the person submitting the change, but they will require the submitter to contact all affected parties for approval, rather than just the IT Operations leadership. They will continue allowing change submissions for an additional day, and they schedule a new CAB meeting for Wednesday. They plan to establish automated rules on Wednesday and leave happy.