For a long time, reading code was the most practical way to build confidence in a system. It was how engineers debugged, reviewed changes, and resolved ambiguity when documentation and reality diverged.

The belief that if the code is read, the system is understood has always been fragile. Modern systems contain hundreds of thousands or millions of lines of code. No engineer truly understands them by reading every implementation detail.

Code inspection mostly worked because code was historically expensive to produce. Implementations appeared slowly enough that humans could keep up.

That economic reality has changed.

LLMs can generate implementations quickly and at scale. Entire features, integrations, and refactors can appear faster than humans can realistically review them line by line. The traditional control mechanism of software engineering, careful inspection of the implementation, stops scaling.

This shift is already visible in what many are calling specification-oriented development. Instead of writing every implementation detail by hand, engineers increasingly focus on describing behavior, constraints, and architectural boundaries, allowing implementations to be generated from those specifications. ThoughtWorks formalized it well.

This article is not about introducing that paradigm. That is already happening. The more interesting question is what follows.

Nor does this mean human judgment becomes unnecessary. The real question is where that judgment should be applied.

Human judgment moves upstream

For years, software engineering relied on a simple mechanism for applying judgment: read the implementation together.

When that mechanism stops scaling, human judgment does not disappear. It moves earlier and higher in the stack.

We got used to asking whether functions were implemented correctly. Those were never the most important questions.

The real questions are about whether the system being built makes sense at all.

What problem is being solved? What assumptions does the design rely on? What constraints shape the space of acceptable solutions? What behavior should the system demonstrate under realistic conditions?

Strong engineering organizations have long recognized this. Design and architecture reviews exist to evaluate the structure of a system before implementation begins.

The reason is simple. Most severe engineering mistakes are not coding mistakes. They are architectural mistakes.

Teams rarely fail because someone wrote an incorrect loop. They fail because the system was structured in a way that made certain failures inevitable.

Generative code makes this shift unavoidable. Spending most of our collective judgment inspecting code becomes a poor allocation of attention. The real leverage comes from shaping the specification, the architecture, and the constraints that define what the system can and cannot do.

This is often where the shift is misunderstood: many observers conclude that AI-assisted coding is useful for prototyping but not for building serious systems. What has changed is not the standard for good engineering, but who can now produce working code.

Implementation used to be a meaningful barrier to entry. Today it is not. When people without system design training develop software, the results reflect that lack of training. That does not reveal a limitation of the tools. It exposes where the real difficulty has always been: deciding what should be built, how it should be structured, and what constraints should govern its behavior.

Seen this way, this shift is less exotic than it first appears. Other disciplines have long relied on specifications and behavioral validation rather than inspecting artifacts directly.

Engineering already works this way in other domains

This shift may feel unusual in traditional software development. But many engineering disciplines have been working this way for years.

When artifacts become too complex to inspect directly, engineers stop trying to understand them line by line. Instead, they rely on specifications, generation processes, and behavioral validation.

Machine learning engineers rarely inspect the internal structure of trained models. They define datasets, training procedures, and evaluation metrics, then evaluate the model based on how it behaves over validation data.

Compilers generate enormous volumes of machine instructions from relatively small programs. Engineers do not read the generated assembly to confirm correctness. They trust the language specification and validate behavior through the compilation pipeline.

Database systems generate execution plans that few developers analyze deeply. What matters is that queries return correct results with acceptable performance.

Across these domains, the artifact itself is not the primary object of scrutiny. Confidence comes from specifications and observable behavior.

Software development is now entering a similar phase.

Architecture defines what the system cannot do

Engineers often describe architecture as the structure of a system. In practice, its real power lies somewhere else: architecture determines what the system cannot do.

Good architecture defines the constraints within which implementations must operate. Services cannot access arbitrary databases. Certain operations must flow through specific interfaces. Critical data paths are isolated behind carefully designed APIs.

This becomes even more important when code is generated by systems trained on the statistical average of existing software. LLMs behave much like competent junior engineers. They can produce locally reasonable solutions, but they tend to replicate common programming patterns, including many common mistakes.

Left unconstrained, generated implementations will often drift toward those average patterns. Architectural decisions counteract that tendency by shaping the space of acceptable solutions before any code is written.

Architectural constraints shape the solution space in which generated implementations can exist. They do not guarantee perfection, but they prevent many of the most frequent failure modes.

Systems of evidence

If code is no longer the primary object of inspection, correctness must be established elsewhere.

Some of those signals we have already discussed. Clear specifications and architectural constraints define what the system is supposed to do and, just as importantly, what it is not allowed to do. They shape the space in which implementations can exist.

Other signals are already familiar to most engineering teams. Automated tests and invariant checks encode expectations about behavior and allow systems to demonstrate that they satisfy those expectations repeatedly. Observability plays a similar role in production. Mature organizations instrument their systems so that behavior can be tracked through metrics, logs, and traces, allowing engineers to detect when reality diverges from the intended design.

But another source of evidence that historically underperformed is system-level validation environments.

The idea was always sound. Observing a system under realistic conditions before production should provide strong signals about its behavior. The problem was rarely conceptual. It was economic.

Building faithful system-level validation environments was expensive. Populating them with realistic data was difficult. Infrastructure often drifted away from production in subtle ways. As a result, many teams relied more on reading code and unit tests than on observing systems in realistic conditions.

That constraint is now weakening.

Generative capabilities make it dramatically easier to provision infrastructure, synthesize data, and recreate environments that resemble production systems. As these production-like environments become cheaper and more faithful, it becomes possible to validate behavior directly rather than reason about it indirectly through the implementation.

Instead of asking whether the code looks correct, engineers can increasingly ask whether the system behaves correctly when exercised with realistic data and traffic patterns.

Death to code review. Long live code review.

Nothing in this shift means code reviews were a bad idea. The intuition behind them was always sound: multiple perspectives surface problems that individuals miss.

What changes is not the value of review, but the dynamics of the code production chain. Humans became the bottleneck mostly because we had no alternative. Code appeared slowly enough that reading it together was the most practical way to apply collective judgment.

That historical limitation is now fading as code generation accelerates and automated tooling can perform much of the inspection work.

Agentic reviewers can analyze implementations far more aggressively than we can. They do not get tired, they do not skim, and they do not hesitate to explore a change from multiple angles.

A reviewing agent can patch the code locally, generate additional tests, execute them, and probe edge cases the original author never considered. It can search documentation and external sources while reviewing the change, compare patterns across the repository, and inspect interactions that would be tedious for a human reviewer to reconstruct manually.

Things become even more interesting when different models review each other. Different models tend to care about different things. In practice, they sometimes disagree in amusingly polite ways in pull request comments. One model worries about performance implications, another pushes on edge cases, and Gemini, in my experience, behaves like a particularly enthusiastic DRY enforcer.

Paradoxically, stepping out of the way of implementation review may actually improve quality. Machines can inspect code more exhaustively than we ever could, while humans concentrate on the decisions that shape the system itself.

Containment instead of inspection

At this point, an obvious objection appears: if generated code is pushed without careful inspection, the consequences could be severe. Data leaks, outages, corrupted systems, or failures that threaten the business.

This concern is legitimate.

But manual inspection was never a robust safety mechanism. Human reviewers routinely miss subtle issues in complex systems, not only related to safety but also to functional correctness and other non‑functional properties.

Mature engineering organizations solved this problem in a different way: not by perfect inspection, but by designing systems that contain failure.

Architectural boundaries prevent services from accessing sensitive data arbitrarily. Deployment systems gradually expose changes through feature flags and canary releases. Monitoring systems detect unexpected behavior quickly. Automated rollback mechanisms limit the blast radius of failures.

These mechanisms do not try to guarantee that mistakes never happen. They assume mistakes will happen and ensure that their impact remains limited.

The goal is not to eliminate mistakes entirely. That has never been realistic.

The goal is to design systems that prevent mistakes from easily escalating into systemic failures or severe consequences.

When implementations become cheap

The deeper shift behind all these changes is economic.

When implementation is expensive, engineering effort concentrates on writing and reviewing code. When implementation becomes cheap, the scarce resource becomes clarity: clarity about the problem, the architecture, and the system's expected behavior.

This reveals something that was always true but easier to ignore. Reading the code was never a robust way to establish correctness. It was simply the most practical one when code appeared slowly enough for humans to inspect it.

As implementations become abundant, that limitation becomes visible.

Confidence has to come from somewhere else. Clear specifications. Architectural constraints. Tests. Staging environments. Observability. Production feedback. Not from how convincing the implementation looks, but from how the system behaves.

Implementation still matters, but it is no longer the primary object of attention.

Machines writing code is not the real disruption. The disruption is realizing that reading the code was never how we truly knew whether a system worked.

As you get back with the product team, you discover that one of the most junior teams is working on marginally related changes, and the tech team is not very engaged when the sales team asks for help because they have cried wolf too many times before. It's clear you will fail, and the client will churn. It's time for drastic action: a Code Yellow.

Such interruptions are commonplace in many tech organizations despite well-established quarterly prioritization cycles and the use of OKRs (Objectives and Key Results). These tools are intended to keep teams on track but often fall short when unexpected challenges arise, as illustrated by the crisis with Client A.

In this article, we'll explore Code Yellows and how the process can transform chaotic interruptions into structured opportunities for growth and learning, ensuring that the most pressing problems are not just addressed but solved comprehensively.

Interruptions are a part of the real world. They come from failures in critical (frequently considered legacy) systems that stretched to their limits; other times, sudden changes to the market or critical clients can make strong pushbacks or requests not aligned with the short-term roadmap. Some incidents happen so frequently that it's usual to meet teams that have mastered the drill of emergency palliative support ("Let's just run that script that deletes that file and restarts the service once again and move on"). Teams and businesses can get trapped in a doom loop, struggling to overcome the root causes for good.

Interruptions can also be self-imposed when the leadership team realizes that the current course of action does not show evidence of progress or move the business toward the declared goals, deciding to change the plan to increase the chances of success. Leadership interruptions are polemic and can be especially disturbing for tech teams, which tend to have a high cost of interruption. This political component can lead teams to engage half-heartedly on the redirection in a self-fulfilling weak results prophecy.

Given these interruptions' frequent and varied nature, traditional methods often fail to address them effectively. While at Google, I witnessed a highly effective process called Code Yellow, specifically designed to tackle such critical interruptions head-on. Its name was drawn from hospital emergency color codes and adapted for the corporate tech world. I experimented with it in the startup world and believe it applies well to their reality.

In response to such critical situations, a Code Yellow assembles a strong team for a few weeks with the mindset of performing intense and excellent work to solve a critical problem. This addresses the immediate issue and sets a precedent within the organization for handling crises efficiently, serving as a live case study in teamwork and decision-making.

Declaring a Code Yellow

Understanding what a Code Yellow entails is crucial for recognizing when such a measure is necessary and fitting. Deciding to initiate a Code Yellow (CY for short) is a strategic choice, not made lightly. The tool will only remain effective if used consistently well over time. Declare too many CYs; teams will see them as pompous names for interruptions and work half-heartedly. Declare too complex or ambiguous CYs; they will take too long, teams get tired, and cynicism about their effectiveness creeps up.

Here are some of the legitimate reasons for which I have declared Code Yellows before:

• A critical Enterprise client was about to churn while resources and plans were completely misaligned with what mattered to avoid the loss. For the attentive reader, the anecdote from the beginning of the article is a true story. As a result of the CY, we developed a trusted advisor relationship with the client, founded a new Sales Engineering department, and translated several of the requested changes into significant improvements for the client base.

• Launching a new service that should solve critical production issues was bound to be delayed on its third quarterly OKR rollover. As a result of the CY, we brought together a senior advising team to make quality and robustness evaluations of the new technology more pragmatic, negotiated more resources to speed up quality evaluations by operators, and resolved newly discovered issues quickly to roll out the new technology to clients.

• The company's quality of service was deteriorating in what seemed to be a death by a thousand cuts, and there was no clarity on how to reverse the trend swiftly. As a result of the CY, we made adjustments to the quality metric to contemplate a new class of failures, established new routines with the operations team, and prioritized issues to recover quality standards for most clients, identifying a new class of issues that were of shared responsibility with a few clients.

These diverse situations, though challenging, were turned around by implementing focused interventions, demonstrating the potential effectiveness of the Code Yellow approach.

Drawing from these experiences, consider declaring a Code Yellow only when:

• The cost of not declaring it is high (clients are churning, high-performers are leaving, or essential business goals are threatened).

• The leadership team will support and sponsor the work throughout the organization (e.g., internal communications, performance evaluation cycles, and protecting the team's focus).

• The company hasn't gotten addicted to frequent Code Yellows.

• The scope is feasible within a few weeks of work.

An example of when it was inappropriate to use a Code Yellow was one instance at a previous company when an aspect of the B2C strategy was not progressing well, and senior leadership decided to declare a Code Yellow with the goal of "launching something.” In addition to the very ambiguous exit criteria, given that this was a new value that demanded significant exploration, there were several organizational-level reasons for which progress was questionable, including strong misalignment at the leadership level of what the goals were, shared ownership and redundant bets. While this situation was important to address, a more fitting approach was in the senior management sphere. In this situation, the Code Yellow intensified the confusion and dragged on for a long time without a clear target.

Code Yellow leader

A Code Yellow needs a leader, ideally one or two levels more senior than it would take to solve the problem. The leader should align sponsorship with the broader senior leadership and own the CY end-to-end. The CY leader's first responsibilities are to define the scope and the initial team and declare the CY across the organization.

Due to the intense coordination and communication demands, the leader is typically someone from the context's senior product management leadership. Still, sometimes, it can be the engineering management leader when the scope requires less cross-organizational coordination.

Code Yellow's scope

The leader should assemble a senior team to discuss the CY scope and communicate it as a set of exit conditions. The team will remain in CY mode until all exit conditions are met.

The exit conditions must be feasible, clearly measurable/observable, and focused on the root causes that triggered the CY. Only include critical goals and nothing too ambiguous. The aim is for the exit conditions to be met in at most five weeks, ideally less than 3. It's hard to keep the intense dynamics for longer. Even though it's essential to be mindful of the CY's duration, avoid promising a deadline. Work with the team to make fast and robust progress and even set milestone dates internally from a project management perspective, but keep hard dates within the CY.

A set of exit conditions could look like this:

* Migrate 70% of all transactions from service A to B.

* Set up alarms and on-call rotation to speed up recovery. Run a complete smoke test with at most five minutes of downtime.

* Cover root causes from the last three incidents with system-level integration tests.

It's prudent to underpromise on the exit conditions to increase the chances of guaranteeing structural progress while avoiding getting the team stuck at a high bar of expectations during a first iteration. The focus should be on the structural progress and solving the root cause rather than elevating the standards of the current solutions.

Code Yellow's initial team

It's part of the practice that the leader can invite anyone in the company to help with the CY, so it's ideal to keep the initial team small and senior to move fast over robust foundations.

Since the CY leader oversees communication and represents the senior leadership sponsorship, technical experts and solid individual contributors can fill the remaining team. Evaluating and recruiting people who haven't been in too many CYs or could see it as an exciting career opportunity is a nice touch.

Code Yellow's declaration

Communication around a CY should be excellent, starting from its declaration. Everyone in the organization should know there is a new CY, the reasons for declaring it, and who its leader is. Hence, it's a good idea to communicate in multiple formats and channels: Slack, email, and company-wide videoconference are the most common.

A CY is a corporate ritual; just like all other corporate world rituals, it can develop cynicism. The quality and discipline of the communication around it are essential traits that help validate that the organization means it, is vested in it and wants to see it through. Getting the senior leadership involved is also good practice, and the declaration of the CY is an excellent opportunity for VPs and the C-level to weigh in on their sponsorship. If no one from the C-/VP level is the CY leader, have one communicate the declaration.

The declaration should include the CY's name, the motivation for declaring it, the exit conditions, and the leader's name. It should be objective and transparent, reminding people the leader might summon anyone to help. It's a nice touch to wish the CY team good luck and thank everyone in advance for their support.

Executing a Code Yellow

The weeks in a CY should feel like the most productive weeks of the year. The team should work intensively but within a reasonable setup and not cross the lines of work-life balance with extra hours. The work in a CY should be like the one we all wish to perform all year but understand it's virtually impossible to sustain for such a long time. It draws from peak motivation.

CY team members should work only on the CY tasks. The definitive prioritization helps with focus and pace. It also contributes to communicating the costs of running CYs and avoiding addiction: several other tasks are interrupted in favor of the CY.

Three coordination components are essential for a good CY execution: team syncs, stakeholder syncs, and Code Yellow updates.

Team syncs

The team should meet briefly most days each week to discuss progress and challenges and explicitly align each member's next steps.

The CY leader should decide when the regular syncs happen. The frequency can vary over time depending on the work and the need for coordination. Sometimes, teams find it helpful to meet at the beginning of the day and a second touchpoint in mid-afternoon.

At each sync, most team members should report their progress and blockers. The leader can summon other people from the organization to help remove such blockers if necessary. The leader should also feel comfortable dismissing team members who are done with their contributions. This practice helps keep the team agile and laser-focused.

The team should produce notes from every sync containing explicit action items for each member involved. This list of action items can guide the discussions for the following sync.

Stakeholder syncs

Since the CY is communicated widely and typically addresses a prominent problem, many organization members might want to be involved. It's good practice for the CY leader to hold a separate meeting, which happens less often, to inform whoever wants to join and help address tangent concerns, which may include communication with external clients, legal considerations, questions, and general suggestions.

Holding stakeholder syncs is an excellent way to keep the sales and support teams involved without drawing from the focused execution of the CY team.

Code Yellow updates

The leader is responsible for keeping the organization well informed of the progress, especially regarding the exit conditions. It's a good practice to set a particular time of the day to share updates and keep the content objective and focused on each exit condition.

The clocklike regularity and discipline of communication can be a great motivator for keeping the team moving quickly. The update should state this even if the leader has nothing new to share.

It is helpful to post regular communication in a public Slack channel and to an email group that anyone can subscribe to. The leader should auto-subscribe the senior leadership and primary stakeholders to receive the updates.

Wrapping it up

A successful CY means more than the team meeting the exit criteria. It's a testament to the company's resilience and capability to course-correct or adapt to unexpected challenges.

It's essential that the leader be mindful of what CYs represent and care for a firm closure as if they are relaying the baton to the leader of the following CY that might come ahead. Impacts should be made clear, or risk a lack of sponsorship from senior leadership the next time the company would benefit from a CY. Individuals should be widely recognized or risk half-hearted engagement from the following team members.

As the CY ends, it's essential that the leader dedicate energy to a comprehensive analysis of the impacts and visual presentations. This will validate all the team's efforts and reinforce the value of such initiatives for the entire organization.

The communication about the CY's conclusion should echo its declaration: use the same concepts and channels and involve senior leadership, who are crucial in highlighting the CY's success to a broader audience and celebrating the achievements.

The careful impact analysis and presentations become valuable assets for team members to demonstrate their contributions in the following performance evaluations. This content is a strategic resource since the CY will probably be the main undertaking of the performance cycle for the team members who deserve recognition.

Lastly, gather the team and primary stakeholders to celebrate the results and consolidate the CY as an opportunity for team bonding and trust-building.

As we bring a Code Yellow to a close, the focus shifts from immediate resolution to long-term impact. Successfully concluding a CY unfolds the opportunity to integrate the lessons learned and the momentum gained into ongoing organizational practices.

Successful Code Yellows set a precedent for how challenges are addressed, turning what could be a disruptive interruption into a demonstration of proactive problem-solving and excellence. Code Yellows can turn pressing challenges into opportunities for the business and team members.

Making interruptions worth it was originally published in Desirable Difficulty on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Working Backwards book, by two ex-Amazon executives, made waves as it illustrated how Amazon built a deep dive culture using metrics, review processes, and communication schemas. The book coined some valuable concepts for the business world, such as input metrics (controllable metrics that can be directly associated with actions) and output metrics (higher abstraction metrics that represent results or outcomes for the business). The book makes a clear case for how working with input metrics is better and discusses how to navigate selecting the proper input metrics to work with.

A fascinating insight is that you can design input metrics to make the deep dive culture easier to attain.

A paradigm shift: opportunity-oriented metrics

Metrics (input or output) often compute accomplishments and communicate a level of success achieved by the organization. Most well-known metrics (let's call them motivational metrics) follow this pattern: monthly active users, conversion rate, monthly recurring revenue, and page views are some examples.

As teams work and communicate using motivational metrics, there are layers of indirection between their analytical work and the information about where the opportunities for improvement lie. Fundamentally, when you decompose motivational metrics to their units (when you list their computed instances), you see examples of solved problems.

One of the most striking traits of a deep dive culture is that teams scrutinize problems to their root cause and often pick a few concrete examples of opportunities as favorites to expose in discussions or test solutions. You facilitate deep dive work when you design metrics to compute opportunities or unsolved problems (let's call them opportunity-oriented metrics). Abandoned carts, funnel drop rate, and recurring revenue churn are famous metrics computed that way.

Designing metrics for a deep-dive culture

So, what happens when you take opportunities as input?

Clear definition of which opportunities are relevant

Priorities are a vital component of any good corporate strategy and communication. Only a fraction of opportunities for a well-run organization deserve attention at any given moment.

Depending on the corporate focus, some non-successful events could be out of scope and better left aside. When a metric computes unsolved problems instead of accomplishments, its definition communicates what should be considered a meaningful opportunity for the organization.

For example, assume a parcel delivery company's focus is to decrease delayed deliveries. Computing all failed delivery attempts would include attempts that failed before the promised date (and therefore did not impact delayed deliveries, the focus). The company could take only failed delivery attempts at the promised date or later as input, thus declaring that early delivery attempts are out of scope.

An important caveat is that computing unsolved problems defines the problems or dimensions that matter. It is a critical decision, often influencing the direction of an organization more than the product itself. And as with any other product decision, it may be incorrect. If the most impactful solution is in a cross-dimension, you may become utterly blind to it. The authors of Working Backwards refer to this as the tricky trial and error part. Incorporating a process that regularly takes a step back to explore the unfiltered set of non-successful events is prudent.

Concrete opportunity examples are easier to access

In a deep-dive culture, it's critical to have zero friction in accessing opportunities to scrutinize. A fresh list of relevant opportunity instances should be trivial to access by everyone. This tip is the most important insight of the article.

Let's take the space of online user acquisition funnels as an example. The typical motivational metric would be conversion rate: a count of user sessions that reached the bottom of the funnel in proportion to all user sessions. The computed units would be successful user sessions.

While drawing insights from successful user sessions is a viable approach, keeping the teams focused on examples of abandoned user sessions would likely produce a more impacting roadmap over time.

The opportunity-oriented version of the conversion rate could be the share of abandoned sessions among the subset of the market the organization could be focused on at the time — like the share of abandoned sessions for English native speakers between 21 and 30 years old. The computed units would be examples of users the funnel failed to convert, ready for analysis.

It could be as simple as a dashboard tool that exports a spreadsheet file with the computed instances.

Communicating categories of opportunities

When counting instances of unsolved problems, you'll have a chance to classify them. Moreover, it's possible to implement simple classification rules to group instances in valuable ways.

For example, in the context of user engagement (think Monthly Inactive Users), one could group users based on how long ago their last interaction was or group them by the feature they use more frequently or by user acquisition channel. The organization could communicate the same metric in charts with different classification criteria for a broader perspective.

The obvious benefit from communicating opportunities grouped in categories is that the most impactful classes of opportunities to address become public upfront information instead of the result of an eventual analysis. As a result, teams will have a clearer perspective about where the highest potential yields lie.

A not-so-obvious insight is that naming the categories in specific ways can drive more engagement and promote a general sense of ownership. Considering the organization structure and gearing the names of the categories of opportunities towards the scope of specific teams help them focus and feel responsible.

For example, suppose the subject is a chain of processes, with teams organized into groups responsible for each chain step. In that case, one can name categories about the first step where the failure was identified (such as "Delayed at the cross-docking facility").

Dashboards quickly evolve into diagnostic tools

Once there has been an investment into defining the relevant problems to tackle and implementing simple rules to group failure instances into categories, the core components of a diagnostic tool are ready.
Besides applying these rules for consolidating information for charts and KPIs communication, it could be possible to reuse the definitions in internal tools for debugging or customer success operations, democratizing the ability to diagnose failure instances within the organization.

Bonus: define a unit that can evolve with the company

As mentioned earlier, determining what the metric considers relevant events is a powerful way to focus the organization. As the problem space evolves, the organization's focus will change. There may have been good progress in the previous direction, and it will want to tap into a different class of opportunities, or it may have found a solid product-market fit and want to go deeper into one of many classes of opportunities considered earlier.

Much effort goes into communicating a new KPI if you're doing it right. And if it sticks, references and dependencies will unfold throughout the organization.

Consider naming the input instances as units (they will likely end up sharing the name of the KPI). When you name a unit, you can evolve its definition over time without compromising its ecosystem and communication framework. But, of course, backward compatibility would be compromised.

For example, assume a sales organization is looking for ways to improve the last step of the sales funnel for its core revenue stream. It defines a KPI for missed sales opportunities while leaving out developing revenue streams that still need development.

Fast forward a year, and now one of the developing revenue streams has matured and deserves just as much exposure and focus. Having two separate KPIs might not be ideal, as one would need to communicate heavily to introduce the new KPI and then wait for the ecosystem to catch up while the first KPI continues to get all the attention.

If, at the first moment, the organization named the opportunity instances as Near Miss Sales (NMS for short), the newer and expanded focus at the second moment could tag along with all that was built around NMS. The organization could then communicate an update on the direction and that the KPI now reflects this new scenario.

Loggi, a tech startup for express parcel delivery, was growing very quickly, and its operations were starting to feel the burden of hypergrowth from a quality of service perspective. Logistics processes are arduous to mature when growing at +3x YoY.

Our only quality metric was SLA, typically used for external client communication. Due to its contractual binding nature, SLA pivots on delivery attempts, not consummated deliveries. For example, contracts stated that 98% of all parcels collected from a client had to have a delivery attempt happen at least once until their promised date.

Loggi chartered Pedro Passarelli, Lucas P. C. Branco, and I with leading the work on quality improvement. Initially, leadership expected us to work on a diagnostic so that teams could prioritize specific improvement opportunities.

As we looked at examples of delayed parcel deliveries, we understood that this was a rich exercise and that we should look for ways to enable everyone in the company to do it routinely. It also struck us that we were not dealing with atypical problems and that there were probably standard metrics to support us. So we decided to study (check out a previous article about studying at work) and found an excellent 1994 MIT paper about logistics metrics. We gathered a few compelling concepts from this study:

• Behavioral soundness discusses that metrics shape how the organization thinks and behaves and must be thought out and communicated to incentivize the expected behavior. As we went through examples of non-delivered parcels, we could detect some ineffective operations driven by the delivery attempt based on SLA KPI. Over time, people tend to play a game against their target metric.

• Internally focused diagnostic metrics. This was a critical confirmation in favor of our feeling that SLA was not a useful KPI to drive an increase in quality of service as it was too conservative and targeted to an external audience. The idea of defining acceptable standards to describe perfect deliveries was under discussion, but the fact that non-adherence to standards could seamlessly become diagnostic information was a breakthrough.

• Metric scale can affect behavior, and percentages are inadequate for excellence expectations when 0.1% matters to the business.

Contrary to what I expected when going after literature on the topic, I concluded that we needed to define our internally focused diagnostic metric to promote the company's desired quality of service culture.

Defining acceptable standards

We described our perfect deliveries and concluded that the parcels were delivered on time and at the first delivery attempt. So, we looked at examples of packages that did not match a perfect delivery, and they all described relevant improvement opportunities. We were thrilled.

Next, we tried to define our acceptable standards at a higher resolution, thinking of rules such as:

• The time of day a parcel should have left a delivery station for last-mile delivery to be considered on time.

• The time of day a parcel should have left the cross-docking facility towards the delivery station to be considered on time.

• The various reasons why delivery attempts failed on the last mile.

Some rules ended up being approximations due to the lack of intermediate data. Establishing the rules as priors to the following phases and grouping examples by the first standard non-adherence along the chain started to surface examples grouped by classes of operational errors. Finally, we seemed to be onto something.

Communicating a diagnostic

We understood that naming the non-adherence groups in a directive and somewhat judgemental way mattered as it would call for ownership and action. It took us many iterations to get it the way we wanted. For the groups before the last mile, we followed the naming template "failed at cross-docking," "failed at the delivery station," etc.

We were on the right track when we presented the metric, and operation managers wanted to look at the examples and even felt uncomfortable with some of the conclusions, considering the diagnostic a bit unfair. However, if we communicated that way, they would care.

Setting the bar for excellence

When we started putting it all together, something felt off as some relevant classes of opportunities were reading, like 0.6%. In some cases, we were dealing with complex problems with already developed solutions, such as incentives to drivers for delivery consummation. We expected teams to improve on them by 10% over a year, which could mean a change of 0.06pp in the metric. Motivating people to diligently pursue that relatively small contribution for a long time seemed challenging.

We took the cue on the scale remark and communicated the metric proportionately to 1000 instead of 100. But, even though everyone understood %, we had no good way of communicating per thousand other than defining a new unit.

Wrapping it all up and shaping the culture

This new metric became a product in itself. So we put much effort into communicating it, from defining the unit, the classes of opportunities, and the dashboard (what goes above the fold and below, etc.).

We named the unit NOT, which is short for Not On Time. The catchy name helped a lot in getting traction.

As we went around the company communicating and working with teams on using NOT, we got a lot of traction and engagement. Operations leadership demanded deep-dive analysis, and teams suddenly had a practical tool and a communication framework to support them.

Quality has improved consistently since then, with observable impact (global NOT improved by +30% on an excellence grade problem space). We could declare which initiatives were promising or not beforehand, point out regional-specific quality challenges, and rally the whole company towards actionable goals with little communication effort.

Years later, we learned that using a perfect delivery as the target was no longer valuable. Specific aspects of our operations needed more attention and exposure, such as the recovery process after a failed delivery attempt. In retrospect, defining a unit was a great idea. As the focus and maturity of the company evolved, we adjusted what the unit accounted for, taking advantage of the previous communication framework: OKRs, governance rituals, and training materials; everything that referred to NOT as the Northstar quality metric still made sense, even though what the unit represented evolved. (Hat tip to Natacha Alencar, who helped us on NOT 2.0)

I did not see it coming, but NOT was one of the most robust pieces of work I have ever accomplished. Everything in the environment was favorable (team, leadership, problem space), and I'm thankful to all. Also, I hope that what we've learned can be applied elsewhere.

Shaping the c̶u̶r̶v̶e culture was originally published in Desirable Difficulty on Medium, where people are continuing the conversation by highlighting and responding to this story.

However, it’s not very common for teams to study about more fundamental problems, that could make a critical difference on how products are conceived or designed, or even in how a problem could be leveraged as an opportunity within the company’s strategy. Here are a few questions that illustrate what I mean by more fundamental problems:

• What conversion rates should we expect from this product? What are the most probable main drivers?

• What are different pricing approaches to explore and how to experiment with them?

• What is the most adequate metric we could be using to track our progress at this point?

• Do we have all the data we need in order to check for our process conformance?

For those that went through college, discussing fundamental problems and the frontiers of knowledge around them was routine and there were text books to guide us deep and wide. But, what about work?

Business books typically cover topics at very high levels, so they can reach a wider audience and succeed commercially. They can help you learn concepts, way of thinking, best practices, but rarely at enough depth to map directly into execution.

As it turns out, academic research is a lot about diving into specific problems and publishing to the world about their learnings. Many think that consuming from academic publications is reserved to those dealing with highly complex or cutting edge problems described through puzzling mathematical equations, such as route optimization, natural language processing, etc. However, there’s a lot of business oriented research out there, even in less mathematically intricate fields such as marketing or ecommerce conversion optimization.

A great way to get started on leveraging academic studies at work is to explore review papers when getting started on a new class of problems. Review papers, referred to as secondary sour­ces in the academic work, are studies that explore a field or problem horizontally, collecting and organizing concepts and discussing what advancements other research work on the topic have accomplished.

Studying and discussing relevant review papers with the team on early stages of a project may payoff in several ways (in addition to being cool):

Discovering the relevant concepts and aligning communication

Review papers are typically very good at organizing the relevant concepts related to the problem domain: different categories of the problem, different categories of solution approaches, metrics, solution evaluation criteria, most common input requirements, etc.

This can help the team not only prune the problem space, but also build a common internal jargon that improves communication.

Benchmarking and adjusting expectations

Such works typically make reference to industry standard numbers (conversion rates, latency, precision vs recall, etc depending on your problem). If you’re lucky to find recent enough studies, these numbers can help you and your team set the expectations about how much benefit the considered approach could yield.

Being able to tell early in the process whether an approach can or cannot bring the result standards you’re looking for means a lot of time not wasted.

Maturity of current solutions and solution insights

There’s hardly any fundamental problem known to humans that hasn’t been studied extensively. For some, we’ve found strong and highly applicable solutions. But some times academia hasn’t advanced enough on a problem space in order to propose usable solutions.

Also, it can be rare for your problem to match perfectly to those of published studies and the solutions presented might be useful as a source of insights for solving your own instance.

Prioritization

A better sense of implementation complexity vs potential yields can be very useful for prioritization purposes. You might find out it’s a low hanging fruit that’s worth pursuing or that the problem seems too hard for the benefit and explore other opportunities.

References

Review papers are literature reviews, so they are honed to help you navigate existing studies and prune your search for what are more specific works you could read.

Personal development

For some team members it could be very impacting to take a measure on how deep one could go on exploring a problem or where are the bars set for someone to become an expert on a field.

Studying as a group also fosters collaboration and creates opportunities for team members to step up and share knowledge with other colleagues. A classic approach is for the team to take turns on who will present a paper to the team (and often other guests) for a discussion session.

For others, like me, it could be a striking reminder that smart people have already thought about a problem and it’s often worth it to take some time to learn from them before jumping into problem solving.

I was lucky to meet some great people back in college years that went on to become true friends and workmates. This strong present tie with academia by working with Davi de Castro Reis, Bruno Fonseca, Akio Nakamura and Andre Paim Lemos (all who I hope will contribute to this blog at some point) helps me keep an attitude of exploring problems in depth and learn not only the tricks of the trade, but also what fundamental problems are behind the challenges we face in business, tech or people management.

A few years ago, we kicked off a service quality task force at Loggi and, since I was new to the logistics field, decided to learn more about the basics. So I stumbled upon this amazing review paper titled “A Review and Evaluation of Logistics Metrics” by Caplice and Sheffi (MIT). It defines a list of criteria for evaluating logistics metrics. It was just perfect for that moment when we were looking for ways to measure our progress and spark a corporate culture. Essentially all wisdom I have gathered since then in terms of how to think logistics management was inspired by a single (shorter than a page) section titled “Behavioral Soundness”. I became such a meta-metrics nerd after reading this paper that I’ll probably end up writing more about it.

Here’s a toast to friends, science and learning (and metrics). Cheers.

The practice of studying about a problem at work was originally published in Desirable Difficulty on Medium, where people are continuing the conversation by highlighting and responding to this story.