The Red Queen’s Race: Lessons from Biology for IT

I'm not a biologist, but I'm going to discuss a concept in evolutionary psychology anyway. The Red Queen Hypothesis, proposed by (actual) biologist Leigh Van Valen, suggests species must constantly adapt and evolve to maintain their ecosystem status. This idea draws its name from Lewis Carroll's "Through the Looking Glass," where the Red Queen tells Alice:

"Now, here, you see, it takes all the running you can do, to keep in the same place."

In nature, this translates to a never-ending arms race between predators and prey, parasites and hosts. Species that fail to adapt risk extinction, while those that evolve faster gain an advantage. Essentially, a species needs to constantly adapt and evolve to maintain their status because all other species are also constantly adapting and evolving. If they don’t, they go extinct. This translates to evolutionary success (uh not going extinct) depending on being the most adaptable and generally speaking, increasing your odds of survival by embracing change.

Escaping this feeling requires seeing the different races in your organization’s ecosystem. While it might be tempting to fit everything into one large race, every organization is running multiple races in various domains. Recognizing these races is the first step in escaping them. Here's a couple examples to help frame this idea:

From an information security perspective, organizations are racing to keep up with hostile actors trying to exploit system weaknesses. These hostile actors want access to your data and technology assets and will attempt to gain access however they can. Meanwhile, on the other side, your IT team patches vulnerabilities, prevents new intrusions, and tries stays ahead of them. They evolve, you adapt, and so on and so forth. This constant back and forth doesn’t leave much space between the two,  but the moment your IT team stops running this race...you might find yourself in a bit of a pickle.

Another perspective comes from the infrastructure side of your organization. Leaders and technologists within an organization are always looking to improve their infrastructure's capacity and efficiency, automate processes, and handle more demand wherever possible. However, as soon as an organization implements an efficiency improvement or develops more efficient technology to handle that demand, this improved standard then becomes the new baseline. This baseline then encourages more demand, continuing the race. For example, if a company automates its server provisioning process, competitors will likely do the same, and users will come to expect this level of speed from everyone. This then creates pressure to find the next efficiency gain, which will then become the baseline. The race never ends, as there's always a new level of efficiency to strive for.

Another external example to consider is in the generative AI space, specifically amongst the hyperscalers and foundational model developers. After Google’s initial research into transformers, OpenAI released ChatGPT, opening the Pandora’s Box of LLMs, with competitors like Anthropic and Mistral releasing their own versions. Each subsequent release from these providers has required more compute, capital, and resources to avoid falling behind. These races exist in various parts of an organization’s ecosystem and domain. Once you see them and identify them, you can start taking the steps to avoid running in the same place forever.

A) Embrace a portfolio approach

Take a probabilistic perspective and organize your organization’s technology assets and projects as a portfolio. View and value your applications, infra, and projects as parts of a whole, not as individual or isolated assets. This approach is about more than simple diversification; it’s about systematically managing and optimizing your technology investment with an approach that prices in the potential risks and costs of failure (and the value of success).

The core ideas behind this approach are:

• Risk-Reward Balancing:
  • Categorize projects by risk level (e.g., low, medium, high) and potential reward.
  • Aim for a balance that aligns with your organization's risk tolerance and strategic goals.
  • Example: Allocate 60% to low-risk maintenance projects, 30% to medium-risk improvements, and 10% to high-risk innovations.
• Fail Fast, Learn Faster:
  • Conduct quarterly portfolio reviews to assess each project’s performance.
  • Use data-driven metrics and clear milestones (more in the Tactics section) to evaluate progress.
  • Be willing to kill projects that don't meet those metrics or milestones.
  • Conduct post-mortems on failed projects to extract valuable lessons.
• Categorize Assets:
  • You can think of your portfolio as consisting of various classes of assets: existing applications, infrastructure, and new or ongoing projects.
  • Managing these different assets can be similar to a financial portfolio, where each class requires unique perspectives and decisions that all add up to the overall performance of the portfolio
  • With that in mind, look for opportunities where different assets can leverage shared resources or technologies.

By managing technology investments like a portfolio, you can optimize overall performance rather than getting bogged down in individual projects. This should increase your odds of survival, while accounting for the different moving pieces of an organization.

B) Break Down Information Silos

Breaking down silos goes beyond improving communication or buzzwords like Agile and DevOps. It involves restructuring your organization’s operations and decision-making.

Core ideas include:

• Cross-Functional Teams:
  • Form agile teams around products or value streams instead of traditional departments.
  • Implement rotational programs to expose team members to different business areas.
• Knowledge Management Systems:
  • Embrace the “bus” concept: if someone got hit by a bus tomorrow, what critical knowledge would be lost? Whatever it is…make sure it gets documented somewhere (before a bus comes into the picture).
  • Use asynchronous documentation like wikis, blogs, or knowledge bases to capture + share information across the organization.
• Continuous Feedback Loops:
  
  • Action produces information: analysis and brainstorming in isolation can only go so far. Take action (code, design, write), and then implement tools for feedback (e.g., code review systems, A/B testing, design critique sessions).
  • The faster the loop between action, feedback, and action again, the more adaptable your organization becomes.

These approaches increase organizational agility and the flow of information, allowing faster response to changing environments and improving your survival odds.

C) Specific Tactics

With the above strategies in mind, here are tangible tactics that can be implemented to enable these approaches:

1. Embed IT Team Members within Business Teams

• Create "pod" structures where IT professionals are permanent members of business units.
• Rotate IT specialists through different business units quarterly or semi-annually.
• Example: Spotify's "Squad" model, where cross-functional teams (including developers, designers, and product owners) work together on specific features or products.

2. Create a Project Rubric and Portfolio KPIs

• Develop a comprehensive set of metrics for a holistic view of your technology portfolio's health and performance.
• Project-level metrics include:
  • Return on Investment (ROI): measure of the financial benefits gained from a project relative to its costs.
    • Example: If a new IT system costs $100,000 to implement and generates $150,000 in savings over two years, the ROI would be 50%
  • Time to Value (TTV): duration between project initiation and when it starts delivering measurable business value.
    • Example: A cloud migration project that takes 6 months from start to when cost savings are first realized has a TTV of 6 months.
  • Technical Debt Ratio: proportion of effort required to fix issues in existing systems compared to the effort needed for new development.
    • Example: If a team spends 30% of their time addressing legacy system issues and 70% on new features, the technical debt ratio is 30:70 or 0.43.
• Portfolio-level metrics can include:
  • Project Churn Rate: percentage of projects cancelled or significantly altered within a specific time period.
    • Example: If 2 out of 20 IT projects are cancelled within a year, the annual project churn rate is 10%
  • Portfolio Efficiency Index: aka the scorecard. This is a composite score measuring how well the overall IT portfolio aligns with and contributes to business objectives.
    • Example: Combining factors like average project ROI, alignment with strategic goals, and resource utilization into a single score from 0-100, where a score of 75 indicates good overall portfolio performance.

3. Implement Continuous Learning Programs

• Host quarterly internal workshops for teams to share learnings, challenges, and innovations.
• Allocate a specific budget for each employee to spend on courses, books, or conferences.
• Implement a "20% time" policy for employees to work on innovative side projects one day a week.

The Red Queen race in technology can feel relentless, but it's not unbeatable. The goal isn't to win a single race, but to build the capability to adapt and evolve faster than the competition across multiple domains. The first step is recognizing the multiple races you and your organization are running, specifically in areas like IT and more specifically, information security, infrastructure, and even your user-facing applications. After identifying these races, taking a portfolio approach and breaking down silos of information will give you a wider perspective into the various assets in your organization and allow you to move quickly and decisively with all the data and information required. Embracing these ideas can help organizations do more than keep up—they can pull ahead.

If you are struggling to keep up with the pace of change, specifically when it comes to IT, please reach out as we'd be more than happy to help!