Should You Build an IDP?

The concept of Internal Developer Platforms (IDPs) has emerged as a promising solution to streamline and standardize the software development process. At first glance, IDPs offer an attractive proposition: a shared, consistent infrastructure that all developers can utilize that encourages collaboration and efficiency. However, the implementation of such a platform is not without its challenges. It requires a team of experienced infrastructure and DevOps engineers, making it a significant undertaking. So, what could be the alternative? This article examines IDPs, their benefits (real and perceived), and looks at some potential alternatives.

What is a Platform?

A platform, specifically an Internal Developer Platform (IDP), is a sophisticated construct of tools, technologies and workflows designed to streamline the process of software development. These are brought together by an engineering team to provide self-service (or golden paths) to its users - the software developers.

At its heart, an IDP is a shared computing fabric that offers a self-service model for its primary users - the software development teams. The beauty of an IDP lies in its ability to automate and simplify tasks that were once cumbersome and time-consuming. For instance, instead of having to open a ticket to provision test or development hardware, developers can now simply make a few API calls or click a few buttons to automatically access the resources they need. In an ideal scenario, a code commit is all it takes for the necessary infrastructure to become available.

However, it's important to note that the journey to this level of efficiency and automation comes with complexities and challenges. The path to achieving a fully functional IDP can be quite messy, but the end result is a powerful tool that significantly enhances the software development process.

Why is it Hard? 

Building an Internal Developer Platform (IDP) is like constructing a product but with a unique twist. You're essentially creating the same product, only this time your customer base is significantly smaller. This might seem like a less daunting task, but it's far from it. The challenge lies in the maintenance of the product. The world of technology is a dynamic one, with a fiercely competitive landscape.

Developers are always on the lookout for platforms that offer the latest and greatest tools and features. If your IDP falls short in this regard, developers won't hesitate to seek alternatives. They have tasks to accomplish and goals to meet, and they need a platform that can facilitate their work efficiently.

Therefore, maintaining an IDP is not just about keeping it functional, but also about ensuring it stays relevant and competitive. This requires a deep understanding of the evolving needs of developers and the ability to adapt the platform accordingly, which is no small feat. 

Here are 4 reasons why IDP may be too complex for your organization:

1. Complexity of Requirements

One of the most daunting challenges that an Internal Developer Platform (IDP) faces is the escalating complexity of requirements. As an organization expands, the IDP must cater to the needs of multiple applications or teams. Each has its unique set of demands and expectations. Complexity grows when a new feature is needed; the introduction of a new feature often needs a complete overhaul of the existing system, This requires careful planning and execution to ensure the current platform customers aren’t disrupted during ongoing operations. This constant evolution and the need to adapt to changing requirements make IDP management challenging. 

2. Resource Limitations

Another challenge in building an internal developer platform is dealing with resource limitations. In an ideal world, the platform team would be able to scale indefinitely, bringing on board those elusive 'purple squirrel' DevOps engineers with the specific operational, development, and Kubernetes experience to meet the growing demands of the organization. However, the reality is often far from this. The platform team can quickly become a bottleneck, blocking progress for every developer in the organization. This is because the team's growth is constrained by the availability of skilled engineers and the budget allocated for hiring. Furthermore, the platform team needs time to build out automation before delivery. Without this crucial step, they risk being stuck in a cycle of manual deployments, which not only slows down the development process but also prevents the platform from ever reaching completion. 

3. Maintenance and Support

Managing an internal developer platform (IDP) requires a significant investment in maintenance and support in the platform. Once the first version (V1) is deployed, the platform becomes a fixed point in time, with all applications and future designs locked to the dependencies and services bundled within it. This rigidity can be a hindrance to feature iteration. For instance, if a new cloud feature emerges that could enhance a platform's functionality or efficiency, tough luck. Users have to wait for the second version (V2), which could be months or even years out. This delay can lead to the creation of shadow infrastructure. This is the result of developers seeking workarounds to bypass the limitations of the current platform. This not only undermines the consistency and integrity of your platform but also adds another layer of complexity to the overall infrastructure management. 

4. Integration and Deployment

Deployment automation can be difficult to implement effectively with a platform. The process often depends on twice as much Continuous Integration/Continuous Deployment (CI/CD) configuration compared to traditional designs. The complexity of this configuration is a result of not only the deployment of the platform itself, but also the deployment of software on the platform. This dual-layered deployment process can be a significant technical hurdle to overcome during implementation. Platform teams will need to be able to closely monitor deployments and health of their platform, as well as vend observability data about deployments that are occurring on the platform.

Platform teams must also determine how opinionated deployments on the platform will be. Lots of guardrails and default behavior can make getting started easier, but will ultimately prove inflexible as applications grow in scope and complexity. Conversely, giving the developers too much freedom upfront results in increased pressure, as they are forced to design their own deployment automation.

What are the Alternatives?

While there are different approaches to providing infrastructure for developers to build on, one of the more developer-friendly options is: Platform-as-a-Service (PaaS). PaaS is a cloud computing model that provides developers with a platform to develop, run, and manage applications without the complexity of building and maintaining the infrastructure typically associated with developing and launching an app.

For most development teams, the primary requirement is a scalable platform to ship code to, and not much more. This is where PaaS shines. It abstracts away the complexities of infrastructure management, allowing developers to focus on writing code and creating applications, rather than getting bogged down in the intricacies of server management, storage, and networking.

One of the key advantages of most PaaS is the inherent support for Docker containers as build and deploy artifacts. Docker containers package up an application along with its environment, making it easy to move and run anywhere. This can be on a developer's local machine, a test environment, or a production server. This portability is a significant advantage for development teams, as it ensures consistency across different stages of the software development lifecycle (SDLC), reducing the likelihood of environment-specific bugs and issues.

Moreover, the use of Docker containers also means that the application can always be ported to an Internal Developer Platform (IDP) when it or the organization is more mature or ready.

PaaS isn’t perfect however, and like any major software architectural choice, it comes with its own compromises. PaaS offerings are often very opinionated, a tradeoff of the simplicity needed to make iteration faster. Developers will often be limited in customization choices should they need to support systems or architecture beyond the platform services. Organizations porting legacy infrastructure from traditional servers or VMs will need to take on additional migration work to port their applications to compatible container runtime. Despite these issues, PaaS still represents a significantly less complex and operations-dependent computing fabric when compared to a platform.

IDP: Don’t Re-invent the Wheel

An IDP can be a powerful tool, but its effectiveness is highly dependent on the context in which it is used.

An IDP is an opinionated platform, meaning it comes with a predefined set of rules and structures. This can be a  real advantage for development teams, as it eliminates the need to design their own infrastructure or security measures. These elements are baked into the platform by default, allowing teams to focus on the core aspects of their projects.

For development teams that are just starting out with a new application, or migrating from on-premise to the cloud, an IDP may simply be too much complexity and management overhead, and as an overall initiative has a much higher chance of failure.

Developers that want to focus on delivering new features should take advantage of a PaaS like what Divio offers; the burden of scaling and operations are handled beneath the surface, and the developers are free to focus on writing code and creating software applications.

Curious to learn more about our solution? Let’s talk!

Stay informed! Join our LinkedIn and X/Twitter community to access exclusive insights and be the first to know about our latest blog posts.