Gartner has identified Platform Engineering as one of the Top 10 Strategic Technology Trends for 2023[1] that can accelerate software delivery at the pace at which it provides business value. It is also forecast by Gartner that by 2026, 80% of software engineering organizations will establish platform teams as internal providers of reusable services, components, and tools for application delivery.
What do we observe from this? We need to elucidate what Product Engineering means, how it modernizes application delivery and the need to adopt digital transformation with high value outcomes.
Platform engineering is the discipline of designing and building toolchains and workflows that enable self-service capabilities for organizations and reducing the cognitive workload. This workload is identified as a major by-product of DevOps, complex microservices architecture, technologies like Kubernetes, approaches such as Infrastructure-as-Code, etc all of which warrants developers to acquire thorough knowledge of the toolchain. This complexity is encapsulated and provisioned in the form of “Internal Developer Platform” (IDP) which comprises of curated set of tools and processes defined by subject matter experts based on the specific requirements of the software under development. The platform goal is to enable frictionless, efficient and experience centric development leading to increased productivity.
The early 2000s was the era of single gatekeeper, the System Admin. Developers would code and as often called, throw over the fence, to Admins which was well understood to be inefficient in multiple forms. With the advent of Cloud, DevOps came to the fore giving rise to cloud native development, empowering organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. The approach was exemplified by containers, service meshes, microservices, immutable infrastructure, APIs, serverless architecture and features such as observability.
In all this evolution, software companies tried to enforce one golden standard – Setup, institutionalize and practice ‘True DevOps’. Developers should be able to deploy and run their apps and services end to end. While enterprises like Google, Amazon were able to master the concept, adoption fell short for many other organizations. The main reason being limited access to the talent pool, expertise on complex toolchains and embracing emerging architectural standards within the context of DevOps. This resulted in emergence of a series of anti-patterns. As an example, shadow ops, where the tasks done by operations professionals are offloaded to developers. This leads to less experienced developers leaning on senior backend engineers for assistance.
DevOps research findings, community learning and continuous improvement measures taken by leading organizations seeded the concept of Platform Engineering which started to differentiate between high and low performing organizations. This entails IDP, managed by a dedicated team, and catering to the needs of software development lifecycle.
Platform engineering teams are responsible for setting up and managing IDP by undertaking the following tasks:
The best IDP are like private PaaS packaging all capabilities into interfaces such as CLIs, web UIs, and automated pipelines.
While DevOps and Platform engineering might seem similar, but they are quite different in many ways and related. DevOps is a broad philosophy while platform engineering is a creation of a centralized platform comprising of tools and exposed workflows. Platform engineering is often implemented after DevOps principles are put into practice. The tools enforcement and cognitive load brought by DevOps is therefore abstracted by Platform engineering.
The platform should be developed by adopting product-based approach. A product vision should be defined along with clear mission statement to fit overall goals of the organization. There should be a charter in place with clear roles of the platform within the enterprise as well as the comprising team members. Stakeholders should be identified, user research conducted, and feedback solicited by the platform team to derive comprehensive understanding of the objectives, developer pain points and shared challenges across the organization. There should be actionable KPIs, such as productivity, lead time, deployment frequency, developer happiness, stability, risks, etc, identified and tracked to measure performance and continuous improvement. Additionally, adoption KPIs like collaboration index, % age usage of platform, effort savings, etc should be captured.
Platform engineering is about architecting IDP which is combination of technologies and tools that the team binds to pave a golden path or paths. There are many ways to build the platform and lot of it is dependent on the enterprise IT standards, technology guidelines and the software under development. The tooling landscape will normally comprise of CI, Security, Orchestration engines, Monitoring, Deployments, Messaging, database, Cloud providers and native toolset /services, Infrastructure control pane, Logging, Database & Storage. The platform is not just about setting the toolchain. The tools are orchestrated, and workflows made available to developers in a way that complex time-consuming activities such as adding environment variables, adding services / dependencies, rolling back deployments, spinning new environments, etc are all encapsulated into the platform and available as consistent self-service experience.
Seldom do the best solutions need to be built from scratch. The tooling landscape is growing progressively to address a wide range of problems. Platform teams can save time and create more value by tailoring off-the-shelf solutions whenever possible. Commercial competitors are more likely to optimize for more generic needs of the industry.
The Platform Engineering community [2] started in 2021 with a handful of meetup groups in Austin and Berlin. Today, it boasts over 10,000 active platform engineers across 19 meetup groups around the globe.
Platform engineering has established itself as a proven approach to unlock DevOps success at scale in the enterprise. It is not just increasing developer autonomy but has broad impact on organizations across multiple areas, such as improving system reliability, productivity, workflows and process standards, overall security, and standardization. Platform engineering is usually implemented as part of a DevOps strategy and is another reflection of rapid technology advances. Organizations should look to use or leverage this approach, and in particular larger enterprises, which have greater inclination towards accelerated digital transformation.
[1] https://www.gartner.co.uk/en/information-technology/insights/top-technology-trends
[2] https://platformengineering.org