BT’s head of software engineering excellence lays out the telco’s route to streamlining development, modernizing applications, automating deployments
From monoliths to containers
Slowly but surely BT is looking to consolidate and modernize its existing workloads. This is where containers and Kubernetes enter the equation, but Premchandran urges caution to anyone who sees the container orchestration tool as a silver bullet.“What really happens is you’ve got to tease out what is containerizable,” he explains. “Say you have a monolith and you’ve got 50 components, all intertwined with hard-coded dependencies and some legacy stack. First you ask, what do I containerize? Where is that unit that I can tease out without impacting the others? That is the biggest challenge that most firms which have been around as long as we have face.”
To overcome this challenge, BT has established a platform team, dedicated to helping application teams identify these containerizable elements and find the best environment in which to host them, be it in the public cloud or on a platform-as-a-service (PaaS).
This has led to a common set of considerations BT engineers take when looking to modernize their workloads, namely how loosely coupled it is and whether you can effectively isolate a service and containerize it, without impacting the existing stack.
All roads lead to PaaS
In terms of infrastructure options, there is some choice, but the idea is to simplify these decisions to gain some level of consistency.
Of course all three major infrastructure-as-a-service (IaaS) options — Amazon Web Services, Microsoft Azure, and Google Cloud Platform — are on the table. Where possible the platform team will push developers to use a managed Kubernetes service or, better still, the Tanzu Application Service, a PaaS from the vendor VMware, and its own Tanzu Kubernetes Grid (TKG).
“You have to handhold them, otherwise they will take the biggest unit they can handle, put that into a Docker container and then lo and behold you’ve got the same monster on better infrastructure — that doesn’t solve your business problem,” Premchandran says of his developers.
That being said, when push comes to shove, the preference would always be for TKG, which promises a consistent managed Kubernetes layer across on-premises and public cloud environments.
“You may want to swap out your cloud provider tomorrow, you might want to move your workloads from on prem to cloud, so you need a control plane for Kubernetes that makes the underlying IaaS irrelevant,” Premchandran explains.
Why drive stick when you have an automatic?
That doesn’t mean that a certain subset of developers at BT aren’t still looking to get their hands dirty with Kubernetes.
“It’s like driving an automatic car, there are aficionados who like the feel of changing gears and the clutch. With TKG available it’s more important to ask what is your specific need, not your want, and if you’ve done it the hard way, here is the easy way and it’s your choice,” Premchandran says.
“Kubernetes is hard… we won’t enjoy doing that at scale, so that is where we ask developers to focus on the application logic and let the hard things be automated for you,” he adds.
That doesn’t mean that self-managed Kubernetes is off the table, but really it should be reserved for use cases where developers really need that fine-grained level of control. For everything else, let the vendor take care of it.
“This is a constant tussle,” he admits, “where people want Kubernetes by default, but I think you’ve got to be a bit more prescriptive in the beginning to developers and then as you see them maturing, have them make those independent choices.”
Training and development
Once those principles and frameworks have been established across BT’s developer community, the next task is to scale it out via documentation and word of mouth buzz, both for in-house engineers and with external partners.
The advantage of having some developers getting their hands dirty with Kubernetes does mean that a new area of expertise starts to spring up across your organization. This is never a bad thing, especially when you are looking to evangelize a new technology, where a few respected voices can go a long way to getting that buy-in.
“If you look at how standards are democratized, you’ve got enterprise architecture that looks at the overall architecture standards and policies, then you have solution architects, who are a level down and are specific to a product, and finally we have distinguished engineers — we call them expert engineers — who are also key influencers for other developers, who look up to them,” Premchandran explains.
For partners, BT embarked on a pre-pandemic roadshow, running events and sessions to lay out the new direction and get them up to speed with the latest documentation, including a new Software Engineering Handbook and internal wiki.
“You can’t have a top-down diktat saying ‘march towards the cloud and let developers figure out their own learning part’,” he says. “We’ve invested heavily in certification programs, getting online e-learning platforms that allow career paths to be developed around the cloud and agile and DevOps and all of the associated skills.”
Now it is about scaling that knowledge and those skills across the organization. “We’ve learned the ropes, what happens when workloads are portable, now developers need to be full-stack engineers understanding hybrid cloud workloads and have the skills to train other people to modernize their own stacks,” he says.
Once all of this has been achieved, Premchandran hopes that he is overseeing a happier developer community deploying at much higher velocity than was previously possible.
“Infrastructure provisioning needs to be at a much faster clip to allow DevOps to happen,” he says, but it’s not just speed that he wants.
“This is an interconnected problem set. You want to go fast because your customers need things fast, but you’ve got disconnected teams,” he says. Kubernetes and other cloud native approaches promise a rote to bringing all of these teams together onto a common methodology.
Over the next couple of years Premchandran will focus on getting more applications onto a modern, cloud native, microservice-based architecture; implement DevSecOps practices for “completely zero touch deployments”; and, finally, increase the number of reusable software components across the group.
For example, if someone decides to build a billing component, can this be used by anyone needing to bill customers within an application? In an ideal world, yes.
The goal for Premchandran is a developer community that can move as fast as it needs to, without being encumbered by legacy or infrastructure concerns.
“Agile is not just about going fast, it’s about having the choice to go fast. There’s a difference. Just because your car can hit 250 miles-per-hour, that doesn’t mean you’re at that speed all the time. But I want a car that can go that fast if need be,” he says.