A security researcher who has previously exposed GSM and SS7 security issues has warned that telco adoption of cloud based operations for IT and network functions leaves them open to a host of new security vulnerabities.
Karsten Nohl of Security Research Labs, speaking to the May Contain Hackers conference, said that his team of researchers had found ways to move from an initial penetration of a cloud instance, gaining enough credentials to spy on user communications, extract user data, gain system admin status and ultimately to take down a network.
Although Nohl’s speech was headlined as exposing Open RAN vulnerabilities, it was in fact by Nohl’s own admission not really about telecoms protocols, still less about Open RAN itself. There is nothing specific about the Open RAN architecture, protocols or interfaces that his researchers have called out. Instead, Nohl focussed much more on the generic vulnerabilities that the increased use of cloud software and automated processes can entail in any environment and, as such, the vulnerabilities found would apply to other private cloud instances, he added.
However, with telcos increasingly adopting containerised functions deployed on Kubernetes and automating processes via APIs, the attack surface arising from cloud adoption is increasing and needs to be addressed, Nohl said. Open RAN, such as it plays a role, does so because it is one further driver of cloud function adoption, and of automated processes. He also described one potential exploit that could target the Radio Intelligent Controller, via its cloud host platform.
Nohl described how his “red team” hackers had, over days and weeks, been able to find credentials to get access to one part of the network cloud stack, and then moving from that, to extend their position of control in the network.
The exploits showed that the security risks from moving to a cloud ops, CI/CD pipeline are social and well as technical.
One of these is deployment of container configurations that don’t segregate physical resources between functional components. That means that if something that is considered less secure gets hacked, then the hacker can break out of that environment and influence something below them (Kubernetes) or into other neighbouring containers.
“It turns out there are multiple paths for doing this kind of container breakout,” Nohl said.
Most commonly these involve a configuration that assign a Privileged or sys_admin capability to a container which is not considered security-critical, but which can then be exploited to gain access to other containers or the Kubernetes host.
Another vector was to to exploit host_PID namespace access that has been set for containers, and use that to kill processes on the host machine. “If you combine that with another capability, ptrace, you can also inject code into the process,” Nohl said. “That’s basically root level access, where two benign looking things are combined for a full exploit.”
A final hack was to use network access so that a guest can access the local host. “At the very least given a shared interface between guest and host, the guest can tcpdump everything from the local host, unless they use SSL. But Kubernetes clusters are built so that everything in a cluster is trusted, so admins assume they don’t need SSL: giving network level access to a guest usually means the host gets hacked.”
All of these, Nohl said, are exploits that his team have seen in real assessments, and are not just theoretical.
A second risk for telcos moving to cloud ops comes about due to the nature of the people and processes that involves. There are simply “many more developers involved”, and with the increased use of automation and software being pushed through CI/CD tools, some threats are amplified.
“Instead of five Linux system admins you can now phish hundreds of people across various companies, all somehow contributing code and if you get to any of them there’s a good change that ultimately you can influence a mobile network. There’s an ecosystem of software development tools that now are part of the development of a network: somebody commits something to Github, that is packaged somewhere, imaged and the image deployed – if any part of that chain gets hacked you are at risk.”
Nohl described how his team had found sensitive code being posted within a developer query to Stackoverflow, old crackable passwords left discoverable, old development sites left online.
One journey involved finding an old development website that was itself isolated from the production network. But it was running in a Docker container on Kubernetes, and those access capabilities were assigned to the container. “So we break out of the container on Kubernetes, and now we are not constrained any more.”
In this instance Nohl’s team “very slowly” started looking on the internal nework, finding hundreds of services and APIs connecting network microservices. “Then if you send ‘wrong stuff’ on one API you get debug info back and in there is a cred of one of the developers. That then allows us to access data lake systems and in there we find customer text messages. It was a multi week hacking journey to do what took us one minute in 2G, but now we have the text messages of a whole country”.
“That is one journey and it is not targeting telco standards. What matters is this is a virtualised network with lots of automation fragments floating around.”
A third hack saw the researchers get into the telco domain, specifically the RAN Intelligent Controller. “The RIC is an optimisation piece of software in each one of hundreds of dockers in hundreds of Kubernetes and sure enough, again, they didn’t configure the Docker sufficiently – so we can break out into all of these can hundreds of K8s environments and take down the network. You can see how few steps are required – and again none of this is telco specific,” he said.
Nohl said his team’s findings showed that security is still not being done “by design” within telco teams. Patching and hardening needed to be done at an absolute minimum on a near-continual basis. Netflix, he said, has a 72 hour self-destroy on its Docker containers, after which thet are rebuilt from the CI/CD pipeline that has all the new patches built in.
He also asked hackers to take also telco security seriously, as networks move into the cloud. “We invite you to consider that. We rely on them [telco networks] and is it important to keep them secure.”
