We've spent the past several decades using duct tape, string, and chewing gum to chain the world's computers together into this thing we call the internet, so there's really no good reason to be surprised that most of it is full of software containing vulnerabilities that are practically just as old.
You can listen to the latest episode of The Kettle right here on this page, as well as on Spotify, Apple Music, or YouTube. Those platforms also let you subscribe to Kettle, so you are always notified when the latest episode goes live.
The world is still reeling from the PHP2Shell revelations that hit the news cycle this week, and on The Kettle there's no better topic for host Brandon Vigliarolo, cybersecurity editor Jessica Lyons, and FOSS reporter Liam Proven to tease apart.
Join us to discuss what exactly is going on, how AI was key to finding another issue no human likely ever would have discovered on their own, and why exposing more bad code in the backbone of the world's computers might be just what we need to rethink the whole thing for the better.
A lightly edited transcript follows:
Hi everyone, and welcome to the latest episode of 's Kettle podcast. I'm your host, reporter Brandon Vigliarolo. And hey, there's only one thing we could focus on for this week's episode, given all the hubbub around it, and that's the PHP apocalypse. It turns out most of the internet is running software that could allow remote code execution on host systems via a pair, or as of today apparently three, vulnerabilities that have come to collectively be known as PHP2Shell. So that's not great. With me to talk about this is our security editor Jessica Lyons and open source reporter Liam Proven, who has a few things to say about PHP itself and why it's probably not surprising we're finding ourselves in this mess. Thanks to you both for coming on.
Absolute pleasure.
Nice to be here.
So Jess, let's start with the basics. What is this? How big of a problem is it? And where do we stand on getting things fixed?
Great. The short answer is it's a very big problem. So we have these two related memory safety vulnerabilities, and they are in the PHP core. Collectively they're called PHP2Shell, and they affect every version that's been released in the last decade affected by one or both of these, going back to about version 5.4. So the general consensus is this is about 70 percent of all of the websites on the internet are running this code. So it basically affects everyone at this point.
What happened here is that this dates actually back to August. An independent researcher first spotted this, and then we can talk a little bit more later about who found more of the exploits and what the (PHP) Foundation did with that. But so he found this vulnerability. There was a coordinated disclosure process with the PHP Foundation, and then that broke a little bit starting over the weekend, and then onto Monday, there were some leaks. There was a video posted that purported to show this exploit. And so we get this emergency warning from the PHP Foundation late Monday saying, "Hey, we are going to be releasing these emergency patches, so everybody get ready so you can implement them as soon as possible. It's very serious." So by Wednesday, they did release these, and there's this mass attempt to patch before any of the attackers get a hold of this and are able to attack websites with it. And one of the issues too is that some websites are running really, really old versions and so they aren't able to patch. So they're just sitting there, these legacy servers are still vulnerable.
So there's no direct update path for those old versions, right? It's basically shut this down and reinstall it kind of thing.
Exactly. So basically they're both these memory corruption vulnerabilities and by exploiting them, an attacker can kind of shape the memory that PHP trusts. And then they can use that to hijack what happens next. And that can mean remote code execution. They can take over the websites, they can steal data. And so people are visiting these websites they trust and they don't realize that the underlying code is stealing their data or delivering malware to their computer. And again, this is potentially 70 percent of the internet. So it's a big deal.
Quite a few sites were offline. I know even some big services. I stopped playing because I had a baby, but I used to play Final Fantasy XIV pretty religiously. And I know that the entire login server was offline a couple days ago
Right.
They were like, "You've got to fix this. Sorry, no one can play the game. If you're not logged in right now, you're not getting logged in for the foreseeable future." It's just insane how much of the internet was affected by this. Now It's under exploit, is that correct? But maybe not successfully exploited yet?
Right. So after the patches were released on Wednesday, later Wednesday, AWS came out and said, "We've seen exploitation attempts at least four days before the patches were released." So that kind of caused another frenzy that, "My God, it's under attack, the whole internet is on fire." There haven't been any successful attacks to date. There's been a ton of attempts.
There's a lot of POCs; some work, some don't. But as of Friday, according to the Foundation and according to all of the security experts that I've spoken to, we haven't seen any evidence of successful exploitation. I don't think that means we're in the clear. I think it just means it's taking a while for people – and honestly AIs – to develop successful attacks, but that's where we are right now.
Because it's a complicated attack chain, correct?
Right. Yes.
So basically, right. This is not working yet, but especially with some of these older websites that are running older versions of PHP that are going to have to take a more thorough approach to getting this fixed and possibly going down for some time, this could be one of those problems that remains serious for a while. And let's not forget, 70 percent of the websites on the internet, that's almost three-quarters of the entire internet. I don't know how many hundreds of thousands or millions of sites that is, but some of those are probably never going to get patched.
Right. Exactly.
Just the way it goes. Right. This is going to be the sort of thing that we end up hearing about months or years from now.
I would say years.
I'm sure there'll be some bank somewhere that eventually ends up getting – we'll hear about some breach disclosure letter from the state of Maine in four or five years.
And local governments, federal and state governments, I think these are the ones that we're going to be hearing about quite a while from now.
Because that's one of the things I was thinking about even for me in my tiny little town in northern Michigan. I wonder if my township or the county is going to be all over it to update their PHP server. Does their IT guy, who probably is a retired dude who lives in the area, even know this is a problem? It really is one of those things that is a big deal, but if you're not kind of following the news on this, it could probably go completely unnoticed. And for many websites, I'm sure it will.
Right. Exactly.
So, it's pretty crazy. Liam, you're working on a story, I think, as we are recording this, about some of the history of PHP and how it really probably wasn't very well designed or intended to become so ubiquitous on the internet. What's going on there?
I've seen one angle on this, which is people saying, "This is 1990s code and it has 1990s problems," which is one way to look at it. But I'm a 1980s internet guy; that's when I started exploring it. My primary email address is older than the web, and it still works.
My gods, it gets a lot of spam, but it still works. I've watched the rise and rise of PHP since the first releases. Now, I have no personal interest in web development. I can program, but I don't because I'm really bad at it and nobody needs to have my code inflicted upon them. The guy that wrote PHP in the first place, Rasmus Lerdorf, is my age. I've never met him; I'm sure he's a nice guy. But he said in a very early interview about it, "I really don't like programming. I built this tool to program less so that I could just reuse code." And that was about 1993. He built a little tool that allowed him to put macros in HTML for his own website, and it's just grown. In a later interview, he said something along the lines of, "I don't know anything about writing programming languages. I'm just improvising this as I go and people seem to like it."
It originally stood for "Personal Home Page," I think, and later was given a recursive acronym of "PHP: Hypertext Preprocessor." This is a programming language that you mix in with the HTML of a website. It predates the modern model of building websites where a database contains the text and a content management system of some kind assembles on the fly the pages that each person sees. This is an older model where you're running programs that are part of the markup language, which was not a good idea, but that was how he did it.
There was a lovely blog post. I've got an old friend who's an extremely skilled and talented programmer who has built a career on PHP, and he said to me, "Look, those criticisms that you've mentioned are of the really early versions and it's changed now." And if he hears this, he's going to be cross with me. But my favorite summary was from a blog called Eev.ee way back when that called PHP "a fractal of bad design."
Whatever level you look at it, it was done badly. If you look at the whole plan of "let's just intermingle program code in with the markup language," not a good idea. If you look at the details of how it was implemented, not a good idea. If you look down at low-level details of how the functions return values, not a good idea: inconsistent syntax, inconsistent functions, functions that do really weird things, by default no stack traces, not great error reporting.
Now, I'm sure a lot of this has been papered over now, but really there is one reason PHP is still around and still so big: Zuck. Facebook was built in PHP. And Facebook makes lots and lots of money and it's been throwing money at making PHP faster and more reliable and more scalable for decades.
Have they been big contributors to it?
Yeah. They even built a compiler for it, which I suspect is still in use. Now, how much of Facebook today is in PHP? I don't know, possibly none, but for a long time basically the world's biggest and busiest public-facing website that wasn't owned by Google – only because Google have failed to ever make a successful social network – was built on PHP. And it's big, it's complicated. Like Linux itself, nobody fully understands how the whole thing works anymore and hasn't probably for decades.
And now people have got bots that they can throw against it and ask them to analyze it. I think this is probably just the first, and I think there's going to be many more. A friend of mine who's a senior developer had a look at the patch, and he's not a C guy, but he said – I'm quoting from comments on Mastodon – that the patch to the current version is 13,000 lines of C. That's a doozy of a patch.
That's huge.
A bunch of people have been working like crazy for quite some time. Some people who've looked at this say it looks handwritten to me. Others have said, "No, I think this is bot-generated." I think the truth is probably a bit of both: some from one, some from the other.
Bit of both. I mean, you can't pump out thirteen thousand lines that quickly and have it tested. God, I hope they tested it.
Right. But that's just for the current version, and this hole affects every published version. So if they do manage to fix this one, how long till the next one comes along? Because I think there's going to be –
Especially with the speed at which vulnerabilities are being discovered now. I feel like every time I turn around, I've written one Patch Tuesday article in the past few months, and that one broke the previous two records. And I think since then every month has included more patches, and a lot of those are probably –
October at this point is the record: over a thousand for Microsoft alone. So, yep.
That's insane. Like you said, it is a fractal of bad design all the way down, and is going to probably have more vulnerabilities discovered.
Right.
Seventy percent of all websites on the internet are using this. What can we do? Are we stuck?
Well, there's a reason for that, and it's one that doesn't get talked about an awful lot: Facebook aren't the only people that use it extensively, because WordPress is also built on PHP, and it's far from alone. Another senior Linux techie friend of mine gave me a free shell account on the machine that he runs for a few friends years ago, and I thought of resurrecting my old 1990s hand-coded HTML webpage for fun. So I found it on the Internet Archive and found a tool to scrape it and turn it back into HTML again. I looked at some CMSes and asked him if there was any CMS on the machine.
He went, "No."
And I went, "Well, would you install one?"
He went, "No."
"Well, why not?"
He said, "Well, all the big ones are implemented in PHP, and I won't let that near any server I run."
"Well, that's a bit dramatic, isn't it?"
And he goes, "No, that's pragmatic. I run a bunch of public-facing servers on my own, and this is how I keep them secure. No containers, no Docker, no PHP, nothing built in any of these technologies."
And I went, "Wow, what about..."
So I went looking for non-PHP CMSes. There are a lot of CMSes out there. I got a commission about a decade ago to write a comparative assessment of the CMS market versus a company's proprietary Windows product. I went and did a survey and came back and said, "I've found 1,300 so far."
And they said, "OK, we can give you five thousand words instead of a thousand words."
And I said, "You realize I'm going to have to go and look at 1,300 pieces of software here. Even if I only write three or four words about each one, 'Mostly harmless,' I'm still going to have to look at over a thousand bits of software. It will take me a year."
And they went, "Well, how can how can we do it quicker?"
And I said, "You can't. I cannot do this survey you want. Give me a full-time job and I'll get it back to you in a couple of years' time."
I found a few CMSes that weren't built in PHP. Most of those were built in Ruby instead. And he went, "Yeah, I'm not keen on having that on my server either."
So it's not even a case of people that have PHP deployments needing to update. They don't know. They built it on something else which was built on something that was built in PHP.
I was just going to say, Liam, you said, "How long until the next one happens?" And we found out today from the PHP Foundation that there is a third one. They won't be pushing an emergency release for this. They said it'll be published with the next regular release cycle. The Foundation told me that this one, quote, "requires a very unlikely combination of configuration and specific PHP application code, which we do not expect to occur in practical use." So, there's already another one.
So weeks then.
Yeah.
I mean, and even with this, it's hard to exploit. You can mitigate it with a configuration setting. But that was part of the thing we've heard with these other two. And then we did have researchers yesterday who developed this new exploit; this was Vulncheck. And that shows how even ASLR can be bypassed to attack these vulnerabilities on x86 machines. So saying it can be fixed with a configuration setting isn't always a guarantee.
That brings me kind of to the elephant in the room of this whole story: AI. because of course it all comes back to AI nowadays. OpenAI and two other security firms reported this. I think Jess, one of your stories or one of the ones Lindsay wrote said that they all used AI to find this, right?
Right. So the companies: Calif, they are a newer startup doing AI pen testing and they're part of OpenAI's partner program, so they used OpenAI models working with OpenAI to find this. And then independently, Wiz, which is owned by Google, also did. So this happened back in August, and then they have been all working with the PHP Foundation on the patch since that point.
They have all said these are vulnerabilities that probably wouldn't have been found by humans alone. By all accounts, they've existed for decades and wouldn't have just been found by human researchers.
Right. So either these are things that have never been discovered and would have never been exploited theoretically – the exploit path is complex, they're not obvious vulnerabilities, so they needed an AI to find them – or perhaps this is the kind of thing that was known about but was privately held in the hands of some bug hunting company that sells exploits to governments. Who knows? Maybe that's the case. I remember talking to a guy one time at Black Hat who basically said his company's in the business of finding really sensitive vulnerabilities and then selling them to the highest bidder. So there are companies like that that exist. So there's probably plenty of vulnerabilities in software that are not well known that governments and high-level functions are aware of. But in this case, this probably wasn't the case. This is an incredibly difficult-to-find vulnerability.
I've seen a lot of chatter online in the past few days about how the internet's on fire and the AI bug-pocalypse has arrived. It seems like we're having another case of people getting all freaked out because this is the end of time, history is over, and AI is going to kill us all. It just seems to me like this is another example of: well, maybe not. This has been getting hammered by potentially bad actors or pen testers, and no one's managed, as far as we know, to successfully use this yet. Is this as serious as it seems, or is this another case of a serious vulnerability that is so complicated and convoluted that it's not really going to be that big of a deal?
I want to note parenthetically that you described OpenAI as a security company, and if I had been drinking coffee, that would have been very bad for my keyboard.
Okay, whoops. Let me take that back.
Ha ha ha.
Calif is interesting because I just wrote about them. They found a hole in OpenBSD, and there aren't many of those. And only about a week ago, they found a hole which they dubbed "Big Fish" in OpenBSD, and it was just days before OpenBSD 8 shipped. It looks like that had been there for 37 years, which is a neat trick because OpenBSD was founded 30 years ago. So this is in – going to pronounce this carefully – Berkeley Software Distribution Unix in the eighties. But there are doubtless more, and I'm an arch AI skeptic.
Some people will claim that this will be the sort of problem that you can solve with AI by getting it to rewrite your stack. Well yes, but then you'll replace it with a new stack that nobody ever understood any of. But there are ways to design, host, and run websites, forums, and chat channels without using any of this stuff. And it could be a net positive if holes like this keep getting found that people will be forced to go back to – sounds crazy, but – static websites serving pre-written code off a disk.
One of my favourite websites – recently I discovered there's a whole chain of them, and I hope you won't mind me swearing; maybe you have to beep me. But there's a marvellous website called "This Is a Motherfucking Website and It's Perfect." And recently somebody rewrote it and the new one looks almost identical: "This Is a Modern Motherfucking Website and It's Perfect." He's written it with more CSS or something, but apparently there's a whole chain of these things and they are extremely low on programming code. They contain interactive elements that you can do with just HTML, and you don't need a whole web framework behind it. And the same AI companies are busily scraping the whole web multiple times a day to feed their training engines. Maybe we just have to suck it up, scrape some of these big websites, and redo them as simpler tech with fewer interpreted languages. We are all collectively victims of the sunk cost fallacy. We've got millions of websites. We spent billions and trillions of dollars building all this. We can't just throw it all away. Well, maybe you have to.
Sunk cost was something that kept coming to my mind as I was writing up the script for this episode and just thinking about this. It seems like a whole big case of, "Well, all the websites in the world are now on PHP, so I guess we're stuck."
We have a script?
Well, not a formal script, just notes.
Haha. Well I didn't get the memo.
I think that's part of the problem though: it illustrates this larger issue with AI in general. AI bug hunting and exploit detection kind of illustrates this bigger problem about the internet, especially these legacy systems that underpin government services and critical infrastructure. It is literally held together by duct tape and string. So it's a massive problem. Again, it takes some work to actually make anything of these vulnerabilities and chain them together. AI is getting better at that. So whether or not this ends up being the end of the world for the internet as we know it, it does illustrate a real problem.
Like Liam said, maybe this is the sort of catalyst that the wider internet community needs to realize that these systems – this duct tape and string – simply won't keep working forever. We need to maybe modernize all the underlying infrastructure and make it simpler and less kludged together with all these different things over the years. PHP is poor design all the way down, right? It's like: well, maybe we need to tear it out and start over. Even if these vulnerabilities are impractical to exploit and hard to find, the fact that they're being discovered does illustrate that most of our code is a mess.
I am reminded , the other day I was talking in the comments about A/UX, Apple's original Unix for the Mac, and Apple paid another company to develop A/UX because the US military had a tick box on the form that said to buy systems, they have to be able to run POSIX. The US Army at one point hosted its website on Classic Mac OS. And it wasn't exactly security by obscurity. They said virtually all web server attacks in the mid-1990s were based on getting a remote shell, and Classic macOS didn't have a shell at all, anywhere. There was no way to get a remote shell because you couldn't get a shell if you were sitting in front of it. So they said, "They try and inject bits of x86 code: well, it's not an x86. They try and get a shell: it hasn't got one. They try and get it to run scripts: it doesn't run scripts." They said, "We just sit there and watch people throwing themselves at it. All the best attacks, and none of it works because the operating system's kind of too primitive to do any of that stuff." I don't believe many customers actually ever used A/UX, but it got Apple some lucrative deals. Maybe we're going to see a new trend of super primitive, super basic websites. There'll be a new difficulty in trying to scale the things because Kubernetes expects a lot of this to be dynamically generated by microservices. But bring it on; I'd love to see that.
That's a Kubernetes problem, right?
Yeah.
I feel like I would love a simpler internet. It would be amazing if I could just wake up in the morning, go to a news site, and just get the headlines. So often, I feel like – and I think is good at having a relatively simple design, but – so often I feel like the news aggregators that I go to... I look for ones that are as minimal as possible. I look for websites that are simple. I don't need all this in-your-face stuff; it's not necessary. I think we probably are all pretty sick of it in a lot of ways.
But do we really think this is going to happen though?
No.
I feel extremely pessimistic about this. I think it's much more profitable to just keep marketing solutions to the problem as opposed to a safe coding overhaul.
Unfortunately.
Geopolitics might yet get in the way. I wrote a piece about OpenEuler, one of the big Chinese enterprise server Linuxes, and one of only two companies actually showing off an OS at the Open Source Summit. The other was also Chinese: OpenKylin, which is more desktop focused. China makes its own chips, China makes its own operating systems, and it's got its own programming languages. They're all open source, taken from European and American developed software by and large, but they could still invade Taiwan and cut off much of the world's chip supply. And suddenly it won't just be that memory is way more expensive than it used to be; it will be that you can't get more memory anymore and you have to keep using 10-year-old chips. And people will be potentially forced to find ways to keep getting more out of the same hardware until we can build new chip fabs, which will take a lot of time and a lot of money. And then quite possibly they'll get blown away in a climate-change-induced hurricane and then we'll have to start again. So, I hope they're going to start building them in the north of Norway and places like that. But yeah we've got bigger problems coming down the pipe and it's all a giant snowball of superficially unrelated things: AI attacks on overcomplicated stacks on one side, and the other kind of attack which involves whistling noises and large explosions on the other side, which cuts off the hardware market. It's hard to plan for this, but nobody's even trying yet.
Who would have thought a talk about a PHP vulnerability would have turned into geopolitical chaos and the endof modern technology as we know it? Break out the sundial, people.
Excuse me, I've just got to go and wind the web server. Hang on.
All right, guys. Thanks for coming on. I think this was a really valuable discussion and it shows just how interconnected all these issues in tech are right now. I mean, it is interesting how we're kind of hurtling toward something in a lot of ways. AI will surely be a part of it, as will expensive chips and lots and lots of vulnerabilities being found every day. Hopefully we'll still be here to cover it as that continues to happen.
If AI doesn't kill us.
Right. If AI doesn't kill us all. Well, if it doesn't, we'll be here on the Kettle to talk about it. And we hope that you, from whatever bunker you're hiding in, will join us on future episodes.
Yeah, we'll be publishing a telephone number so you can phone into future Kettle episodes.
All right, folks, we'll talk soon. ®