Interview w/ Nat Friedman on the Mysteries of the Scrolls
We are on the cusp of a New Renaissance
I recently got the chance to interview Nat Friedman, former CEO of GitHub and creator of the Vesuvius Challenge which aims to crack the riddles of the Herculaneum Papyri. Nat has become a fan of the show, and I had a great time talking to him.
In this episode we discuss:
The Genesis of the Vesuvius Challenge
Early Attempts to Open the Scrolls
Using a Particle Accelerator to Scan the Scrolls!
Partnering with Daniel Gross and Brent Seales
Nat’s Childhood experience with Open-source Communities
How to Design Prize Incentives for a Complex Contest
Doing Crazy, Strange and Risky Projects
A Possible Resurgence of Epicureanism?
This episode is sponsored by Ancient Language Institute. If you’re interested in actually reading the newly unlocked scrolls, you will need to know the languages of the ancient world. The Ancient Language Institute will help you do just that. Registration is now open (till August 10th) for their Fall term where you can take advanced classes in Latin, Ancient Greek, Biblical Hebrew, and Old English.
Stay Ancient,
Alex
Transcript
Alex Petkas:
Julius Caesar's father-in-law owned a villa on the shore of the Bay of Naples in a little town called Herculaneum. This is the father of Calpurnia, who was the longest standing and most respectable of Caesar's wives, his final wife, the one who is a character in Shakespeare's play, Julius Caesar. Well, this man, Calpurnia's father, Piso, was his name, Calpurnius Piso. He was a personal enemy of Cicero. He was the consul in 58 BC. Cicero wrote a speech against him, delivered it in the Senate. We still have that one.
But Piso was also a dedicated student of Epicurean philosophy, and he invited a famous philosopher named Philodemus to come and reside at his villa in Herculaneum, and Philodemus brought a vast library of Greek philosophy with him from Athens.
A hundred years or so later, that library Philodemus curated was still in that villa when Mount Vesuvius erupted, and along with the city of Pompeii, the Vesuvius eruption buried that villa in Herculaneum too—carbonizing those scrolls in pyroclastic mud and ash, and thereby sealing a treasure trove, a time capsule, to be uncovered centuries later.
Scholars have known about this library and these scrolls for more than two and a half centuries. Hundreds of scrolls have been unearthed and possibly thousands more still lie buried in that villa. But for the most part, we've been unable to read these scrolls, or we've only been able to read certain bits of them and with extreme difficulty.
Until now.
We are on the verge of some of the most incredible discoveries about the ancient world since the Renaissance. And in this interview, I am talking with one of the people who's at the center of the story of decoding the Herculaneum scrolls. His name is Nat Friedman. He's not an archaeologist or a historian, but a technologist, a company founder, a former CEO, and we'll get to the details in the interview.
I'm Alex Petkas. You are listening to The Cost of Glory, where it is our mission to retell the lives of the greatest Greek and Roman heroes following the lead of the ancient philosopher and classic writer Plutarch. In order to try to unearth the characters of these men and their insights, if we can, to resurrect some of the ancient spirit of greatness those real-life heroes exhibited, and with which they have, over the centuries, periodically in times of great trouble, inspired the more active classes of society.
I'm happy to announce, by the way, that the Cost of Glory is now an Infinite Media collaboration.
Well, without further ado, here is the interview with Nat Friedman, where Nat tells the story of Herculaneum, the Scrolls, and this fascinating contest that he co-engineered to help push the scientific research and technological innovation past what seems to be a tipping point in the effort to decode these texts. And we'll talk about some of the other very important people involved, modern people, and also about Epicurean philosophy, about Philodemus, and many lessons that you can apply for bringing out the best in your own work in life. I hope you enjoy it and find it as inspiring as I did.
Alex Petkas:
Nat Friedman is an entrepreneur and investor. He's a co-founder of Simian and Xamarin. I hope I said those right. The latter of which as acquired by Microsoft in 2016. Nat stayed on at Microsoft, and in 2018, he led Microsoft's acquisition of GitHub, an important open source platform of which he became the CEO of in 2018 until 2021. Then he moved on to other projects, which brings us to maybe the number one reason and occasion for our conversation.
One of these cool projects that's come across your desk, Nat, the Vesuvius Challenge, which is in sum... We're going to get into the details... a mission to scan and digitally unroll and read a large trove of ancient scrolls that are preserved through the famous Pompeii destroying eruption of Vesuvius, but it's never been possible to read them before until now. So at the Cost of Glory, we're all about bringing to life great ancient figures and texts and ideas from the past, from the Greek and Roman world that you can use today. So for that reason, I'm really thrilled to have you. Thanks for joining us, Nat.
Nat Friedman:
Thank you for having me. I love your podcast.
Alex Petkas:
Great. Well, why don't we start by just you telling us, for those who don't know the story, how does a guy like you get interested in a project like this?
Nat Friedman:
It was really pretty random, honestly, one of those sort of sequences of events that seemed so contingent and unlikely that you have to wonder how predetermined anything that ever happens really is. But yeah, I mean, it started back in early 2020. I was out here in California. I was running this company called GitHub, and COVID struck and the lockdowns kicked off, and we were in that early period of COVID when there was so much uncertainty about what it was. I don't know if anyone really remembers, obviously COVID was not good, but there was a lot of concern at the time that it was actually much, much worse than it ended up being. People were disinfecting their groceries, and there was just a lot of worry in the world.
And the company I was running was a great company. GitHub is a fantastic company, amazing people, but man, not always easy. And so I was just found myself looking for an outlet, looking for other things to occupy my mind in the evenings. And I stumbled upon this book that I think Amazon recommended to me, which was called 24 Hours in Ancient Rome. And as a kid, I'd never actually been very interested in history, to be honest. I was a math and science kid, and learning about math and learning about science seemed to me like you were in pursuit of universal truths of the universe and these things that would be important and true in any world. And history seemed very contingent and parochial to me, and something that just was like a sequence of events that had no universality to it. That's what I thought at the time.
Alex Petkas:
Aristotle would've agreed with you actually. He says history is all about particulars rather than the philosophical truths. It's very interesting.
Nat Friedman:
Yeah. I mean, I did later, I think after college, I did realize that even if that were true, this world with this history is the only one I would ever live in. And so even if all these things were contingent, I would only have to learn them once. And so that did wake me up to the fact that maybe it was worth paying some attention to, but I didn't really do it. I mean, I took some classics in high school and I did enjoy reading Plato and Homer, and it was just fun food for thought, but to me it almost seemed like reading fiction.
And so anyway, for whatever reason in this period in early 2020, this book appeared my Amazon recommendation list, and I decided to give it a shot, and I ended up staying up all night reading it. It is fantastic book actually, 24 hours in Ancient Rome. I really recommend it. Someone later told me that it was written for schoolchildren, which might be true, I don't know, but it doesn't matter. It was fantastic. It really drew me into thinking about that period.
And so I found myself over the subsequent evenings in these different Wikipedia-chasing-and-surfing exercises. And I think there was something about reading about ancient history at a time when the world was unsure about its own future that was just deeply calming. In either large amounts of time or large amounts of space, your problem seems small.
Alex Petkas:
I love this image of you as a CEO of this $7 billion acquisition, huge enterprise, taking the time to reflect on the past. Maybe it's the kind of chaos of COVID, really not sure what you're supposed to do, but maybe that's the answer, that it was somehow calming.
Nat Friedman:
It was. It really was. Your own issues or problems seem pretty small if you realize that your life is short and there have been many lives before you, and there will be many lives after you. And the other thing that I think has a similar effect is astronomy, just thinking about the size of the universe. You maybe hopefully become slightly less petty if you pay attention to these things. So maybe it was that, or maybe it was just interesting too.
So there's this chapter in the book that talks about the way Roman bread was made and everyone was into making bread back then. So I went out and sought out the heirloom wheat varietals, this red fife flour, and tried to make my own Roman bread. And anyway, so one night I'm in Wikipedia and I stumbled on this article about the Herculaneum papyri, and I'd never heard of them before. I thought it was just really incredibly fascinating. Obviously, Vesuvius covered a whole region of the Bay of Naples in mud and ash and incredibly hot gases. And everyone knows about Pompeii, and I'm sure I'd heard about Herculaneum before, but I'd never heard of this.
And so there was this enormous villa, private villa that was just on the outside of Herculaneum that had been buried by the eruption as well. And it was thought to have been built and owned by the father-in-law of Julius Caesar, this guy Calpurnius Piso. And what made it really special is that in the mud, as it was excavated in the 1700s, late 1700s, these lumps of charcoal were discovered. And I think at first no one knew what they were until someone noticed on a broken off flake of one of them a little bit of writing. And the excavators realized that they'd stumbled upon a library, a library of papyrus scrolls made from the Egyptian papyrus plant.
And this was a unique find because, somebody gave me this great phrase, "Humidity is the enemy of history." And so to the extent that there were libraries along the Bay of Naples at that time, if they had not been buried by a volcanic eruption, they would've just eventually rotted away. And so therefore, we have really almost no intact books or works of writing from that period unless they were copied by a cascade of monks through the Middle Ages and preserved through copying, and anything that was original is gone. There are some exceptions in really arid places like Egypt, I later learned. But those are unfortunately the exception and not the rule.
And so hundreds, actually thousands of scrolls were excavated from there, and then there were attempts to open them. And most of those attempts were in the late 1700s were quite destructive. People did things like they took a dagger and they would just scrape off the layers of papyrus or slice it, and maybe they'd see a few letters and they'd try to write those down along the way, but this was very, very destructive. It's-
Alex Petkas:
It's horrifying to think now with what we can do. Yeah.
Nat Friedman:
Yeah. Well, the worst part about it is, as I later learned, the scrolls that were attempted to be opened in these really physically destructive ways were the best ones because obviously the excavators looked at the ones that were in the best shape and thought these will be the easiest to open and read. And so it's probably just through the selection bias the worst ones that we have now were not destroyed along the way.
There was a monk, this guy named Piaggio in the late 1700s who ended up designing this machine to attempt to unroll them. And it used a cat gut and pig bladder, and it was semi-successful. So it would, in the case of the best scrolls, be able to unroll parts of them. But it was also pretty destructive overall too, so layers would stick together and there'd be gigantic holes in the unrollings that he did, but we were able to apparently determine from that that these were primarily Epicurean writings of this Epicurean philosopher who's a follower of Epicurus, this guy named Philodemus.
So there had been a little precedent for looking inside these things, but what was left couldn't be opened. And what was left was huge, hundreds and hundreds of scrolls, and in fact is the largest library that we know of from antiquity and is a mystery. I mean, I loved that. It's a mystery. What's in there? No one knows. And there were no known techniques for either opening them or figuring out what was in the papyrus without opening them.
Alex Petkas:
It's amazing. I did not know the part about cat gut and pig bladder.
Nat Friedman:
Yeah.
Alex Petkas:
So before the last five years, there were some number of maybe the easier ones to open, many destructive losses of texts in the process, but I guess in recent years, it seems like the technology has sort of... And I remember when I was in grad school hearing whispers about this, like, "Oh, they're getting close, they're getting close," but it always seems like asymptotical, like we're not actually getting closer.
Nat Friedman:
Yeah. Well, first I think it's just intrinsically exciting to have this opportunity. And I think everyone had these dreams about having a time machine and going back to some period that's particularly interesting to you and being able to see what it was actually like, or I remember when I was a little kid either thinking of or reading about this theory that maybe you could extract audio from pottery wheels, and if there had been some reed that was brushing against the pot when it was being spun, then maybe it would've like a phonograph needle recorded audio from the room, and just being totally fascinated by that idea. I had this idea that maybe as paint was drying, the pressure waves from the sound in the room could end up being encoded in the physical structure of the dried paint. I no longer think that makes any sense at all, but when I was 11 or something, I thought that was a neat idea.
So we all have had these theories or fantasies about how cool it would be to get a real window into the past. And so I think that's what fascinated me about this was like, wow, if you could actually get all these scrolls and read them, it's a lot. It's a of text and it would add, in terms of the total megabytes of information we have about the past, it would add a really material amount.
And so I think that was one of the things that excited me. And then I think it was also just this anti-authority impulse of there are these mystery scrolls, they're unopenable, you can't open them and someone tells you you can't open them and you think, "We've got to find a way to open them." So I just thought it was really cool. I'd never heard of it before and started talking to my friends about it, and it turns out none of them had either. And so somehow this was like... My friends are from tech and software, and so maybe we just aren't educated on this. But I thought it was just such a neat mystery that people have worked on for hundreds of years.
And then I think maybe later that night I found an article about an effort that was underway to try to read the scrolls using 3D imaging. And that there was this professor at the University of Kentucky named Brent Seales, who had been working for years, actually, I think at that time for about 18 years, to scan the scrolls in 3D at high resolution with something like a CT scanner... If you have an injury, you might've been in a CT scanner or part of your body was... To be able to look inside them and then hopefully be able to use software to not physically, but digitally unroll them, and then recognize the ink and read what's in there. And he'd been working on this for years and years.
And so that was when I found about the Herculaneum papyri. I thought that was cool. When I found out about Brent's project, I thought that was the coolest thing I'd ever heard, and I just became a fanboy and thought I'd follow along. So that's how it started.
Alex Petkas:
Interesting. Yeah. So had he scanned a scroll, a full scroll at that point, or had it just been fragments?
Nat Friedman:
Yeah, no, he had over several years. I think his first scans he did were in 2009 and they were on a low resolution. I think maybe it was a medical grade CT scanner. And when you do a medical CT, you don't really need to see things at the millimeter or sub-millimeter level most of the time in order to see if your wrist is broken or whatever. And so that was an important step, but those were not usable to try to read the scrolls.
And then the thing that really excited me, because I found this in early 2020, was that I read that in 2019 he had scanned two of the scrolls in a particle accelerator. And the particle accelerator was helpful because it allowed him to get much higher resolution and theoretically a resolution that was good enough to be able to read the text. And so when I discovered this just months after those first scans, I thought, "Okay, a breakthrough is imminent." And I went and set up some Google alerts just for Brent's name and for the papyri so that I'd get an email if anything was announced and I wouldn't miss it, and that was where I was in early 2020.
Alex Petkas:
And when we're talking about particle accelerator, was he using the one at Oxford?
Nat Friedman:
Yeah, so it's a synchrotron at Oxford, near Oxford called the Diamond Light Source. And the thing you'd get with a particle accelerator that's helpful is it's really bright, so you get a high flux of photons from that, and that makes the scanning faster, which is helpful. And then the other thing you get from a particle accelerator that's nice is the beam is parallel. Because the scrolls are pretty big relative to most things, you would scan in a bench top CT scanner at high resolution, the parallel beam allows you to get uniform resolution across the entire scroll as opposed to a conical beam, which would give you high resolution near the source of the photons and much lower resolution on the other side of the scroll. And so the particle accelerator, well, first it's just freaking awesome to be able to use a particle accelerator to do something-
Alex Petkas:
This thing is gigantic, right?
Nat Friedman:
Yeah, yeah, oh yeah, they're like buildings.
Alex Petkas:
It's like the size of many buildings. Yeah.
Nat Friedman:
Yeah, it's like one donut-shaped building. I don't actually know its diameter. I think it's certainly hundreds of meters. Maybe it's a... Yeah, I'm not quite sure. We can look it up. But yeah, it's a building. It's a big building that's a big circle. And particle accelerators use magnets to accelerate electrons, and then at the end they hit something and they cause it to shoot off photons. And if you do it at the right energy, you get X-rays coming out of it. And those X-rays then pass in a parallel beam through the scroll, and then you have a little detector on the other side of the scroll that's sort of like a camera, like a CCD sensor in a camera, and that can see the variations in light intensity that the photons have after they've passed through the scroll.
And so then what you do is you shoot a bunch of those and you spin the scroll in place. You just slowly rotate it, and that allows you to get these variations and intensity at different cross-sections through the scroll, and then you do some math. There's an algorithm called the inverse radon transform, which allows you to reconstruct based on all these different images the 3D structure of what's inside the scroll. And that whole process is called tomography, and that's where the TNCT comes from. It's a computed tomography scan.
Alex Petkas:
Yeah, I love the technical explanation that you do it so simply and well. It's just amazing to think how much human brain power had to come together to make this thing remotely possible, and yet it is possible. As you said in a tweet a while back, how would the ancient Romans feel if they knew that 2000 years later we would be using particle accelerators and supercomputers to read their words, preserve them for eternity and whisper them into the ear of a baby God?
Nat Friedman:
Yeah. Yeah. I mean, they might be horrified by everything about us, actually, but there's probably... I think, I mean, you would know a lot better than me, but it seems to me from listening to your podcast and reading a little bit that the ancients were pretty concerned about posterity and legacy. And so maybe the idea that we're working incredibly hard to resurrect their words and then preserve them on the internet and maybe put them into AIs, maybe they'd like that. I don't know. They'd certainly find it interesting.
Alex Petkas:
Yeah, I think it's very much in the spirit of antiquity. People look at, they say, "Oh, well, Plato has already had this classical culture. They've had Homer for 300 years." Homer himself is already classical. He's speaking this language that is very old and archaic, and there's a deep desire to preserve this and to pass it on, and Immortal the Bard is kind of like the vector of immortality for these heroes. And I always think that people look as far back into the past as they want to look into the future, so there is something really classical and frontier spirit I think about bringing this stuff to life.
Nat Friedman:
Yeah. So anyway, just picking up the story, that's how it started. And then a couple of years passed and my Google alerts did not inform me of any major breakthrough. We didn't know what was in the scrolls yet. And I had this excuse, which was that my friends and I organize these get togethers occasionally in California, small groups of people who get together to share what they're working on, interesting projects. And we use it as a way to just try to invite folks who are doing cool things and whatever we think is fascinating that's out there in the world.
And I thought, "Gosh, I should really invite the scroll guy, Brent, and he can talk about his projects. I know none of my friends know about this, and I think everyone will find it as fascinating as I do." And so I reached out to Brent and invited him to come out to California for a weekend and spend it with us and chat about what he was doing. And it was actually a little bit hard to get him on the phone, but when I finally did, he was game, and he came out and I was really excited.
I got him to do a talk, and I had lured him out with this promise that if he came, there'd be a bunch of great AI people there, maybe they would get excited about the project and want to help or have some ideas that could be useful for him. And there were some successful tech founders there, and I knew his project was funded, but relatively small team and maybe somebody would get inspired and write him a check or something like that. And so that was my bait that I used to get to meet him. I was just starstruck and thought he was doing this cool thing.
But the weekend passed, and though it was a lot of fun, nobody else was as excited as I was. I was totally scroll pilled by the end, but somehow it hadn't worked on anyone else. And so then I basically just felt bad because I had invited Brent out and promised something would happen and nothing had happened. So before he left, we got together and I'm just trying to salvage his trip, and so out of embarrassment, basically I suggested to him, "Why don't we work together and we could launch a contest and try to get people on the internet who were smart to work on this?" Because it seems like once you have the scans, which he had done, it's just a software problem and someone who's working from their laptop anywhere in the world might be able to make some progress and contribute to this and get it done faster.
I was partly aware that as an academic who'd been working on this for almost 20 years, Brent might not be open to that because academics want to finish their work and receive credit. But I think Brent's a really unique person, and I think he was probably also getting impatient and just wanted to figure this out. And so he said yes, and I got excited and I promised I'd chip in the prize money. I offered to put up $100,000 or $125,000. And then my friend Daniel Gross, who was with me, said he'd match that.
So we had a handshake with Brent and a quarter million dollars, and I left. I drove home pretty excited about this, but then it pretty quickly dawned on me what a huge project this was going to be. I mean, the scans are big, the techniques are complicated. There's a lot of things you need to know to make any progress on this at all. It's not like a simple math puzzle or something like that. There's how the scanning works, how the tomography works, the file format of the scans, the tools that need to be used to unroll the scroll, some of which Brent's team had already built that were instrumental.
They had made enough progress that I was confident enough that this could work, it might work, but there was just a lot that hadn't been done yet. No one had yet found a single letter inside of a scroll, and there were lots of technical challenges in even identifying the ink. And I should probably say something about that. So interestingly, if you get an X-ray of your arm or something like that, it's pretty easy to see where the bones are because they're denser than the other tissues in your arm. And so usually on an X-ray, they show up as white and then your other tissues are darker.
And so that's what you would hope for the scrolls is that you scan them and the letters show up like bones, they show up white against a dark background. There'd be some contrast. But unfortunately as it happens, the scrolls in Herculaneum were written using a carbon-based ink, and then on papyrus, which itself is also carbon-based, and so the radio contrasts of those things was nil basically, so when scan one of these scrolls, you see no letters, you see no ink in X-ray. You don't see it at all.
Alex Petkas:
One scholar described it to me as the stuff that collects at the bottom of your lamp. Stick your finger in that, add some gum Arabic, and that's your ink, and it's actually not that different from a burnt scroll. It's already carbonized stuff.
Nat Friedman:
Yeah, someone told me that too. There was soot, soot on papyrus basically. And so Brent had some results from not the whole scrolls, but broken off fragments that indicated it was possible though. He'd done some painstaking work to show that it was possible to do using machine learning and that the machine learning algorithms could possibly pick up subtle patterns in the shape of, not in the contrast, but in the shape. The papyrus surface has got all these ridges in it from the fibers in the papyrus plant, and then they crisscross, and so there's ridges and divots, and the ink, it turns out, kind of fills those in a little bit and smooths them out. And so you have in parts a little bit of a smoother surface. Maybe it stands slightly out of the papyrus.
And what we later learned was that when the ink dries, when it's thicker, we think when it's thicker, it will form this kind of pattern of what we call cracked mud or crackle, which looks sort of like the surface of a dried lake bed a little bit. People who have been part of the contest, like this one guy Casey Hanmer who made this critical discovery of this cracked mud pattern, and is a friend of mine, he spent so much time looking at these things that now whenever he's going around, if he sees cracked mud, it's a game of Tetris to him. He sees letters everywhere, so yeah, that sort of after image.
But yeah, so we had all that, but no one had found letters in the scrolls before. And so anyway, I realized it was a super complicated problem and it was going to take a lot of work and there's no way I could do it myself. And Brent said, "Look, I've got a couple grad students, but they're incredibly busy," so we needed to get help. And I actually put out this very mysterious job description to try to hire someone to work on this project, and I ended up getting actually about 60 applicants. I didn't say what it was. I just said, "This is a Indiana Jones style adventure for a programmer."
Alex Petkas:
Dangerous mission, return not likely.
Nat Friedman:
Exactly.
Alex Petkas:
Think of the Shackleton thing.
Nat Friedman:
Yeah, I promised, "If it works, this will be a story you tell your grandchildren." That was sort of my-
Alex Petkas:
Oh, nice.
Nat Friedman:
... my promise, and I think we've lived up to that one, so that's really cool.
Alex Petkas:
In spades. Yeah.
Nat Friedman:
But I ended up hiring this great guy, JP Posma, who helped basically me put together the contest at the beginning of last year, of 2022, and I guess the rest is history. We launched the contest and it was a heck of a year. We'd set a deadline for ourselves of December 31st. And I was pretty confident it could be solved, but not at all confident that it could be solved in the nine and a half months that we gave ourselves. And so I was sort of biting my nails the whole year and constantly asking, "What can we do to increase our probability of success? And is there anything we can do to move the wall forward?", basically.
Alex Petkas:
And so the structure you guys came up with I think is really interesting because GitHub is to Microsoft, it's like open source versus the kings of IP and patents and suing you if you infringe. So I think there's an interesting paradigm shift there. And then the way that the Vesuvius Challenge works with the incumbent way is institutional funding, apply for a grant from a government or a foundation, and then it has to get channeled through academia. And obviously that has come up with some good for society, but it also can be a little bit slow and inefficient and politicized. I think just this idea of doing a competition open source like you did is fascinating. Yeah. So how did you come up with that? Tell us about that.
Nat Friedman:
Yeah, people think this was really a creative idea. For me, this was the most obvious idea because that's what I'd done my whole life. I mean, my whole life I basically worked in online internet communities. I grew up on the internet. I got on the internet in the early '90s, and I was an introverted science kid. So by the time I was 15 or 16, most of the people I knew were on the internet. This is much less unusual now than it was then. Then it was really strange.
But the people who were on the internet then were not normal people. They were obsessive technology people and programmers. And so we had this emerging phenomenon of what we now call open source already then, which was super smart people working on projects together, writing software, sharing their work, learning from each other, and I loved it. And as a 15-year-old in a small town in Virginia, I could join these incredible communities online and learn from them and try to contribute, and so that was amazing to me.
And then I was running GitHub, so GitHub basically hosts now a lot of these open source projects. And so to me it was like the most obvious thing was, why don't we open it up to the world and many eyes can make the problem shallower potentially, but there were some serious contest design problems that needed to be solved. So the main issue was that the problem was, though Brent and his team had laid this extremely good foundation and had pioneered a lot of key techniques, there was just an unknown amount of work left, and it turned out to be a pretty decently large amount of work left.
So there's this continuum of collaboration and competition, and the problem was to pick a point along that continuum. Our original idea when I first pitched to Brent was we should do a big XPRIZE style, just like a grand prize for someone who can read the whole scroll or something like that. As I was starting to try to figure out how to put that together and looking deeper into the problem, a friend of mine said, "Nat, there's four or five PhDs left in this project. This is not a weekend hack. This is not something somebody could do in a few evenings. This is a really hard problem," and I agreed.
And so the concern there with doing a grand prize and only a grand prize for people working on this part-time is that, let's say there's 10 teams and each of them makes a little bit of progress, maybe they get 5% or 7% or 15% of the way there. If that's all you do, you're probably going to give up. You're going to feel like your chances of winning are low. And in the case where this is just a contest, you have no incentive to share your little bits of progress with anyone else.
And so we decided that actually first I went looking for a solution to this problem. I ended up calling the folks at XPRIZE and talking to people at other online contests, distributed contests to try to find an answer. And basically, I couldn't find that anyone had solved this. I mean, it's possible that I missed some antecedent for this where someone had already come up with this solution, but we ended up just banging our heads against it and then struck upon this idea, which was to basically have two systems of prizes. One would be this gigantic, charismatic grand prize for doing something amazing and cracking the problem seriously, and we knew that a grand prize would just excite people. It would have prestige if you won it. You'd get famous, and if we put a lot of money-
Alex Petkas:
Big check on the stage.
Nat Friedman:
Yeah, big check. Exactly, yeah. We could put a lot of money behind it as an incentive too. But then I had this idea of having progress prizes, which would be a series of smaller prizes that you could win for making little bits of progress towards the goal conditioned on you open sourcing your work. And so the idea was that if you were one of those teams that got 5% of the way there and you thought, "Well, I can't get any further, I got to go back to work or school," you could release that 5% and then if you qualified for the progress prize, we would give you an award and recognize you and you'd get some money and you get some dopamine along the way, and everyone would benefit. Your little 5% of progress would accrue to everyone.
Alex Petkas:
And you open source it after you win the prize, right?
Nat Friedman:
After you win, yeah, yeah. That would-
Alex Petkas:
It's not like as you're going, you can peer over people's shoulders.
Nat Friedman:
Yeah. I mean, you can open source it before and still be eligible, but most people would choose not to because it's this blend of competition and cooperation. You're in an adversarial relationship for the grand prize and you're in a collaborative relationship for the progress prize. And so most people would choose to wait until after they had won to release their progress as open source, but actually not all did.
So that was the design, and we hoped it would work, and it turned out to work much better than we expected. It was incredibly powerful as a mechanism design for creating this community and creating a ton of progress that I think happened much faster than it would have otherwise. The alternative might've been to have an enormous grand prize. If there was a $20 million grand prize or something like that that might've worked, but we didn't have that kind of money, so that was not an option for us.
The other challenge we had is that a lot of the talent, the new skills that were needed to work on this are also the same skills that are needed to work on AI. And AI is having this gold rush right now. And so with our little archeology project, we were competing with vast riches and commercial success that all these other people could go chase for themselves as well. And so the progress prizes created a community and little bits of prestige and progress. And actually what we didn't anticipate is that they were really encouraging. And so the people who won progress prizes early would become progressively more obsessed with the project, and they would've received this dopaminergic reward for having done something, and then they would want that again.
In fact, all the people on the grand prize winning team, all three of them, Yusef, Luke, and Julian had won multiple progress prizes before. And I think maybe if they hadn't, not only would the work not be out there for others to build on, but they might not have been so encouraged.
Alex Petkas:
Yeah. And I think that that's one of the striking things about the project. And I do encourage people to go to scrollprize.org to check it out, check the website, and right now you have another round of competition going on, and we can talk about that more if you'd like, but one of them is go back and reproduce the results that were done before, but more efficiently, more or less.
Nat Friedman:
That's right.
Alex Petkas:
Something like that.
Nat Friedman:
Yeah, exactly.
Alex Petkas:
So it seems like so much of the hard thinking leadership, and let's say dictatorial control, in it comes into the very careful thinking about the incentives, I'm trying to think of lessons that other people might use to construct their own competitions and rethink how we get-
Nat Friedman:
I think, yeah, I mean maybe the-
Alex Petkas:
... projects done.
Nat Friedman:
... meta thing here is we treated the contestants as our customers, basically. And so we didn't want to just succeed, but we wanted them to have this really good experience and have a high chance of having a good experience. And so I do think a lot of people think with contests or bounties that it's fire and forget, you just put it up there, it's a million bucks and that's all you have to do. And it was nothing like that at all. I mean, I think that's a very bad way of thinking about it.
For example, we knew that it was really hard to get into this and that people have a lot of other things they could do with their time, especially the people who could make progress on this. And so we actually, before launching the contest, worked really, really hard to produce a website that made it exciting and told the story and then also had technical tutorials that provided an on-ramp for someone to get the context they needed to start working on it at all. And that was a lot of work. There was a lot for people to learn.
And one of the things we realized was your reasoning in 3D, and that's hard to even describe in text, and so we needed to make animations. I ended up hiring an animator and we spent a couple months with him making explanatory animations, which are now all over the Scroll Prize website. And so I do think I've built products before and I've run companies before, and this actually felt remarkably similar even though it was a contest. You have to think, "How do I make it so when someone comes to my webpage, they're interested and then they know what to do next?" And so yeah, contests are a really high leverage tool, but they don't give you a reason that you can be lazy and not think about all of these things.
Alex Petkas:
Right, it's not all about the money. And it is like that Patrick Collison speaking in an interview recently about how beauty's important in design because among other things, it shows that the people in charge really care. And that's really motivating for a customer, to say nothing of a contestant.
Nat Friedman:
Yeah. We wanted to speak to them too. We wanted to write the text on the web page so it was clear that the people on the other side of the contest were like them, were technical and also excited by this cross-domain project.
So anyway, yeah, I mean, the big couple decisions we made along the way were after figuring out this basic design for progress prizes were two. The first is we realized that there was this one step in the process that was just really, really, really hard. And the contestants a few months in were making no progress on it.
And so just to describe how it works, the first thing you do is you scan a scroll and you get this, basically this 3D image of it. The next thing you have to do actually is to... The scroll is a wound spiral of papyrus. They can be really long. They can be 15 meters long, and then they get rolled up and the writing is on the inside. And then unfortunately, when Vesuvius erupted, the hot mud really distorted them. They're super messed up. The heat caused the papyrus to blister in places or to delaminate or to fuse together some of the layers, and then they got bent as well. And so it's not like this perfect spiral that you can just easily unroll the software.
What we have to do instead is to trace the surface of the papyrus inside the scroll. And unfortunately, we don't have good algorithms for doing that yet, and so that process ended up being really, really manual. There's some tools that Brent's team had built before, this guy, Seth Parker, and those were a good start, but those tools also needed to be greatly improved. And so what we ended up doing basically was, and this was a controversial decision inside the team, what we did is we ended up hiring a team of people full-time to just trace papyrus all day on these giant monitors. And this is-
Alex Petkas:
This is the segmentation, right?
Nat Friedman:
Yeah. The term we use for this is segmentation. It's not in the spirit of a contest, right? Like, "Wait, hold on a second. You're supposed to be the referee. Now you're running out on the field and kicking the ball a little bit. What are you doing?" But I was just desperate to make sure we were successful by the deadline, and this seemed essential. It was just not going to happen otherwise. And it turned out to be really, really important. And we ended up spending actually a lot of money on this, hundreds of thousands of dollars on paying people to do this meticulous work.
We hired 15 people initially. It turned out to be so difficult to do that really, most of them were not good at it. And we ended up with a full-time three person team of segmenters who were awesome, David and Ben and Conrad, and they worked until their eyes hurt, honestly, on this. We were in a race because we knew in order to meet the grand prize threshold, we needed a certain amount of papyrus segmented quickly.
And so that was one key thing. And then the other key thing we decided was that we had this grand prize, which was for finding four passages inside of a scroll of sufficient length, which we picked 140 characters as that length, and then we realized that was a pretty high bar. We needed some intermediate milestone. And so we decided to have the first letters prize for finding the first 10 letters inside the scroll in a certain four square centimeter region.
I think that was early summer when we put that progress prize out there, and that was critical. I mean, that's when the whole avalanche started to run essentially, because the first thing that happened was that Casey Hanmer found this cracked mud pattern, and he found a couple of letters. He didn't find 10, but he found a couple of letters in the scroll, and he decided to share that with the community. And that kicked off a race. He found those letters by looking just manually with his own eyes. He called it direct persistent inspection, just looking at it a lot in the x-rays. And we had thought-
Alex Petkas:
Taking time off from his Nat Friedman backed space-related energy startup, right? He's not just doing this in a basement.
Nat Friedman:
No, he's a father of three and is running a startup. It's also very hard. And I don't know how he did it, honestly, but he's amazing. So he found this, he put it out there, and then subsequently, Luke Farritor, who was at the time 21-year-old University of Nebraska undergrad, saw Casey's pattern that he'd found and said, "I can train an AI model on this to find more instances of this," throughout the flattened segments that we produced with our segmentation team.
And absolutely miraculously, he found the first word that had been seen in 2000 years from inside an unopened scroll. He was the first person to see that handwriting in millenia. And then even more incredibly, it was like a cool word. It was this word porphyras, which I hope I'm saying it right, but-
Alex Petkas:
Yeah, close enough.
Nat Friedman:
Okay, thanks. It means purple. And it could have been a super normal word, like although or something, but it was purple, and that was so mysterious to us. So that was the first word that was found.
Alex Petkas:
The color of royalty. Yeah.
Nat Friedman:
Yeah, exactly.
Alex Petkas:
So we're going to have Brent, Dr. Brent Seals on here in a little while on the show. Everybody stay tuned. Coming soon. So we want to get more into the technical details, but we could zoom out a little bit. One of the things that I think is so exciting about this project is this meeting of humanities and great knowledge from the past, but activated with a whole lot of high agency determined people. Dr. Seals, very determined person. I mean, he's been working on this for 20 years.
Nat Friedman:
Definitely.
Alex Petkas:
Yourself and Daniel, all the people on the prize team. I think of what Paul Graham says about being relentlessly resourceful as a key for a startup founder. I wonder if this is a project like this just has to filter for that kind of person and train that in them, which to me is just this classical spirit of competition that you see in the Olympic games.
So if I could ask, what are some of the... Because when I think of the great men that we study on the Cost of Glory podcast, somebody like Julius Caesar, okay, he's very ambitious, but he's also interested in technical matters, he rewrites the calendar, he's a great field engineer in warfare, he's a poet, he's a writer. He's this kind of Renaissance man in a lot of ways. And it's like Andrew Roberts was saying on a show recently, Napoleon would've been a tech founder today. I think you'd probably say the same thing about Julius Caesar. What are some of the qualities that you would like to see this project inspire in people today that you'd like more of?
Nat Friedman:
That's a good question. Well, I think, yeah, it would be great if more people did crazy projects that might fail. And I think, at least in my world, this was a really unusual thing to work on. And I got a lot of funding from friends who just thought it was crazy and they hoped it would work, and that was key. And so yeah, I do think people should do more things that just seem strange, but that they have a lot of personal enthusiasm about and should be slightly more impulsive about that.
I mean, the story for me psychologically of doing this is that I got into it impulsively, and then once it was out there and I had my name on it, I really wanted to do whatever it took to make sure it succeeded. And so I had this relatively low threshold for starting this project. And then once I was in it, I was, in a way, I was stuck. I trapped myself because it was going to be really embarrassing if this didn't work. If it didn't work, people would say, "Of course it didn't work. This was an insane idea. People have been trying for hundreds of years to open these scrolls and you had horrible hubris by getting involved at all."
And so I do think that's part of it is just you should do something like that. And then I think the other thing is there's a lot of areas of the world that are just not picked over by people at least of my tech background, so I would say it's probably a good idea to look in the just less obvious areas to some extent. I mean, bringing software to archeology, maybe it's obvious to a lot of other people, but at least in my world it wasn't. And so I think that there's a hundred dollars bills flying on the sidewalk there. And then yeah, just do whatever it takes. Do whatever it takes and do it quickly. I think that's really important too.
But yeah, I hope this will be good if it inspires other people to do more crazy projects, not because you can tell the story of super high impact. The classic thing for a tech founder to work on after they have a slight amount of free time is something in health, cure diseases, et cetera. I think that's very important, obviously. But I would like to see more surprising projects from people and things that are slightly stranger and only make sense in retrospect.
I think that's what you want to do. You want to do things that make sense in retrospect, whether or not they make sense at the time, and the ones that make sense at the time don't always make sense in retrospect. And so that would be my... If this is inspiring to people from my world, I hope it inspires them in that way to do something that seems strange, doesn't make sense now, but if it works, will make a lot of sense in retrospect.
Alex Petkas:
Do you worry that people... I think about this project could unearth another Renaissance worth of ancient texts. Would you think people will be more or less receptive to such a discovery than they were in the Renaissance today?
Nat Friedman:
I don't know. I don't really know exactly. I guess they were pretty receptive during the Renaissance.
Alex Petkas:
Maybe we overestimate it with the rosy colored lenses in the past.
Nat Friedman:
My guess is they'll be slightly less receptive now just because we have so many sources of new information every day, and there is a sense of freshness from a lot of that information on Twitter or whatever. And so we're not as bored is my guess as we might've been in a lower information era.
However, I think we will be very interested in it now. I do think there's a way in which people... The world is changing a lot and society has fewer bedrock spiritual values that we really believe in and that we all have in common. And so there's a way in which I think some people at least are looking to the past and the ancient world for comfort and guidance and maybe other ways of thinking and living in a search for something that's not a fad diet or ephemeral.
Alex Petkas:
And so many of these texts that we've already discovered really deal with some of the hardest questions of life. One of the best passages of Philodemus is on how do you meditate more on death so you live a better life, and so when death comes... And he's living in this period where everybody important is dying violently, actually. Julius Caesar dies violently, Pompey dies violently, Cicero dies violently. All these guys in a period that are Philodemus' peers, his clients... His patron Piso, I don't think died violently, but maybe because he took Philodemus' advice.
I think some of these texts really... I would love to discover dialogues of Aristotle. I would love to discover letters of Julius Caesar. I think we really could, but the Villas secrets, I mean, and there's of course, there's great art, and we're going to do more of this content on the show to talk about why this is so cool. Even what we've already discovered to me is very inspiring because it tells us about these people that are some of the most fascinating figures in Roman history.
A guy like Philodemus is trying to help those great leaders connect with some of the profound questions, and I don't know, the values that are really going to make their society and their own lives better. I really think that's his mission. And so I find it's already an inspiring story to me as I'm getting into it. I'm going down the rabbit hole with you guys as a non-technical person.
Nat Friedman:
Yeah, there's a way in which it seems random that what we've got is a bunch of Epicurean stuff. But I was wondering, maybe Epicureanism is what we need right now. What if actually there's some amount of meaning that we can find in Epicureanism? Stoicism's super famous, and a lot of people out here in California are into Stoicism. But I like the idea of just exploring this alternative frame of Epicureanism too. What makes life worth living from an Epicurean point-of-view?
Alex Petkas:
Yeah. And in times of political turmoil, like they lived in Philodemus' Day, maybe especially so. Well, Nat, thank you for taking the time. Where can people find you before you go?
Nat Friedman:
Oh, sure. Well, you can read more about the Vesuvius Challenge at Scrollprize.org.
Alex Petkas:
Right.
Nat Friedman:
We've read 5% of one scroll. We have hundreds of full scrolls left to read, and we're working really hard on doing that. And I hope we do uncover a dialogue of Aristotle or a new history or something incredible. But whatever happens, it'll be great.
And then, geez, where do you find me? My website is nat.org and so you can just read a little bit more about me there, or I'm on Twitter as Nat Friedman.
Alex Petkas:
Great. Well, thanks again. We'll see you soon.
Nat Friedman:
Thank you for having me.
I liked it so much, I linked it here: https://www.libertyrpf.com/p/502-boring-quality-investing-pepsis
Yay! 💚 🥃