Social coding revolutionized how we share code with others. Bundler, npm, and Github made publishing and consuming code so convenient that our dependencies have become smaller and more numerous.
Nowadays, most projects quickly resemble a Jenga tower, with layer upon layer of poorly understood single points of failure.
This talk is an opportunity to pause and reflect on our relationship with open source. Convenience and ego drive most open source adoption, but these shortsighted motivations raise long-term problems we need to clearly identify if we can ever hope to solve them. This presentation is just a first step in that direction.
The video above was presented at RubyConf 2014 on November 19, 2014. The slides are up on SpeakerDeck, as usual.
Transcript of the talk
[00:00:00] Good morning. Let's roll. My name's Justin. Fun fact, I get paid by the tweet. So if you could follow me on Twitter and say hello, I'd love that. Or if you want to drop me a long form line, you can reach me hello at testdouble. com. Open source is good, right? Companies working with competitors, other companies on common tools and then turning around and sharing that for free.
[00:00:22] Startups now they can stand on the shoulders of giants and build great new things with just adding a little code on top. Companies that couldn't exist otherwise. And then never before in the history of the universe has an individual that's not state sponsored or company sponsored been able to just do a little work of their own and then literally change how the world works.
[00:00:42] But is open source good, really? Companies love consuming open source, but if you ever want to share an upstream patch, much less open source a library, they suddenly are very stingy and skeptical of this open source thing. And then a lot of startups keep falling into the same trap of hoovering up all of this free stuff without understanding how it works.
[00:01:00] And building maintainability nightmares. As they get successful, they can't add new features anymore. And most of the maintainers I know are pretty burnt out, right? They don't like the fact that they're doing something for fun in their free time. Companies are running on that stuff and then expecting customer support on nights and weekends.
[00:01:19] So today my goal is just to bring to light a handful of issues affecting open source. And my only objective here is to encourage you to do the same thing. Because maybe if we can start to build a broader awareness of some of the systemic issues in open source, we can start to have ideas of how to fix them.
[00:01:34] And then maybe somebody will come along and start to create new creative solutions for those things. And then we can, start to live and realize the promise of true openness, whatever that means. And then we're done. But again, today, very little, minor, just looking at a handful of things.
[00:01:51] Topics such as dependencies pulling back the curtain a little bit to show what it's like to be a maintainer issues of trust, adoption, security, and then some deep thoughts about how we interact with each other as humans as well as where I think the future is heading. I like to start off with term definition, the word ideology.
[00:02:08] Most of us think of the word ideology as like a political subscription or affiliation, what you believe, but I like this definition more. They do not know it, but they are doing it. Ideology has the negative space that's driving our actions without us even realizing it. It's a quote from a dude named Karl Marx.
[00:02:26] Open source fans are a bunch of hippies, so I figured I'd start with a Marx quote. That comes from the book Capital. And Capital is an interesting book because as a work, it sits at the intersection between philosophy and economics. I think it's an interesting subject to start with today because so does open source, right?
[00:02:40] We share all this code altruistically as if to earn karma from people we don't know. And yet, there's all these companies out there making bucket loads of money off of open source. And every company that even doesn't contribute to open source needs it to get by. So thinking of capital and traditional economies, I want to look chart the march of progress of economics.
[00:02:59] In the beginning, everything was shitty. Everyone was just trying to get by survival day to day. But then, as groups of people started to form, specialization emerged. You could go to one market for your veggies and another person for your meats. And through efficiencies, this opened the door to the development of human culture, recreation and art emerged.
[00:03:19] Industrialization further optimized this, because now we could go to one place and get all kinds of goods. The internet totally inverted that. So now from my bed, on my iPad, I can order things from anywhere in the world, and have it all shipped to my door. This is progress. But where does it lead us?
[00:03:37] This year there were rumors that Amazon is actually using big data to predict what you're gonna buy before you click one click. And they're actually shipping it to distribution centers near you in advance. So that they can send it to you the same day or the next day. And some people are starting to ask questions like, is this march of progress actually somehow taking away something about, that's been core to the human experience?
[00:03:58] So it's an unintended consequence. Another example of unintended consequences in the progress of economics? Food Inc. This documentary was a big hit, and look at that aggressive tagline, you'll never look at dinner the same way. But I think it would be more honest if that said, for at least a month.
[00:04:14] Because we can't change these. These are systemic issues. When you chart progress over time, there's a natural accretion of awfulness. It just mounts up until it gets to a point where we all freak out about it. And when we freak out about it, we think, we're going to fix this, right? But we can't just turn on a dime.
[00:04:29] Nothing stops this strain. It's going to keep on getting worse before it gets better, and maybe we can reign it in to current panic levels later. Speaking of awfulness, let's chart the same march of progress about all of the tools that we use to suck in new dependencies from the open source world.
[00:04:50] Charting open source's progress over time, in the beginning, there were just files out there on the internet. And if I wanted to build a system and, Pull in some open source, I'd have to go find it first, download it, and then literally check it into my version control, depend on it, and then logically, it continued on as part of my application.
[00:05:07] If it broke, I had to fix it, and that was a guard against pulling in too much, because I didn't understand it, and it was just more and more for me to maintain. Makefiles and common build system tools emerged as a great way to depend on stuff logically. So now I could build an application and depend on something like libxml, and it could be built on each of the systems that needed to be able to compile, but, it existed as a separate entity.
[00:05:28] I could upgrade it separately and I could view it as apart from my application code. Java and its JAR files were another great innovation because now instead of having all these build systems configured appropriately, a single compiled bytecode could be distributed and then run anywhere.
[00:05:44] So with Java, I could literally go to a website, download a JAR, put it on my classpath, and it would work. And this was so convenient that it actually opened the door to those same sites saying, Oh, and by the way, we depend on this third thing. That's what we call a transitive dependency. And what that allowed was really all of these libraries that we depend on to become small, more focused, and even, when you think of transitive dependencies like Apache Commons emerged as almost an alternative language stack within a language ecosystem.
[00:06:10] It was extremely novel. Ruby, you guys, have done great work making this even easier with RubyGems and Bundler. Now, when I'm writing my gem file, I only say the things that I explicitly depend on, and those transitive dependencies are discovered for me, and their version resolution is handled for me automatically without me even knowing.
[00:06:29] And I can have arbitrarily deep dependency graphs, only really thinking about the stuff that I directly depend on. NPM has just taken this a step further, because the Node. js runtime doesn't require it allows you to load the same library multiple times in a single process. Which means, I declare my dependencies, and then it just, Naively sucks up all the dependencies of my dependencies and so on and so forth to these gigantic trees.
[00:06:51] And it gets really broad and really deep. Up to the point where a very common thing for a Node. js library to run into is a support ticket saying, Hey, I can't install this because it's literally longer than the Windows Max file path limit of 256 characters. Wish that was a joke. We, this march of progress is optimizing for convenience.
[00:07:12] Getting somewhere more quickly. It's short term progress and it's available to us for the low price of long term fragility. The comedian Louis C. K. talked about this recently. It's true everything that makes you happy is going to end at some point and nothing good ends it's if you buy a puppy you're bringing it home to your family saying hey, look everyone We're all gonna cry soon.
[00:07:33] Look at that what I brought home. I brought home us crying in a few years Here we go Countdown to sorrow with a puppy
[00:07:48] our communities Louis CK a guy named Gary He told me to build a small but non trivial Rails app. An empty app will have 50 gems, but yours will end up with 75 to 100. Now go away for six months. Come back, update all your dependencies. Your app no longer works. I know from experience in the Ruby community that this is true.
[00:08:08] But it's easy to start a Jekyll blog. It's easy to install Sass. It's easy to generate a Rails app. It's always easy right now, never in a year. The reason, I think, is that when somebody asks us what our application is, We think of the code that we write as being our application. Even though, upon inspection, all of us would agree that our app is really the full stack of everything that we ship into production.
[00:08:33] It's never been easier to ship something to production, but the things that we're shipping to production have never been more complex. I'm guilty of this all the time. I say, oh, it's a Rails app, because that conveys a lot of information all at once. I never think to say, oh, and Rails depends on Thor, this very specific version specifier.
[00:08:48] I don't even notice that. I didn't even know it until I made that slide. Even though 272 gems can no longer be Installed in the same project because of version resolution conflict. So Bundler, it hides some of this from us. I think it could actually promote some of this information. It could tell us, Hey, we just installed 10 direct dependencies and that conferred 43 transitive ones.
[00:09:10] Or it could say, the version specifiers on all those gems preclude the installation of 1, 300 other gems out of our ecosystem of 88, 000. It could tell us, Hey, and by the way, if you were to run bundle update right now, it would be unable to update five gems to their latest version. I'd love to know that proactively.
[00:09:26] Of course, if you're an NPM, a Node. js fan, you might point out, Haha. Version resolution doesn't affect NPM, and you're right, but there's other issues. I can depend directly so let's pretend orange triangle is a dependency. I might direct directly depend on version C of it. One of my dependencies might depend on version B, and one of its others might depend on version A, all at the same time.
[00:09:47] Now this is all well and good until I start to think about it, and I think what if I call my triangle, and it gives me back a domain model object at version C's library understanding, And I foolishly, admittedly, I pass that into my dependency who passes that into version B of the same dependency.
[00:10:02] As a library author, did I think about that when I was like writing the library that was possible? Probably not. So is it going to blow up or is it going to work? The answer is nobody knows. And that's not great for understanding the software that we're building. Another issue is that, as a library author, I can specify the exact version that I want of something, but I can't control the specification underneath.
[00:10:24] If Orange Triangle depends on a very loose specifier, like Red Square, for example, at version star, which means that the user's install time, they're just gonna get the latest and greatest version things could potentially break. In fact, on LinemanJS, one day, we just started to get issues that every new installation was failing.
[00:10:41] And we hadn't changed anything for a month. What had happened was that one of our direct dependencies broke because it had too loose of a version specifier on something else and so there was a breaking API change. And then to fix it, we realized that because the maintainer was out to lunch, we actually had to fork that dependency and then specify explicitly, I want version D, the last compatible version of this thing, and then push that up to NPM Push up a new version of Lineman to NPM.
[00:11:07] And now we're saddled with the maintenance and ownership of this NPM module that we don't understand, that we'll never look at, that if there's a security thing or whatever, we're just, it's just baggage now. Here's a video that the team took of me that weekend. I
[00:11:26] feel like that a lot. We all understand that the code that we write for applications is the code that we need. And the code that we depend on directly is there for convenience to help us get started. And if we think about it, we can all agree that the stuff that our dependencies depend on, that's, brings some complexity.
[00:11:41] But when we think any deeper than that, it just starts to feel like risk. And how often do you ask yourself about your transitive dependencies? Not often, it's just anything deeper is mysterious. That's why I think a lot of people have been lampooning the phrase full stack developer this year, because nobody knows everything that they're shipping anymore.
[00:12:00] So this makes me long for the good old days sometimes. Of like makefiles. Sure they were painful, but maybe that was a healthy pain. 30 years later, a lot of these old C projects still build correctly. Now who's confident that when I run npm install on my project 30 years from now, it's still gonna work?
[00:12:20] Let's talk a little bit about all these people ruining everything like me, open source maintainers, and what it's like to be one. I think the most important thing to know is that open source maintainers are not rock stars. They're just humans, and in fact, they're just like extra early adopters.
[00:12:34] The way I envision it is that a maintainer is like On Google, looking for the thing that they need, and then realize, Oh, it doesn't exist yet. I'll turn around and go make this thing and work in the open and share it. An early adopter, and slightly less might just Google the same thing the next day and then find it.
[00:12:48] And then their reaction might be, Oh, sweet, I found this cool brand new thing. I'm gonna go share it on Hacker News and talk about it because it's still hip and underground. It's like finding a cool indie band and you want to share with people. And this starts putting the maintainer up on a pedestal, right?
[00:13:02] Promotes them. And they might get stars in their eyes and get excited that people are using their thing. But because they're on that pedestal, other people on Hacker News don't mind at all pointing out, Oh, it doesn't do X or Y or Z, and now the maintainer realizes their ego is all wrapped up in the adoption of their tool.
[00:13:16] But because early adopters are usually just as competent as maintainers at building stuff, maybe they'll send in pull requests that fix those issues. And now the maintainer's really happy again. It's super up and down. The initial release of any new library, new open source stuff, is usually just scratching the itch of a person who has a need.
[00:13:32] So there's gonna be rough edges. And early adopters are great because they, as developers, often will send in pull requests or submit issues to help round things out. And what emerges is usually something that's pretty simple. Pretty ready to be consumed by the masses. Maybe you call that a version 1.
[00:13:45] 0 candidate. Or if you're like afraid of the implications of what 1. 0 means in semantic versioning, like I am, that's your version 0. 84. 0. So this is the, at this point is I'd love for the conversation to happen between maintainers and early adopters. Sharing ownership, Hey, let's own this thing together so that the early adopter, we can broaden the con contribution base of this project.
[00:14:05] That typically doesn't happen. I'd even settle for just a maintainer saying, Hey, let's make you a committer. That way you get notification of issues and stuff, and early adopter agrees like that. They'll help sometimes, but because those conversations don't happen at this point, the maintainer may as well be saying, Hey, let's never communicate again.
[00:14:19] Sounds good. Bye forever. And then they're off. Off to the next thing. Because early adopters are always after the new shiny. So why don't maintainers just share control? I think the reason is that they misjudge how much happiness this library is going to bring them. Because up until now, they got to scratch their own itch and build a thing, and that was fun.
[00:14:35] And then, they started to get acknowledgement for it, and that was great. Then they got to version 1, and now they can only imagine the sky's the limit, this thing's gonna make me super happy, I'm gonna be like DHH, and people are gonna start holding conferences after me. Fear not, because late adopters will disabuse them of this happiness.
[00:14:52] So from the maintainer's perspective, if it scratched their itch initially, version one is probably mature enough to do what they need. So they might go a week without any commits or activity. They might go a month. And because they're also like early adopters, some new shiny thing's gonna come along. And, that might distract them and then whisk them away.
[00:15:09] And they're off into the next thing. A late adopter sees that, and then their response is usually excitement to say, oh, look, no recent commits. This must be stable. 800 stars. This sounds like a safe bet. Open source, that's free, that's good. And so there, like very often from years old projects, I'll get an email being like, Oh, we're so glad we found this.
[00:15:28] Thanks. So a maintainer's needs start small, but then we, layer them on, round things out. And if we conceive of each library as a thing, and then we layer on top of it what a user needs, like what I need out of that library, it's never a perfect fit. There's gonna be some things I wish that library did that it doesn't do.
[00:15:45] And in that gap becomes a negotiation. I have to ask the question do I submit a PR for this? Do I work around it? Do I open an issue and ask them to implement a feature for me? And so a lot of those late adapters, like literally two days later, will come to the project, put a stern look on their face, and then start saying, wait a second, this thing doesn't enterprise my enterprise thing at all.
[00:16:04] How could they ignore such an obvious and important use case? And so this is where entitled GitHub issues come from. Missing obvious and important feature. Our team thought this was a well written and good library, except that it doesn't do this crazy nonsense. And so I try to be polite and I respond I didn't know it had to do crazy nonsense, could you please explain?
[00:16:21] But instead of a real reasoned conversation, all their co workers just jump in and start plus one ing it. And the implication is that I'm bad and I should feel bad for not doing this stuff for them and having anticipated their needs. So odds are, it's a weekend, I'm at the park with my wife, I'm having a beautiful day, and now I'm glued to my phone, getting really nervous and feeling inadequate.
[00:16:39] So I run home real quick and I start working, hacking, hacking, hacking on the stuff. I come back and I'm like, hey guys, I just spent all weekend building this thing, can somebody please just verify it works? So I can close out the issue. And now what happens at this step? I never hear from them again. And now I have a SAD, because I just did all this work for free for strangers that they didn't appreciate, that they might not ever use, and now it's, I'm saddled with it forever, and I'm gonna have to work around that little edge case in my code going forward and make the thing that I love less maintainable.
[00:17:11] This is how the, projects tend to bring me less happiness as a maintainer over time, to the point where I hate most of my projects after they're three years old or more. Late adopters, because these well rounded version 1 things do the happy path well enough, late adopters typically are, when they're asking for something, they're asking for more niche features than early adopters.
[00:17:27] This leads a lot of people in our community to assume that late adopters make better customers than users. And that's why you see things like donate buttons, people talking about dual licensing, GPL plus commercial, pro feature add on packs that they can pay for, maybe paid support contracts or consulting hours.
[00:17:43] All of this, I think, misses the point. Cause it's a motivation impedance mismatch here. People are building open source out of intrinsic motivation and drive that they have, and money isn't going to solve that problem. And so if you want an open source library to do something, and the other person doesn't, just throwing money at it isn't going to make them any more motivated to do it necessarily.
[00:18:03] I would much rather if culturally we all got in the habit of letting maintainers feel free to say no. I feel a jerk when I say, like a jerk when I say, no I can't do this feature for free for you. But, even now, like when I ask for a pull request or something like that, what I really want to say often is I don't think that this is a good idea for the curation of this library long term, I'm sorry.
[00:18:23] And I feel like a total asshole when I say stuff like that. But I would, I think that we'd be all a lot healthier if we were a little bit less entitled and when we opened an issue, we were more complimentary and asking for advice on how they might approach the same problem instead of assuming that the library needs more code in it.
[00:18:37] Trolls are a totally different category of people who make us unhappy. Obviously they spit out hate, sometimes threats whatever it is, we know that the effect that they have on other people is they make them want to give up and walk away. And remember, maintainers, they're not rock stars. But because of the asymmetry like a lot of famous maintainers have 15, 000 followers on Twitter, we assume that they have their stuff together.
[00:18:59] And we might even understand if they got that many followers, there's probably a couple of trolls in the room. But it's not that big of a deal. But because of the asymmetric nature of communication, we don't see the fact if those trolls are the overwhelming majority of who's interacting with them on the Internet.
[00:19:12] And they have an outsized impact on a lot of open source developers psyche. So when the maintainer quits, it's We're all like, whoa, that's weird, that's what happened? I didn't see any of that. Seth Varga recently left the Chef project. He was a great contributor, did a lot of awesome stuff, but just a handful of trolls in the community were able to sap all the joy that he had from that project.
[00:19:31] Further making us sad. And so maintainers at this point, this is where I often see the cries for help, saying, hey, I'm burnt out, can somebody please help me maintain this thing that has all these features that nobody wants anymore? Hello? Anybody? And at this point, no one's there. The early adopters who might have helped, are long gone at this point.
[00:19:48] And this is exactly how a lot of projects stagnate and die. But even well maintained projects up until, by maintainers who do all the things right, no maintainer lasts forever. Big invisible one this year was TJ Holloway Chuck left Node. js for Go. And in the process he handed off a lot of his infinity billion NPM modules to other maintainers.
[00:20:08] But he actually sold Express. js to Strongloop, even though he wasn't the primary maintainer on it anymore, and he didn't tell any of the maintainers. It's really easy to screw this up. Why? Of course, beloved in our community, he left, and he had every right to close down all of his stuff and leave.
[00:20:24] And because he was highly visible, we all had forks of his stuff and we were able to put it all back together again. But that's not true of most maintainers. So I like to think what if there was an application for this that recognized that it's very human, very natural for people to leave and stop contributing at a certain point.
[00:20:38] And I think it would be neat to be able to broaden the base of people who are contributing. So what if I had a service where I could authorize RubyGems and NPM and GitHub, and it would like, aggregate all of my projects, and I could actually like, explicitly say, I need help on this one, and here's the type of help that I need.
[00:20:50] That way when somebody goes and logs in, there's like a dashboard where they can see all the projects that they used that have asked for help in specific ways, and then a call to action to say, yes, I'll offer to do that. That's a lot easier. Strikes me as way more successful than just sending off an email in the dark to a maintainer being like I like your thing, can I help you?
[00:21:08] The reason that I did this talk today, the reason that I thought to put it together was because our friend and beloved colleague Jim Wyrick passed away. And when Jim died I thought a lot of things, and it was very hard on me, but one of the things I thought about was like, he's got this tremendous legacy and I want to see if I can help.
[00:21:24] In particular, I have a a favorite gem of his is rspec given. And I wanted to see if I could go about helping adopt it. And that process was very complex and difficult. And it made me realize that when people go to their lawyer to do estate planning, very few people ask what's going to happen to my GitHub repositories when I die?
[00:21:39] And even if they had, the law hasn't really caught up to technology, and it never will. But maybe we can work around this. So an application like this that already has all of our authorizations could effectively serve as a dead man's switch. That is if you fall off the pedal, the train stops. So maybe I could list beneficiaries, like Todd and Brandon.
[00:21:57] And after, I go absent for a certain amount of time, maybe it sends me little tickler emails every month. And if I go away for 60 days, it knows to add Todd and Brandon as owners on all of the projects that I've authorized it to. I, by the way, I like to come up with a name for a thing before I build the thing.
[00:22:12] I'm not gonna build this, but I like to call it somebodypleasemakethis.
[00:22:18] io. Really, somebody please make this. I would be very appreciative. So that's all that's just about dependencies that are like, binary dependencies or runtime dependencies that we package with our applications. What about all these cloud services that we use, right? This slide is a slide of services that are going to shut down someday.
[00:22:41] Next slide. Can any centralized service, probably written volunteer on less money, if the corporate ones are all going to shut down, what about the purportedly open ones. And can they be truly open? Like a maximally open system is rife for abuse, but if you make it too curated, then it could exclude people and limit the amount of contribution.
[00:23:03] And I ask this, of course, because almost all open source infrastructure is entirely centralized. Out of convenience, right? Because the march of progress just wants to make it easier and easier to adopt stuff. It's a hard question to answer. How do we decentralize this? A lot of people in the room contribute to RubyGems, and I think they do great work, and I think they're underappreciated.
[00:23:19] But what if RubyGems were to disappear? How many businesses would that affect? Do the businesses relying on RubyGems realize that as liability? What if NPM were to fail and lose a month of backups? How many things would no longer be able to install and work? I ask the question because I want to know what would a decentralized dependency service maybe look like?
[00:23:37] We have BitTorrent, apparently, right? We have cryptography. It seems like the sort of thing that we'd be able to figure out without having a single point of failure. Speaking of single points of failure, one of my favorite things to do is wait until the next time GitHub goes down. Because people freak out on Twitter and then they make the kind of snarky joke of it's distributed so why do you guys care?
[00:23:59] Because I can still work locally and in fact I can SSH to my buddy and we can still collaborate. Good thing that's all we use GitHub for! It's not like we use github to pull down our dependencies or we use github to test all of our code or we use services that get that depend on github to analyze all of our code.
[00:24:16] We even have most of the continuous delivery services that ship our code also depend on github being up and even if you finish your work, your next issue. is in GitHub issues, because that runs on GitHub. And you can't find the next thing to do, because now it has also co opted all of our project management.
[00:24:32] So how can we connect numerous services to our code, which is admittedly the source of record, the thing that everyone's going to want to integrate with, but still avoiding that single point of failure? Fortunately, Git is very fast and very portable. And it seems to me like a distributed transport layer or mechanism that would let these people hook into it without having a single company needing to be up for everything that we do to work seems possible.
[00:24:54] Somebody please build this. Let's talk about trust. So open source requires adoption, obviously. It's optimized for it. And in order to adopt something, people have to trust it's good. Now there's explicit trust, which is the dependencies that we directly depend on. But there's also implicit trust. The stuff that they depend on.
[00:25:14] We just trust that the people we trust are trusting the right people, etc. It's a big web. But as a maintainer, how do I get people to trust me and use my thing? The answer, of course, is marketing. Let's look at marketing over time in open source. So consider Linus Torvald's 1991 announcement of Linux.
[00:25:33] He did it on the Minix mailing list. Here it is. And look at it. Got a few paragraphs, but no catchy name. He makes a self deprecating remark in the first line, and then he goes totally off message and talks about how it'll only work on his niche particular hard drive. This is bad marketing. Linux, if it were announced today, this thing runs most of the servers in the world.
[00:25:56] It wouldn't even make the front page of Hacker News if it came out today. I think that's reason for pause. A decade later, the Ant project for Java, this is trying to get into a lot of big corporate environments that don't trust open source. Look at all the things they had to do. They had a fancy logo.
[00:26:12] Look at all this website stuff. They had a mission statement. And then of course, Apache, they had a foundation affiliation to try to confer trust. But the more dependencies that we're hoovering up over time, the less time we have to vet them. And so nowadays, the standard GitHub markdown readme is expected.
[00:26:30] And here, you expect a quick, catchy intro, a whole bunch of real easy steps to get started, and then some mostly green badges up at the top to tell you things are working. And it's gotten even further because now there's all these corporate backed sponsored projects. This one is Yeoman, where a diverse group of engineers are building a rocket with gradients and an authoritative tagline.
[00:26:53] And they even have a one liner install with a wizard that walks you through how to generate a thousand different types of projects. It's a pet idea. Separate talk. Everything in open source is optimized for adoption. It's the natural selective pressure that we're under as maintainers. So we have all these rock stars, people in the community we look up to, like when Aaron Patterson releases a new gem, I trust Aaron, I know he's, good at programming codes, and so I trust that his gems are gonna work.
[00:27:21] I could look at how many stars or how many forks a project has. I could look at how much activity there is and how many open issues. I can see how many times that project's been downloaded. Now, even though 90 percent of those are Travis CI downloading them over and over again, my lizard brain tells me it's still important.
[00:27:35] Semantic versioning. We went like 40 years without any sort of like proper formal way of versioning software. But why did we need one now? The answer is we have so little time to vet our dependencies that we need to know at a glance like is this ready for use or is this safe to update?
[00:27:50] And who's got time to vet transitive dependencies? Nobody. The more people that you explicitly trust, the more people you don't even realize that you're trusting. I encourage everyone to recognize, too, that all these projects are marketing to you. I write open source, and I want people to use it because I love it and I'm passionate about it.
[00:28:08] I want you to engage with my brand and use my stuff. But I hope that you have a head of discernment on and you realize that every new thing you pull in adds complexity to what you're doing and more stuff can fail. Speaking of failures, let's talk about security. I actually have been, for a long time, a believer that you can do worse than security through obscurity.
[00:28:29] You could have a whole bunch of stuff up in the open and nobody working to secure it. Of course, the Free Software Foundation would point out open source code is accessible to everyone, right? The cathedral and the bizarre argument of many eyes and shallow bugs. But that's only technically true.
[00:28:41] People have to read the code for that to be true. So who reads the source code? First of all, there's the people who claim to read the source code, and then there's a tiny minority of them who actually do. There's people who fork projects, but so few people who fork do anything. There's the people with commit rights, but then the actual committers are usually only a handful of that.
[00:29:00] There's people who send a pull request, but only a handful of pull requests are anything more than just drive by single tactical things. And then there's the people who are hunting for exploits. And this is the only community where a hundred percent of them you can rely on reading the code. So in absolute terms, an uncomfortable proportion of people reading the source code are actually out to exploit it.
[00:29:20] Bash is a computer program. It's in everything. In the Internet of Things, all those things run Bash, like your smart fridge, etc. So it's unfortunate that Bash had a terribly big security exploit because a lot of those things are not upgradable and patchable. Somebody went and looked at that code and what they saw was, shocked them.
[00:29:37] It was global variables everywhere. There was all these void methods that took no arguments. Basically, that means that all that method could possibly do is A, nothing, or B, muck with global state. And then here's the vulnerable function. I'll give you a second. If this looks like nonsense to you, I agree.
[00:29:56] Let's just read, we're going to read a line of open source as a group. This is the beginning of a for loop. And I look at this. And I'm a Rubyist, so I don't spend a lot of time with for loops. But, I look at this and I'm like, why is that called string index? Why are they doing assignment in the second thing instead of a Boolean operation?
[00:30:13] Why are they incrementing in the second clause instead of the third clause? Surely they did that for a reason, I think, because I would write that like this. And so I always trust, I always assume that the person who wrote some other code is as smart or smarter than me. So clearly they must have done that for a reason.
[00:30:28] Until I can prove that they didn't. And so the amount of cognitive depletion that occurs as I read this method was so great I only got like halfway through it. So the Free Software Foundation, who maintains Bash, says, The solution is not proprietary software. The solution is to put energy and resources into auditing and improving free programs.
[00:30:45] And I think that's a pretty agreeable thing. So who here wants to audit the quality of code that literally everybody depends on?
[00:30:55] None of you, right? Why is that? It's because as a library becomes more popular, The importance that we audit that it works, and that it's safe, and that it's secure goes way up. But our motivation at an individual level to audit it goes way down. If I use a gem and only three people are using it, I'm probably not going to trust that it works, so I'm going to read the source code.
[00:31:13] But if I'm using Bash, I'm not going to think, I should probably read this. I'm going to assume somebody else did. The downside is that everyone else is assuming the same thing and no one's actually reading it. This is an example of tragedy of the commons. And in this case, it's an example that it's nobody's problem until it becomes everybody's problem.
[00:31:28] And at that point, it's too late. My hope is, there's this recent group together, I think that companies can solve this by investing, because they have the money, and if they collaborate, the ToDo group is an example, that they came together, they want to form some norms about open source governance.
[00:31:41] One of the things I hope they look into is sharing notes when they're auditing the security of stuff, working together to patch exploits. I don't know if that's one of their goals, but I think that's probably where the solution lies. If you work at a smaller company, it's your responsibility, I think.
[00:31:56] It used to be the case that we were trying to get open source into the door, at wherever we worked, and so we didn't want to say anything negative about it. But now everyone needs open source, and I think it's our responsibility to to tell our bosses that open source is not a free lunch. It's not some altruistic thing to make sure that it's working, right?
[00:32:12] It's a massive source of potential liability, and it's worth vetting it out.
[00:32:18] I want to share some thoughts about our interactions as people as well as where I think the future is heading. Unfortunately, all of these stick figures that I drew painstakingly were a lie. Because what they intimate is that open source is a thing that happens between people face to face in human interaction.
[00:32:34] But it's not like that. How we actually communicate is email, GitHub issues, Twitter. Almost all of it is just asynchronous text. We're all no more than an avatar, a username, some emoji now and then, and text on a screen. That means that when you get riled up about open source, no one can hear you scream.
[00:32:54] I'm screaming all the time and none of you ever come to say hi. And that's a big part of the problem. Evie had a tweet recently I liked a lot. She said, It's effed up that a lot of modern discourse is optimized for whoever has the fewest feelings and the most free time. Who would design that system? Us, apparently.
[00:33:14] So when there's uncertainty, ambiguity, or disagreement in a team, and if that leads to simmering disdain, Is email the right use? The right communication channel? Probably not. I think that it makes more sense to escalate to a higher fidelity form of communication. So if there's ambiguity, maybe real time text makes sense.
[00:33:32] Could clear things up. If you're having a voice conversation, you can emote and empathize with the other person. If you're making somebody cry, you can see it on a video chat. And if all else fails, you can always meet up in person. And there's something about human biology when two people get in a room, they tend to walk out with some kind of compromise, and that doesn't seem to be true on Twitter.
[00:33:51] I don't know why. Also, this strategy can be great troll repellent, right? Trolls get their power through anonymity and the perceived lack of consequences for their action. But if you level up the fidelity of the communication and the discourse, that means that their anonymity is also going to erode, and they lose that.
[00:34:08] And when you if somebody's trolling and you're like, alright, cool, let's have a hangout, suddenly they go away. So what I'd love to do is make opting into these higher forms of communication easier. So if I'm in a GitHub thread, I'd love a button that would open a new chat in Gitter, which is a tool for chatting on GitHub.
[00:34:23] Or build a live example in place, like through JSbin. Or s schedule a pairing session with Screen Hero, or AirPair, or hop onto my Nitrous. io box, right in place. I think, to anything to encourage us to have those more real communications, the better. I also want to think about where the future's heading, cause I, I showed a graph that it's gonna get worse.
[00:34:45] So on that graph, I said, no, okay, we're here, and, it's gonna get worse before it gets better. So what does this look like? And I was stewing on this while Gogaruko was happening, and I was thinking if we extracolate from our current culture of dependence, where would that take us? And I didn't come up with a lot of great answers, but then I saw this tweet.
[00:35:01] Yehuda was giving a talk about the Rust programming language, and he says, Rust enables a whole new generation of high level programmers, us, to write systems level code. And my initial reaction was, ugh. We're this huge pyramid of stuff we don't understand. That's scary. But as I explored it and started working on this, I actually realized I'm okay with this.
[00:35:24] There actually might be some benefits to this, and let's talk about those. So if you think of innovation over time, low level system innovations happened before high level ones, naturally. You need low level systems before you can have high level ones. And then they crept up. In the last 20 years, most of this open source innovation that we've experienced has happened in high level programming languages.
[00:35:42] Now only very recently have we started reinvesting in low level systems like Go and Rust. But today's open source dependency culture, really, is steeped in a lot of assumptions about high level code. And so the question is, if we were just to cargo call that down to low level systems engineering, how would that translate?
[00:36:00] How would it work? When I think of my systems programming friends, they tend to be very conservative and cautious. And that's because they're a mix of isolated from innovation and just curmudgeons. Put it in a slightly nicer way. That means that there's a certain amount of accidental or incidental cautiousness.
[00:36:18] A certain amount of intentional learned cautiousness. And maybe I can learn from that cautiousness too. Embedded in real time failures have very grave consequences in some cases. And that's probably where that cautiousness comes from. So if you think of a high level system, what's the worst that could possibly happen?
[00:36:37] Healthcare. gov starts failing and people can't get their insurance. That's really bad. That's 60 nightlines in a row of drama for our nation. For So let's take the same technology, and put it in charge of this thing, and see how that works, right? Low level systems, there's way more rigor required in terms of fault tolerance, because people can literally die if you don't get it right.
[00:37:01] My, my system's friends have a different perspective on dependencies than we do. They view adopting a dependency as outsourcing our understanding of how to do something. That means that if we have an application responsibility, that orange blob over there, and some particular dependency, we can view that dependency as solving part of our application's problems for us.
[00:37:19] We still know what what it's doing and why, but we might not know how. And that represents what I call understanding debt. Understanding debt is typically paid down by iterating. Any dependency we have, we learn about it through usage and through changing. How it works and what it's, what we need to work around, what's not great about it.
[00:37:38] But, if iterative releases aren't possible, then you shouldn't outsource that understanding. If we think of high level systems versus low level systems, a high level system is highly centralized, might only run one instance. Low level systems tend to be built, fabricated, shipped, and then they're out the door.
[00:37:53] High level systems have very short lifespans, typically. Low level systems might have to live and be raided for very long periods of time. High level systems usually have a full staff maintaining the thing for its entire life, but low level systems might just have a support chain. High level systems usually have way more than enough overhead in terms of memory and CPU in order to absorb the impact of some inefficient dependencies, but low level systems have very hard and real caps.
[00:38:18] High level systems are highly connected. We have all these tools to automatically deploy patches and new stuff, but low level systems might have intermittent connectivity or no ability to be patched at all. So clearly, we've made it very easy to iterate high level systems. But at this point, it's still very hard to iterate low level ones.
[00:38:36] So these concerns require that systems engineers have a deeper, upfront understanding of what they're building. If you consider if we were to graph out the depth of understanding necessary to be successful in a high level web app versus, say a low level plane control, like a landing gear or something, You only really need to know how browsers work.
[00:38:54] HTTP, a little bit of JavaScript the request response life cycle. But you might need to know a lot up front about how planes work and everything that can go wrong and everything that you're going to be plugged into and you need to account for that on day zero. So this traditional we write a handful of little tiny domain objects on top of this mountain of open source stuff that we don't understand, that might be an appropriate level of depth for a high level app.
[00:39:17] But if we were to simply cargo cult that over to a low level system, there's this danger zone in the middle of stuff that we don't understand, and it's inappropriate, and might put us at risk. And so systems engineers have a couple options. They could either build more richer, deeper domain stuff, or they can do, go through a process of qualifying those tools, and literally, rigorously proving that they work.
[00:39:38] This reflection caused me to realize that modern tooling is just a product of high level web development. If we look at this chart again this represents just today's perspective on how dependencies work. And if we were to do the same thing at the low level, rather than worry about Cargoculting ruining systems code, I'm pretty sure they're gonna still make sure that stuff works.
[00:39:55] Maybe together, collectively, we can have a new and broader perspective on what is currently our, culture of dependence. In fact, I'm hopeful now that systems innovations, bringing the same convenience that we have in open source to low level systems, they might reciprocate some cautiousness desperately needed cautiousness, as well as emphasize the importance of understanding the things that we're shipping.
[00:40:21] So we covered a lot of ground in these four areas today. But I want to part with the message that, I think that the open source world can be a lot better, and there's a lot of room to improve, and it's all actionable, and most of it's fixable. Companies, I think that we need to do our job of convincing companies that they have a responsibility to be contributing back, and to be auditing the security of stuff.
[00:40:40] I think startups need to be cautioned that, there's no such thing as building a real app in a weekend and being done. Maybe that means that like they should invest the time in understanding the things that they're standing on top of or if they have to get out the door in 30 or 60 days or whatever.
[00:40:53] Understanding that you probably want to throw away version 1 when you go back to rebuild version 2 after you validate it after you've been successful. I like to call this the slow code movement.
[00:41:07] Individual maintainers, I think that they can learn a human lesson from this that is, I want to be more inclusive of the platform, where we could aggressively try to pull in new owners and contributors as early as possible while there's still excitement to do that. And then I think we can solve with tools, make it easier for people who want to participate and get involved in open source.
[00:41:24] Make it very explicit, and invite them, as opposed to just waiting for them to work up the courage to email somebody. Once again, my name's Justin. Please say hello. We're, I come from Test Double. We're an agency of really terrific software developers. And so if your company needs developers and some help, please let me know.
[00:41:41] And I'd love for our great people to work with your great people. Like everyone else we're hiring too. So feel free just to drop a line, join at Test Double. I'd love to chat with you during a break today. And if you're too shy for that and you just want a free sticker, I got stickers too. And most importantly, I just want to thank you all for your time and thank the organizers for having me.
[00:42:00] I'm really humbled and privileged to be here. Thanks.
Related Insights
New Stanford research shows 9.5% of engineers contribute almost no visible work—but blame won’t solve the problem. Here’s how leaders can fix the root cause.
