What is DXOS? July 2023

In this episode, Jess and Zhenya discuss the unique privacy-preserving way of building local-first, collaborative web applications with DXOS.

With DXOS, developers get access to a reactive, peer-to-peer database embedded in the browser, which is designed specifically for private, real-time, and offline collaboration. CRDT techniques and passwordless cryptographic identity are combined to enable real-time multiplayer apps which support offline writers without any special work on the part of the developer. The interface to the database is just regular JavaScript objects, they can be mutated directly, and those changes are persisted locally and propagated to connected peers via secure, peer-to-peer WebRTC connections. No backend servers. No API tiers. No conflict resolution code. Applications work offline and synchronize with other devices directly, peer-to-peer, with no loss of privacy.

React and TypeScript APIs are available. Read the DXOS documentation here: https://docs.dxos.org.

If you'd prefer to listen to this conversation, here's an audio version:

DXOS Fireside Chat

0:00

/2574.721451

1×

Transcript

Jess: Hello and welcome to DXOS Fireside Chats, the inaugural edition.

So I was thinking since this is the very first one and this is gonna be DXOS related, I thought maybe we could start with kind of an obvious topic, which is what the heck is DXOS? 'cause I think a lot of people watching this will have that question.

Each of us has different kind of perspectives on DXOS, and I know that my perspective shifts even from day to day and so talking about how we're thinking about it right now could be a good place to start.

Zhenya: Yeah. To me it's a way of building apps, the way you'd rather be building apps.

And we're the dev tooling to help you do that.

Jess: Let's talk about the new way and then let's talk about the dev tooling 'cause I think how to characterize the dev tooling is a whole branch. How do we think about that? How do we communicate it to ourselves, to others?

And then the new way is also really important. What is this new way? How's it different from what exists today?

Zhenya: First of all, users should be able to hold onto their data and, keep that private, or keep control over it. Folks out there called this credible exit, where when you leave an app, that app doesn't own the stuff that you made with that app. Think about all the people getting de-platformed, for I'm sure legitimate reasons in many cases, but it's still not compatible with the vision that we had when we built the internet, which was about sharing and openness. But we've sacrificed on privacy over the time, unfortunately.

Jess: So why? I mean, I don't believe that most software developers have commercial incentives for lock-in, especially in larger companies. But setting that aside, when an average React developer sets out to build an app, I don't think they actually have that.

I want to own the user's. I want to violate their privacy and prevent them from credibly exiting my system. I don't think they have malicious intent. So why are most applications built that way? Why do we go down this path automatically? And not enabling credible exit.

Zhenya: You're right. It's totally not something we think about doing intentionally. I think it's just an outcome of how we've been building things and you know, people tend to copy each other over time as technology gets passed along. But our environment has changed. Like we now have a lot more horsepower on the client.

This has been true for the last 20 years. We've had a ton more compute, a ton more bandwidth. We're just saying, hey, there are principled reasons to make software more decentralized. But there are also functional reasons that come out of how databases are built and how cloud systems run, they're tending in that direction, in the backend anyway. The way they do replication, leadership election, all of the problems are actually concentrated in the database.

And looking at it, we're like, why do you even need the hop from the client to the database through this API tier? What if we just put the database into the client? It turns out you don't need a lot of things if you do that and you have really nice things like privacy. You know, you're peer-to-peer, connected to all of the people you collaborate with.

And you get real time collaboration for free, offline consensus for free. You know, you could work offline and then come back a couple days later and rejoin the swarm and be fine. . And, and it turns out, all you have to do is put the database right where you're using it, which is in the app.

Jess: That's one of the things I love about like the shape of the DXOS software is that in many ways the architecture is much simpler than the modern web app and, and web app framework. So I think when you take the database and you locate it centrally, Like far from the client that's interacting with that database, now you have this distributed systems problem of, okay, I make a change over here in the client and then I have to like, maintain that state somehow and update the local state.

But then I like send a something over to the database and then hey, database update yourself. And then the database like tells me whether it was able to or not. And then the intern, like the application has to be smart enough to know how to handle that. Of like, Ooh, database, database said that didn't work.

So now I've gotta revert to the previous state, you have multiple state management solutions, and it ends up being this huge problem. If anybody's in the React community that's listening to this, I don't have to tell you that. Like you can probably name five state management systems off the top of your head that you've used in the last 12 months, right?

So it's, it's a huge issue. But when you have the database locally, like running on the client, You state management is just display what's in the database. And then to update the ui. It's the thing changed in the database, redraw the ui, and so you state management becomes this much simpler problem. And that's just one example of a, a lot of sort of architectural changes that make applications much simpler.

Make some simpler to build, simpler to reason about. Simpler to maintain in production that kind of stuff. We can talk about some of the things that we're having to do behind the scenes that, that like are more complicated or that are like, where does that complexity go? Would be another way of, of asking the question, but that's one of the reasons I.

I like things that are of the shape. So in building DXOS apps 'cause you've been working with DXOS for a year. I've been working there, here for a couple months. How, how has been your personal experience building and like maintaining apps versus in other frameworks that you've used or worked on in the past?

Zhenya: Oh man. Yeah, it's, it hasn't been easy. Like it, we for sure are in the middle of building the plane while trying to fly it, right? So it's really difficult in, in some places. But you do hit these moments of magic. We hit these moments where we feel we're done (building our app) a lot sooner and quicker.

And then we're like, oh, wow. It works now? Oh, and I can do this and I can do this? And then you just get all this stuff kind of for free once you do hit these magic moments. It's been really interesting. Like the first thing that blew my mind was how the data is actually on another domain from you and when you make your database space, you create a space to talk to with, with ECHO the data actually lives on a different domain that is up to the user to decide, you know, we offer them halo.dxos.org where you would keep your wallet of identity and your, your application data, but what's cool about that is multiple apps can interact with that same data under the user's purview. But just that you, you no longer really own or control the storage of that data to, to the application developer is really interesting. That was the first kind of mental shift to go through.

There's fun and, and it makes sense once you're, after that point, you'd say, okay, like, I get it.

Jess: Yeah, that makes a lot of sense. I hadn't really thought about how I had kind of thought of the database as like running alongside each of the different each of the different clients. Or sorry, each of the different apps.

But I think what I hadn't really thought about the reasoning there of the fact that the database runs on its own domain. And that that is also something that is portable. Or you could run the DXOS vault on your own infrastructure and you know, on your own domain, and then have the data be tied to that domain particularly important in the browser context.

Right. That's, I think that's why my brain is going down there. Because yeah, within the context of the browser, the way you have to store data is you have to store it attached to some domain. And then the, an app that is, you know, being served from that domain is the only app that has permission to access that data.

Zhenya: Right. And then we sort of, it, it, that would be the, the vault itself is the one that owns that, that, that payload of data. And then it would, it would sort of grant access to these other apps to, to look at that data and that's where the identity piece comes in. Like we give you turnkey identity.

It turns out that when you make a database, identity needs to be built kind of baked into in some of the layers. If you plan to do this real time collaboration and this peer-to-peer stuff. So we give you like this cryptographic identity strength where devices belong to users and each device has a separate private key and when they mutate the world, like you can track who's, you know, who's doing what without loss of privacy, right? So you can have any number of keys for devices and we're gonna think about. Rotating them for you and all that stuff. But the idea is that if you're an app, you're welcome to the user's data and the user can let you have that data and, but it's not yours to like own and delete it's the user's. It's really interesting.

Jess: . Yeah, that definitely the user owned data. I've been there's a thread on Twitter, which maybe we can link in the show notes that I've been asking, like, how do we talk about this idea?

Because there's not really a term for user owned data. We've talked about you know, some, some of the ID ideas that people have thrown around are like user sovereign data or . Geoffrey Litt has this article called Bring Your Own Client where he, instead of making it about, Who owns the data?

It's about the relationship that the client has to the data. So you can bring your own client to the data. BYOC has a nice like abbreviation. But if you think about how computers used to work, where you had files on the file system, and any application that was running on your computer could open those files as long as it knew what to do with them, as long as it understood the protocol or whatever the file format was, it could open edit, Save, overwrite, you could duplicate the files, that kind of stuff. So at that point when we had files on your local machine, we basically had user owned data, but no one called it that 'cause it was just data. It's just files in a file system actually, we didn't really even think of it as data at that time.

Yeah, this moved to the cloud. It's kind of like now our data lives in data stores, which are essentially located, and now we're going back to this model where the data is living on your local device.

Like it's literally stored index dd indexed DB in your browser. You could actually go and inspect it. We could, we could even synchronize it with your local file system. With through, and we, that's something we'll probably do at some point, like replicate that data to your local file system outside of the browser context.

But yeah, I don't think we have, we don't have concepts for this right now to describe it. And it's, it is such a mental model flip. You have to think about it differently.

Zhenya: Yeah, absolutely. I would say concepts that haven't spread yet. I'm sure smart people out there have the same thoughts and have written them down even.

And maybe we're, I'm just not, we're not reading them or we're not finding them yet, or, 'cause we're busy building this. And by the way if any of this like, sounds scary to anyone, like the whole what the weird stuff we're doing. It's still a big experiment and we do offer, we do like leave it open, like we leave a gradient of what you do with this.

So for example, the thing I, I mentioned about where the data lives and how it lives on this halo separate domain, that's sep, you know, run by DXOS by default. But you can run it on your own if you want it. Like we do that. Another one is where you can run it in monolithic mode, where your app will own the, like the entire data.

So this vault where ECHO stores its data. It's all open source. And it's freely customizable to run anywhere you like. And, and we're just doing this to kind of set the stage to experiment and take feedback essentially. We want you to come and tell us if, if this makes sense to you, if it worked for you.

But there, I, I've heard some interesting chatter about people doing similar things with these like centralized vaults or these like places where the data actually lives separate from the app. Right. And I think there's desire for more interop and more standardization around that. So like I'd love to get in touch with anybody who like, caress about that.

It sounds like really interesting.

Jess: Definitely. Please reach out. We should definitely do a podcast on the ecosystem at some point. Because there are, like you mentioned, there's a small group of people doing things that I call DXOS-shaped. And some of them are taking on the whole enchilada the way we are.

So, but a lot of them are doing like point solutions and have made it pretty far. I. With, you know, biting off, taking a smaller bite of this whole software stack. And so it'd be really interesting to like, go through, talk about some of the ones that we have been inspired by, you know, what they're, what they're working on.

Yeah, yeah. You can stop there.

Zhenya: Yeah. We owe that explanation, to ourselves and everybody out there to explain how we relate to these other pieces. Ideally, we're interopping, with all of them, and not like reinventing wheels and stuff. So yeah, we're gonna do that. Soon.

Jess: All right. So Zhenya, that question of what is DXOS, you've described a little bit like the shape of the software that DXOS creates or this new way of building software privacy, preserving peer-to-peer, local first, real-time collaborative and user owned data. Sort of these sort of tenets of the way that the architecture of DXOS worked.

And you described DXOS at the very beginning as a developer tooling that allows you to build. New types of software. So I was thinking as we focus, think about like this developer tooling angle, the what is DXOS angle. I've got this little stack of cards that has some prompts on how is DXOS like a blank or not like a blank.

And I thought we could just go through them at random. I'll pull a card at random and then we can ex opine. On how is DXOS like that or not? So I thought for our first one it would be obvious because it's in the name of like, how is DXOS like an operating system? I mean, it's literally.

If you go look at our web, our website DXOS, the "OS" is definitely pointing towards operating system, and I think we use that in some of our website copy. So how is DXOS like the operating system? How is it not?

Zhenya: Well, let's, let's try to agree on like what we think about when we say operating system first, right?

When I say operating system, what comes to your mind first? Like what's the, what's the key attribute?

Jess: Someplace where all my apps and all my data live. It's a container for all of the things that I wanna run and all of the files that I have and that kind of stuff. So all of my data.

Zhenya: And you're sure you're not saying that because now you work here and we're all about apps and data. But Okay.

Jess: No, no. I mean, like, yeah, I mean it, you know, you could, the other way to think about operating system, which is a slightly different ship, but it's the same thing, is a a browser for your local apps and operating like so finder on macOS, right?

Or the iOS experience where you're dragging things around and that kind of stuff. That's an operating system as well. That's the sort of the view into an operating system. And then the third way that I think about operating system is a runtime environment. So it is literally where the code executes.

And you know, it's the vehicle for that. And a lot of times, I'll go even one step further, an operating system also gives you a, a layer to interact with the hardware that's connected to that computing device. That's the way I think about it.

Zhenya: Yeah. That's perfectly agreeable.

And it's hard to add anything more valuable to that, except that I, I recall that. Yeah, I think it was about like multi-processing and like multitasking is why you couldn't, 'cause the first machines out there just ran your code, like on the hardware, right? Like if you wrote in punch cards, if you were around, I guess you would just write in machine code. And then the operating system was like the first program we designed to like, help multiple developers run their code on that machine at the same time together, roughly speaking. And then it evolved to single user machines that run multiple apps and so on, like you said. And now it's the, it's the browser. When you said browser, I'm sure some of the browser people, like the Chrome oss people were like, yes. Like we're, we're an operating system too. So, okay. That's an operating system. I think it's, it's the thing that is the home for all developers to build their software and have it run at the same time for all of the users that consume that software.

We're, we're like super interested in the class of software that has to do with just information exchange between users, right? That's like a whole pile of software, productivity, software, et cetera. So how is DXOS similar to that? Well, it offers enough tooling and like a pattern for how to build apps for developers, right?

So that's at least the developer stack portion of an operating system. That's the frameworks and all the stuff that makes it work. We're sitting on top of a browser and, and a browser figured in your definition already, right? That it, it's the place where users come to put their data. So in that sense, I think we're complete.

We're trying to add to the web platform. To kind of have it lean further away from things that you don't need about centralization. You wanna have freedom to like put your code anywhere. You wanna have freedom to put your data where you think it should belong, right? . So we have tooling for you to build apps with, and because those apps, it turns out can interrupt in this new way because all this local software, like you said, is just sharing local state, and that's just the same as we had files back in the day.

Well, that completes the story for like users. Living in there and sharing all of their data roughly in that ecosystem. . So I think we kind of cover the users, the data, the developers, you know, with, with solutions. I think that's why it's a system of operating between all these people.

Jess: Yeah, that's interesting. So what are some ways that DXOS is not like an operating system? As I defined it .

Zhenya: It doesn't talk directly to hardware. I think. Okay. I don't know what I'm doing right.

Jess: No, I think that's right. Right. Yeah. Browser APIs do, but I think DXOS is a sitting on top of a an execution environment that sits between DXOS and the hardware.

Right. We're not writing like drivers. To communicate directly with, you know, the signals coming in from a USB port. So in that sense, we're not an operating system in that we're not the execution environment where your code runs. Except that you can write, you know, code that runs within the context.

Like you can write functions, for example, that run on some DXOS client that's running. And so there is a sense in which DXOS is a kind of restricted execution environment for your code. And it has access to the user's data and it's able to run arbitrary code against that data.

And so there actually is a sense in which it is an execution environment without the "interacting directly with hardware" piece.

Zhenya: Totally. Yeah, totally. I think even more so in the future. We're looking at that seriously about like what it's like to just run code if it's, if it's not in the browser.

So far, like a lot of people use node. It's easy to write something in just node, . But there's that vault concept, which makes it a lot more like this operating environment almost because like your, your app in node. Likely, you know, if the user is to retain control over their stuff, it's likely that they need this, you know, separate vault entity, which is part of that environment.

. But of course, node offers the bulk of it, right, and you don't have to do much to build a DXOS app on the server, like a headless one. That's doable.

Jess: I mean, we have a TypeScript SDK, you could do that today. And run it on the server. Yeah.

It's interesting too. I think the other thing that makes DXOS operating system like is returning to a point you made earlier of if an operating system is a place where the users go to get their applications and like they store their data there and then they have their applications or views, I'll, I'll leave aside the views for a second, applications which they use to, you know, manipulate their data.

DXOS is. A little bit different in that by separating the user's data from the application, it encourages you to make many different applications potentially that access user data. And so there, yeah, so then if you think of DXOS as like a sum of all of the applications written in DXOS then in that sense it's similar to an operating system.

Zhenya: Yeah, right, exactly. The data's shared. It's, it's one database for all the stuff. The magic of that is like one app can go build a to-do list, and then a totally different app can use that same to-do list, to render a document with the tasks built in. And they can be written by different developers, and the experience using all of that is real time multiplayer, offline capable by default for all developers for free just because they're using the,

Jess: the database.

Yeah. No, man, that's, I'm, I'm, so one of my, we should put a pin in this as a future fireside discussion. But one of the philosophical questions I'm curious about is, does DXOS trend towards building smaller, Unix- shaped tools, tiny little tools, like many, many, many, many apps that are all much, much smaller and have a much more constrained feature set. Does DXOS, does the grain of the DXOS technology trend towards that, or does it trend towards a master app that does everything, like all in the kitchen sink. And it's so easy to make these little views that it's even easier just to put 'em all in the same thing so that you have this, you know I think you've used the term master app or meta app . And so I think we should come back to that as a future topic, but yeah, it'd be interesting.

We don't really know, like we're speculating right now.

I got another card for you though. I'm gonna preempt it. Here we go. I should not have like offered that. So here's my card. All right. So how is DXOS like, or not like a social network?

Zhenya: Oh, interesting. Social network is, A place where people can trade specific kinds of messages, let's say It's okay. It's where people can communicate, it's where users can communicate. And I think, does that mean any Messenger app is now a social network?

Jess: Maybe? Right? It's sort of trending that way.

Yeah. If you think about Telegram, the way people are using Telegram these days.

Zhenya: Yeah. I think there's like some aspect of discovery. Beyond just messaging where, like your phone, that, that had phone numbers, and so it's really hard to discover people. There's no directory of phone numbers.

That's like the directory of phone numbers was broken, it was like a book that had no numbers. It's hard to use. But I think social networks bring this discovery aspect and if you squint, I think they all also bring some kind of ad revenue and newsfeed play that I think is not interesting.

But if you're saying social network and it's about users communicating and discovering each other . Then I would say we have a lot of overlap with the way the wallet works, like the, the vault and works. Because it turns out when you collaborate with others over your data, you end up exchanging security information just to authenticate people into your data 'cause it's really important to you that you establish people's identity when they're allowed to interact with your vault. That exchange is more risky the first time you do it with somebody you don't know. And so you want to exchange some kind of side channel secret, like a one time code. All kinds of auth systems have a concept for this, right? But it's through the fact that you've already seen that identity key before that you can sort of trust them in subsequent interactions for other spaces. And maybe even trust their network of friends, like whoever they trust you might be able to trust as well.

So for that reason, like the thing we have in the vault that like deals with the invitation flow for your peer-to-peer exchange is quite similar to some of the aspects of a social network.

Jess: Yeah. It's kind of a social graph, right?

Zhenya: It is, yeah. It ends up being.

Jess: This device, which is my device, is related to this other device, which is owned by someone else.

Yep. So yeah. Yeah. No, it's interesting. You could build up a social graph from this like, Relationships between devices which is kind of interesting to think about.

Zhenya: That, that ends up being data that your vault knows about and if an app is allowed, like if you allow an app to access that part of your vault, then you could build software that visualizes that. And I think we've done that with our stuff and we'll roll out more samples that show that.

Jess: Yeah. One of the things that's interesting there too is if you boiled the social network all the way down to, it's a social graph. People to to communicate with people across that social graph.

Then yeah, DXOS in some ways does resemble that. I think the big two things that are seem to be missing from social media, social pla, social network today that in DXOS world, one would be the notion of sort of like a global timeline. Or some sort of global town square that's, unfortunately, that is kind of like baked into the way that people understand social media right now of like, I'm gonna put something out there and then the entire world can access it and discover it if they want to.

And we don't have that kind of you might call that a publishing step. That DXOS doesn't currently have, although we could potentially build it. And then the second thing that we don't really have, like you mentioned, is we don't have discovery on a sort of, let me browse a directory the way that you might do in a Twitter or something like that.

Discovery works on a peer-to-peer, like direct device to device relationship of "Hey, I wanna share with just you, Zhenya, I'm gonna send you a special link.". Once that link is established I mean, like you said, we, our devices know about each other and then there's, we can now share data seamlessly between each other.

We could share messages. We could have a decentralized social network within each individual space.

Zhenya: Yeah, so your note about the feed is critical, right? It's the actual instrument of putting ads in your face, right that everyone's using it for.

But like, if you squint at it, I think the feed might start with some kind of pre-rendered static page. it's not quite the, data store you're looking at there. You know, all of these big apps have to like pre-bake some of the feed headliners, or maybe they used to have to, maybe today it's totally different.

But I'm trying to say is like our database is not quite for that purpose quite yet. Like, I think we're still quite focused on the protocols. And the main use case is collaboration between dozens of people. It's not for hitting that database with thousands of requests per second from anonymous users to like expose one entity.

You typically don't really do that with these high cost data stores, right? You would probably bake, bake that into some kind of static asset and put that on a CDN or close to the edge. And in that sense, that's also a different store in more like a blob store or a cache or an edge cache.

And our argument is that isn't the hard part right now, like, everyone's doing that pretty well. People aren't good at doing collaboration real time and offline in composition with that. So like, when it's online, it's real time, but when it's offline, things are okay too for like hours or days, you can just deal with rights from multiple offline parties and then reconcile them when you come online.

No one gets to that in their build for their software today. But if they started with what's inside a normal database that knows how to replicate they would get it for free. And that's what we're doing. There's a ton more complexity to solve to bring it to the client and that's what we've tried to do. You need to bring an identity, you need to bring in all this stuff. But yeah, the, the point is it's things should be easier this way. We hope. Yeah, right.

Jess: For sure. So I actually, that brings me to one of the cards that I had. So in what sense do you think of DXOS as a database?

Zhenya: Well, I think we're saying the core of a developer platform here is the way state is being synchronized between clients. We're, we're calling that a database, so it's not everything 'cause you also need to figure out how to host, you need to figure out how to do identity, you need to write UI tooling. To bring that to the developer, they don't wanna have to rebuild all this stuff from scratch. So our vault comes with UI flows that, you know, for joining spaces and finding peers, and that's where the social network graph will eventually materialize, you know, or the community will help build it there.

But yeah. It's quite centered around a database.

Jess: So traditional database, right? You have it's persistent in the sense that I can, it, it's not a cache or an in-memory thing only. But like I can, I can quit this application, bring it back up, I can shut down the database, bring the database back up, and the data is still there.

Persistence, querying, some ability to get things from the database as I want them, find them in there, that kind of stuff. And then there's usually some idea of performance as well. Whether that's like, you know, performed via indexes or whatever. But in other words, it, it'd be faster for me to put this in here than it would be to put it in a text file and do free text search across the entire text file, right? Like, I'm not gonna maintain my own database. And then there's like SDKs and DX developer experience type things around it.

But if we, if we just settled for persistence, performance, and what was the other thing I said? Persistence, performance. And the third thing I said, how does, how does DXOS stack up?

Like what, tell me about the database aspects of it. Yeah.

Zhenya: Like persistence, querying,

Jess: persistence querying. That was the third thing.

Zhenya: Oh, right, right. Querying persistence is a challenge actually in the way that we think it should work. It turns out that if you don't plan to have any central servers or any central authority on where your stuff is, at all, and things depend on the storage of all the clients involved in the swarm, and some of them may go offline and, and perhaps all of them may go offline forever, then you're kind of lost. You know, you, you, as long as there's, you know, there are no peers left in the swarm, you're, you don't have the data anymore.

So persistence has a higher cost. And we're finding ourselves returning to something that feels like using a server. But it's totally not like using a server, but more like using a reliable peer. Like a reliable or dependable, let's say dependable peer, something that you can self-host.

You're welcome to self-host, or you can put it on a server somewhere. Or eventually there will be people running services that help you do this for a fee or whatever, but yeah, it's like your vault duplicated in this other safe place that's always online. . That helps with a lot of things, like, you know, keeping track of who's online and offline and making sure all their rights are properly ordered but also doing this key persistence backup function. Yeah. But yeah, it's kind of like the way BitTorrent works, right? And the way peer-to-peer . Streaming would work is if, if all the peers or seeds vanish, then yeah, persistence is a problem.

So, but you know, we, as far as a client is concerned, things are persistent on that client.

Jess: That makes sense.

Zhenya: What else? Querying and performance? Querying is still under construction. But we are at the point where we can now build simple apps with it. I think a key thing to note about the difference between static files on disk and your object model in memory when you're working with your client app and JavaScript or TypeScript is, is that like you don't want to deal with the serialization and transformations that you have to do to, to put something from memory to disk in a text file. You wouldn't like to work with a text file directly as much as you would like working with a JSON payload or a proper object in memory. . So I think that's a big reason why things like mon, you know, the NoSQL movement was, you know, partially successful because it's much easier to work with because I'm already in a memory space with objects. So that's what we give you. In terms of The way the database feels to use you're just getting regular objects out of it that don't need any more mapping, and you can work with them just like you would with JavaScript objects.

And to query them it sort of feels like matching selectors on fields, it's like, Hey, can you give me all of the objects that have this fee field in this value? It, you know, the capabilities right now are on par with the capabilities of a key value store, I would say, for like how easy it is to query.

Obviously we're gonna add to that, you know, as soon as we bump into it when we're building apps, we're gonna go, oh, now we need this type of querying that we wanna do. Under the hood, it is a graph store so it already does give you the ability to like traverse edges between documents and assemble document trees and make them reactive in the sense that if other people change that document, it'll tell you, like it'll fire events and stuff and change on you.

And then in terms of performance, we're, we're working hard on making it scale. So at the moment I think we're good at maybe 10, 20 people collaborating in a given space where you'll, you'll start to see funny things happen unless you're running your agent which is the, that dependable peer I spoke about, which helps to like control the size of the database.

One thing to realize about this whole model and this is true about how databases already work under the hood, but we're just making it a lot more visceral, is that your data isn't just the cell in the table your data now has a feed of changes on that cell. And the feed contains changes as granular as possible from all the peers in the swarm. So if you add more peers, every single peer is replicating all of those feeds, so that can grow pretty quickly. And so we've solved that problem as well. And there's a protocol and a mechanic for keeping that under control and not being exponentially expanding. But yeah, that's, that's one of the things to, to think about when you're putting data in ECHO.

I think that covers performance, right? I mean, we're working on pushing that further. Right? I mentioned a number around 20 peers per space is our current benchmark. But we're gonna be putting that higher and higher as we go.

Jess: Right. And then the size of the data too, if the data is actually being stored in IndexedDB or in your browser, and so, you know, there is data that exceeds the hard disk space of your local client, would not fit. And so, you know, if you have a 128 gig machine you're not gonna store more than a hundred gigs in that in that browser. Probably it's gonna be a lot smaller than that before you run out of space. But there are some limits here in terms of we tend to think about centralized data stores in terms of storing many gigabytes or many terabytes of data, and sometimes even larger than that and this is not that kind of thing.

The interesting thing that I've noticed too is, is when you think about Centralized data stores versus this sort of, or like a Postgres or on MySQL or a big table or some of these other data stores that we're familiar with, and then ECHO, which is working within this, peer-to-peer decentralized space, one of the reasons those data stores get so big is because they're storing the data for literally millions of users.

And it's unlikely that you would have millions of users all in the same space within an ECHO application, all in the exact same space. And so right now the space is the unit of replication. And you know, I think there's probably some Dunbar number type stuff where the, like the largest space is gonna be a hundred, a hundred twenty five, a hundred fifty people, and there'll be some weird outliers that have a thousand or 1500 or, or even 10,000, maybe the biggest space, you know, ever. But not millions. Right, not, not user bases the size of Twitter or Facebook or that kind of stuff.

And so the characteristics of the database can be very differently tuned for these sort of smaller sets of users, smaller sets of data, knowing that it's not gonna get that big because we only have so many users sharing one of these little ECHO instances.

Zhenya: You touched on a really big interesting thing here with this whole scalability and how much you need it.

If you look at, for example, any modern, like high performance client application, even if it's working with a lot of data . Like I would point to maybe something like Azure DevOps, let's say as an example I would call a data heavy client application, right? That's really all about looking at tables of data and it's, you know, very interactive with other, with other apps or. Something like Airbnb, even like Airbnb is, is an app that needs to perform quickly and like render a whole, you know, bunch of data about listings, right, all kinds of different ways.

The client itself, when that client loads it isn't gonna be handling the entire database on the server. For any one given user, clients are always bounded to some manageable amount of data. Turns out users don't actually need access to that entire data that you have on the server, and that's just an aspect of centralization as it turns out, like the, all those servers and backend tier are holding a linear combination of multiple users worth of stuff, and the togetherness of it isn't always the primary thing to do.

The primary thing to do is to just serve those users with the Airbnb listings and the, the togetherness of it is actually value to other people who want to mine data and who want to, you know, train AI on that data and do whatever else they want with data. .

But like, So the end users, it actually doesn't matter that it's all together and it's all up there in the cloud. To them all they would want is just their data, where they are, like on their device. You only collaborate with a number of teams of people. And those teams aren't bigger than like 50 people, A hundred people, right.

Jess: It makes me want to try to build an application that would have a sort of central directory or centralized data data store. And then also have the user data be stored in ECHO. So something like a Google Maps or an Airbnb where they're like, there is actually just one set of locations, right, that are, you know, that are defined for the whole globe.

And when you're accessing Google Maps data, you are accessing like the set of locations that Google Maps has defined. And then, you know, and user data then could be layered on top of that, but stored on the client side. Right? So you could actually have reviews or like favorites would be one thing or places that I visited.

So instead of that being stored on Google Maps server that's now stored on the client side, it's private to me, but I'm still kind of looking into that centralized store for this central directory that, that would make sense.

Zhenya: Yeah. Yeah. A hundred percent. Exactly. That's, that's kind of the realization we're making in some places that like the, the data you're collaborating on and that's private to you is user data is okay to live completely on the client.

And there are these special things that do want to be in the middle between everybody and they do want to be singletons. Like DNS is a good example of this, it's like the way we name all of our servers. That is this one, global Singleton, and you have to pay the Piper to like get your SSL certificate and play that game.

So, Yeah. That's an interesting area. We have a lot of thoughts about also, maybe we should talk more another time. Right. But you're absolutely right this private data doesn't need to be in the middle.

And you know, composer, the composer GitHub extension is kind of similar. It like privatizes some state that lives in GitHub and lets you collaborate on your GitHub issues, like in private with your team before committing it to GitHub.

It's the same kind of thing.

Jess: That's really interesting. Yeah. Like you said, we, we grab that data from GitHub, from a GitHub issue.

We bring it into ECHO Collaborate on it. And once you're done collaborating on it, you save it back to the centralized data store, which is GitHub's database, and update the contents of the issue. That's interesting.

Yeah, it's there's so much to explore here. I think one of the really interesting things to kind of go back to the beginning of this conversation DXOS enables people to build a software that has a different shape or a different way of building it.

And there's so many unsolved problems when it comes to how you would architect applications like this. In many ways, it's simpler, like we talked about, of like you can store state and things locally, which like simplifies some of the client logic. And some, and it makes, it sometimes makes it easier to reason about the overall system, but there are other, like, there it is trade-offs too.

There are other things that it makes more challenging to reason about. One of the things I'm excited about over the next several months is starting to get a feel for the real the grain of DXOS applications of like, okay, when you encounter this shape of problem, here's a general pattern that you can apply to solving this problem.

And I think one of them is this sort of a central directory problem. And maybe we can like dive into that more and like a future episode of how would you architect a system like that. We can talk about our GitHub composer extension, but there'd be many other examples like an Airbnb or a Zillow or a Google Maps.

And we could give some other examples of those kinds of things. That, yeah. How would you architect that in a decentralized way?

Zhenya: Sounds like fun.

Jess: Cool. Well, I noticed we're coming up on time. We've talked about DXOS as, and a lot of things about its attributes. Any closing thoughts?

Zhenya: Oh boy. Yeah, stay tuned. We're like super nervous. I think. At least I am, because I'm excited to like open the doors and show everyone like what we're working on. But that just means we're like, couldn't, couldn't be more keen for your feedback. So if, hopefully some of this was provocative and if you have thoughts, please hit us on our discord and look at our code.

You know, try to run it. Tell us what's wrong with it. 'cause we wanna know. Thank you.

Jess: Cool. Yeah. And we'll try to do this again. Thanks y'all for joining.

Zhenya: Adios.