C3D, or contract controlled content dissemination, is one way in which content could be stored and passed between networked nodes using decentralized architecture. It is part of a stack of infrastructure which would allow for serverless email, decentralized dropbox, user-owned twitter feeds, and a whole series of decentralized applications.
This is only at proof of concept stage, but nonetheless we are getting there.
C3D – What is it?
We have set up a DAO stack using modified versions of the three traditional layers of software design: model, view, and controller. The Ethereum contracts layer acts similarly to a controller (although not exactly like a controller in that some data flows between the model and the view without passing through the controller first). This is more or less what DOUG is meant to do and does quite well at this stage. The view is any user interface running on client side javascript with pretty much any server structuring.
This post will focus on the model layer of this stack – which is what C3D is all about. There are two aspects to C3D, there is the client cache layer which runs on users computers and there is the contract layer which runs on the blockchain. These two aspects break up what is used by a traditional relational database to separate the raw data storage and dissemination from the establishment of the relationships between the blobs of raw data. The contract layer establishes the relationships between raw blobs of data and the client cache layer works on saving, getting, and disseminating the raw data.
The model layer understands blobs and groups. Blobs are individual pieces of content. They can be individual emails; they can be individual files; they can be forum posts; they can be proposed rule changes to a DAO framework. It is immaterial to the C3D system what is actually in the blob. Blobs can be a part of groups. Blobs can be part of multiple groups. Blobs do not know if they are or are not a part of groups. Blobs can be encrypted and thereby have reduced (controlled) access. Blobs can be sent in the clear and thereby have open (truly open) access. Blobs can be cryptographically signed thereby proving authorship. Blobs can be not cryptographically signed thereby enhancing psuedonymity. Blobs are files. Whatever you can do with a file, you can do with a blob.
Groups are ordered sets of blobs. They can be threads in a forum; a series of proposed rule changes (and changes to the proposed changes using a gitlaw functionality or something similar); a folder containing emails; a git repository. It is immaterial to the C3D system what the group actually does. The C3D system will track the order of blobs within a group via a linked list which resides in the contract layer.
C3D uses trackerless torrenting technology to disseminate content using simple magnet links to file blobs. All file blobs have a computed magnet link and a filename which is a slightly truncated SHA1 checksum of the file components.
The client cache layer controls the following one-off tasks without reference to the contract layer:
- Make blob
- Destroy blob
Note that the second task will not destroy a blobs existence within the network but will destroy it within the user’s cache.
The client cache layer controls the following one-off tasks between the cache on the users computers and the DOUG compatible c3d contract (contract layer):
- Add blob to group
- Remove blob from group
The model also controls the following ongoing tasks based on querying of the contract layer after each block:
- Watch contract
- Unwatch contract
- Watch group
- Unwatch group
- Ignore group
For POC 1, which should be finished in the coming two days, this is all that the model layer will be able to do.
For POC 2, I plan to add encryption and signature functionality which will marry with DOUG’s access restrictions nicely so that users can have much finer grained control over who can access the contents of the blob.
C3D – How is this working?
In the blockchain there is a DOUG-compatible contract which is a tightly packed, linked list of groups and blobs.
Each contract on the chain is a major grouping of blobs. Each contract in turn has Groups which are ordered by submission time. Groups are minor groupings of blobs.
The contract allows for four transactions noted above. These transactions can be sent either from the UI to the contract directly, or from the UI to the model to the contract. Because the transaction is standardized and because the functionality of subscribing to content and publishing content are isolated it need not matter which component operates on the relational layer of the data model. There will likely turn out to be optimizations in favour of one method or another for different use cases but those will likely only show up over time.
Publishing Content
When the UI wants to add a file blob (because its user just made a post to a forum for example) it sends a publish command to the client cache component of the model layer to make a blob. The contents of that blob are also send to the client cache component of the model layer as parameters of the command.
Here is the sequence of events which the model goes through in order to publish content. The model layer:
- takes the SHA1 checksum of the file,
- truncates the checksum to 24 bytes,
- saves the file in the cache with the truncated sha1 as the filename,
- creates a torrent file without trackers,
- submits the torrent file to a running torrent client which uses DHT rather than trackers,
- gets the btih (bittorrent information hash which is 40 bytes long) from the torrent client and builds the magnet link,
- sends a transaction to the contract with identifying the magnet link which is the btih (40 bytes) + the sha1 checksum filename (24 bytes). This transaction takes exactly two memory slots (64 bytes) and thereby reduces the gas of the transaction somewhat. The gas costs are not currently optimized.
In POC 2 there will be the addition of an update blob command which will remove an old blob from its group(s) and then republish the updated content.
Subscribing to Content
When the UI wants to watch content it will send a subscribe command to the model layer with params of either a contract address or a contract address + group identifier. For instance, if a user is interested in a series of forum posts, the UI should have that content available to the user as soon as possible. This will require caching on the user’s hard drive for efficiency and responsiveness, and in order to do that the UI can send the subscribe command so the model layer will automatically grab the content and have it available within the cache to the UI as soon as it can. The model layer will keep remember what needs to be watched based on criteria established by all of the UIs which are currently interacting with it.
Watchers will get more sophisticated around POC 3, but for now (POC 1) it is quite raw. The model layer will query a running Ethereum client each block to see if anything has changed within the watched content. If content has been added then the model layer will add the magnet link (which was published by the original poster) from the contract to the client cache’s torrent client which will download the client from peers, cache that content, and reseed that content to the swarm. If the content matches the subscribe command boundaries but has either an ex post or ex ante ignore command the client cache layer will remove that content from the cache and from its seeded content. For POC 1 this will be very rough and will be the focus on POC 3 to make this component smarter and better optimized.
The opposite command to subscribe will be the unsubscribe command to the model layer with params of either a contract address or a contract address + group identifier.
Managing the Cache
When the UI wants to stop watching some content (because the user is a spam bot, because the content is not important, or for any other reason), it will send an ignore command to the model layer. Ignore commands will override subscribe commands.
POC 3 will build a unignore command which will remove any previously submitted ignoring commands.
When the UI wants to simply get content for a one off, it will send a get command to the model layer along with the magnet link to the content (because the UI is connected to the contract|controller it will know the magnet links). The client cache layer will add the magnet link to the running torrent client. Get commands do not respect ignore or watch commands and are meant for one off things rather than ongoing interactions with data sets.
The Components of the Cache
Functionally, the model layer is working on the ~/.cache/c3d folder. Within that folder it builds a JSON file of watched components called watchers.json. Watchers are simply JSON objects identifying contracts and then within the contracts identifying groups via an array of groups. There is also a JSON file of ignored components called ignored.json. This file uses the same format as the watchers.json file.
Within the c3d folder there are three folders, one for blobs, one for torrents (the ‘.torrent’ files), and one for content that is currently in the process of being blobbed and published. The original poster of content will need to seed the content blob for a sufficient time to get their information into the network swarm, but afterward if it is popular (heavily subscribed) content then the original user could delete it if they wished.
C3D – What can we do with it?
- Server-less email
- Decentralized dropbox
- Github 2.0
- A resilient pirate bay
- Legal rule changes backed by gitlaw principles
C3D – How do we use it?
This is not really a strictly standard MVC architecture but it is analogous to an MVC architecture. One of the key different is the attempt to leverage the advantages of file system caching vs. contract establishment of relationships as explained above. On this basis developers can build a whole variety of components with very little effort other than modifying the model layer to suit their particular needs.
This project is currently at POC ~0.7 and can be found on the github repo for the client side and in the other github repo for the contract side. I am hoping to have POC1 (v0.3.0) pushed by Tuesday with further POC rounds to follow.
Currently the project is in ruby mainly because that is where I’m the quickest and most comfortable. Ideally this should be written in Go to utilize Go’s concurrency model and ability to cross platform binary compile and should probably utilize Taipei Torrent (a Go torrent library) to abstract away everything from the user, but for now I would rather push out the concepts to see where the flaws are and spend the time a little later when we have proven the concept to port it to Go.
Would you be interested in helping us test this or build this? Please fork and contribute whereever makes sense for you.
If you would like a demonstration of the dissemination side of the platform, just add the following link to a torrent client: magnet:?xt=urn:btih:1648bc0269db160521bacc85bb3327ed36494a49&dn=5e8a167bc76982df528580cf
Happy Contracting!
~ # ~