Background: On Friday, I started reviewing the RSS produced by some of the BitTorrent sites. What I found was pretty great. There were some problems, but nothing that can’t be easily sorted out. What was really exciting was that through Twitter, some of the developers of the feeds and apps that use them, got in touch. As a result of the discussions, I agreed to outline ideas for a possible BitTorrent namespace. That’s what I’m doing here.
These are just ideas. Please don’t implement anything based on what you read here. If you have observations, please post a comment below. If this yields a namespace that people use, there will be a fixed URL that contains the docs for the namespace. It won’t be on unberkeley.com. 
All elements in this namespace are optional. It’s perfectly valid to have an RSS file that represents torrents in enclosures without using any of these elements.
Each RSS item describes a single torrent. The enclosure element describes the torrent file itself. The length attribute is the size, in bytes, of the torrent file. The type attribute is the MIME type of the torrent file. The url attribute is the address of the torrent file.
While the namespace is designed for use in RSS 2.0 feeds, it may be used in any XML-based file format that allows extension through namespaces, such as Atom or OPML 2.0.
For terminology, I used the BitTorrent vocabulary Wikipedia page as a guide.
Now for some of the possible elements of the namespace.
torrent:contentLength
- The total number of bytes in all the files the torrent makes downloadable.
torrent:contentFiles
- The number of files the torrent makes downloadable.
torrent:seeds
- The number of clients that have complete copies of all files made available by the torrent.
- It’s a guide to how quickly the files may be downloaded.
torrent:peers
- The number of clients that have a partial copy of the files made available by the torrent.
- Note: This was originally “leechers” but was changed because there was a consensus that it should be called “peers.” DW 12/6/09
torrent:verified
- A boolean value, if true, the torrent has been verified — it’s a real working download, not a fake or scam. If false, it has not been verified.
Posted by thermal on December 3, 2009 at 5:35 pm
Hi Dave,
Perhaps the scope could be broadened to allow for meta-file formats in general, not just BitTorrent .torrent files.
The concept being simply a file that includes references to other files or media using metadata such as a SHA1 or mtag . The reason I’m suggesting this is because I’m currently working with others on such a format, similar in ways to Bram’s torrent file.
With this approach, the type of meta-file, such as bt, or even the MIME type, could then be specified. And since a lot of attributes are likely to be shared, this might prove advantageous over having a different implementation for each format.
What are your thoughts on this?
Posted by Dave Winer on December 3, 2009 at 5:54 pm
In general you should re-use as much as you possibly can. Makes everyone’s life easier.
I’ve seen the other comments on the original post. I need some time to digest all this and then hear people’s reactions to these ideas.
I like to move deliberately, esp when I’m new to an area as I am to BitTorrent. I’m just a user here, but I’d like to do some development, and getting the RSS straightened out is key to doing that development.
Posted by IH on December 3, 2009 at 6:05 pm
I’d suggest using “BT” as namespace instead of “torrent” both conciseness and correctness, as torrent is a somewhat ambiguous word.
Posted by Dave Winer on December 3, 2009 at 6:19 pm
At first I was using “bt” as well — but as I thought about it, I felt that the elements in the namespace would be describing torrents. I used the Wikipedia terminology page as a guide…
Posted by Mason Lee on December 3, 2009 at 6:29 pm
Don’t confuse xml namespace prefixes with xml namespaces. In this case, “BT” or “torrent” would be a prefix, local only to individual xml files. Conforming XML parsers will ignore the prefix string except to find the corresponding namespace URI. As Dave points out, the actual xml namespace will be an http URL, and it’s that which matters for cross-app compatibility.
Just one of the many reasons it’s not recommended to parse XML without a library. Luckily there are lots of good open libraries.
Posted by NovaKing on December 3, 2009 at 6:09 pm
not sure if verified has a point to it… if you are providing an rss feed to it, you would assume it’s already verified, and if that is not the case, you don’t want to change the rss feed once it has been verified.
Posted by Dave Winer on December 3, 2009 at 6:17 pm
It’s in the kickasstorrents.com feeds — and I put it in to see what people would say.
At least some of the torrent sites include files that aren’t verified.
Posted by NovaKing on December 3, 2009 at 6:29 pm
yes, i understand that, but you can say all torrents are not verified when first uploaded, then get changed to verified at a later stage, this i can’t see working well with torrents… unless you change a rss feed after it is first created?
Posted by NovaKing on December 3, 2009 at 7:47 pm
Just thinking about this some more, i would rather have 2 weeks, one for “not verified” and one for “verified” having it in 1 rss feeds still just makes no sense in my head… am i the only one thinking it odd to have?
Posted by NovaKing on December 3, 2009 at 7:50 pm
2 feeds.. i think i need more sleep right at this moment
Posted by thermal on December 4, 2009 at 12:07 am
Nova,
I agree. Since a “verified” field would’t be static, each item in the feed could be subject to change. There could be issues with this, like caching etc.
Is each RSS item generally supposed to be fixed by design and not updatable once published?
Posted by Stefan on December 6, 2009 at 3:58 pm
The s should be updateable as long as they have unique s.
With torrent:verified, I have a question. As I assume that most sites actually delete the whole torrent file and everything associated with it when it is verified as spam, I suspect that RSS aggregators will not pick this up unless the RSS is updated to somehow mark this. Any thoughts?
Posted by DeathfireD on December 7, 2009 at 7:16 am
RSS is a dynamic thing so it would make sense that a site would update the verified field. As more social networking type torrent sites pop up that rely heavily on the user to rank and decide if a torrent is valid, this field is a must.
Posted by MarcusT on December 8, 2009 at 6:26 pm
It’s not just the “verified” field that might change state over time, wouldn’t the “seeds” and “peers” fields potentially change every time the RSS feed is retrieved? If not, are they meaningful?
Posted by Stefan on December 6, 2009 at 11:50 pm
Looks like my post got manged up, it just stripped the html instead of textifying it.
What I meant to write was: “The items should be updateable as long as they have unique guids.”
Ugh.
Posted by Twitted by john95959 on December 3, 2009 at 6:16 pm
[...] This post was Twitted by john95959 [...]
Posted by Mason Lee on December 3, 2009 at 6:29 pm
This looks like a great idea.
Posted by Reviewing BitTorrent RSS feeds « UNBERKELEY on December 6, 2009 at 3:05 pm
[...] by Ideas for a BitTorrent namespace « UNBERKELEY on December 3, 2009 at 5:10 [...]
Posted by Mike on December 6, 2009 at 3:29 pm
Would recommend torrent:leechers be changed to the proper term torrent:peers.
Posted by Kol on December 6, 2009 at 4:22 pm
Agreed. Naming conventions should be based on original definitions instead of nicknames adopted because of “cuteness”.
It’s exciting to see this level of co-operation regarding standardization. It would do my heart good to learn about an invitation-only brainstorming session after the fact.
Thanks for this spark, Dave.
And many thanks to the numerous developers making it happen.
Posted by Dave Winer on December 6, 2009 at 4:33 pm
Okay — sounds like a bit of consensus here, so I will change “leechers” to “peers.”
It was confusing to me because kickass used both terms and they often had different values!
About “invitation-only” — everything has been out in the open from the beginning. I don’t know any of the leaders in this field, so I couldn’t have invited them to anything. I’ve pissed a lot of people off in the RSS world about not being willing to go to invite-only confabs. I say that’s a contradiction, you can’t have something be open and work on it in an exclusive environment.
In this context I’m just a note-taker. I don’t know very much about BitTorrent except I think it’s an important technology and it needs some care and feeding.
Posted by Kol on December 6, 2009 at 5:00 pm
My thought about the “brainstorming session”:
1. I imagine a chat room or voice conference organized and hosted by a developer.
2. In such an environment, distractions by trolls would decrease the value of the effort and waste many people’s time.
3. Moderation of some sort would be absolutely necessary in order for ideas to flow freely.
4. I don’t see any reason why the general public couldn’t join in as an audience. In that light, I would hate for my cat to jump on my keyboard while I’m watching the discussion and send whatever it is she’s trying to type.
Just a thought.
Posted by Stefan on December 6, 2009 at 3:54 pm
As magnet links are increasing in popularity now, I think it would be a mistake to leave them out of this. I’m thinking of something like torrent:magnetLink.
Also, I think “torrent” is better than “bt” as the name for the namespace.
Good work.
Posted by Dave Winer on December 6, 2009 at 4:00 pm
I didn’t know about magnet links until I read your comment.
http://www.google.com/search?q=magnet+link
Of course if it’s data people want to distribute, then it should be in the namespace.
Posted by Kol on December 6, 2009 at 4:33 pm
For reference, here is a very good writeup:
http://torrentfreak.com/bittorrents-future-dht-pex-and-magnet-links-explained-091120/
It seems that many end users are having difficulty wrapping their heads around magnet links recently. It’s not for lack of information, though.
Posted by Mike E on December 6, 2009 at 5:11 pm
Rather than explicitly specifying a magnet URL, the components that would be used to construct it should all be individually provided.
Posted by Adapa on December 6, 2009 at 7:45 pm
So that would be:
torrent:infoHash
- Is the torrent hash, hex encoded, for a total of 40 characters.
torrent:tracker
- zero, one, or more tracker URL’s; comma delimited
[torrent:displayName would be unnecessary since name is already available elsewhere within the feed.]
Posted by James on December 6, 2009 at 7:12 pm
I also agree, the magnet URL should be included into the RSS data.
Posted by NovaKing on December 7, 2009 at 2:55 am
Having support for Magnet URI would be rather handy in case the torrent download link goes down, this was the reason we added magnet links to most of our sites.
Posted by Peter on December 6, 2009 at 5:03 pm
Hi Dave, Do you know about Metalink (http://en.wikipedia.org/wiki/Metalink , http://www.metalinker.org) it is also a metadata file format that that describes the location of content files. From the Wikipedia entry “it stores the multiple download locations for a file (FTP/HTTP/P2P) in a single metafile with the extension .metalink.”
It seems to me that a bittorrent namespace is a subset of the greater problem of using meta-data files (.metalink, .torrent, etc) to point to the location of the actual content over an RSS feed.
I don’t known much about using .torrent or .metalink files over RSS but I suspect that they have core requirements that heavily overlap. Could result in a general namespace for Metadata files in general and a specific name space for each Matadata file type (eg.torrent , .metalink etc)?
Posted by Kol on December 6, 2009 at 6:15 pm
I’m under the impression that Dave is facilitating our discussion and that he will chime in when he has something to add. He’ll read and respond as he has reason to do so.
In the meantime, please keep sending links to information and weeding out the conflicts.
Some of us in the audience are your students.
Posted by Kol on December 6, 2009 at 6:41 pm
I forgot to clarify.
“your students” is in reference to everyone adding their knowledge.
I dare say that a good portion or the respondents here are self-taught.
Impressive.
Posted by Matt on December 6, 2009 at 5:21 pm
You guys should look at the work Tribler is doing with RSS and Buddycast that is totally distibuted .
http://www.tribler.org/trac/wiki/Buddycast3
http://svn.tribler.org/bt2-design/proto-spec-unified/trunk/proto-spec-current.pdf
Posted by Kol on December 6, 2009 at 6:28 pm
Agreed again. Tribler should be in on the discussion about standardization.
Hopefully, the admins have seen the relevant threads by now. If not, it’s a simple task to get their attention.
Posted by NovaKing on December 7, 2009 at 2:55 am
I will try and get in contact with the tribler team and get them into this conversation.
Posted by Mike E on December 6, 2009 at 7:47 pm
So that magnet URI’s may be used, perhaps adding a “rel” attribute to the “enclosure” tag (just like in “link” and even “a”) could be an option. This would mean that multiple “enclosure” tags within an “item” element would be allowed.
e.g.: [enclosure rel="magnet" url="magnet:?xt=urn:btih:6TUXQVU37YU72BSHISLEWO47F4OAYQCC" length="X" type="Y/Z" /]
[enclosure rel="file" url="http://x.y.z/file.torrent" length="X" type="Y/Z" /]
The only potential problem with this naming convention is that a magnet is technically a URI, not a URL.
Also note that I’ve used a base32 encoded info_hash value. From what I remember, the ad-hoc standard that exists specifies that a base16/hex value is to be used. I don’t understand why both base16 AND base32 aren’t considered valid since their size is indicative of the encoding used.
All clients should be able to determine the encoding used from the size of the value alone, however I recall there being a few clients that only supported either base16 or base32.
Posted by Sebastien BORGET on December 7, 2009 at 5:24 am
Dear Dave,
I wanted to notify you of my previous work on the same topic. I indeed had published in July 2006 a white paper
http://www.borget.net/technical-articles/bittorrent-rss-20-feed-specifications-white-paper
http://www.borget.info/download.php?file=bittorrent%20rss%202.0%20feed%20specifications%20revision%201.pdf&action=go
I would love to chat with you further, feel free to contact me at my email address
best
Sebastien
Posted by Dave Winer on December 7, 2009 at 6:27 pm
Got it, but I’m not doing any back channel conversations. Say whatever you have to say here for everyone to see.
Posted by myxal on December 7, 2009 at 6:13 am
Not sure about torrent:verified but another possible true/false property could be something like trusted/vip. See piratebay – the VIP torrents have the pink skull, trusted have green skull. Another application would be the past mininova setup, where some of the torrents came from registered authors etc.
Anyway, I haven’t yet used torrent RSS and didn’t know there was no consensus here. Hopefully by the time I get to building a headless home server/ automated torrent downloader there will be no problems
Posted by Torrent Sites Get Feedback from RSS Inventor – FUCK THE RIAA on December 7, 2009 at 6:20 am
[...] His writing was soon picked up by isoHunt’s Gary Fung and EZTV’s NovaKing, who have already implemented some of Winer’s suggestions and started discussing a more standardized RSS output, as well as a BitTorrent RSS namespace. [...]
Posted by Torrent Sites Get Feedback from RSS Inventor @ blog.idtorrent.org on December 7, 2009 at 3:01 pm
[...] His writing was soon picked up by isoHunt’s Gary Fung and EZTV’s NovaKing, who have already implemented some of Winer’s suggestions and started discussing a more standardized RSS output, as well as a BitTorrent RSS namespace. [...]
Posted by LMZ on December 7, 2009 at 6:18 pm
add please magnet url and the future is here =)
Posted by uberVU - social comments on December 8, 2009 at 2:12 am
Social comments and analytics for this post…
This post was mentioned on Twitter by davewiner: Ideas for a BitTorrent namespace. http://r2.ly/q5yu...
Posted by John Doe on December 8, 2009 at 3:35 am
Internationalization.
Every movie/series torrent will produce, in external webs, several subtitle files.
Subtitles are specific to one archive timing (is very annoying when your only Spanish subtitle don’t match with your downloaded episode)
Any idea to solve this will be greatly user-appreciated.
Posted by NovaKing on December 8, 2009 at 8:02 pm
One needs a source for subtitles, and as far as i am aware there is no good source for them, most are user generated and are rather questionable. but if something more standard starts to appear for subtitles then it would be a good idea.
Posted by John Doe on December 9, 2009 at 1:59 am
If I am a subtitle generator web that uses 3rd party torrents then it makes perfect sense. RSS will say: download my subtitle(s) and this torrent (the media source).
Posted by NovaKing on December 9, 2009 at 3:47 am
Ahh, you do make valid point, but wouldn’t that be more of a subtitle namespace instead of a torrent namespace? then you can have structure rss feeds with multiple namespaces outlining what you have to offer. as i’m sure subtitles will need a fair bit of information to be shown in the feed as well.
Posted by John Doe on December 8, 2009 at 3:38 am
Internationalization (Subtitles).
Every movie/series torrent will produce, in external webs, several subtitle files.
Subtitles are specific to one archive timing (is very annoying when your only Spanish subtitle don’t match with your downloaded episode)
Any idea to solve this will be greatly user-appreciated.
Posted by DrNo on December 9, 2009 at 2:07 am
Why not a general rss for multimedia archives? If it is valid for a music storage like itunes then it will be more used and more known standard.
Extra fields needed:
price (x$, free, …)
how-to-pay (type & url)
how-to-download (torrent, direct-download, human-interface)
torrent (…)
direct-download(server, url)
type-of-media (mp3, movie)
mp3(use mp3 tags)
movie(imdb tags)
…
And I miss DHT in bittorent. Is it equivalent to Magnet?
Posted by NovaKing on December 9, 2009 at 3:50 am
Magnet is just a hash code of a torrent which it gets from DHT.
not sure if all of your fields listed are useful for a torrent namespace, i agree itunes and such should probably have a namespace for the data they can provide, but this discussion is more for the bittorrent namespace
Posted by DrNo on December 11, 2009 at 3:23 am
An only bittorrent makes sense?
Perhaps yes.
I have mldonkey, that is a multiprotocol [ http://en.wikipedia.org/wiki/MLDonkey#Features ] downloader.
I think that average user uses multiprotocol downloader managers, and that implies a multiprotocol rss it’s better than a only-torrent one.
Is it difficult to keep it open to other protocols in the same rss channel?
I have no deep technical knowledge of RSS. Could it be a flag, make it a xml node, a sub-namespace, … ?
Magnet handles it all? (edonkey, torrent, direct download like rapidshare).
Summarizing: Keep it easy to multiprotocol if you can.
Posted by NovaKing on December 11, 2009 at 12:46 pm
I’m not saying ditch the other protocols, what i am saying is they can have their own namespace with things specific to their genre. this discussion is about the torrent namespace, let’s focus on getting one completed before moving onto another.
Posted by MarcusT on December 11, 2009 at 1:12 pm
Yes, surely what is needed is a *generic* extension to RSS which can be used by many protocols/applications/formats, most of which will share many attributes/fields – and some potential applications may not yet have been invented – as opposed to just a BitTorrent-specific extension.
Posted by Stuart Taylor on January 5, 2010 at 7:55 am
To follow on from Dave’s initial suggestions, I think that a Bittorrent namspace is long over-due, but should include the following elements.
– Checksum hash for the content files
– URL of a tracker for information, with appropriate attributes Seeds, Peers, and Updated. An item could contain multiple instances of a tracker element
– URL of magnet link to suit compatible download clients
– URL of link for compatible direct download. This is offered by some indexing sites (i.e. kickasstorrents.com)
– Optional name of the uploader (this would need to be at the discretion of the uploader)
On reflection, the Magnet and Direct links could take the form of a common element with appropriation relationship attribute values (i.e. , ).
Posted by Michael on January 5, 2010 at 1:06 pm
A better solution would be access interface descriptors/classes. For example, an FLV file is better suited for streaming, whereas a DivX/XviD encoded file is more appropriate for downloading first and then watching. A torrent file is different again, being a metadata format that describes how to go about getting it, not where to get it from.
If a quasi-formal list of access classes could be thought up, it could be used in conjunction with an enclosure element to describe how the file/MIME type/enclosure reference is to be used.
These could be mapped to MIME types, however a magnet URI is a different scheme, not a file. I don’t think it has a MIME type.
Posted by Stuart Taylor on January 5, 2010 at 7:57 am
To follow on from Dave’s initial suggestions, I think that a Bittorrent namspace is long over-due, but should include the following elements.
contentHash – Checksum hash for the content files
tracker – URL of a tracker for information, with appropriate attributes Seeds, Peers, and Updated. An item could contain multiple instances of a tracker element
magnet – URL of magnet link to suit compatible download clients
direct – URL of link for compatible direct download. This is offered by some indexing sites (i.e. kickasstorrents.com)
uploader – Optional name of the uploader (this would need to be at the discretion of the uploader)
On reflection, the Magnet and Direct links could take the form of a common element with appropriation relationship attribute values (i.e. rel=”magnet”, rel=”direct”).
Posted by Dave Winer on December 6, 2009 at 6:22 pm
Okay — but there haven’t been any trolls yet.
You’re welcome to have whatever chats you want, but don’t organize them here if they’re not open to everyone who wants to participate. I want to say completely public and not have any back channel at all, for myself. That doesn’t mean that other people can’t do whatever they want.
Remember I’m just a notetaker here, I have no stake in any of this, except as a user and a fan.
Dave
Posted by Kol on December 6, 2009 at 6:33 pm
Your notes are appreciated and well taken. Please continue to scrutinize and respond as you see fit.
You’ve started something here that’s been sitting in the backs of many minds.
Let’s watch it develop.
Posted by NovaKing on December 7, 2009 at 2:57 am
I believe this should be an open discussion, there is no requirement for this to be done in secret. if trolls wish to come here, we can easily clean it up as we see them.