First, let me say that it’s great that RSS is so widely supported by BitTorrent sites.
When I met Bram Cohen at a Wired awards ceremony in 2003 (BitTorrent won, Skype was also nominated, as was RSS), I told him I thought the two technologies were a great match. Both were low tech, open formats for content distribution, that solved parallel problems.
So it’s great that six years later so much is being done at the intersection between BitTorrent and RSS.
That said, there are some obvious improvements possible. I’m going to take some notes here, and hopefully add to them over time.
ezrss.it
Looking at the feeds produced by ezrss.it. An example.
1. The length attribute on the enclosure must be the length of the torrent file, not the length of the file the torrent describes. For example, the correct length of this torrent is 14680, not 367450064 as the RSS feed indicates it is.
2. It would be great if the feed items had <category> elements. Obviously everything on eztv.it would be TV Show. Use the same categories that Mininova uses?
3. It’s a good idea for <item>s to have <guid>s. It’s the foolproof way for clients to tell if they’ve seen an item before. It might make the file a bit larger, but not by very much. It’s really worth it. ;->
isohunt.com
Looking at feeds produced by isohunt.com. An example.
1. Add some carriage returns and tabs to the XML text. Makes it easier to read the feeds.
2. The content-type that’s being returned is text/html. That’s pretty hokey for XML data. Try text/xml for a better result.
3. <item>s have <guid>s — good! you can leave off the isPermaLink=true, it’s the default value. Any client that isn’t defaulting this way has a bug.
4. They make the same length mistake that ezrss.it makes. The length should be the length of the torrent file not the length of the file it describes.
5. The category is provided, but it’s encoded in HTML in the description. That’s okay but there should also be a <category> element. It’s esp important for a site like isohunt that handles multiple types of torrents (not just TV like eztv.it).
Here’s an example feed produced by kickass.
This feed is interesting because it provides good hints for extensions to RSS that might prove interesting for BitTorrent applications.
But first, a list of things that need immediate attention.
1. There should be an enclosure element on each item, that’s how the client knows what to download.
2. The author element must be an email address.
3. The following elements are not part of RSS, and probably should be part of a namespace just for BitTorrent clients: torrentLink, hash, peers, seeds, leechs (probably should be spelled leeches), size (good idea, since the enclosure length would be the size of the torrent file, but it’s probably better to call this length for consistency), verified (use true/false instead of 0/1, slightly easier for a non-programmer to understand).
Good stuff…
1. They use the category element! That’s good.
Here’s an example feed produced by torrentzap.
1. No enclosure element, would be hard to write a client that processed this RSS feed.
2. Good — it uses the category element correctly.
3. There is no size element in RSS. Again highlights the need for a BitTorrent-specific namespace.
4. Would make it easier to debug if there were some carriage returns in the file.
5. Please use guids.
Update: Here are some ideas for a Torrent namespace.
Posted by IH on November 28, 2009 at 2:53 am
Dave, not everyday our RSS get critiqued by creator of RSS, so thanks!
Good tips on , that is now added on isoHunt.com and isPermaLink is dropped to save a little bandwidth. We are already using Content-Type: text/xml in our headers however, not sure how you got text/html.
On , since we are using the same link there as in , is that good practice? Or only is enough? It seems to me most clients still use for display, so I assume including both is good practice.
Posted by Dave Winer on November 28, 2009 at 8:32 am
Thanks for coming by!
1. I’ll check the content-type again. I just ran a quick script to see why it was displaying funny in the browser. Probably my mistake.
2. It’s pretty common for link and guid to have the same value, but they serve different purposes so both should be there, if it makes sense. I feel guid should *always* be there. It’s hard to write a good aggregator if you can’t depend on them being there, and being unique.
Posted by IH on November 28, 2009 at 2:58 am
ok tags are stripped instead of html escaped. Using [] in place of .
Posted by IH on November 28, 2009 at 3:05 am
On length in .torrent enclosures, I know it’s not correct to spec. But considering .torrent themselves are nothing but metadata links to the actual content and BitTorrent clients parses them, I find it more useful to include length of content files being pointed to rather than length of metadata.
If BitTorrent clients would adopt a standard attribute in enclosures to use, say, contentLength to specify length of pointed to content while length to specify length of .torrent file itself, that would be a more proper extension of RSS. So it looks something like:
[enclosure url='http://isohunt.com/linux.torrent' length='.torrent length' contentLength='content files length' type='application/x-bittorrent' /]
Posted by Dave Winer on November 28, 2009 at 8:36 am
No doubt it’s useful to know the size of the media file the torrent file is pointing to.
You can if you want adopt a convention among BitTorrent servers and clients, as you describe, with one difference. You should define a namespace, say “bt” and then refer to your attribute as “bt:contentLength.” The rule in RSS 2.0 is that it can be extended, but the extensions must be in their own namespaces.
I’m doing to do a writeup of another Torrent site’s RSS in a few minutes. They invent lots of illegal stuff. That’s both good and bad. It’s a starting point for what might be an excellent BT namespace. But their feed is not very useful as presented, and it totally invalid RSS.
I have some ideas for new clients assuming we can upgrade the quality of the RSS produced by the BT community (of which I am just an observer, btw, you guys make the decisions).
Posted by RSS of BitTorrent, Programmable Twitter client « News, Software and All you need on December 1, 2009 at 4:31 am
[...] Reviewing the RSS generated by BitTorrent sites. The goal to help improve the quality and uniformity of their metadata, perhaps to allow more [...]
Posted by Greenbid News » RSS of BitTorrent, Programmable Twitter client on December 2, 2009 at 1:49 pm
[...] Reviewing the RSS generated by BitTorrent sites. The goal to help improve the quality and uniformity of their metadata, perhaps to allow more [...]
Posted by NovaKing on December 3, 2009 at 1:43 pm
Hi Dave,
I’m amazed we would even get a mention, I wouldn’t have thought our site to be popular enough.
In response to your notes. The reason for the filesize being the size of the content and not the torrent file is actually due to third party software, I have had a number of developers ask for this to be implemented. and as no one else would even notice the change i implemented it. and i didn’t really want to create new tags as i didn’t exactly want to break the RSS RFC too much
In regards to the Category tag, I originally did not see a need for it, but if standardization is the key, well then I’m sure i can add it
with guid, i am actually working on this right now, so this should be implemented soon. thanks again to even knowing we existed
Posted by Dave Winer on December 3, 2009 at 1:57 pm
Are you kidding? EZTV is god-like. (I’m not worthy I’m not worthy.)
Don’t worry about “creating new tags” — I’ll start a BitTorrent-specific namespace, probably later today or tomorrow, and we can put the tags in that space. It won’t break the format, since RSS 2.0 specifically allows for adding new elements in namespaces.
When you’ve updated the format of your feeds, could you post a link here so we can review them?
It’s really great that you guys are tuning in!
Posted by NovaKing on December 3, 2009 at 2:08 pm
You can thank TorrentFreak for informing me about this page.
I’ll try and have the changes applied today. and when you create that new namespace i’ll implement those changes too, then i’ll have to talk to the developers to get them to update their software.
Posted by NovaKing on December 7, 2009 at 2:46 am
Had a hectic weekend… RSS has been updated. all new feeds will contain guid/comments
http://ezrss.it/feed/
Posted by M. Bedda on December 3, 2009 at 4:47 pm
Hello Dave,
I’m one of the developers who uses both isohunt and ezrss’s feeds. I respect that you’re seeking to fix the use of RSS, and i’m all with standardization, as it’s actually the idea of RSS in the first place.
Actually NovaKing was using the length attribute to store the length of the torrent file itself, till he changed it lately when i asked for the length of the content itself, where it is the thing i need not the torrent file itself.
I would have no problems if any changes occur as long I’ve been informed, so i could make my updates.
Best Regards,
M. Bedda
tvtrigger.com
Posted by Dave Winer on December 3, 2009 at 4:54 pm
Excellent.
I’m starting on the namespace definition right now.
I will include an element that contains the length in bytes of the content that the torrent file points to.
Posted by Ideas for a BitTorrent namespace « UNBERKELEY on December 3, 2009 at 5:10 pm
[...] by Dave Winer in Lunacy. Leave a Comment Background: On Saturday, I started reviewing the RSS produced by some of the BitTorrent sites. What I found was pretty great. There were some [...]
Posted by thermal on December 3, 2009 at 5:20 pm
Hi Dave,
I’m currently working with NovaKing on several specifications, one being a ‘mediatag’, or ‘mtag’ that serves the purpose of uniquely identifying a “multimedia object”. A multimedia object could be one of several classes, for example: a series, season, episode, movie, album, song et al.
We haven’t quite figured out the best way to integrate this with RSS yet. The most obvious way would probably be an element, such as: 37d0aee3-90b508e8-30dd3b6d-b8858cfe. At present an mtag is a 128-bit identifier represented in hex, similar to: http://en.wikipedia.org/wiki/Portable_Unique_IDentifier
Would be great to get your feedback on this and some other ideas we have.
Posted by thermal on December 3, 2009 at 5:40 pm
The angle brackets of the mtag example were stripped. Here’s the example again using square brackets:
[mtag class="episode"]37d0aee3-90b508e8-30dd3b6d-b8858cfe[/mtag]
Posted by Dave Winer on December 3, 2009 at 5:51 pm
I just posted some ideas for a “torrent” namespace.
http://unberkeley.com/2009/12/03/ideas-for-a-bittorrent-namespace/
Dave
Posted by Torrent Sites Get Feedback from RSS Inventor | We R Pirates on December 6, 2009 at 3:00 pm
[...] move forward, Winer started by reviewing the RSS output produced by a few torrent sites and wrote up some comments and suggestions on how the various implementations can be [...]
Posted by Torrent Sites Get Feedback from RSS Inventor - P2P Talk? on December 6, 2009 at 4:20 pm
[...] move forward, Winer started by reviewing the RSS output produced by a few torrent sites and wrote up some comments and suggestions on how the various implementations can be [...]
Posted by Sebastien BORGET on December 7, 2009 at 5:32 am
Dear Dave,
I wanted to notify you of my previous work on the same topic. I indeed had published in July 2006 a white paper
http://www.borget.net/technical-articles/bittorrent-rss-20-feed-specifications-white-paper
http://www.borget.info/download.php?file=bittorrent%20rss%202.0%20feed%20specifications%20revision%201.pdf&action=go
I would love to chat with you further, feel free to contact me at my email address
best
Sebastien
Posted by Torrent Sites Get Feedback from RSS Inventor – FUCK THE RIAA on December 7, 2009 at 6:18 am
[...] move forward, Winer started by reviewing the RSS output produced by a few torrent sites and wrote up some comments and suggestions on how the various implementations can be [...]
Posted by EZTV’s RSS gets guid, comments and category elements « UNBERKELEY on December 7, 2009 at 8:21 am
[...] reports in a comment on Unberkeley that they have added support for guids and links to comments to their feeds. They [...]
Posted by Ideas for today’s show « Rebooting The News on December 7, 2009 at 8:35 am
[...] BitTorrent and RSS. [...]
Posted by RSS for BitTorrent, and other developments « News, Software and All you need on December 7, 2009 at 10:08 am
[...] has a piece today talking about my efforts to sort out the differences in the RSS used by various BitTorrent [...]
Posted by RSS of BitTorrent, Programmable Twitter client | Twitter News - Twimmer.com on December 7, 2009 at 1:37 pm
[...] couple of days I’ve written three pieces I think Scripting News readers would want to see:1. Reviewing the RSS generated by BitTorrent sites. The goal to help improve the quality and uniformity of their metadata, perhaps to allow more [...]
Posted by Michael Nutt on December 7, 2009 at 5:46 pm
Last summer I was working on a project to p2p-accelerate podcast feeds; we ran into similar issues:
http://wiki.limewire.org/index.php?title=P2P_in_RSS
Posted by Miles Redford on December 8, 2009 at 2:17 am
About 2 years ago I got tired of spending time looking for torrents. Even though Mininova was great, they didn’t have everything.
Even though there is RSS capabilities in some torrent programs like uTorrent, due to some of the inconsistencies and other fun things the programs have crashed my pc’s several times….
So I wrote a little program for getting the feeds (in Vb).
I thought mininova was the best RSS feed and was easy to code for, Isohunt as well.
I have RSS feeds from sites that need you to DL a rar or zip file(RSS 1.0, 2.0 Atom), and sites with feeds like torrentzap.
I made filter routines for the Feeds I do get like that which simply(not heh) convert their feed to RSS 2.0.
And some sites have made huge changes to their feeds over time causing more entertainment..so I am constantly making adjustments to the filtering, timing…no wonder uTorrent used to crash on me.
I used to take the [??S/??L] info off the filename at mininova so I wasn’t duplicating the torrents in my little database, then they removed it and placed it in heh…
I could go on and on as even though I no longer spend a lot of time searching for torrents, I now spend that time and more adjusting and adding to my program.(not a complaint as I love programming)
And since mininova left the scene I had to create a number of new filters for the new feeds I have now.
Anyway, thanks for the RSS.
Posted by Torrent Sites Get Feedback from RSS Inventor - News from the technology world - Technology News on December 8, 2009 at 4:30 pm
[...] move forward, Winer started by reviewing the RSS output produced by a few torrent sites and wrote up some comments and suggestions on how the various implementations can be [...]
Posted by Torrent Sites Get Feedback from RSS Inventor « My blog at Servage :) on December 12, 2009 at 3:13 am
[...] move forward, Winer started by reviewing the RSS output produced by a few torrent sites and wrote up some comments and suggestions on how the various implementations can be [...]
Posted by Dave Winer on December 3, 2009 at 2:27 pm
Ask the developers to come by here and participate in the discussion?
Posted by NovaKing on December 3, 2009 at 2:30 pm
I’ll give it my best shot, not promises though