Tuesday, March 3, 2009

File Sharing using Stratus (Part 2: Send file example)

As described in previous post, I was experimenting with the new Stratus service introduced by Adobe with the relase 10 of Flash Player.

After going over the VideoPhoneLabs example by Jozsef
Vass, I was curious about trying to use the same approach for a different need, in this case a binary file transfer, having in mind P2P file sharing in Flash.

Here it is the resulting app:


Remember Flash 10 or newer is required. Test your current verision here.

Notes:
By a quick search, I came across this forum post where someone asked about the possibility of doing this and there was the confirmation by using the NetStream’s “send()” method to send any kind of content.

A guy called Kris in Duesseldorf did also introduce his work in that thread, and these posts about transferring Classes between peers were also quite interesting.
communicating arbitrary data via rtfmp (pt. 1)
communicating arbitrary data via rtfmp (pt. 2)



So hands on, let’s code a proof of concept that is capable of sending files through peers!

I. Use case:

A user will log in into the application using a nick name and he/she will get a List of other users connected to it. He/she will be able to choose one of the connected users, select one of the files in his/her local filesystem and send it to the target peer. He/she (the destination user), will be prompted to accept/deny the new incoming file.

This is just a proof of concept and is not intended to work as a production application, so there’s no work related to security, performance or other possible improvements.


II. User management service:

As described in the previous post, Stratus is a service used to manage the connection between 2 peers and allow them to communicate directly, but the way one discovers each other, how they meet, is out of scope of the service.

In Josez Vass example, a “reg.cgi” file with the structure of a python web app is provided, with the idea of people interested in running the example deploying it into his/hers own systems. It’s great he provided this script, b
ut it was a subject and a technology totally disconnected from were the example is about, Flex and Stratus.

I started looking how to deploy this script when I thought that it would be great if I could offer access to it to anyone trying my example, to avoid this non related subject/technology barrier. So it was a great excuse to st
art with Python, Django and Google’s AppEngine, and my first dummy Google AppEngine application came out!

If you run the example, you’ll be connecting to a Django application hosted in http://stratus-demo-service.appspot.com/ . It’s so simple and, probably, so bad coded, so I’ll keep the code, but feel free to ask for it if
someone wants to check.

It just implements the same API as Josez’s python script, so if you want to use it to try VideoPhoneLabs application, just go and change the WebServiceUrl constant in VideoPhoneLabs.mxml file to

private const WebServiceUrl:String = "http://stratus-demo-service.appspot.com/"


This app has three possible calls:

  • http://stratus-demo-service.appspot.com/?username=myUserName&identity=12345678901234567890 will register the requesting usename with the nearID passed as “identity” in to the system. This just stores a triple username-nearID-updatetime into a Django Model (in this case, a google.appengine.ext.db.Model object), that contains those three files. Successive calls with the same username will result in updating this updatetime field, used to timeout unactive users.
  • http://stratus-demo-service.appspot.com/?listUsers=true will retrieve a list of all currently conneceted users. Are considered connected users those that made a “PING” call to the system (a user register call) before than 2 minutes ago (so user connections expire after 2 minutes).
  • http://stratus-demo-service.appspot.com/?friends=username1&friends=username2 will retrieve the list of just the given usernames with their ids. This called is provided for compatibility with the VideoPhoneLabs example but it’s not used for the File Sharing one.



As said before, this is my first Python, Django and AppEngine app, so please don't be very strict if something is not working ;-)


III. File Sharing itself:
So, with the user management problem solved, I proceed to creat a project to illustrate the file sending functionallity. The structure is similar to the VideoPhoneLabs application, but without the camera/mic setup and managing the user connection to the user management system all in a single .mxml file. I just wanted to make the code easy to follow, so I tried to write it in the execution order, and avoid adding any auxiliar file.

Then in the file, you’ll see the following structure:

0. Stratus service connection code. Here, a NetConnection object that handles the connection to the Stratus service is set up.
1. Registering to the user management service. A call to the AppEngine service is done here to register the new nearID provided by the Stratus service in section 0. This code will be called regularly to maintain the user session alive. Once the user is registered, a NetStream “listenerStream” is created to recieve incoming file transfers.
2. Userlist request. The result is parsed into an Array used as binding source for a ComboBox of users.

This first sections are common for both, sending and recieving users.

3. Makes the handling for File selection from the browsed file system.
4. Waits for the “Send” button to be pressed. If it is the case and a destination user has been chosen, a NetStream “outgoingStream” is published. This is stream is the one the receiving user will register to to download the transfered file. So once it is published, the sending user asks the receiving if he/she would be so kind to ask it “please”. This sounds ironic, but is the only way of the transfer to start... if the destination user asks it, that is, if the destination user registers to a origin’s user NetStream (in this case, the “outgoingStream” we’ve just created).
5. The recieving user is pinged in his listenerStream then, and he/she answers subscribing to the sender’s “outgoingStream”, published with the name “fileRequest”.
6. The sender recieves some events, and we’ll wait until a NetStream.Play.Start to invoke the desired function:

switch (event.info.code){
case "NetStream.Play.Start":
log(INFO,"Sending "+sendFile.name+" ("
+sendFile.size+" bytes)");
outgoingStream.send("receiverFunction",
sendFile.data as ByteArray,userName.text,sendFile.name);
break;
}

Receiver “receiverFunction” is called, and the user is prompted to get the file. If he/she answers affirmatively, a new File with the transfered content is stored were the user decided.



So that’s much it! Try the app at the top of the post, entering a username and waiting for someone to connect... or being more realistic, opening a couple of browsers and simulating being two different users :-P

Here the code. Any kind of feedback is appreciated :-)

31 comments:

Anonymous said...

Awesome!!!

Thanks guy, you're god!!!

Mike Parsons said...

Hi there ... great sample any chance you want to share your appengine code?

Thanks
Mike

Marc Baiges Camprubi said...

sure Mike, mail me and I'll send it to you... but it's just an ugly Python function answering to http calls!

Unknown said...

Hello, its possible you to send me your app engine code?

Unknown said...

Thanks for the article. It's very nice.

My question is on data integrity. This approach relies on UDP protocol. How can I be sure that data is not broken?

Marc Baiges Camprubi said...

Hi Alexey,
yes, you're right, this tecnology is inteded to be used to transmit media info such as video or audio, we're losing packets won't affect the overall message or turn it unintelligible.

This is just an example or proof of concept. First idea I have is to use as3corelib MD5 classes or any other class that let's you get a Hash of a bunch of info, and send also this value between peers, to check if the received file makes the same Hash as expected. But what if it's not? Re-transmit all the info again? Depending on the file size we could break it into smaller packets then, to try to minimize this re-transmition overhead... I suppose it's a logic you have to think about, not provided by this technology.

Aaronius said...

I'm trying to view your source but I'm getting a 404. Would you be so kind as to fix the link? I'd really appreciate it. Thanks!

Also, I posted a message in the developer forums about data integrity and here's the response I received from an Adobe employee:

"You could implement this also with RTMFP and Stratus if you use the data messaging API (shared objects or NetConnection.call). Those guarantee data integrity, in order to be compatible with RTMFP (RTMFP is a connection oriented protocol on top of UDP, same as TCP but with the ability to drop packets that are "droppable", like video frames and such)"

I think he meant NetStream.send() would also ensure data integrity but I'm still waiting to hear back on clarification.

Aaronius said...

Also, I've done a few tests transmitting files with your app and checksums have matched up so far so it seems like data is staying in-tact. The only thing is I'm unable to try it out with a large file because I'm getting an end of file error. The file probably just needs to be cut up as discussed here:

http://forums.adobe.com/thread/20279

Looks like you've already been to that thread though and just haven't implemented the file splitting yet. I'd love to see it work with large files though. Very cool.

Nick Bilyk said...

Nice post. I'm writing an AIR App that uses P2P to send large files if anybody is interested. As far as I can tell there is no way to send files over 100M with flex without AIR.

-N

Marc Baiges Camprubi said...

thx for your note Nick!

Anonymous said...

Who knows where to download XRumer 5.0 Palladium?
Help, please. All recommend this program to effectively advertise on the Internet, this is the best program!

Nick Bilyk said...

The utility I mentioned earlier now live. http://www.belugafile.com - it's an open source AIR utility that allows you to send large files P2P using Stratus. It's using AIR so the files can be loaded in chunks with a FileStream.

Marc Baiges Camprubi said...

just watched the video and code, looks pretty clean! congrats!

Anonymous said...

Can you send me the appengine code?

Marc Baiges Camprubi said...

sure, leave me a contact address and I'll try to find it :-)

Rage said...

dkrasikov@gmail.com Thanks!

John said...

Hi Marc:

Thanks for your great work. Could you please send me the app engine code to kewlfungi@yahoo.com

Appreciate it.

Thanks.

John

Fabien Saulier said...

Hi Marc! I have the same request like the other mate, could you send me your appengine code, please ?

Thanks for this and for your post,

Fabien

Fabien Saulier said...

aaahh my bad, my mail is fsaulier@gmail.com :}

Anonymous said...

Hello, its possible you to send me your app engine code? Thanks.
sarahroberts2010@gmail.com

Unknown said...
This comment has been removed by the author.
Unknown said...

great sample~! Could you send me ur app engine code ? ^^ Thanks.
rn5347@naver.com

Terry Hardin said...

I'd be very interested in reviewing your Google App Engine code for this solution (thardin137@gmail.com)

Anonymous said...

For simple p2p rtmfp, can I use Flash media interactive server or do I have to use the enterprise server?
I don't need group just 1-1 p2p.

Marc Baiges Camprubi said...

Sorry, I don't know about Flash Media Server software! Never used it...

Unknown said...

Thanks for the code. Could you send me your appengine code, please ?
jorge.giro@gmail.com

Kalyani said...

Great work!
But I just needed your app engine code. Could you send me that please?
guddi19@gmail.com

Thanks:)

Kalyani said...
This comment has been removed by the author.
Kalyani said...

Hello,
I was trying a p2p code myself and I see that in my office network I just can't make a connection using netConnection.connect("rtmfp://p2p.rtmfp.net/[my dev key]);
(It works at my home though :))

However, I can connect using netConnection.connect("rtmfp:");

But just connecting using 'rtmfp:' doesn't allow me to do any file transfers or video calling.

Any idea what exactly the problem can be? I don't see any exceptions or errors but the transfer just denies to happens

Corey Gwin said...

Can you please provide me the Google AppEngine code?

Anonymous said...

Can you please provide me the Google AppEngine code? my email: fsdevpoint3@gmail.com

MAXwidget