File upload using cURL

[danh]

Hi Mark,

I am trying to get a file to upload to my target using cURL instead of a web browser. It seems to almost work; that is, the entire file is transferred, but the target does not seem to recognize this. I am using this command line:

curl -F datafile=@1000.dh 10.10.5.132/HS.bin

I've attached a 'good' Wireshark capture (successful upload from web browser), and a 'bad' Wireshark capture (unsuccessful upload using cURL).

Can you see what's wrong with this approach? (I do see the extra 'Expect: 100-continue' in the bad capture, but I don't know how this might affect things.)

Thanks,

Dan

[mark]

Hi Dan

I have had a look into this and found two points.

1) The Expect: 100-continue seems to be a request to the server to send the response 100 back and then the data will be sent. I looked around and there seems to be a flag System.Net.ServicePointManager.Expect100Continue property which controls this behavior, but I wouldn't know where to find it.
In your case the client is waiting 2s and then sending the data anyway so this is probably not a huge issue.
The uTasker HTTP server doesn't look for this and so it is ignored. I would be possible to check for it during a post and respond accordingly but I don't know whether this is worth worrying about or not. Standard web clients don't use it.

2) The reason why the post is not working is due to the header construction. The uTasker HTTP server expects the content length to follow the boundary in the header. I believe that Chrome also does something similar when posting parameters, but not when posting data.
The result is that the data is all accepted but is 'dumped' since the server never actually synchronized to it (it needs to find the start of data content and know its length to know when to stop. So TCP looks OK but the data has landed in the trash bin...

This is what I did:
- I used your bad trace and played it through the simulator and has a few adjustments which allow the header ordering to be accepted
- then I checked the content of the file, which is not saved to the file "H.bin" in FLASH

I am not absolutely sure whether it works since I don't know the exact content of your test file - so you can maybe confirm the following:
a) The content length seems to be 69'998 bytes (but it may be off by +1..+4) - can you give me the exact length?
b) The end of the file contents looks like this "88888 99999 \r\n0001000 0000011111 22222 33333 44444 55555 66666 77777 88888 99999 ". The last byte in 0x20 (a space). If this is correct (would also confirm the length) it looks to be handling the length correctly. If you happen to have "\r\n\r\n" at the end (which I believe is a delimiter sent after the data content, but not sure) it means that teh result may still be a little bit off.

It is a bit of a nuisance that the different clients construct the headers differently because it means that the server has to do more work to ensure getting all possible combinations correct (which is what an embedded system wants to be able to do with as little overhead as possible) but we can certainly get it fixed for the cURL case too (using a define to avoid extra overhead when this is not a requirement).

Please check your file content and, if it looks good, I'll send you a new HTTP.c with the option.
Do you think that point 1 is serious or not???

Cheers

Mark

[mark]

Dan

For people wanting to use cURL, do you recommend a download source? Any tips about installing and using?
I assume that you would like to use it to automate posting of data - in what type of application?

Regards

Mark

[danh]

Hi Mark,

1) There must be some cURL option to turn off that Expect100Continue thing, I'll look into that. But as you said, it goes ahead with the transfer after a couple seconds anyhow, so not a big deal.

2) The data file is exactly 70,000 bytes, and ends with \r\n. I've attached the original file for your reference. This data is just some test data so that the data content and boundaries are easily recognizable.

I started looking into cURL for testing, but if I can put my update file on the target using cURL, without having any web pages yet stored on the target, I can eliminate the need for the FTP server and save some resources. (My update file contains both the application code and the file system content.)

Here's where cURL lives:

http://curl.haxx.se/

You can build from source, or download binaries for just about any platform.

Thanks,

Dan

[mark]

Hi Dan

I am 2 bytes off at the moment so need to check where this is happening.

I like the idea of using cURL to automate (command line web transfers).

Will get back when the solution is working correctly.

regards

Mark

[mark]

Hi Dan

I have worked a bit more on this and have a temporary solution which I will send you by mail.

1) The didn't study the reason for the 2-off error but instead put a fudge-factor in. I think that it has something to do with the fact that there are two boundaries received before the data and a boundary compensation is no longer necessary (?) In any case it should sort out your case for the time being.

2) I added a 100 continue response. This should remove the delay which you are seeing so it is worth giving a try.

The problem that I have seem is that the various cases which are handled can probably be done better by using a general logic, rather than following the cases which have been observed up to now. I would be good if the present solution (hopefully correct for all known cases) could be reworked to improve this logic and also its handling efficiency (not that this is really a practical issue, but if it can be improved it would be good).

As you probably known, the headers need to be parsed and basically the server is looking for various standard strings and following parameters which it either interprets of ignores. My idea is to use a string index with flags for each string to allow the parsing to take place in a more central manor without the use of string compares. This would also allow them to operate over TCP frame boundaries (although this has never been observed to be necessary). It would be interesting to cut down the complexity which is growing, increase parsing efficiency and flexibility all at the same time.

Hopefully I will get a chance to do a rework and test all the presently known cases for compatibility. If, in teh meantime, you find any issues with the temporary solution just mail me...

regards

Mark

[danh]

Hi Mark,

I am not having any luck with this yet. I've posted you some private email about it.

Thanks,

Dan

[mark]

Hi Dan

That is a shame. I will get back again once I have done some more complete testing with cURL.

Regards

Mark

[mark]

Hi Dan

Just a quick update.
A rework of the HTTP POST has been performed and has resulted in a more flexible solution, including capability to post parameters over multiple frames.
It was possible to increase code size efficiency and also speed by using a new technique, also resulting in more maintainable code.
File posts with IE7, Firefox, Chrome and cURL seem OK, as do parameter posts from IE7, Firefox and Chrome. More testing with cURL will be performed to see whether its other features are compatible.
I sent you an early version which may allow you to work with it.

Regards

Mark

[danh]

Hi Mark,

I tried the new stuff with cURL and it seems to work fine. I'll try to test it more extensively in the next few days, and I will also try the capability to post parameters over multiple frames (since a need recently came up for that).

Thanks for your efforts!

Dan