Author Topic: Slow FTP download  (Read 15872 times)

Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Slow FTP download
« on: October 30, 2008, 01:11:45 PM »
Hi,

I have tested the FTP server in the uTasker and found that when i upload to the unit i get a transfer rate of 130 K byte/s.
But when i try to download the same file i only get a transfer rate of 5 -6 K byte/s.

The data are stored on a SD card, but even when ii try not reading data from the SD card and only send a "empty" buffer I get this low connection speed.

Have I any of you see this on yours system ???

Regards
Steen
« Last Edit: November 07, 2008, 03:36:09 PM by svl »

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3236
    • View Profile
    • uTasker
Re: Slow FTP download
« Reply #1 on: October 30, 2008, 05:27:04 PM »
Hi Steen

The reason for the slow FTP download is due to the operation of the TCP stack on the PC. It is using delayed ACKs (see the following for a short explanation and an example calculation of transfer speed: http://www.utasker.com/forum/index.php?topic=4.msg26#msg26).

The FTP server is designed primarily to upload web pages, but does also support (basic) downloads - not optimised for speed as you have found out.

Other protocols, like TELNET, support buffered TCP operation and can achieve fast rates - see http://www.utasker.com/forum/index.php?topic=25.msg101#msg101

The HTTP solution uses a special technique to achieve fast rates without requiring additional memory buffers and so will also give fast transfer speed.

If you can use HTTP (rather than FTP) to get the files you will be able to greatly improve the transfer speed.

It may be possible to adapt the FTP interface to (optionally) make use of the HTTP windowing technique - is the speed an issue or just an observation?

Regards

Mark

Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Re: Slow FTP download
« Reply #2 on: October 31, 2008, 07:15:47 PM »
Hi Mark

Thanks for your reply ;D

I have read the short explanation of the problem and think that it can be compared to the HTTP server case you mention.

The reason that I mention ask this problem was that I was expecting the same performance for up/down load and was interest in knowing if I was doing something wrong now that I was modifying the code to support SD card.

is the speed an issue or just an observation?

Well yes it is an issue, my basic idea with the FTP connection, was to use a standard way to transfer files between the system and a PC, and in many cases will the settings saved to the system will have to be read out from the system again later.
I don’t fully understand the window buffering, but do you have any ideas on how difficult it is to make the changes that increase the speed without using to much memory?

Regards
Steen         

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3236
    • View Profile
    • uTasker
Re: Slow FTP download
« Reply #3 on: October 31, 2008, 07:44:11 PM »
Hi Steen

I have received the same question about the differences in upload and download times several times and up to now I have been told that it is not a practical issue since FTP downloading is something which is rarely needed (of course upload speed is more important since this is its main use).

I haven't specifically looked into the extension needed for FTP but I would guess that it is not that complicated. HTTP is rather more difficult due to the fact that it supports multiple users (several transfers at the same time), has to insert dynamic content and regenerate any lost frames (that is the basic complexity - if no regeneration were even needed it would be really simple) plus the inserted content from as little information as possible (to avoid having to buffer everything in RAM). FTP is single-user and can send RAW data so this should be possible with very little saved information.

Maybe I can add this as an option - give me a day or so and I'll get back with more info.

Regards

Mark



Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Re: Slow FTP download
« Reply #4 on: October 31, 2008, 09:48:01 PM »
Hi Mark

Lokking forward to hear from you and hope that you have good news. ;)

Regards
Steen

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3236
    • View Profile
    • uTasker
Re: Slow FTP download
« Reply #5 on: November 01, 2008, 04:22:32 PM »
Hi Steen

I have a quick solution to easily improve transmission speed in a LAN. It will also improve performance in the Internet (where delays are longer) but not as dramatically. It is however extremely easy to implement.

Here are results with a 128k file in the simulator. The results on the target will not be as fast (partly due to memory access delays) - you can post your results if you like as reference. Note that the times are measured by FTP client and probably include times for link establishment, which will make the throughput also data file size dependent (due to fixed overhead). Larger files will be more efficient.

Original:
- FTP upload 128k at 1.44MB/s
- FTP download at 7.2kB/s

With option FTP_DATA_WINDOWS active
- FTP upload 128k at 1.44MB/s
- FTP download at 2.98 MB/s


The change is very simple so I will post it here:

1. In config.h add the option define in the FTP configuration block
            #define FTP_DATA_WINDOWS            2                        // allow transmission of FTP data with windowing support


2. In ftp.c add the following at the start of the file:
#ifdef FTP_DATA_WINDOWS
    extern TCP_CONTROL *present_tcp;                                     // global pointer to present tcp socket's control structure
#endif


3. In ftp.c, in the data listener case TCP_EVENT_ACK modify the following block:
(add the conditional code for FTP_DATA_WINDOWS)

#ifdef FTP_SUPPORTS_DOWNLOAD
        else if (DOING_UPLOAD == ucFTP_action) {
    #ifdef FTP_DATA_WINDOWS
            present_tcp->usOpenCnt = 0;                                  // no outstanding data to be acked
    #endif

            FileOffset += usLastSentSize;                                // block was successfully received so we can send subsequent ones
            if (!fnSendFTP(MSG_UPLOAD)) {                                // send next block
                fnQueueSendFTP(MSG_DIR_OK);                              // completely transmitted
                ucFTP_action = DATA_INACTIVE;
                fnTCP_close(FTP_TCP_Data_socket);
                return APP_REQUEST_CLOSE;
            }
        }
#endif


4. In ftp.c in the function fnSendFTP() modify the following:
(add the conditional code for FTP_DATA_WINDOWS)

    case MSG_UPLOAD:
        if ((usLastSentSize = (unsigned short)uGetFileData(ptrFile, FileOffset, FTP_Data_Tx.ucTCP_Message, FTP_DATA_BUFFER_LENGTH)) != 0) {
    #ifdef FTP_DATA_WINDOWS
            unsigned short usNextSent;
            signed short sSend = fnSendTCP(Socket, (unsigned char *)&FTP_Data_Tx.tTCP_Header, usLastSentSize, TCP_FLAG_PUSH);
            if (sSend <= 0) {                                            // if transmission error
                return sSend;                                            // return transmission status
            }
            present_tcp->usOpenCnt += usLastSentSize;
            usNextSent = (unsigned short)uGetFileData(ptrFile, (FileOffset + usLastSentSize), FTP_Data_Tx.ucTCP_Message, FTP_DATA_BUFFER_LENGTH);
            if (usNextSent == 0) {                                       // end of file reached
                return sSend;
            }
            usLastSentSize += usNextSent;                                // complete outstanding data to be acked
            return (fnSendTCP(Socket, (unsigned char *)&FTP_Data_Tx.tTCP_Header, usNextSent, TCP_FLAG_PUSH)); // send second buffer
    #else
            return (fnSendTCP(Socket, (unsigned char *)&FTP_Data_Tx.tTCP_Header, usLastSentSize, TCP_FLAG_PUSH));
    #endif

        }
        else {
            return 0;                                                    // end of file reached
        }



Note that the complete change list is a little longer to ensure that the TCP stack has also a few thing set correctly, but as long as you have also the following set it should work correctly (these are already active in the reference demo project):
HTTP_WINDOWING_BUFFERS and SUPPORT_PEER_WINDOW

What I haven't checked is the reaction to a client which is overwhelmed by the transfer speed. I can test this using an old Windows 98 laptop since it can't keep up with the speed and its TCP input buffer closes. Then the uTasker TCP stack should start probing the connection but I think that the FTP server may have to help out a bit (at least to get an optimum implementation). You will almost certainly not have such a problem on the hardware due to the fact that the file accesses will also be slowing things down, so I think that you will be able to work without any problems. I will try to test the probing case later and ensure that this is all correct before adding this officially to the uTasker project.

In the meantime I hope that it helps in your project.

Regards

Mark



« Last Edit: November 01, 2008, 04:28:12 PM by mark »

Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Re: Slow FTP download
« Reply #6 on: November 02, 2008, 07:12:30 PM »
Hi Mark

Thanks for the quick reply (and solution).
cant wait to try it out;-)

I will first have the opetunity to test your code at the end of this week.
Will post my result as soon as i get them..

Regards

Steen

Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Re: Slow FTP download
« Reply #7 on: November 07, 2008, 03:35:43 PM »
Hi Mark

What a lovely day. :D
Have just tried your code modification and what a differences.
Instead of 6Kbyte in download I now get 160K byte.
It was defiantly the kind of improvement I was going for, so thank you for the help. :D
Keep up the good work.

Best regards
Steen


 

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3236
    • View Profile
    • uTasker
Re: Slow FTP download
« Reply #8 on: November 07, 2008, 05:59:34 PM »
Hi Steen

Good news.

I still have the tests with a slow client to do and will report back with any changes so that this feature can be wrapped up.

Regards

Mark

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3236
    • View Profile
    • uTasker
Re: Slow FTP download
« Reply #9 on: November 08, 2008, 11:02:20 PM »
Hi Steen

I had some time so decided to test the FTP with an old Win98 laptop - based on the fact that I found it quite easy to overload it with fast TCP data (running a browser).

However I found that I couldn't get such an effect with FTP (DOS), presumably because it is a much more efficient client - even on an even older laptop that I found in the cellar. Therefore all that I could do was to add some code to ensure that no more data was sent than the client could really accept - this was missing in the previous version.

Based on this experience I would suggest that nothing more is actually needed in normal use.

Thinking about it, I don't actually expect that even a very slow embedded FTP client, with limited memory, should have any problems with the present implementation. So I think that we can leave it like this.

If anyone does ever find that a data transfer does fail due to the client's window closing (not continue) please send a recording to me so that I can play it through the uTasker simulator. The only reason that I could think of it failing - even if someone manages to find a client that can't handle the rate - is if the client's TCP stack doesn't send a correct window update, or if this gets lost (would hardly be repeatable..).

So here's the change (and final solution..... probably..):

Code: [Select]
    case MSG_UPLOAD:
    #ifdef FTP_DATA_WINDOWS
        {
            unsigned short usTxLength = present_tcp->usTxWindowSize;
            if (usTxLength > FTP_DATA_BUFFER_LENGTH) {                   // if the destination rx buffer can accept a full TCP frame send a full frame
                usTxLength = FTP_DATA_BUFFER_LENGTH;                     // full frame size
            }
            if ((usLastSentSize = (unsigned short)uGetFileData(ptrFile, FileOffset, FTP_Data_Tx.ucTCP_Message, usTxLength)) != 0) {
                unsigned short usNextSent;
                signed short sSend = fnSendTCP(Socket, (unsigned char *)&FTP_Data_Tx.tTCP_Header, usLastSentSize, TCP_FLAG_PUSH);
                if (sSend <= 0) {                                        // if transmission error
                    return sSend;                                        // return transmission status
                }
                present_tcp->usOpenCnt += usLastSentSize;                // the destination receivers input buffer size after receiving teh previous frame
                usTxLength = (present_tcp->usTxWindowSize - usLastSentSize);
                if (usTxLength > FTP_DATA_BUFFER_LENGTH) {               // if the destination rx buffer can still accept a full TCP frame send a full frame
                    usTxLength = FTP_DATA_BUFFER_LENGTH;                 // full frame size
                }
                usNextSent = (unsigned short)uGetFileData(ptrFile, (FileOffset + usLastSentSize), FTP_Data_Tx.ucTCP_Message, usTxLength );
                if (usNextSent == 0) {                                   // end of file reached
                    return sSend;
                }
                usLastSentSize += usNextSent;                            // complete outstanding data to be acked
                return (fnSendTCP(Socket, (unsigned char *)&FTP_Data_Tx.tTCP_Header, usNextSent, TCP_FLAG_PUSH)); // send second buffer
            }
            else {
                return 0;                                                // end of file reached
            }
        }
    #else
        if ((usLastSentSize = (unsigned short)uGetFileData(ptrFile, FileOffset, FTP_Data_Tx.ucTCP_Message, FTP_DATA_BUFFER_LENGTH)) != 0) {
            return (fnSendTCP(Socket, (unsigned char *)&FTP_Data_Tx.tTCP_Header, usLastSentSize, TCP_FLAG_PUSH));
        }
        else {
            return 0;                                                    // end of file reached
        }
    #endif
#endif


Regards

Mark

Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Re: Slow FTP download
« Reply #10 on: November 13, 2008, 12:21:28 PM »
Hi Mark

Great work.

Will test your final code and get back to you with the result.

Have you concideret to implement a SD card with FAT file system on?
As you properly can se form my other post this is the way I am going and it looks like a good/cheap (1 GByte SD card 1.78$) way to store data!
I know that it takes more code space but so fare i only see that I gain new function when i use HTTP server an so on ... :)


Regards

Steen Larsen
 

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3236
    • View Profile
    • uTasker
Re: Slow FTP download
« Reply #11 on: November 13, 2008, 12:53:48 PM »
Hi Steen

There are a few developments on the 'official' list. I don't want to promise anything but one of these is an interface which will allow mass storage via USB/SD, based on a PC-compatible file system. It is not the first on the list but quite high. There are no dates defined for releases - it depends on when we think that it is complete enough and well tested enough. There will be a 'secret' new package announced fairly shortly containing an interesting new module (particularly for industrial use) and after that mass storage may then become the main focus. (...but no promises...)

regards

Mark

Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Re: Slow FTP download
« Reply #12 on: November 13, 2008, 01:04:23 PM »
Hi Mark

No problem, just want to give my input to a great system.
Lokking forward to the "new" release, and will continuing my work with the FAT system I have found..

Regards

Steen