TCP Buffer

[neil]

Hi Mark,
  I have a tcp connection in which text is sent to my server application. I do the following:

void SendMessage(unsigned char *Ptr,int Length)
{
TCP_MESSAGE test_message;
uMemcpy((CHAR*)&test_message.ucTCP_Message,&Ptr[0], Length);
fnSendTCP(TCPsocket, (unsigned char *)&test_message.TCP_Header, Length, TCP_FLAG_PUSH);
}

which works fine when doing 1 SendMessage(..). But If there is another SendMessage() soon after, characters are lost. Is there a facility within utasker to have a buffer for sending data (so a few SendMessage() can simply fill the buffer)?

Regards
Neil

[mark]

Hi Neil

When using a simple TCP socket it is necessary to wait for an ACK to the previous frame before sending a second. If a second is sent before the ACK to the first has been received it can cause strange things (since sequence counters are not correct). i don't know exactly why characters are lost in your case but it may be a consequence of sending a second message too early and causing strange things to happen at the TCP level (?).

It would be possible to add a queue of multiple buffers and send the next only when the previous has been acked, but there is a also the buffered TCP method (as used by TELNET for debug output messages). See the following:
http://www.utasker.com/forum/index.php?topic=25.msg101#msg101
and also http://www.utasker.com/forum/index.php?topic=91.0

If you use a buffered TCP socket rather than a simple TCP socket it should help handling buffering for you (up to the buffer size). A "TELNET-like socket "
see http://www.utasker.com/forum/index.php?topic=25.msg106#msg106 is the most luxurious method since it should handle really everything for you.

Regards

Mark

[neil]

Hi Mark,
  Thanks, the buffering looks the best way.

I will be having a few TCP connections, and only one needs buffering . I assume there shouldnt be any problems doing this?

Neil

[mark]

Neil

Buffered TCP can be performed on a socket basis so can be mixed as required.

Regards

Mark

[neil]

Hi Mark,
   I cant seem to get anything sent using the buffer method.

My previous routine, had :
      uMemcpy((CHAR*)&test_message.ucTCP_Message,&Ptr[0], Length);
      fnSendTCP(TCPsocket, (unsigned char *)&test_message.TCP_Header, Length, TCP_FLAG_PUSH);

which works fine with 1 send. So I simply done the following with buffers , in its place:
in config.h:
            #define SUPPORT_PEER_WINDOW                                 
            #define WINDOWING_BUFFERS      4                             
            #define CONTROL_WINDOW_SIZE                                 

placed:
          #define USE_BUFFERED_TCP  outside USE_TELNET define, as not using TELNET.

commented out WAKE_BLOCKED_TCP_BUF, as will never reach 2800 bytes.
left TCP_BUFFER, and TCP_BUFFER_FRAME as default.

within callback placed:
case TCP_EVENT_PARTIAL_ACK: fnSendBufTCP(Socket, 0, usPortLen, TCP_BUF_NEXT);break;
case TCP_EVENT_ACK: fnSendBufTCP(Socket, 0, 0, TCP_BUF_NEXT);break;

and in place of previous fnSendTCP() and uMemcpy(), I placed:
      fnSendBufTCP(TCPsocket, (unsigned char *)&test_message, Length, (TCP_BUF_SEND | TCP_BUF_SEND_REPORT_COPY));      

but nothing gets sent to server.

Regards
Neil

[mark]

Hi Neil

I assume that the TCP connection is working correctly before you send the data.

Check that you have enough HEAP for the defined buffer. The following discusses this and indicates where to check in the code:
http://www.utasker.com/forum/index.php?topic=480.msg1905#msg1905

Regards

Mark

[neil]

Hi mark,
  If I replace the  fnSendBufTCP() with  uMemcpy() and   fnSendTCP(), then it works fine. I will have a look at the link to see the heap, but my application has approx set aside 5k of RAM for my code.

Neil

[neil]

Hi Mark,
  It looks like its a heap problem, the first uMalloc returns 0 in the fnSendBufTCP().

How is the RAM set aside in utasker ?
As the processors I am using has 32K, after the static is used for own global variables, and arrays, Is the rest not set aside for functions  that usethe memory, uMalloc for example?

just checked the fnHeapFree() functions etc . I got back
Free heap = 0x0810 from 0x4c00
Unused stack = 0x09db

Regards
Neil

[mark]

Hi Neil

See the following:

http://www.utasker.com/forum/index.php?topic=96.msg384#msg384
http://www.utasker.com/forum/index.php?topic=468.0

You can control the HEAP size with OUR_HEAP_SIZE (in config.h). Assuming you have enough free memory (to ensure that the CPU has adequate stack for its routines) you can increase this value to allow the TCP buffer to be set up.

Good luck

Regards

Mark

[neil]

Hi mark,
  My application uses approx 6k(after doing some reductions) ram , including in TxQueueSize/RxQueueSize for serial/II2 etc, and for global variables.

fnHeapFree()=2308
fnHeapAvailable()= 19456 bytes. leaves a lot from the available 32k for other use.

so this means 17148bytes of ram has been used, taking away my 6k approx, is 11148bytes used by utasker. Isnt this a lot for ustasker?

Regards
Neil

[mark]

Hi Neil

Don't forget to include Ethernet buffers, which uTasker also puts on HEAP:

Default:
#define NUMBER_OF_TX_BUFFERS_IN_ETHERNET_DEVICE         2                // allocate 2 buffers for transmission
#define NUMBER_OF_RX_BUFFERS_IN_ETHERNET_DEVICE         6                // allocate 6 (full) buffers for reception

which gives about 12k.

Otherwise uTasker probably uses only a couple of hundred bytes for task queues, timers etc.

Regards

Mark

[neil]

Hi Mark,
  12k is a lot for my application.  I assume this is buffers for all ethernet connections, I dont have ftp,TELNET  etc, only a few TCP and a UDP connection.

How can I work out how the 12k is split, so I can reduce this.

Regards
Neil

[mark]

Neil

These are physical buffers for Ethernet (shared by all sockets).

2 for transmission is necessary due to a slight problem in the chip. When 1 is used it tends to send the buffer twice since the controller sometimes recognizes the same buffer as waiting, even when it has just been send.

6 for reception is advisable so that the amount of reception overruns in kept to a minimum in busy networks (with lots of broadcast traffic). This has always given good results. This can be reduced (each buffer reduction saving 1.5k of SRAM) to 5, 4, or maybe 3 (NUMBER_OF_RX_BUFFERS_IN_ETHERNET_DEVICE in app_hw_m5223x.h). If the RAM is otherwise not needed (would otherwise not be used for important things) it is however best invested in Ethernet reception performance.

Regards

Mark

[neil]

Hi Mark,
 Thanks for that.

When you say physical buffers, do these buffers capture all traffic for the chip only in an interupt, then pass them to the relevant callback function?

There will be never that heavy traffic within the processor, so I think I can reduce it.

When traffic flows around the network, does the chip ethernet module filter out the traffic only for it, then pass it onto the buffers?


Regards
Neil

[mark]

Neil

The Ethernet controller works only with the MAC address.

It receives frames which are addressed to its MAC address and broadcast addresses (eg. ARP resolves and various other broadcast based things). In a busy network there tends to be a lot of broadcasts which have to be received, although they may not be relevant.

When programmed for promiscuous operation the Ethernet controller receive ALL frames, irrespective of the MAC address (usually only needed for monitoring functions but may also be needed for IPv6 - watered down in the form of multi-cast addressing).

Since the SW needs to look at all received Ethernet frames (to see whether relevant or to be discarded) this is a load for the CPU. If a busy network is passing say 5 fast broadcast frames, the CPU needs to look at them all. It probably can't do it as fast as the burst is arriving but the buffers ensure that all 5 are queued. If 6 uninteresting broadcast frames are queued and a valid frame for the address is received as 7th frame then the first received one must have been dealt with before the 7th is received (otherwise it is an overrun and gets lost). The more buffers you have, the more fast frame bursts can be queued without an overrun.

Note that the M5223x project has also an option called USE_MULTIPLE_BUFFERS.
This works slightly differently because it doesn't reserve say 6 full size frame (9k) buffers but instead a larger number of 256 byte buffers (36 x 256 = 9k in this example).
The difference is that 6 full size buffers can hold only 6 frames. The frames can be from 60bytes - often the case for fast broadcast frames - up to 1500 bytes in size, but one frame requires one buffer. After 6 fast small frames all buffers may get full.
36 x 256 byte buffers can hold up to 36 small (up to 256 byte) frames and so can better withstand a bombardment of small frames without an overrun. But, to receive long frames multiple buffers are required and the management of frames spread over multiple buffers (not necessarily contiguous in memory) is more difficult. To overcome the difficulty there is one 'extra' full size buffer (here some memory saving is lost again) which is used to piece the received frame fragment together into a linear buffer before being passed to the stack. This requires an extra copy which is also time consuming.

Although the multiple buffer technique can be useful in environments where lots of fast small frames are to be expected, the fact that it requires an extra 1k5 overhead (although the overall size can still be reduced) and the extra memory copy required means that the standard technique with 6 rx buffers tends to be preferred unless the SRAM restrictions call for a "maximum memory size reduction with highest tolerance to fast frame overrun" compromise.

Regards

Mark