µTasker Forum

µTasker Forum => µTasker general => Topic started by: sharning on July 09, 2007, 03:46:52 PM

Title: External EEPROM alternative to M95xxx
Post by: sharning on July 09, 2007, 03:46:52 PM
Hi All,
I have a suggestion as an alternative to Marks example M95xx SPI eeprom implementation for external expanded file system memory. Whilst not a true EEPROM as such the M25PExx series has a sub page granualrity of just 256 bytes , and can still update less then this. it is less than 1USD for the 8 M bit device and can work at 50MHz SPI clock. It has the same package and pinout, and on inspection most cammands are the same as the M95xx, with the exception of the write command and of course an extra byte needed to address the greater than 64K bytes data!. An 8M bit device is of course 1 M bytes, giving as an example 64 pages of 16K bytes. The file system would thus use upper and lower case letters as the first char and 16K would be ample for most web applications etc. As a beginner, I am just trying to debug this at the moment. Interested to hear what others have to say?
Dean Sharples.
Title: Re: External EEPROM alternative to M95xxx
Post by: mark on July 10, 2007, 08:52:45 PM
Hi Dean

I have plans to do the same in the very near future with the ATMEL AT45db041 (there is one on the ATMEL SAM7X EVAL board). I expect that they are more or less equivalent.

Here are my tips for your attempt.
1. Configure the project to use SPI File system rather than FLASH based file system (SPI_FILE_SYSTEM rather than FLASH_FILE_SYSTEM in config.h).
2. Set up a new EEPROM size in app_hw_xxxx.h. Eg.

Code: [Select]
#ifdef EEPROM_64K
    #define EEPROM_PAGE_SIZE 128                                         // respect larger page size in SPI EEPROM

    #define FILE_GRANULARITY (2*1024)                                    // File made up of 2k byte FLASH blocks
    #define SINGLE_FILE_SIZE (1*FILE_GRANULARITY)                        // each file a multiple of 2k
    #define FILE_SYSTEM_SIZE (32*SINGLE_FILE_SIZE)                       // 64k reserved for file system (including parameter blocks)

3. Note that the SPI EEPROM support is actually 'emulating' a FLASH device so it shouldn't actually matter whether the device is really an EEPROM or a FLASH type.

4. The simulator will probably not actually work correctly yet because the larger devices probably require different addressing (more bytes to address more address lines). This will mean getting into the driver routines:
- first you will have to change the actual SPI accesses if they are not compatible.
- then you will need to add the simulated FLASH in WinSim.c. You will see that the low level SPI routines call fnSimM95xxx() and so they will need to call fnSimM25PExxx() instead. In fnSimM25PExxx you need to look at what the SPI is sending and try to react as if it were the real device. This will enable both sides to be developed together and fnSimM25PExxx() also checks that the SPI driver is really doing that what you want to do with it. Depending on the commands, the simulated FLASH writes to a local memory of the same size as its FLASH or returns data from it.

5. Once this is operating correctly it is best to save the contents to disk on program termination and read them back on on start - see the routines fnPrimeFileSystem() and fnSaveFlashToFile() in FileToDisk.c.

The last job would then be to check that it really all works like that when you test on your target with the real device connected. Often it requires an itteration somewhere but sometimes it is possible to get it right first time - in any case it usually is much faster than trying to do the whole job directly on the target.

Good luck


P.S. I would also like the possibility of operating with both internal FLASH and external SPI based FLASH at the same time. This is probably a bit more complicated since the project is presently set to do EITHER / OR...
Title: Re: External EEPROM alternative to M95xxx
Post by: sharning on July 11, 2007, 11:58:22 AM
Thanks for the tips Mark, I have it up and running fine now with a 25Pe80 (8 M bits).

It was a good learning experience for me and I nneded to change a few more things.
As there are about a dozen changes it is probably not appropriate I post them here but I will PM you in case you can use the changes in future versions. Another suggestion is to use the program memory based error 404 message as uninitialised eeprom would result in a totally blank page.
For now I just createrd an error 404 file and as I use 64 file entries now the last file starts with a "{" (which cannot be hyperlinked anyway). Also I had to change the way you calculated the last file entry as it doesnt work out for eeprom. The same applied to the FILE_GRANUALITY and FLASH_GRANUALITY, as I chose 8K granuality so the 2 paramater blocks fit into the first file slot.
I test the FTP speeds and the HTTP speeds and noticed with a 28K jpeg file FTP is greater than 30K Bytes  per second, but the web page takes about 7 seconds to load in IE7. Not that I will be doing many 30K pages but I thought I would mention it.
It certainly is now quite impressive with 1 M bytes and 63 files for less than 1 US dollar!

Title: Re: External EEPROM alternative to M95xxx
Post by: mark on July 11, 2007, 12:58:44 PM
Hi Dean

Congratulations. I will study your code changes in detail since it will be useful to ensure that the solution is generic for small and large chips. This will certainly save me some time on my project too!!

Another suggestion is to use the program memory based error 404 message as uninitialised eeprom would result in a totally blank page.

This restriction is historic and also due to the fact that presently only FLASH or SPI FLASH can be used at the same time. When using SPI FLASH it is only possible to serve pages from there and not to set some to be served out of program flash. That is the reason why the last file block defaults to 404 error file position, but of course requires that the equivalent html file is really loaded there. This is what I will also have to solve in my next project.

but the web page takes about 7 seconds to load

This is a bit longer than I would expect (ensure that there is really no repetition somewhere) but serving large pages is not that fast. The reason is that the HTTP server doesn't send TCP frames using windowing. This is typical for small TCP/IP stacks and has two reasons: firstly it is very simple and reliable; secondly it doesn't need any ram buffers. Using traditional TCP windowing it would be necessary to have several k buffer space for each session - and if say 5 users could access the same web server at the same time this makes 10..15k of RAM, which the small footprint chips have problems to deliver.

First the calculation:
28k will be served in about 28000/1400 frames = 20 where 1400 is the approx. TCP frame payload (check that you have set this (HTTP_BUFFER_LENGTH in app_hw_xxxx.h) since it can be set smaller to save memory on smaller devices (like NE64).
The PC running teh browser will be working with "delayed acks", meaning that it will ack TCP frames after about 200ms delay, in the hope that the application will send data earlier on to which it can 'piggy back' the ACK - this saves sending short frames in some circumstances. Since the Web browser is not sending anything, it results in each ack being delayed by about 200ms and therefore each served frame also being delayed by 200ms.
20x200ms = 4s. I would expect about 4s rather than 7 but this explains basically why it is not that fast. (this is however content specific as well. The dynamic web parser does sometimes have to cut some frames in length if it is not possible to pack in the generated data in the space available).

In the other direction there is no delayed acks in operation and so uploads are (adequately) fast in a LAN.

Typical web pages of only 2k or so are not noticably slow but of course if you have more memory you can afford to have flashy web pages, where this then becomes apparent.

It just turns out... that I also have the same problem with my new project and so my plan is also to add a special windowing technique to the HTTP server where two frames may be sent at once. The TELNET server uses buffered TCP and this does full windowing and achieves fast bulk transfer but it has to have a buffer to work with (its size is defined by TCP_BUFFER in app_hw_xxxx.h). This buffer is absolutely necessary since data is of a random fashion - for example debug outputs from code or data passed from another input like UART. If there were never frames lost it would not be necessary but in order to be able to repeat TCP frames it must have a backup of the data previously sent. In fact most TCP implementations will require a buffer size of twice the data length due to the fact that TELNET can perform byte stuffing if it is operating in ASCII mode which in the worst case doubles the length of the transmitted payload which needs to be backed up. This is not the case in the uTasker since its solution never saves byte stuffed data (which is a bit more complicated) but reduces the maximum buffer space requirement by factor 2.

But back to HTTP... The fact is that once the transmission side actually sends two TCP frames without waiting for the ack of the first, this is already adequate to reset the delayed ack algorithm used at the PC with the browser running on it. Allowing just one outstanding ack is not very difficult and, due to the fact that HTTP served data is essentially static (not of random nature) it is possible to regenerate lost frames without having a buffer, but rather with maybe one or two additional pointers saved in each HTTP session object. My hope is that it will then achieve high speed serving at the cost of only a few bytes RAM and probably a couple of hundred bytes extra program space. Plus the added development complexity until it all works correctly...

I will tell you as soon as there is something ready so that ou can verify its performance.



Title: Re: External EEPROM alternative to M95xxx
Post by: sharning on July 11, 2007, 01:48:08 PM
Thanks Mark for you detailed explanation, and the suggestion you are considering looks most promising.
Considering the low cost of this try of pseudo EEPROM, it certainly frees up substantial space for code rather than data, and the simple file system you have becomes easy to deal with once the data sizes increase. Indeed , most users will never need to consider skipping first letters with 16K files. I also did some experimants regarding compiling with registar paramater passing, and traced it to as far as it gets into an infinite look on init
extern MULTISTART_TABLE *uTaskerSchedule( void )
-then it crashes with an unimplemented opcode exception.

I think that the combination of A6 frames and register paramaters may be the problem but I havent looked further yet.

As a suggestion in case others encounter the dreaded while(1) unimplemented vector point, it may be useful to add a litte code to parse the exception stack frame into a few registers as the frame structure is mess to work out the vector number after tracing down the stack pointer etc.

I am loking forward to doing the "easy" debugging on application code (not drivers) using your simulator, as this I am sure will be a great timesaver, and the poor quality of the metrowerks debugger compared to their HS08 and HS12 versions does little to help in the hardware "nitty gritty" !

Title: Re: External EEPROM alternative to M95xxx
Post by: mark on July 15, 2007, 04:34:18 PM
Hi Dean

I am presently going for the ATMEL 45DB041 (512kByte). I am based in Europe and a search has shown that I can get these fairly easily here (I have just ordered 4 in SO-8 and have one on an ATMEL EVB in a leadless package). I didn't find a local supply of the M25PExx types to order a couple for development work and ST doesn't seem to sample them - although they could be ordered through bigger distributors.

I don't have any results just yet but I have been experimenting with windowing on HTTP transmissions. I will still need to test various error cases to ensure that they are all really handled correctly but the increased speed is certainly very useful. My (smaller) web pages are built up (in LAN tests) very quickly and the impression is already much sleeker. The amount of additional RAM is mimimum (I think I have about 3 bytes more per HTTP session at the moment) and the code is not as complicated as I envisaged. As I wrote, it requires more work and tests to be sure but it will certainly be a worthwhile improvement.


Title: Re: External EEPROM alternative to M95xxx
Post by: thamanjd on July 26, 2007, 09:58:03 AM
Sorry this might be slightly tangential...

I also intend to use spi flash for some data storage and logging with NE64. Although my implementation is not required for use as flash files system for ftp or http etc. I'm not too worried about implementation, i 'm just having trouble finding the part i want. I'm trying to get some samples of m25pe16 but i cant seem to find small quantities at online stores etc. ST doesnt do online ordering of samples of this type yet. I cant get any samples out of local distributers either. I see the m25pe80 is available but i really need 16Mbit or greater and dont want to use two m25pe80's to get it.

If anyone knows an online component store i can purchase small (<30) quantities of m25pe16 please let me know.

John Dowdell
Title: Re: External EEPROM alternative to M95xxx
Post by: mark on August 13, 2007, 12:02:49 AM
Hi All

I have managed to add a boot loader to the demo project (presently for the M5223X Coldfire) which allows SW uploads to an external SPI FLASH.
It posts the file to the start of SPI FLASH and the boot loader reads this and programs it to program FLASH.
I believe that it is compatible with AT45DB011B, AT45DB021B, AT45DB041B, AT45DB041D, AT45DB081B, AT45DB081D, AT45DB161D but I have only actually tested with AT45DB041B.
An encryption option is also included so that the distributed code is not readable at any stage until it is extracted by the boot loader to its final destination in FLASH.

I have had to increase the boot loader space to 4k since the SPI driver and FLASH driver are needed at the same time and the 2k mark was easily passed…

What I haven’t done is used the SPI FLASH for anything other than the SW upload so the application will still have to sort the general use out itself. My solution was to define a file (HS.bin in the demo project) which is diverted to the SPI FLASH rather than the internal FLASH file system. This works fine for the SW upload but is far from a general solution for application data saving and retrieval along side an internal file system!!

Nevertheless it enables full 256k (minus Bootloader and any file system space) SW uploads and so opens up some interesting new possibilities. This shoudl be included in the next M5223X SP.