Filesystem on NE64 using dataflash

[fieroluke]

Hi Mark,

first of all, congratulations on a most remarkable project! I have only lately had time to start using uTasker, and I am very impressed!

I'm not sure if I should post this here or on the N64 section, but I think the topic might be of general interest, so I posted it here.

I have added an Atmel Dataflash '161 to the NE64 on my project's board. The uTasker runs great, but I want to make use of the DB161 instead of the internal flash obviously. Will the SP5 improvements also available on the NE64 in the future? Currently the latest NE64 service pack is SP1. Can I incorporate the changes from SP5 into the NE64, or should I wait?

I want to add a (slightly) more flexible filesystem using the external flash while still maintaining compatibility to the existing code, so I have a request/proposal: most uFile functions use the address when referencing an open file. It would be quite easy to add something a typedef const char* to something like HANDLE or FILE. The existing code would remain identical in most places, namely where the pointer is being used as a kind of handle when passing it to uFile in subsequent calls. Only when the pointer is accessed directly, the "handle" could be "processed" by a function or even macro to use it as a pointer when necessary.

The effect would be that the uFilesystem could be replaced with something more powerful without either flooding the remaining code with #ifdefs or breaking compatibility with the uFilesystem. I'm currently flooding the code with #ifdefs wherever the filesystem is accessed, but if the "handle" proposal makes it into the code trunk, I could remove most of them.

My suggestions for the "new" filesystem (if anyone cares to discuss them):

The new filesystem only makes sense if an external memory of sufficient size is present, so the basic requirement is a "logical" sector size of at least 512 bytes. On the larger dataflashes this is the physical sector size of 512/528, on the smaller devices a logical sector consists of two physical sectors. Even though the memory is quite large, I will limit file size to 64K, because else the code gets bloated and things get complicated. This is still an embedded device after all.
If necessary, things can get expanded later.

The filesystem should support dynamically adding and deleting files, so I reserve one sector as a 
cluster map, each bit representing one cluster. The cluster size is determined by the number of logical sectors available on the device divided by the number of bits in the sector allocation map. The '161 for instance has 512 bytes per sector, so 4096 clusters are supported, so for this device each cluster consists of one logical sector. This way the filesystem can easily find a free sector when it needs to. The map only needs to be updated when a file is written or deleted.

Since sectors are now no longer subsequent, I plan to use the last two bytes of a sector as a link to the next sector in the file. The 528 byte sector size is ideal for this, because the link information can be stored in the extra 16 bytes of each sector, keeping a payload size of 512. A 
backward link to the previous sector can also be stored here. The downside is still that seeking through a file takes more time, but since accesses are still sequential, the number of the following sector is always read automatically.

I will use a directory, but the directory also fits into one sector. The one letter filenames are a burden if more than a few files are used and if a structured approach is chosen for the website. Therefore I will calculate a 16bit hash value of the filename. This way things look like directories are supported while they are actually only part of the filename. The downside is that the old filename can not be recovered, same as for uFile. Each directory entry consists of:

- 2 bytes Hash of filename
- 2 byte start of file on flash device (# of first cluster)
- 2 bytes filesize
- 2 bytes Mime type and reserve.

This way 64 files fit into the directory, and if necessary more than one directory sector could be used as well should the need arise. The point is that I can make better use of the 2MB memory in the dataflash, more flexibility with filenames, and yet no MSDOS like complex filesystem with FATs, bootsectors, directories, subdirectories, etc.

The index in the directory makes for a great file handle, because using it most information is easily accessible (except for the current read position).

Another idea is to reserve the first few sectors and store the full filename/path in them. For a 64 byte path, 8 pathnames fit into a sector, so 8 sectors suffice to keep all filenames. This is only necessary for listing, yet could be quite handy without excessive effort. The hash approach would still be kept to prevent having to read and compare all directory entries.

A single 512 byte sector buffer is probably necessary to prevent excessive code overhead, but the directory and sector allocation table don't need to be read in their entirety if e.g. a free sector is needed, the software only needs to look for the next free bit or unused directory entry.

This proposal builds on the uFilesystem without adding the complexity of things like FAT, yet allowing more flexibility and generally larger file systems.

I would gladly donate any extensions to you/the community, but I want to prevent reinventing the wheel, so I wanted to ask you first for your comments.

Thanks again and best regards,


Oliver

[mark]

Hi Oliver

I have added an Atmel Dataflash '161 to the NE64 on my project's board. The uTasker runs great, but I want to make use of the DB161 instead of the internal flash obviously. Will the SP5 improvements also available on the NE64 in the future?
Currently the latest NE64 service pack is SP1. Can I incorporate the changes from SP5 into the NE64, or should I wait?

Yes, the support will be added to the NE64 project very shortly, although without the software upload possibility (for the moment at least). It will be in the form of an external memory mapped storage device without the uFileSystem, which will remain internal.


The uFileSystem was developed originally for small devices like the NE64 where the code size is rather critical. However it does offer quite a comfortable solution considering how slim-lined it is. Once you get used to its restrictions it is very workable when you know that you don't have space for more that 20 or 30 small files anyway.
The same applies also to slightly larger devices like the Coldfire with typically about 128k file space, where again a full blown file system would still be over-kill in the majority of projects performed using it.

With the addition of the SPI FLASH making several MByte of cheap external data storage very feasible for the smallest of processors, the uFileSystem is certainly not optimal. It is for this reason that the SPI FLASH support presently only enables its use as a memory mapped storage device rather than incorporating it in the file system. The internal FLASH uFileSystem works as before and any storages reads and writes outside of its valid range automatically access the external memory.

Your suggestions as to extending the present capabilities (but maintaining basic compatibility) are interesting. One unanswered question for myself is whether a second file system (possibly doesn't need to be compatible at all) should be offered - this instead of the uFileSystem or even in parallel to it? What extra features does it need? Should it suppport directories? Should it pretend to be FAT? Should it be a unique solution going a totally new way for embedded work? And, as you ask, should it re-invent the wheel or should it follow, or use, other examples?

I suggest also looking at some other projects:
- Fabio has ported a file system for use on SD cards. I wonder how this would fit? See: http://www.utasker.com/forum/index.php?topic=28.0
- http://elm-chan.org/fsw/ff/00index_e.html is a project with FatFs / Tiny-FatFs which could possibly be used as a base.

Presently I am contemplating the way to go here. It is clear that with large amounts of memory a more powerful option will be useful. Should there however be compatibility issues if the memory is removable? Should it be possible to write an SD card from a PC and read it in the embedded product (or reverse). Or is this not an issue for the projects done with the uTasker? Will there be copyright issues if FAT or FAT like operation is performed???
Should the uTasker have its own solution or should it integrate existing ones?

My present job is to get the basic drivers operating correctly for the present devices because the drivers will be needed by all implementations. In the meantime it would be interesting to hear from others as to which solution would be preferred.

Many thanks for your interesting contribution. I am pleased that you are enjoying your work with the uTasker!

Regards

Mark

[fieroluke]

Hi Mark,

thanks for the reply. Yes, your thoughts are largely what I have been considering too. For removable devices I think compatibility would be a major plus, but copyright is an issue as you point out. FAT source code is available e.g. from SanDisk, but only if you design a NDA and distribute your product with SanDisk devices or something along these lines.

Whatever the solution is, whatever file system one wants to use now or in the future, I think the common thing necessary is a well defined file system interface with a set of required features and a set of optional features. This is what I have been working on so far, and this is the reason I proposed a "handle" or some similar concept even for the existing uFileSystem. It would just make switching to something else (whatever it's going to be) simpler.

So this would be the first step, and it would keep the simple uFileSystem along with all existing code in place. If the uFileSystem somehow would access memory through a set of ReadSector (or something similar) routines, which could even be no-ops for simple memory mapped file systems like uFileSystem, again it would be simple to save precious CPU memory and just switch to external memory like the SPI or something else. This is pretty much already in place with the SPI for uFileSystem.

With a defined simple interface to the stack and application, and a similarly defined simple interface to access storage, the file system can be exchanged easily for something else.

Most of this is already in place, and I spent a 3 hour train ride browsing through your work. BTW, I do find the uFileSystem impressive especially due to its simplicity, yet being quite powerful. I think with a few defines here and there and maybe renaming one or two functions, the goal of paving the road for optional file systems can be achieved.

I'll check out the links you have provide, maybe others have comments as well, which I'd love to hear.

Best regards, and have a nice weekend!

Oliver

[mark]

Hi Oliver

A quick update on the porting work...

Today I completed the SPI FLASH driver port to the SAM7X using a 32M Bit device and I also decided to change the interface extern int fnEraseFlashSector(unsigned char *ptrSector, MAX_FILE_LENGTH Length) where the last parameter is new. The reason for this is that it is then possible to delete areas in memory without the higher level needing to know details about the medium. The lower level driver can then use internal FLASH deletes or external ones and also choose the algorithm best suited [eg. whether SPI FLASH page deletes are made or more efficient block deletes].

This was not as simple as I expected but on the SAM7X it allows assuming a memory map consisting of both internal FLASH and external SPI FLASH, and continuously write across FLASH types as well as page and chip boundaries. Also deletes of areas encompassing multiple FLASH types and chips is supported.
I think that this is important as a base for the subsequent higher level file system developments so I have hopefully got this sorted out.

I thought that I could quickly port to the NE64 - the code is almost identical to the SAM7X in this case - but there may be a complication due to the fact that the cross compiler is not expecting 32 bit pointers to be used. I found when I checked the routines managing the memory map there were various warnings like:
Hardware/NE64/NE64.c:1833: warning: integer overflow in expression

Since the NE64 generally uses either a small linear memory range up to 64k or banked memory, it is possible that a linear memory approach beyond 64k causes complications.

To understand the idea when accessing the the 'virtual memory mapped FLASH memory', it works like this on the NE64 (assuming it can work).
An access from 0x8000 to 0xBfff is directed to the internal file system (in a memory bank).
Any access from 0xC000 is directed to the external SPI FLASH.
If there were 2 16MBit SPI FLASH chips available, they would have the ranges
0xC000..0x20Bfff and 0x20C000..0x40Bfff respectively.
fnEraseFlashSector((unsigned char *)0x8000, 0x404000); would effectively cause all 3 physical FLASH ranges to be deleted.
fnEraseFlashSector((unsigned char *)0x20C000, 0); will cause one page at the beginning of the first SPI FLASH chip to be deleted.

There seems to be no problem using fnEraseFlashSector((unsigned char *)0x84000, 0); on the NE64 - (no compiler warnings), but fnEraseFlashSector((unsigned char *)(2048*264), 0); does. Perhaps it is the GCC preprocessor only(??)
I will have to look in more detail about why address range checking throws up the warning described above before getting the drivers working with the device.

Regards

Mark

[joelw]

I suggest also looking at some other projects:
- Fabio has ported a file system for use on SD cards. I wonder how this would fit? See: http://www.utasker.com/forum/index.php?topic=28.0
- http://elm-chan.org/fsw/ff/00index_e.html is a project with FatFs / Tiny-FatFs which could possibly be used as a base.

Presently I am contemplating the way to go here. It is clear that with large amounts of memory a more powerful option will be useful. Should there however be compatibility issues if the memory is removable? Should it be possible to write an SD card from a PC and read it in the embedded product (or reverse). Or is this not an issue for the projects done with the uTasker? Will there be copyright issues if FAT or FAT like operation is performed???
Should the uTasker have its own solution or should it integrate existing ones?

Having support for removable memory that one can share with another machine would be great for uTasker.  The only reason to consider FAT would be to address SD memory / machine compatibility.

I've ported both EFSL and FatFs/Tiny-FatFs to LPC2xxx with IAR 4x.  I prefer FatFs/Tiny-FatFs.

FAT is still somewhat in the air as to licensing.  If you contact Microsoft and talk to them they mention $10k and $250k for FAT/VFAT licensing.  Also the SD card organization "requires" membership to use their SSP specification, somewhere around $1250 per year.  Both fall under a gray area.  There are all sorts of new devices that fall under this gray area.


Cheers,
Joel

[fieroluke]

Since the NE64 generally uses either a small linear memory range up to 64k or banked memory, it is possible that a linear memory approach beyond 64k causes complications.

To understand the idea when accessing the the 'virtual memory mapped FLASH memory', it works like this on the NE64 (assuming it can work).
An access from 0x8000 to 0xBfff is directed to the internal file system (in a memory bank).
Any access from 0xC000 is directed to the external SPI FLASH.
If there were 2 16MBit SPI FLASH chips available, they would have the ranges
0xC000..0x20Bfff and 0x20C000..0x40Bfff respectively.
fnEraseFlashSector((unsigned char *)0x8000, 0x404000); would effectively cause all 3 physical FLASH ranges to be deleted.
fnEraseFlashSector((unsigned char *)0x20C000, 0); will cause one page at the beginning of the first SPI FLASH chip to be deleted.

There seems to be no problem using fnEraseFlashSector((unsigned char *)0x84000, 0); on the NE64 - (no compiler warnings), but fnEraseFlashSector((unsigned char *)(2048*264), 0); does. Perhaps it is the GCC preprocessor only(??)
I will have to look in more detail about why address range checking throws up the warning described above before getting the drivers working with the device.

Regards

Mark

Hi Mark,


I only have experience with CodeWarrior for the HC12, not GCC, but the compiler should be able to address up to 8M of memory with far pointers. But with all the paging going on it's difficult to understand which areas are fixed and which can contain some other kind of memory. In any case using banked data memory causes lots of overhead.

Maybe the compiler is just getting confused because the parameter is a pointer, what if it's just an unsigned long? When erasing memory you have to calculate the sector number from the address anyway, and then break the sector number down to a command sequence, so there should be no need to pass a real pointer. Passing a sector number and sector count would be more intuitive anyway in my opinion; if a function is called EraseFlashSector(n), I'd expect it to erase physical or logical sector 0 of the device. If two 16 MBit SPI flashes were connected, I'd expect EraseFlashSector to hide this detail and make things look like a single 32 MBit SPI flash was connected.

Best regards,

Oliver

[fieroluke]

I suggest also looking at some other projects:
- Fabio has ported a file system for use on SD cards. I wonder how this would fit? See: http://www.utasker.com/forum/index.php?topic=28.0
- http://elm-chan.org/fsw/ff/00index_e.html is a project with FatFs / Tiny-FatFs which could possibly be used as a base.

I have had a look at these projects, and while they're very interesting, the licensing issue with FATs remains, because both seem to be FAT compatible. I like the FatF approach of multiple files, subdirectories, etc, but even at 1K RAM usage is a little high for an embedded project, and I like uTasker's minimalist approach as well. My approach would be a hybrid of these two: longer filenames and directories look like they're supported, while the physical structure isn't. RAM overhead is little more than 1 sector for the file system proposal above. The key is limiting the sector allocation table to one sector I think, otherwise houskeeping of free sectors gets messy.

One thing I think would be worth considering though is supporting FatF's low level disk-IO interface, which is independent from FatF and suitable for other filesystems too, including uTasker's own.

Best regards,

Oliver

[mark]

Hi Oliver

My present prefence is to use

extern int fnEraseFlashSector(unsigned char *ptrSector, MAX_FILE_LENGTH Length);

This is in fact the same as

extern int uFileErase(unsigned char *ptrFile, MAX_FILE_LENGTH FileLength);

now...

This means that if the FLASH is mapped in memory it can use a pointer to the address.
The lower level routine can then work out whether it is in internal FLASH or external FLASH based on its address (internal FLASH addresses are real addresses in the memory map and external FLASH occupies also a virtual memory address range).
Internal memory can return the data by using a simple uMemcpy().
External FLASH accesses then cause the physical chip to be determined (sub-address ranges in the virtual address area). Basically this selects the CS (chip select) to be used.
The last step is to determine the page (or block, since block erases are rather faster than page erases).
This is achieved by
            int iChipSelect;
            unsigned short usPageNumber, usPageOffset, usLength;
            usPageNumber = fnGetSPI_FLASH_location(ptrAddress, &usPageOffset, &iChipSelect);

Therefore one call is used to work with any physical device (including multiple) devices, whether internal memory mapped or external via an interface (this could also be extended to include accesses via other interfaces such as USB or even Ethernet to collect data from a server...although this is probably outside of the present discussion scope!!).
The caller doesn't actually need to know where the data is - a single file could be situated in different physical memory locations (its start in internal FLASH and its end in external FLASH for instance).

I like the way that this works.

By using unsigned long rather than pointers does of course solve the compiler compaining but I think that a pointer is more logical - I expect that a couple of carefully located castings will get around the problem without having to get involved with banking issues.

Presently I am not totally clear about the FAT licensing implications so will have to do a bit of reading before making any more decisions.

I didn't get to testing the NE64 just yet - I was at a Luminary seminar all of today. I will put a report in the thread on the new Cortex M3 based device later.

Regards

Mark