Messages

1

µTasker general / NE64 service pack progress?

« on: October 19, 2007, 09:16:23 AM »

Hi Mark,

do you have an estimate when the SP for the NE64 will be ready?

Best regards,


Oliver

2

µTasker general / Re: Filesystem on NE64 using dataflash

« on: September 26, 2007, 11:14:28 AM »

I suggest also looking at some other projects:
- Fabio has ported a file system for use on SD cards. I wonder how this would fit? See: http://www.utasker.com/forum/index.php?topic=28.0
- http://elm-chan.org/fsw/ff/00index_e.html is a project with FatFs / Tiny-FatFs which could possibly be used as a base.

I have had a look at these projects, and while they're very interesting, the licensing issue with FATs remains, because both seem to be FAT compatible. I like the FatF approach of multiple files, subdirectories, etc, but even at 1K RAM usage is a little high for an embedded project, and I like uTasker's minimalist approach as well. My approach would be a hybrid of these two: longer filenames and directories look like they're supported, while the physical structure isn't. RAM overhead is little more than 1 sector for the file system proposal above. The key is limiting the sector allocation table to one sector I think, otherwise houskeeping of free sectors gets messy.

One thing I think would be worth considering though is supporting FatF's low level disk-IO interface, which is independent from FatF and suitable for other filesystems too, including uTasker's own.

Best regards,

Oliver

3

µTasker general / Re: Filesystem on NE64 using dataflash

« on: September 26, 2007, 10:57:28 AM »

Since the NE64 generally uses either a small linear memory range up to 64k or banked memory, it is possible that a linear memory approach beyond 64k causes complications.

To understand the idea when accessing the the 'virtual memory mapped FLASH memory', it works like this on the NE64 (assuming it can work).
An access from 0x8000 to 0xBfff is directed to the internal file system (in a memory bank).
Any access from 0xC000 is directed to the external SPI FLASH.
If there were 2 16MBit SPI FLASH chips available, they would have the ranges
0xC000..0x20Bfff and 0x20C000..0x40Bfff respectively.
fnEraseFlashSector((unsigned char *)0x8000, 0x404000); would effectively cause all 3 physical FLASH ranges to be deleted.
fnEraseFlashSector((unsigned char *)0x20C000, 0); will cause one page at the beginning of the first SPI FLASH chip to be deleted.

There seems to be no problem using fnEraseFlashSector((unsigned char *)0x84000, 0); on the NE64 - (no compiler warnings), but fnEraseFlashSector((unsigned char *)(2048*264), 0); does. Perhaps it is the GCC preprocessor only(??)
I will have to look in more detail about why address range checking throws up the warning described above before getting the drivers working with the device.

Regards

Mark

Hi Mark,


I only have experience with CodeWarrior for the HC12, not GCC, but the compiler should be able to address up to 8M of memory with far pointers. But with all the paging going on it's difficult to understand which areas are fixed and which can contain some other kind of memory. In any case using banked data memory causes lots of overhead.

Maybe the compiler is just getting confused because the parameter is a pointer, what if it's just an unsigned long? When erasing memory you have to calculate the sector number from the address anyway, and then break the sector number down to a command sequence, so there should be no need to pass a real pointer. Passing a sector number and sector count would be more intuitive anyway in my opinion; if a function is called EraseFlashSector(n), I'd expect it to erase physical or logical sector 0 of the device. If two 16 MBit SPI flashes were connected, I'd expect EraseFlashSector to hide this detail and make things look like a single 32 MBit SPI flash was connected.

Best regards,

Oliver

4

µTasker general / Re: Filesystem on NE64 using dataflash

« on: September 22, 2007, 05:37:56 PM »

Hi Mark,

thanks for the reply. Yes, your thoughts are largely what I have been considering too. For removable devices I think compatibility would be a major plus, but copyright is an issue as you point out. FAT source code is available e.g. from SanDisk, but only if you design a NDA and distribute your product with SanDisk devices or something along these lines.

Whatever the solution is, whatever file system one wants to use now or in the future, I think the common thing necessary is a well defined file system interface with a set of required features and a set of optional features. This is what I have been working on so far, and this is the reason I proposed a "handle" or some similar concept even for the existing uFileSystem. It would just make switching to something else (whatever it's going to be) simpler.

So this would be the first step, and it would keep the simple uFileSystem along with all existing code in place. If the uFileSystem somehow would access memory through a set of ReadSector (or something similar) routines, which could even be no-ops for simple memory mapped file systems like uFileSystem, again it would be simple to save precious CPU memory and just switch to external memory like the SPI or something else. This is pretty much already in place with the SPI for uFileSystem.

With a defined simple interface to the stack and application, and a similarly defined simple interface to access storage, the file system can be exchanged easily for something else.

Most of this is already in place, and I spent a 3 hour train ride browsing through your work. BTW, I do find the uFileSystem impressive especially due to its simplicity, yet being quite powerful. I think with a few defines here and there and maybe renaming one or two functions, the goal of paving the road for optional file systems can be achieved.

I'll check out the links you have provide, maybe others have comments as well, which I'd love to hear.

Best regards, and have a nice weekend!

Oliver

5

µTasker general / Filesystem on NE64 using dataflash

« on: September 21, 2007, 02:43:14 PM »

Hi Mark,

first of all, congratulations on a most remarkable project! I have only lately had time to start using uTasker, and I am very impressed!

I'm not sure if I should post this here or on the N64 section, but I think the topic might be of general interest, so I posted it here.

I have added an Atmel Dataflash '161 to the NE64 on my project's board. The uTasker runs great, but I want to make use of the DB161 instead of the internal flash obviously. Will the SP5 improvements also available on the NE64 in the future? Currently the latest NE64 service pack is SP1. Can I incorporate the changes from SP5 into the NE64, or should I wait?

I want to add a (slightly) more flexible filesystem using the external flash while still maintaining compatibility to the existing code, so I have a request/proposal: most uFile functions use the address when referencing an open file. It would be quite easy to add something a typedef const char* to something like HANDLE or FILE. The existing code would remain identical in most places, namely where the pointer is being used as a kind of handle when passing it to uFile in subsequent calls. Only when the pointer is accessed directly, the "handle" could be "processed" by a function or even macro to use it as a pointer when necessary.

The effect would be that the uFilesystem could be replaced with something more powerful without either flooding the remaining code with #ifdefs or breaking compatibility with the uFilesystem. I'm currently flooding the code with #ifdefs wherever the filesystem is accessed, but if the "handle" proposal makes it into the code trunk, I could remove most of them.

My suggestions for the "new" filesystem (if anyone cares to discuss them):

The new filesystem only makes sense if an external memory of sufficient size is present, so the basic requirement is a "logical" sector size of at least 512 bytes. On the larger dataflashes this is the physical sector size of 512/528, on the smaller devices a logical sector consists of two physical sectors. Even though the memory is quite large, I will limit file size to 64K, because else the code gets bloated and things get complicated. This is still an embedded device after all.
If necessary, things can get expanded later.

The filesystem should support dynamically adding and deleting files, so I reserve one sector as a 
cluster map, each bit representing one cluster. The cluster size is determined by the number of logical sectors available on the device divided by the number of bits in the sector allocation map. The '161 for instance has 512 bytes per sector, so 4096 clusters are supported, so for this device each cluster consists of one logical sector. This way the filesystem can easily find a free sector when it needs to. The map only needs to be updated when a file is written or deleted.

Since sectors are now no longer subsequent, I plan to use the last two bytes of a sector as a link to the next sector in the file. The 528 byte sector size is ideal for this, because the link information can be stored in the extra 16 bytes of each sector, keeping a payload size of 512. A 
backward link to the previous sector can also be stored here. The downside is still that seeking through a file takes more time, but since accesses are still sequential, the number of the following sector is always read automatically.

I will use a directory, but the directory also fits into one sector. The one letter filenames are a burden if more than a few files are used and if a structured approach is chosen for the website. Therefore I will calculate a 16bit hash value of the filename. This way things look like directories are supported while they are actually only part of the filename. The downside is that the old filename can not be recovered, same as for uFile. Each directory entry consists of:

- 2 bytes Hash of filename
- 2 byte start of file on flash device (# of first cluster)
- 2 bytes filesize
- 2 bytes Mime type and reserve.

This way 64 files fit into the directory, and if necessary more than one directory sector could be used as well should the need arise. The point is that I can make better use of the 2MB memory in the dataflash, more flexibility with filenames, and yet no MSDOS like complex filesystem with FATs, bootsectors, directories, subdirectories, etc.

The index in the directory makes for a great file handle, because using it most information is easily accessible (except for the current read position).

Another idea is to reserve the first few sectors and store the full filename/path in them. For a 64 byte path, 8 pathnames fit into a sector, so 8 sectors suffice to keep all filenames. This is only necessary for listing, yet could be quite handy without excessive effort. The hash approach would still be kept to prevent having to read and compare all directory entries.

A single 512 byte sector buffer is probably necessary to prevent excessive code overhead, but the directory and sector allocation table don't need to be read in their entirety if e.g. a free sector is needed, the software only needs to look for the next free bit or unused directory entry.

This proposal builds on the uFilesystem without adding the complexity of things like FAT, yet allowing more flexibility and generally larger file systems.

I would gladly donate any extensions to you/the community, but I want to prevent reinventing the wheel, so I wanted to ask you first for your comments.

Thanks again and best regards,


Oliver