Hi Mark,
first of all, congratulations on a most remarkable project! I have only lately had time to start using uTasker, and I am very impressed!
I'm not sure if I should post this here or on the N64 section, but I think the topic might be of general interest, so I posted it here.
I have added an Atmel Dataflash '161 to the NE64 on my project's board. The uTasker runs great, but I want to make use of the DB161 instead of the internal flash obviously. Will the SP5 improvements also available on the NE64 in the future? Currently the latest NE64 service pack is SP1. Can I incorporate the changes from SP5 into the NE64, or should I wait?
I want to add a (slightly) more flexible filesystem using the external flash while still maintaining compatibility to the existing code, so I have a request/proposal: most uFile functions use the address when referencing an open file. It would be quite easy to add something a typedef const char* to something like HANDLE or FILE. The existing code would remain identical in most places, namely where the pointer is being used as a kind of handle when passing it to uFile in subsequent calls. Only when the pointer is accessed directly, the "handle" could be "processed" by a function or even macro to use it as a pointer when necessary.
The effect would be that the uFilesystem could be replaced with something more powerful without either flooding the remaining code with #ifdefs or breaking compatibility with the uFilesystem. I'm currently flooding the code with #ifdefs wherever the filesystem is accessed, but if the "handle" proposal makes it into the code trunk, I could remove most of them.
My suggestions for the "new" filesystem (if anyone cares to discuss them):
The new filesystem only makes sense if an external memory of sufficient size is present, so the basic requirement is a "logical" sector size of at least 512 bytes. On the larger dataflashes this is the physical sector size of 512/528, on the smaller devices a logical sector consists of two physical sectors. Even though the memory is quite large, I will limit file size to 64K, because else the code gets bloated and things get complicated. This is still an embedded device after all.
If necessary, things can get expanded later.
The filesystem should support dynamically adding and deleting files, so I reserve one sector as a
cluster map, each bit representing one cluster. The cluster size is determined by the number of logical sectors available on the device divided by the number of bits in the sector allocation map. The '161 for instance has 512 bytes per sector, so 4096 clusters are supported, so for this device each cluster consists of one logical sector. This way the filesystem can easily find a free sector when it needs to. The map only needs to be updated when a file is written or deleted.
Since sectors are now no longer subsequent, I plan to use the last two bytes of a sector as a link to the next sector in the file. The 528 byte sector size is ideal for this, because the link information can be stored in the extra 16 bytes of each sector, keeping a payload size of 512. A
backward link to the previous sector can also be stored here. The downside is still that seeking through a file takes more time, but since accesses are still sequential, the number of the following sector is always read automatically.
I will use a directory, but the directory also fits into one sector. The one letter filenames are a burden if more than a few files are used and if a structured approach is chosen for the website. Therefore I will calculate a 16bit hash value of the filename. This way things look like directories are supported while they are actually only part of the filename. The downside is that the old filename can not be recovered, same as for uFile. Each directory entry consists of:
- 2 bytes Hash of filename
- 2 byte start of file on flash device (# of first cluster)
- 2 bytes filesize
- 2 bytes Mime type and reserve.
This way 64 files fit into the directory, and if necessary more than one directory sector could be used as well should the need arise. The point is that I can make better use of the 2MB memory in the dataflash, more flexibility with filenames, and yet no MSDOS like complex filesystem with FATs, bootsectors, directories, subdirectories, etc.
The index in the directory makes for a great file handle, because using it most information is easily accessible (except for the current read position).
Another idea is to reserve the first few sectors and store the full filename/path in them. For a 64 byte path, 8 pathnames fit into a sector, so 8 sectors suffice to keep all filenames. This is only necessary for listing, yet could be quite handy without excessive effort. The hash approach would still be kept to prevent having to read and compare all directory entries.
A single 512 byte sector buffer is probably necessary to prevent excessive code overhead, but the directory and sector allocation table don't need to be read in their entirety if e.g. a free sector is needed, the software only needs to look for the next free bit or unused directory entry.
This proposal builds on the uFilesystem without adding the complexity of things like FAT, yet allowing more flexibility and generally larger file systems.
I would gladly donate any extensions to you/the community, but I want to prevent reinventing the wheel, so I wanted to ask you first for your comments.
Thanks again and best regards,
Oliver