Author Topic: utFAT - preliminary information  (Read 22488 times)

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3243
    • View Profile
    • uTasker
utFAT - preliminary information
« on: December 12, 2009, 12:11:20 AM »
Hi All

Many users have asked about it and its preparation is nearing completion, therefore it is time to present some first details.

The utFAT (uTasker FAT) is FAT32 compatible and will soon be available for use with SD-cards. It supports LFNs (Long File Names) - which can also be disabled. It supports standard and high capacity V2 SD-cards.

It doesn't support FAT12/16 and also not older MMCs. However the difference is in fact quite small and could be added if there is real demand. The idea behind this simply being that MMCs and FAT12/16 no longer correspond to the present state of technology and so it seems a nice chance to start with that which is relevant today rather than dragging around also remnants of bygone times.

It has a simple and practical user interface. This was not copied from standard interfaces but chosen based on practicality - the calls enclose functions which do things that users will hopefully want to do in real-world application in an encapsulated and logical manner. For example, there is no interface for mounting disks - this is fully automated by the underlying mass-storage task and so, if the SD-card is inserted, it can be used, if not the disk drive is not available... the first SD-Card appears as drive D:/
When no SD-card is available fall-back to the uFileSystem is possible.

It includes an interface including the detailed work to do typical user interface things. For example, to add a DOS type user interface (which is included in the debug task) the code required doesn't need to get involved with how the directory paths work since this is all encapsulated in a couple of easy to use functions. The demo allows this type of interface to operate on the UART, via TELNET and/or via USB. See an example at the bottom.

It is fully integrated into the FTP server, allowing browsing through directories, retrieving files etc. The access point for the FTP server within the directory structure can be set in case such accesses are to be restricted to certain directories on the SD-card.

It is fully integrated into the web server, allowing complete existing web server content to be copied from a PC to an SD card and immediately be served from an embedded web server. The location of the HTTP server's root directory and its default file can be configured.

When working with LFNs the Linux workaround technique can be optionally used to (presumably) avoid potential patent issues in commercial projects.

The uTasker simulator operates with a simulated SD card making testing, debugging and study of the utFAT - and FAT32 in general - very comfortable.

RAM use for single-user FTP and 6 parallel HTTP sessions with LFN support is about 700 bytes. Users interfaces (like DOS type command line) need  a complete directory path, which can be long with LFN and deep directories, but is user configurable to suit the application, therefore 300 byte or so is often adequate.

Lots more technical details be be disclosed as the package is finalised....


That was a first brief presentation, which is hoped to wake interest in some of the new opportunities that it adds to the project use. A first release is planned for the end of the year so I am looking forward to some interesting discussions and then some interesting projects ;-)

Regards

Mark


Below is an example of the DOS-like command line interface in action (it is in fact a copy of the complete uTasker web site on an SD-card. All standard web content is displayed by the processor's web server as if it were the real web site...):

  Disk interface
===================
up           go to main menu
dir          [path] show directory content
cd           [path] change dir. (.. for up)
prt          [path] print file (ASCII)
root         set root dir
del          [path] delete file or dir.
help         Display menu specific help
quit         Leave command mode
>dir
Directory D:\

---A 23.11.2009  22:31                   9 Test1.txt
---- 23.11.2009  22:31   <DIR>             DIR_1
---- 28.11.2009  14:13   <DIR>             uTaskerWeb
---- 29.11.2009  17:13   <DIR>             webPages
1 files with 9 bytes
3 directories, 1496460228 bytes free
D:\>cd uTaskerWeb
D:\uTaskerWeb>dir
Directory D:\uTaskerWeb\

---A 02.10.2009  15:13                6442 avr32.html
---A 03.06.2009  02:43               12026 avr32_1.gif
---A 03.06.2009  02:35               42176 avr32_2.gif
---A 12.05.2007  15:31               15545 bd-cowo-blumen.gif
---A 06.09.2006  22:48                2007 BLogo.jpg
---A 05.04.2007  20:08               13832 chip1.jpg
---A 05.04.2007  20:11                5325 chip2.jpg
---A 05.04.2007  20:12                3369 chip3.jpg
---A 05.04.2007  20:12                2275 chip4.jpg
---A 05.04.2007  20:12                1461 chip5.jpg
---A 12.08.2008  17:18              319367 ColdFire_Graphic_2.jpeg
---A 27.09.2006  23:36               20992 EBV.doc
---A 13.04.2009  20:04                 318 favicon.ico
---A 02.10.2009  15:13                5963 forum.html
---A 19.06.2007  20:03                 413 ie-navi.css
---A 02.10.2009  15:18               21221 index.html
---A 23.03.2008  00:29               17790 index_.html
---A 02.10.2009  15:13                5768 kirin3.html
---A 22.10.2009  19:38                9539 links.html
---A 02.10.2009  15:13                7788 modbus.html
---A 01.08.2009  01:51               59615 modbus1.jpg
---A 01.08.2009  02:03               65343 modbus2.jpg
---A 12.05.2007  15:56                1509 muttertag.html
---A 19.06.2007  19:51                 171 navi.css
---A 22.01.2009  16:30                3731 New.gif
---A 26.02.2007  12:30              326904 PIC_0049.JPG
---A 26.02.2007  12:31              342977 PIC_0050.JPG
---A 26.02.2007  12:32              379115 PIC_0052.JPG
---A 02.10.2009  15:13               15636 support.html
---A 28.11.2009  14:13               23844 SW_Demos.html
---A 06.09.2006  10:38                1719 Text1.txt
-HSA 20.01.2008  00:36               85504 Thumbs.db
---A 04.09.2006  21:38               55038 uTasker.bmp
---A 14.09.2006  16:37               69632 uTasker.doc
---A 04.09.2006  21:17               14865 uTasker.JPG
---A 14.09.2006  16:51               52990 uTasker.PDF
---A 04.09.2006  21:17              128390 uTasker2.bmp
---A 14.04.2009  18:21                1673 uTaskerAv.jpg
---A 14.04.2009  18:30                1669 uTaskerAv1.jpg
---A 04.09.2006  21:48               35887 uTaskerLogo.jpg
---A 07.10.2006  13:41               12971 uTaskerLogoR.gif
---A 04.09.2006  21:48               24610 uTaskerLogoR.jpg
---A 07.10.2006  13:43               11759 uTaskerLogoR1.gif
---A 19.11.2006  18:38                2634 uTaskerLogoS.jpg
---A 03.11.2006  02:07                4943 uTaskerLogoSmall.jpg
---A 06.02.2007  20:24                1600 uTaskerLogoSS.jpg
---A 07.10.2006  13:47               55654 uTaskerR.bmp
---A 07.10.2006  13:50                2498 uTaskerR.GIF
---A 27.09.2006  16:33               69632 uTasker_d.doc
---- 02.10.2009  15:13   <DIR>             Demos
---- 11.10.2009  19:11   <DIR>             docs
---- 06.02.2009  21:32   <DIR>             Forum
---- 02.10.2009  15:13   <DIR>             Licensing
---- 28.10.2008  16:52   <DIR>             mjb
---- 02.10.2009  15:12   <DIR>             Payment
---- 27.10.2009  00:49   <DIR>             Projects
---- 02.10.2009  15:13   <DIR>             software
---- 18.01.2009  21:36   <DIR>             temp
---- 02.10.2009  15:23   <DIR>             test
49 files with 2366130 bytes
10 directories, 1496460228 bytes free
D:\uTaskerWeb>cd demos
D:\uTaskerWeb\demos>dir
Directory D:\uTaskerWeb\demos\

---A 20.01.2009  22:48              109387 Kirin3.jpg
---A 02.10.2009  15:13                6027 Simulations.html
-HSA 20.01.2008  00:36               30720 Thumbs.db
---A 10.06.2009  00:56               68503 uSimAVR32.JPG
---A 31.07.2009  16:08               64484 uSimAVR321.JPG
---A 10.03.2009  23:05               50316 uSimLM3S1XX.JPG
---A 08.04.2009  22:43               46866 uSimLM3S6XXX.JPG
---A 02.12.2007  22:47               41017 uSimLM3S6XXX_old.JPG
---A 11.11.2007  23:59               40090 uSimLM3S6XXX__.JPG
---A 02.01.2007  18:53               48967 uSimLPC23XX.JPG
---A 21.06.2009  14:36               74857 uSimLPC24XX.JPG
---A 14.09.2008  19:49               47438 uSimM52211.JPG
---A 02.01.2007  18:53               47815 uSimNE64.JPG
---A 02.01.2007  18:53               44467 uSimSAM7X.JPG
---A 02.01.2007  18:53               51307 uSimSTR91XF.JPG
---A 10.01.2007  22:16               85076 uTaskerSim.JPG
---- 14.07.2008  16:36   <DIR>             exceptions
---- 21.11.2009  15:59   <DIR>             exes
---- 14.07.2008  16:36   <DIR>             Freescale
---- 21.01.2009  21:25   <DIR>             M522XX
---- 14.07.2008  16:36   <DIR>             Rabbit
---- 03.01.2009  23:13   <DIR>             SAM7X
16 files with 857337 bytes
6 directories, 1496460228 bytes free
D:\uTaskerWeb\demos>cd ../../webpages
D:\webpages>dir
Directory D:\webpages\

---- 16.10.2009  14:07   <DIR>             WebPagesLPC23xx
---- 02.08.2009  19:52   <DIR>             WebPagesM5223X
---- 16.10.2009  16:01   <DIR>             WebPagesM5225X
---- 20.06.2008  13:50   <DIR>             WebPagesNE64
---- 29.07.2009  13:39   <DIR>             WebPagesSAM7X
---- 21.05.2009  19:17   <DIR>             WebPagesSTR91XF
---- 29.07.2009  21:02   <DIR>             WebPagesAVR32
---- 31.10.2009  17:21   <DIR>             WebPagesLM3Sxxxx
0 files with 0 bytes
8 directories, 1496460228 bytes free
D:\webpages>
« Last Edit: December 12, 2009, 12:20:52 AM by mark »

Offline mhoneywill

  • Full Member
  • ***
  • Posts: 173
    • View Profile
Re: utFAT - preliminary information
« Reply #1 on: December 13, 2009, 11:57:14 PM »
Mark,

uFat looks to be very interesting, I have a few of questions

1. Will it be possible to run a uFat with an SPI memory chip instead of an SD card? I'm interested because I would like to use a slightly more flexible file name scheme rather than relying on the first letter dictating the file location

2. Will uFat be able to support userfile collections stored as a single file in the uFat system?

3. Will uFat support similar commands to the uFileSystem making porting easier?

Cheers

Martin

Offline hervé

  • Jr. Member
  • **
  • Posts: 98
    • View Profile
Re: utFAT - preliminary information
« Reply #2 on: December 15, 2009, 09:01:34 AM »
Hi Mark,
It looks nice.
Do you plan to allow simultaneous access to a file (one write access with multiple read access)?
Do you have a pre-release available at this time?

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3243
    • View Profile
    • uTasker
Re: utFAT - preliminary information
« Reply #3 on: December 15, 2009, 10:13:43 AM »
Hi Martin

1) SD cards and SPI FLASH are quite different due to the fact that SD cards have a controller in them handling such details like level-wearing and bad block management (this is invisible to the user but is probably highly developed) - at least I think... (I read a SanDisc technical document where this point seems to be confirmed). Therefore level wearing is something that would be quite difficult to retain. This is however not a problem in applications where the quantity of writes is not too high.

Also the dimensions of SPI based FAT32 may call for a slightly different configuration. ATMEL SPI FLASH chips (with small granularity) seem to me to be good candidates since they fit well, whereas large granularity types will be a bit of pain.
Although I do see that FAT would fit certain SPI FLASH devices fairly well it would still mean a quite different hardware interface to the SD card one. If the demand is there it can be looked at after the SD card work is completed and established.


2) I don't know that there would be any advantage of a user file collection stored in the utFAT. It could also not operate with the existing addressing mechanism. [Reminder: SPI FLASH memory is effectively linear and so can be memory mapped (virtually, as the uFileSystem does). SD cards - and FAT in general - are not linear, they work with sectors (effectively windows into a small block of memory) and and clusters which are like blocks of the files (or directory information) linked together almost anywhere in the memory (not linear)] If utFAT is used I think that standard file addressing is the way to go. However if SPI FLASH is used I find embedded user files a good compromise to allowing it to essentially look like a file system with long file names and even directories (the complete directory path is the file name), complete web pages to be packed into one single file for uploading and fast access (still not as fast as uFileSystem access but faster that FAT32 access since the file search is generally rather less complicated).

3) No, it is not really possible to use compatible commands. The operation of the two are just so different. However it is still easy to use. Here is an example (present state and may change before release...Also no error handling for simplicity.)

FTP file retrieval, uFileSystem case - the file is identified from the path/name. The length is checked and,

    ptrFile = uOpenFile((CHAR *)(ucIp_Data+5));              // get file pointer
    if ((uGetFileLength(ptrFile)) && (fnTCP_Connect(FTP_TCP_Data_socket, ipFTP_data, usFTP_DataPort, FTP_DATA_PORT, 0) >= 0)) {
        ucFTP_action = DO_UPLOAD;
        FileOffset = 0;                                      // start at beginning of file
    }


.. data is put to the output buffer.

    uGetFileData(ptrFile, FileOffset, ptrBuffer, usTxLength));           // read from the file to a buffer
.. (with some management of the length and file offset)



In the utFAT case:

    static UTDIRECTORY utDirectory = {0};                                // directory object for a single user
    static UTFILE utFile = {0};                                          // file object for a single connection

    utOpenDirectory(DISK_D, FTP_ROOT, &utDirectory);                     // set the FTP root (eg. D:/ftp_dir/) - initialisation
    utFile.ptr_utDirObject = &utDirectory;                               // link the file object to its directory object



.. locate the file according to the retrieval name (eg. /dir_1/file_000000012888_long_name.txt
    utOpenFile((const CHAR *)(ucIp_Data+5), &utFile, UTFAT_OPEN_FOR_READ);

..
    if (utFile.ulFileSize != 0) {
        utReadFile(&utFile, ptrBuffer, usTxLength);                      // read from the file to a buffer
    }

In this case the management of the file is content is performed in the utReadFile() (and UTFILE object) since it needs to be able to do seek type functions.

Integrating this into FTP and HTTP was not difficult. In fact, since the utFAT (necessarily) performs extra file location management some of the pointers/indexers used together with the uFileSystem are superfluous. I just set them to the momentary utFAT values to keep them in sync so that it is easier to understand the two when they are side-by-side. Eg.
FileLength = utFile.ulFileSize; // set the local length (as used by uFileSystem code) to the internal value to ensure compatibility with later length use

In an attempt to ensure porting (or using both files systems together - useful as a fall back in case the card is extracted...) I have encapsulated details of mounting disks. The only addition code is setting where the directory is located (can also be root), which initialises everything for its subsequent use by the particular user (see utOpenDirectory()) - whereby there may need to be be multiple directory objects if multiple users have their own relative locations in the utFAT (like multiple DOS windows each have their own independent present location). Then it is easy to move to other directory locations - allowing absolute or relative addressing (relative is generally more efficient for the file searches too). The user code shouldn't need to worry about too much detail (eg.  - D:/dir_2/myFile.pdf or ../../dir_3/dir_4/myOtherFile.jpeg - basically just pass an appropriate string).

Regards

Mark
« Last Edit: December 18, 2009, 08:00:59 PM by mark »

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3243
    • View Profile
    • uTasker
Re: utFAT - preliminary information
« Reply #4 on: December 15, 2009, 10:26:06 AM »
Hi Hervé

File sharing for reads is certainly possible. At the same time I am working on additional file sharing management (also for the uFileSystem), where users can protect files from deletes and modification while they are being used.
I expect that this mechanism can be shared by all file system types.

At the moment I don't have a pre-release that I have made available due to the fact that there is still a lot of activity on it (interfaces change as further tests and experience show that the solution still needs some optimisation). The goal is a first release around Christmas and pre-releases would greatly endanger this - this would be supported by all processors (this needs some work since the SPI interfaces have to be adapted for each) but not have all features. At a minimum all read support will be available but not file sharing management (also not necessary when only read) and not (all) write [delete, create, format, etc.] capabilities. The last bit is left open depending on the progress, but basically it is seen as important to have a well defined interface supporting all read requirements as step 1; if this foundation is solid the second phase of adding write support and other extensions will be easier. Note that some good progress has been made and so I do hope to have some write capabilities in the first release but I am not committing to it, in preference to a mature solution in the long run.

Regards

Mark