Author Topic: utFAT Countdown...  (Read 12142 times)

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3236
    • View Profile
    • uTasker
utFAT Countdown...
« on: December 31, 2009, 01:18:18 AM »
Hi All

The forthcoming utFAT module is of great importance to the uTasker project. It was a major goal for 2009 but was not started until well into the fourth quarter - rather later than originally anticipated.

It is also an exciting development, with new technologies to learn about and new problems to understand and solve. As with all such developments there is always something that could be done differently, better, with more features and efficiency... if only there was the time. But time doesn't stand still and there comes a point at which a new software package moves out of the cosy developer's environment into the more hostile real project environment. And there is no stopping this inevitable step, since the data 3.1.2010 is circled in indelible ink on my calender!

On this date the new utFAT will be made available as module for use in the following projects:
  • - SAM7X
  • AVR32
  • LM3Sxxxx
  • M522xx
  • LPC2xxx
  • STR91XF (yes, even this rather 'unloved' chip!)

This will probably be in the form of new V1.4 releases, containing any other patches to date as well as adapted projects (VS and other target IDEs) pre-configured to include the new module.

Although the functionality is more-or-less complete there is still a lot of work remaining to qualify its operation, plus make sure that it has a "usable" user's guide. Be warned that - even though it is destined for serious use and hopefully great things - it is a very new development which should be considered a Beta version until matured!

Therefore I will be making all efforts to get working packages out on time so that you can start finding its weaknesses and benefiting from its strengths - in return for the understanding that there may be a few rough edges that I didn't have time to file off... the further development will not stop but rather serious tuning will start.

To get you into the mood, I have also added the start of the utFAT document to the pre-liminary document section so that you can also monitor its development during this phase and even start giving feedback during its development; http://www.utasker.com/docs/uTasker/uTasker_utFAT.PDF

Regards

Mark

Offline dkg

  • Newbie
  • *
  • Posts: 48
    • View Profile
Re: utFAT Countdown...
« Reply #1 on: December 31, 2009, 02:46:19 AM »
Hi Mark,

This is exciting for sure. We have a use for this as soon as I get back to work from the Holidays. Looking forward to trying it out.

Dave G.

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3236
    • View Profile
    • uTasker
Re: utFAT Countdown...
« Reply #2 on: January 04, 2010, 02:18:16 AM »
Hi All

I tried but didn't manage to get all ready for 3.1.2010.

However versions for SAM7X, AVR32 and Freescale M522XX have been made available (see the specific processor forums for an overview of the versions).

Luminary LM3Sxxxx, NXP LPC2xxx and STR91XF have been tested (on at least one development board) but the new packages could not be released on the same day due to the extra work to make and test the packages. These will follow - in the order above - and should all be available before the end of the week.

The user's guide is also being developed in parallel and is probably about 65% complete at the time of writing. It should already be useful for first tests with the new features and can be found here: http://www.utasker.com/docs/uTasker/uTasker_utFAT.PDF

This is the present state of the new module's development:

- first of all beware that this a new development and has been release very early due to various requests to start using it as Beta. So it is Beta - it has not been tested outside of a warm laboratory environment and also has a few feature that are not yet fully complete. I am however convinced that its development is fairly advanced and it should already start being useful; at the same time its weaknesses can start being identified with more intensive use.
- today the SAM7X and AVR32 releases were tested on the ATMEL AT91SAM7X-EK and ATMEL EVK1105 boards (compiled with GCC). These board have SD card sockets on them and all worked basically as expected. The SPI speed is not necessarily set as high as it could be for the processors but serving of web content worked well and also viewing large PDF files form the web server was successful.
- the M52259EVB was used to test the Coldfire release (compiled with Codewarrior). All worked normally apart from the fact that PDFs sometimes show some corruption when viewed via web browser - the same large PDFs can however be retrieved without error via FTP. This gives the impression that maybe the web browser is parsing the content and causing come errors in the file (html and graphics worked well) - it also seems restricted to the Coldfire, or possible the compiler, so this will need to be investigated more. Furthermore the Coldfire development boards are not supplied with SD sockets so an external one had to be wired up; thsi is a potential source of interference but probably not a reason for the different behavior.

- Generally there are some things that don't work optimally or with some errors too. The following are the known bugs, short comings, etc. that will need to be worked through for the next utFAT version (since the utFAT module is in its own directory in the uTasker folder and is also a module that can be exchanged without requiring a new project version it should be possible to simply upgrade when new versions of the module are available).

1) Ensure that the SD card is not activated together with the SPI FLASH - this is presently not possible (either one or the other...)

2) utFAT reads directories and contents with short and long file names. However it doesn't support writing long file names yet. This means that only short file names can be used when creating new files and directories. Furthermore one has to be careful not to rename or delete files and directories which already have long file names!! This is because deleting them is rather more complicated and this was not solved before this version - when long file named files are deleted their content is deleted but not their name, which can be confusing. This is probably the first point that will be worked on.

3) Although the mass storage module can detect SD cards it does this by polling the slot (in a background task). The card is successfully detected but sometimes (it seems a little board dependent) it doesn't always recognise the card as being formatted. Therefore insert the SD cards before powering up the board to ensure that they are reliably detected. In addition to the detection, the module can detect their removal - at the moment the card will not be monitored for insertion after the removal; this needs to be worked on. Finally, since not all SD slots include a switch for detection there is presently no use of such switch inputs - this may be added later.

4) When inserted SD cards are successfully recognised during operation (some boards do this quite reliably) they are not necessarily recognised by the HTTP module - therefore point 3) should generally be respected at the moment.

5) The write protection switch that some sockets have is not yet monitored. Assume that write protection is not yet supported...

6) The utFAT module has various opportunities for optimisation - there are a few subroutines that are very similar but were added to achieve something without endangering existing (similar) routines. Such things will be worked on to achieve better code density as the module matures.

7) The size of SD cards with more that 2G are displayed incorrectly. This is due to the fact that the number of bytes no longer fit into an unsigned long (they need long longs to be displayed). For the moment the byte size overflows - a simply solution may be to display them in kBytes. This is not otherwise a problem with operation...

8 ) The relative path addressing doesn't always work perfectly. Specifically creating new directories referenced by a path name (and possibly renaming etc.) [eg. mkdir dir1/dir2/dir3, or mkdir ../../dir1/dir2] doesn't behave correctly. For the moment move to the directory and create the new one there to avoid any confusion.
Update on 31.1.2010: utFAT1.2 (the version used by registered Beta testers) now allows full referenced operation so can create files and directories, delete, rename, etc. as in the examples above. In addition it supports root referencing like "cd \" to set the root, "dir \dir1" to display director dir1's (under root) content from any location. The full flexibility seems now to be available ;-)

9) Up to now the operation with the FAT32 info sector has not been fully analysed - it has been implemented so that it keeps the number of free clusters up to date and also increments the next free cluster. Creating new directories and files can take some time when it is necessary to search through large amounts for clusters, whereby the next free cluster value should help; it is probably not used optimally just yet...

10) Only simple FTP tests have been made to date. It is not sure that the FTP implementation with the SD card support will suit all FTP client types and whether there may be some restrictions due to the open point 8 ). This needs to be investigated. DOS FTP has worked well though.

11) With large SD cards, large amounts of data and missing FAT32 info section support (this support is not guaranteed) searching through FAT clusters can be quite time consuming. A count of the free clusters, for example, can easily take several seconds because it involves already reading about 15MBytes on a 2G card and increasing amounts on larger cards. This count is not needed for normal operation (just for displaying) and has been realised as a background task (as has detecting, mounting and formatting cards). Other accesses are blocking accesses and so can cause the system's overall behavior to be degredated. Some seemingly quite simple SD card operations do sometimes require a surprisingly large amount of data to be read, searched and written [eg. to create a directory requires reading all of the present directory's content - to be sure that its new name doesn't already exist - adding a new directory cluster, deleting the new cluster's content, modifying its entry in all FAT32 cluster areas and also modifying the original directory entry, not to mention upgrading the FAT32 info sector - involving reading several several kbyte of information and writing several kbytes more]. A complete analysis of the read/write times, the SD card internal delays and addition of capabilities to makes these non-blocking tasks is a point which is still open in the development, and its documentation.

12) utFAT already includes some managed file support so that a user can ensure that an open file can not be modified by another user (write protection) or that the user becomes aware of changes made by another user (file synchronisation). Further development of this and point 11 are probably the most challenging aspects involved in optimising the file system as a whole. They will however result in a powerful module in return for this additional development effort.

Good luck with your first tests!

Regards

Mark

P.S. Don't forget that the uTasker simulator now includes SD card simulation support, which greatly helps analysis of program flow. The difference between the SD card content, as opposed to the uFileSystem and general FLASH content, is that it becomes immediately valid and is not saved when the simulator terminates. The main reason is that it doesn't create a memory array (this would be very wasteful for a 16G card...!) but only instead saves only the actually written SD cards sectors directly to a file.

« Last Edit: February 01, 2010, 12:11:28 AM by mark »

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3236
    • View Profile
    • uTasker
Re: utFAT Countdown...
« Reply #3 on: January 04, 2010, 11:30:07 PM »
Hi

Here's an update from 4.1.2010.

Today I made a new LM3Sxxxx package ready for release and tested it on the LM3S6965 (the microSD cards are nice on these boards). All went very well so this will be posted shortly.

There were however four further points which need to be mentioned:

a) This board/processor detects an inserted card very well and also recognises that it is FAT32 formatted. This shows that there are some differences between the processors/projects - compare point 3 in the list of open points.

b) I found that I have to add a pull-up (eg,. I use 1k) from the MISO to 3V3 otherwise the SD card never gets detected and mounted correctly. The same is true on the Freescale and STR91X, but not the ATMEL chips. I didn't actually understand why (attempts to drive the line during switching to SPI mode and activate internal pull-ups didn't seem to help. So when testing these boards please add a pull-up resistor for the moment...

c) VisualStudio gives one warning when utFAT (mass_storage.c) is compiled. GCC gives about 12 warnings. These can be ignored as far as I can work out. The code is doing what I wanted to do at these locations; either the code could be changed slightly to use different techniques to get around them or else it may be possible to quieten them somehow. For the moment I think that they can however safely be ignored.

d) When testing formatting an SD card I was getting some timeouts, which effectively stops the process due to a failure. I think that it is due to the fact that the time waited during sector writes is a bit critical. It was happening during the end of formatting and it seems that the write time of the SD cards(s) is not always constant. So I doubled the maximum wait and then it was always good.

This however brings me to a point that is also something to be further developed; usually I have seen that timers are used to monitor SD card accesses - when the times are greater than some values (I am not sure about all the maximum times yet) there is an error - like the one seen during formatting. I decided that timers were not really necessary since the communication with the SD card takes place serially and this transmission time is essentially also a timer - by allowing a certain number of busy-polls one also achieves a fairly accurate timer to monitor such worst case timeouts.

I did however increase the SPI speed to 12MHz for the final tests and this effectively reduces the max. delays accepted - possibly this is the explanation. So the point is that, although I feel happy with using the SPI transmission as base for maximum delay monitoring the code needs to be set up correctly to respect the actual speed used at the time and also various delays need to be adjuster at certain points in the module to set realistic values. The actual value which is there limiting in case of real card failure during formatting (the only time that I have seen it triggering in normal operation and before adjusting so that it never fired) is static int CommandTimeout = 10000; The idea being that calls can change it to suit the action being actually performed. Presently it is simply set high enough not to disturb when the card is actually present...

This probably fits into point 11 but is worth mentioning in case it does kick in when running the SPI even faster - simply increase it for the moment...

Regards

Mark
« Last Edit: January 04, 2010, 11:54:41 PM by mark »

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3236
    • View Profile
    • uTasker
Re: utFAT Countdown...
« Reply #4 on: January 06, 2010, 01:40:13 AM »
Hi All

Update 5.1.2010.

The LPC2xxx release has been made. Although the IAR tests on an MCB2300 board (with SD card connected to an external socket via SPI1) tests worked well the GCC build is not stable - I think that the Rowley Crossworks project (also using GCC) was however OK (not absolutely sure since I though there was some board problem at the time).

However I do have the feeling that I have neglected the GCC build in this project since thsi doesn't seem to be purely due to the new utFAT... for this reason I decided to release it like that for the moment.

There is quite a difference with the LPC2XXX release due to the fact that some LPC23XX and LPC24XX processors have SD card interfaces. These are connected to the SD card sockets on the evaluation boards. Since the utFAT SD interface is presently SPI based these sockets can not be used - I had to wire up external ones on my board to do the development. This may be rather a disappointment in some way...

The SPI interface does, on the other hand enable SD cards to also be used together with all chips in all LPC2xxx families, not just the ones with internal controllers.

The use of the SD card controllers would make transfer more efficient since they allow 4 bit data mode and DMA but this will require an additional development exclusively for the LPC2xxx chips before it can be made available.

Regards

Mark