Async operations?

[schveiguy]

Hi,

Are there any plans to make async operations possible on the utFAT/SD system?  In an application I'm writing, the config and logs will be stored on the SD card, but I'm concerned that the writing of the log/config will consume too many cycles, since the write is synchronous.  I understand the module is preliminary, but so far, it's worked for me!

-Steve

[mark]

Hi Steve

The present version of the utFAT is indeed synchronous apart from the formatting itself which is asynchronous (or at least highly asynchronous) since formatting a card can take several minutes.

If you look in the code you will see quite a lot of loops like this:
while (utCommitSector(ptr_utDisk, ptr_newDir->ulSector) == CARD_BUSY_WAIT) {}
This means that, writes and such will wait in this code until completed and finally returning from the call. As you know, writes can take some time - about 5ms - and also reads are not instantaneous.

In the development version there is some new support of managed writes, deletes, etc. This means that rather than calling a function which returns when it has completed the user calls it and it returns immediately although not finished. The memory manager then completes the operation in a sort-of background task and informs the user when it has completed.

This doesn't mean that writes are faster and it also means that the user can't do more memory operations until the outstanding one has completed (these are locked - a bit like when semaphores are used) but it means that the user can do 'other' operations while waiting.

The module is prepared with such busy loops at the locations where the waits take places. These need to be extended to pass control to the memory manager and return to the caller to complete asynchronous operation. The memory manager has indeed been added and tested for some time consuming memory operations, like deleting large portions of internal and external FLASH, but has not yet been extended to the SD card interface.

In any use of SD cards for storage it has to be understood that writing takes some time - especially if the writes are not performed in native 512 byte blocks [if the user ensures that write are optimised to be performed in 512 bytes blocks it will in fact help a lot]. There will be a limit to the maximum possible writing rate that can be achieved, irrespective of how it is implemented (synchronously or asynchronously) and if this limit is passed it will not be able to work. If it is approached it is also dangerous. Asynchronous operation will not actually improve on the limit - it will only give a mechanism for the processor to be able to do 'other stuff' while the hardware is working - therefore be careful that this is the actual difficulty found. If you don't have to do the other stuff it would mean that the physical limit has been hit and there is probably not a lot that can be done about it. Do you have any more details about the rates that are involved?

Regards

Mark

[schveiguy]

Hi Mark,

I'm not so much concerned about the rate, it can be as slow as it wants to be while writing/reading.

What I'm concerned about is that while my task is writing data, the OS cannot do anything else (aside from process interrupts) while the hardware is sending the data to the card.  Usually in a pre-emptive OS, the processor starts running other tasks on blocking I/O, but as you pointed out, uTasker just does a busy loop, not letting anything else run.

The managed operations seem to be exactly like what I want (immediate return from a write, then async notification when it is complete).  Is this in the latest download?  I noticed you could open a file "managed", is that what you are talking about?  I thought managed meant that the open file itself could survive across calls to the task.

Note that I have very little experience with low-level FAT stuff, so I'm quite ignorant on timings and limits.

-Steve

[mark]

Hi Steve

It is possible to put uTaskerSchedule() in each while loop. This will not work in the simulator but will on the target. This allows scheduling to continue while waiting on the SD card completion.

However it does introduce further complexities - as also having other pre-emptive methods - since access to the same resources then need to be avoided while the operation is still active, so other things start waiting and then others start waiting on them etc... But to solve one restriction in particular cases it is a method that can be used quite easily.

The application itself can also do a lot to help by reading and writing blocks of 512 bytes (the native size of the SD cards) at a time rather than lots of smaller chunks or larger blocks. This means that each access is optimised in terms of efficiency. Also the application can then return between each block transfer to allow other tasks to be scheduled with a minimum of interference between each individual block read and write.

Regards

Mark

[schveiguy]

OK,

For now I'm just resigned to use a background task that writes 512 bytes at a time, then returns.  If this proves to be too time consuming, I'll revisit this.

I'm interested in the managed reads/writes more, do you have any examples about how these are used to read/write asynchronously?  Is this code available?

-Steve

[mark]

Hi Steve

Please see the other thread for latest code.

The managed file mode in the present version doesn't perform any asynchronous work but it does allow some file sharing operations when a file is (optionally) opened in this mode. What this means is that one user can open a file for read and another the same file for write (for example). Since the file is marked as a managed file all users of the file will be updated on file changes. This means also that it is not possible for one user to delete a file that is opened by another user in managed mode (the fact that another user is working with it - i.e. has the file open - will protect it).

When one user has a file open and reads its size, the size will automatically be updated when another user modifies the file (assuming the users don't restrict such operations on the file). Essentially all objects related to a file are updated when the file itself changes. There are various flags that the user can set to restrict certain operations - these are probably not documented yet though.

This should give a first idea of what the mode means. Until the documentation is updated with the new capabilities you'll need to ask here if such things are of interest and the code doesn't make it clear enough to use it just yet.

Regards

Mark