write to a file after close

[Lukas]

Hallo.
I'm new in the uTasker world. I read the two docs about the filesystem, but still have a problem:
I open a new file:

Code: [Select]

   unsigned char *ptrLogfile = uOpenFile("Ulog.TXT");
   uFileWrite(ptrLogfile, data_log, 6, 0);
   uFileCloseMime(ptrLogfile, &ucMimeType);
When the file does not exist, it works. Otherwise not. Can't find why.
Also I can't write to the file a second time, after i close it.

Want to log the data from UART in a array, then write it to a file, send this file over TCP/IP. When the next value comes from the UART, want to open the file, add the new value to the existing and so on.
Someone can help me?

Regards
Lukas

[mark]

Hi Lukas

First a question about the file you wish to save.

Are you sure that you need to save it to FLASH (file) and not just keep a copy of the data in RAM?

The actual details involved are a bit HW specific (for example you could be saving to internal FLASH or external FLASH). The rules are different depending on the size of the FLASH granularity (eg. the STR91 will have a granularity of 64k or the SAM7X will have a granularity of 256 bytes). I will assume you are using maybe the LPC23XX since you are setting the ucSubFile parameter when using uFileWrite(), in which case life becomes (unfortunately) even more complicated.

Taking your example: you are opening a file with name "Ulog.TXT", which is basically returning a pointer to its location in memory.
Then you write 6 bytes of data to it.
Then you close the file, meaning that its length is written and its mime type (TXT) set as reference to its content type. This file is now visible in the file system as "U.TXT" and its length can be requested - which will return 6.

The first time this works fine since the FLASH was empty. Unfortunately it is not so easy to add data to the existing contents - to change the file's length it needs to delete the present content (at least 32k on the LPC23XX!). The LPC23XX also doesn't allow a second write to a single line of FLASH (16 bytes) so even a change of the length is impossible without first deleting the entire content first. [see the following for some background on the FLASH in the NXP devices - http://www.utasker.com/forum/index.php?topic=136.0].

Therefore due to physical restrictions, to add more data to existing data requires the following procedure:
- First write as performed
- Open, read and backup present file data (copy to RAM)
- Add new data to its image in RAM
- Save the new complete data (this will automatically delete the old file and rewrite to same location with new length).

As you see, the uFileSystem is intended mainly for storing files used by the web server where overwrites is the norm and not additions to existing files. This is however possible by using the lower level FLASH interface, where the uFileSystem doesn't actually need to be respected - you can decide to use a physical address range and control it as you wish (but of couse still respecting the physical limitations of the FLASH technology).

Therefore your application may be more suited to first building up the data contents in a simple RAM buffer (declare it on heap using uMalloc() and keep one length variable so that you know where to copy the next data block to (simply using uMemcpy()). If you do want to save this at the end to FLASH as reference the uFileSystem interface will be fine. If you have however in the meantime transferred the data via TCP to a PC then perhaps there is no need to ever store it locally in FLASH...

Good luck

Regards

Mark

[Lukas]

Thank you for help Mark.
So I will try it as you described with making backups in RAM and writing every time new.

I see its a great forum with fast and qualified help.
Thanks.

Regards
Lukas

[Lukas]

Hallo again.
How is it possible to write files larger than 4kbytes? (on the NXP 2378)
I open a file and write 300 * 32byte.

Code: [Select]

ptrFile = uOpenFile("z_log.txt");
uFileWrite(ptrFile, ucLog, 32, "z")
Of course I  open the file only 1 time.
I tried to name my file "6log.txt", tried uFileWrite..... with "z", with 'z', with zeros.
But every time I reach 4kbyte the system close the file and the values are wrong.

Regards
Lukas

[mark]

Hi Lukas

I think that this behaviour has something to do with the configuration of the file system and the use of the file name.

Check the size of the file system configured (see for example the document in the sub-directory of the WebPages folder for the processor). In the latest LPC23XX SP there are different settings included (one with smaller and one with larger files system - the larger added when set to support firmware uploads).

You may find that the file system (which uses sub-files of 4k size) is not large enough to accept a file with name 'z' - this is the last possible file name in the file system and so will be positioned at the end. The file system will write any files which would be outside of its physical size setting to the very last block and it will also cut of anything which is too large to fit in this last block. As I said, the LPC23XX uses 4k sub-sections so this behaviour would be expected if the file is outside the configured range.

Assuming that this is the explanation, you will need to know the physical area reserved for the file system (in kByte) and the corresponding file naming convention corresponding to this (best see the document above as reference). Any files positioned low in the file system (eg. '8', '9', 'a', etc.) will be able to have a length as big as its starting position to the physical end of the FLASH region, where as files positioned towards the end (eg. 'z', if it actually exists), also having the same restriction, will have rather less room (again to the end of the physical range) and in the most extreme case just the 4k sub-sectiorn size!

The use of the last parameter (special case when using sub-file types) may also be a problem - it should be 0 or 1 depending on whether the file is a sub-file or not. You can use the following to automatically set this fnGetFileType("z_log.txt") rather then "z" which would always signal a sub-file and so could be incorrect in some cases.

Good luck

Regards

Mark

[Lukas]

I'm without any idea.
I read several times the FileSystemLPC23xx_336k.doc. I can copy larger files via FTP without problems. Tried also to different postions (as mention in the doc file I changed the file name), but it is allways the same. When the file reaches 4k it crashes. 

Regards

[mark]

Hi Lukas

When you write "crashes" does the board reset or does the file get limited to 4k?

I will test this in the next few minutes - as I understand you are using 336k file size so I will test this configuration.

Note also that automatic deletes only occur when addressing the files by the 'normal' file names and not their sub-file names. This is due to the physical restriction of the LPC23XX FLASH granularity - only 32k blocks can be deleted; sub-files allow smaller files to be written but the larger block delete restriction still (physically) applies. I'm not sure that this has anything to do with your problem but I will report back once I have repeated your tests.

Regards

Mark

[mark]

Hi Lukas

I have hopefully tested code to do as you want:

Code: [Select]

unsigned char *ptrFile;
unsigned char ucLog[32];
int iRepeat = 300;
unsigned char ucTest = 2;
unsigned char ucMimeType;
MAX_FILE_LENGTH length = 0;

ptrFile = uOpenFile("6_log.txt");
ucMimeType = MIME_TXT;
while (iRepeat--) {
    uMemset(ucLog, ucTest++, 32);
    uFileWrite((ptrFile + length), ucLog, 32, 0);
    length += 32;
}
uFileCloseMime(ptrFile, &ucMimeType);
I open the file '6' since this corresponds to a FLASH sector ('z' works but won't automatically be deleted on the next write to overwrite new contents to the file and thus would need a 'manual' file delete command). I also specifically tell uFileWrite(, , ,0) that it is not to be handled as a sub-file!

Then I write 300 times blocks of 32 bytes (the content is changed each time for easier checking). Note that each write uses an incremented pointer (+ length).
At the end it is important to close the file since this sets the file length.

Afterwards you can view the file (it is displayed as x.txt in FTP client) - it is 9'600 bytes in length and you can also download it via FTP and check its contents.

If the start value of ucTest is changed and the test repeated it will be seen that the file is erased (automatically - overwritten) and the content corresponds after to the new values. As stated above, this is only overwritten 'automatically' when the file is writen using its file name and not its 'sub-file' name.

I am sorry that it is a little more complicated to use due to the sub-file model (it is much easier to work with chips which have small FLASH granularity). However if you can live with file of 32k granularity you can in fact remove the sub-file define and it is then easier, with the restriction that every file - however small - occupies at least one 32k sector on the LPC23XX. This is OK for some projects but mostly it is a bit of a waste of FLASH so the sub-file method at least makes it workable.

Does this solve your problem?

Regards

Mark

[Lukas]

Hi Mark.
Thanks for you great help.
Unfortunately it still does not work. I used your example, without any change. So it obviously must be a config problem?
If it helps here is what I see with DosFTP after doing the write operation and closing the file:
-rwxrwxrwx 1 502 502 143355 Jan 1 2007 y.
Crashes means the file is limited to 4k (when I write not to file "6_...") or the value above (with writing to file "6...").

For now I removed the sub file system.
Reagrds
Lukas

[mark]

Lukas

-rwxrwxrwx 1 502 502 143355 Jan 1 2007 y.

This looks to me as though there is something in FLASH at the address 0x59000 (position 'y' in the 336k configuration). The size if 'seen' as 143355 in length but its type is not recognised. You could download this per FTP and see what its content is like. You should also be able to delete it from the FTP client.

It is possibly something which was copied earlier but doesn't respect the file system contents (corrupted) and, since it is 4k in to the '6' file block, it could also be the source of your difficulties. The LPC FLASH needs to be written in blocks (the driver uses 256 byte blocks) and it can't modify data already written. This means that corrupted locations can cause some difficulties.

Try getting rid of this (if FTP delete doesn't help you could delete all FLASH and reprogram again).

Have you had more sucessess without sub-files? If you don't actually need files, but instead work with direct data in FLASH the sub-file system probably doesn't have any advantages since it only helps enable writing smaller files but doesn't actually change anything about the deleting part of the problem.

If you still have difficulties, compress your uTaskerV1.3 directory and send it to me so that I can try it out as it is (tell me if you have changed anything outside of this directory though).

Regards

Mark

[Lukas]

Hallo Mark.
Working without Sub-File-System works fine and it suits to my project.

But one more question:
Is it possible to read out of a file before closing it (e.g. to copy values back to a buffer)?
I write my values over Ethernet (TCP/IP), when writing fails, I open a second file and write the value into it. Everytime a new value is ready for sending over Ethernet, I check if TCP connection still is down, if not, I want to write the actual value over TCP/IP and then rewrite all the "in-file-waiting" values. But I don't want to close the file in case that during rewriting the connections fails again. So is it possible to read out of a open file (allways the last value) and then put the pointer one value back, read again, and so on, until all values are written back (so the file is "empty")? If the connection fails again during writing, I want to write new values again into the file, but the startpostion should be behind the last unwritten value.

Regards
Lukas

[mark]

Hi Luka

Basically reading from a file before it is closed is not a problem since the start of the file (the pointer to the files location on open) and also the offset in the file are known by the routine controlling the write - this means that it can also read back data based on the same pointer and offset.

However the LPC23XX has a complication which will make it a bit trickier in this case (most other supported processors will not have this extra difficulty). This will require some extra work since this is not performed automatically unless the file is closed.

When writing to the file the data is stored temporarily in a buffer called ulFlashRow[] (see lpc23xx.c) which is 256 bytes long (aligned on long word boundary). This is used to collect the data to be written to FLASH until a row is complete - only when a complete row has been received will it actually be committed to FLASH (see also fnWriteBytesFlash(0,0,0); which is used to commit it and always performed on a file close to ensure that any outstanding row buffer has been written, even when only partly full). This means that if data is to be read back from FLASH before the close, not only the content of FLASH (which can be easily read using a pointer) but also the content of this row buffer (written but not yet committed) to FLASH) will have to be respected - I can image that this is a not that easy to get working correctly.
The 256 row size is the smallest block that can be programmed to the FLASH at a time.

A last extra complication is that the very first line of 16 bytes in the FLASH (see ucFirstLine[] in uFile.c) is also not written to FLASH until the close has been performed. This is due to the fact that the FLASH content can not be modified in a 16 byte line (technical details are here: http://www.utasker.com/forum/index.php?topic=136.0). In order to be able to write the file's length and MIME type on the close, this complete line must be written (as a part of a 256 byte row) whereas only changes inside of the first 16 byte line are possible. Again, when reading before close, this extra line buffer must be read rather than the FLASH content! (this is not so complicated since it is fixed at the beginning).

As I said, it is possible to do what you require but there are some nasty little complications due to the FLASH technology which would be needed to be overcome. The line and row buffers are normally hidden in the FLASH driver so normally the user doesn't need to know that they exist, but to do what you want would require manipulating them which I expect will result in quite complicated code.

I would suggest that you add an intermediate buffer of maybe a few kBytes RAM which you store to before calling the file writes. If you can afford a buffer long enough to hold a few TCP frames you can probably do all the reading back and any necessary modifications there before actually performing writes. If your writes are then always multiples of the row size (256 bytes) they will also always be fully committed to FLASH so that the low-level row buffer can be ignored (it will never actually hold anything between writes). Apart from the first 16 byte line which will never be written until the close - which is not that complicated to respect - you will then have a much easier interface for reading back and perfoming even last minute modifications before it is committed to FLASH.

Good luck with this - I am sorry that your requirements always seem to be complicated by the FLASH technology!!

Regards

Mark

[Lukas]

Ok, so I have to think about the best solution, your suggestion sounds good.

Is is possible to write to two different files? (which actually means to have two different file open at the same time)?

Reagrd
Lukas

[mark]

Luka

No - only one file can be open (for writes, when actively writing) at a time. This is because there is only one set of line and row buffers which have to track the writing. Multiple files open for reads is Ok since an open is in fact only getting a pointer to its physical location for following reads for reads and writes, so is rather a glorified term in this implementation....

Regards

Mark

[Lukas]

Thank you Mark for the fast help.
Great support.