Author Topic: How do I reduce the size of a file  (Read 9197 times)

Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
How do I reduce the size of a file
« on: September 30, 2013, 08:13:31 PM »
Hi Mark

I have a case where the file I am working on is getting smaller.
How do I update the file info to reflect the new size?

Looking forward to hear from you.

Regards
Steen

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3234
    • View Profile
    • uTasker
Re: How do I reduce the size of a file
« Reply #1 on: October 01, 2013, 04:19:31 AM »
Hi Steen

At the moment I don't know for sure but I guess that on a PC a new temporary file is created and the reduced file content copied to it - then the original file is deleted and the temporary one renamed to match the original file name.

In your case I could imagine that you would prefer to overwrite the file length details to remove the end of the file. This would be simple to do by overwriting the file entry but the end of the file's cluster chain would still occupy space and so should be cleaned up (freed). This would presumably require a seek through the file until the end (new end) is reached (if you already have a file pointer to the end of the new file it will save having to seek to it) and then the remaining clusters in the chain deleted so that they can be reused.

Probably the routines to do this are not all accessible from the application so it may be necessary to have a routine in mass_storage.c that allows a file to be truncated in this way if it is generally of use.

Regards

Mark

P.S. Note that a file opened with the attribute UTFAT_TRUNCATE causes its entire content to be deleted so that the same name is used for new data. This is related since it deletes all content (used cluster chain and marks the lengths as 0). Maybe what would be useful is to be able to open (truncate) a file with a certain length; this would then truncate the file to that length and allow additional data to be tagged on to it by writes to the file while it is still open (or no writes made if the file is to be left like that).

Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Re: How do I reduce the size of a file
« Reply #2 on: October 01, 2013, 06:00:14 AM »
Hi Mark

I have been working with FsFat prior to the uFAT. The FsFAT contains a FileTruncate function.
Wen using this function it adjusted the file size to the current position in the file.
The reason for using this way instead of making a copy of the file, is simple 2 things:
  • Speed, Only making the changes a requires is faster, making it unnecessary to rewrite the contents that do not change
  • Wear, Rewriting the entire file will reduce the total life time on the SD card

Are the already a "hidden" function that can do this in the mass_storage.c?

Regards

Steen

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3234
    • View Profile
    • uTasker
Re: How do I reduce the size of a file
« Reply #3 on: October 01, 2013, 01:43:41 PM »
Hi Steen

The (lower level) functions to do this are certainly in mass_storage.c but I think that adding a function such as

extern int utTruncateFile(const CHAR *ptrFilePath, UTFILE *ptr_utFile, unsigned short usAccessMode, unsigned long ulNewLength);

would be the best method.

This would then (internally) call utOpen() and utSeek() to get the information necessary and then delete and cluster chain at the end (when necessary) and adust the file size in the file table - the caller will be able to write data to the file from this location if desired (the file pointer is at the end of the size specified by ulNewLength). If the value ulNewLength is greater than the original file size it will simply be set to the end of the file and nothing will actually be deleted/changed.

Do you agree with the interface to such a new function?

Regards

Mark

Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Re: How do I reduce the size of a file
« Reply #4 on: October 01, 2013, 06:56:52 PM »
Hi Mark

From my perspective I will recommend this interface:
extern int utTruncateFile(UTFILE *ptr_utFile);
For this to work the file need to be open and in Write mode(If not, it return a error).
When calling this function it will truncate the file to the current position of the file pointer (If at the end of the file no changes is done;-)).

The reason for this interface is that changing the file size are in my opinion always done after changing some contents in the file and hereby having the file opened anyway.

This will also give the faster function and the possibility to put the function into a sub function that only have a UTFILE object to work on.

Hope that you can use this input.

Best regards
Steen

Offline mark

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 3234
    • View Profile
    • uTasker
Re: How do I reduce the size of a file
« Reply #5 on: October 02, 2013, 02:03:54 AM »
Hi Steen

Thanks for the suggestion - that seems to be a good method.

Please try with the following code, which I tested by truncating files, and then adding extra data to the end of them, then truncating them again to different points in the file. If you watch the free cluster count ("info" command) you will see that any freed up clusters are deleted so that they can be reused (this was the most sentitive part because it is not that visible - the actual file length change is simple).

Code: [Select]
// Truncate the file to the present file pointer location
// this usually makes the file smaller and cleans up no longer used clusters
//
extern int utTruncateFile(UTFILE *ptr_utFile)                            // {59}
{
    if (ptr_utFile->ulFileSize > ptr_utFile->ulFilePosition) {           // if the file pointer is at the end of the present file there is nothing to do
        if (!(ptr_utFile->ucFileMode & UTFAT_OPEN_FOR_WRITE)) {          // only allow truncation if the file is open for writing
            return UTFAT_FILE_NOT_WRITEABLE;
        }
        else {
            UTDISK *ptr_utDisk = &utDisks[ptr_utFile->ucDrive];
            DIR_ENTRY_STRUCTURE_FAT32 *ptrFileEntry = (DIR_ENTRY_STRUCTURE_FAT32 *)ptr_utDisk->ptrSectorData; // the directory entry in the sector buffer
            unsigned long ulOriginalSize = ptr_utFile->ulFileSize;
            ptrFileEntry += ptr_utFile->private_disk_location.ucDirectoryEntry; // move to the file entry
            if (fnLoadSector(ptr_utDisk, ptr_utFile->private_disk_location.directory_location.ulSector) != UTFAT_SUCCESS) { // ensure that the directory sector is loaded
                return UTFAT_DISK_READ_ERROR;
            }
            // We set the new file size to be equal to the present file pointer location
            //
            ptr_utFile->ulFileSize = ptr_utFile->ulFilePosition;
            fnSetFileInformation(ptrFileEntry, ptr_utFile->ulFileSize);

            while (utCommitSector(ptr_utDisk, ptr_utFile->private_disk_location.directory_location.ulSector) == CARD_BUSY_WAIT) {} // force writeback to finalise the operation
            ptr_utDisk->usDiskFlags &= ~WRITEBACK_BUFFER_FLAG;               // the disk is up to date with the buffer


            // We need to free all clusters no longer needed by content after the new end of the file
            //
            if (ptr_utFile->public_file_location.ulSector >= ptr_utDisk->ulLogicalBaseAddress) { // don't delete cluster chain if there is none allocated
                return (fnDeleteClusterChain(ptr_utFile->public_file_location.ulSector, ptr_utDisk->ucDriveNumber, 0)); // free up all clusters belonging to the file content
            }
        }
    }
    return UTFAT_SUCCESS;
}

If you are happy with the operation and don't find any errors I will commit it to the next utFAT version.

Good luck

Regards

Mark

P.S. Below is the test command that I added to the command line menu. As you see utSeek() is used to move to the new file location that defines the new file size (you will probably get to the new location with seeks and writes but the important point is that the present location at the call to the new truncation routine defines the new end of file.

Example of its use is "trunc 1000 test_file.txt" - this truncates the existing file test_file.txt to be 1000 bytes long and any following writes are added to the end (unless a seek to a different location is performed first).


Code: [Select]
    case DO_TEST_TRUNCATE:                                               // {59}
        {
            UTFILE utFile;                                               // temporary file object
            unsigned long ulFileLength = fnDecStrHex(ptrInput);
            utFile.ptr_utDirObject = ptr_utDirectory;
           
            while (*ptrInput != ' ') {
                ptrInput++;
            }
            if (utOpenFile(++ptrInput, &utFile, UTFAT_OPEN_FOR_WRITE) != UTFAT_PATH_IS_FILE) { // open an existing file
                fnDebugMsg("File not found\r\n");
            }
            else {
                const UTDISK *ptrDiskInfo = fnGetDiskInfo(DISK_D);
                unsigned long ulOriginalFreeClusters = ptrDiskInfo->utFileInfo.ulFreeClusterCount;
                utSeek(&utFile, ulFileLength, UTFAT_SEEK_SET);           // move the file pointer to the end of the file size that is required
                if (utTruncateFile(&utFile) == UTFAT_SUCCESS) {          // truncate to new length
                    fnDebugMsg("File successfully truncated");
                    ulOriginalFreeClusters = (ptrDiskInfo->utFileInfo.ulFreeClusterCount - ulOriginalFreeClusters);
                    if (ulOriginalFreeClusters != 0) {
                        fnDebugDec(ulOriginalFreeClusters, WITH_SPACE);
                        fnDebugMsg(" clusters freed");
                    }
                    fnDebugMsg("\r\n");
                }
                else {
                    fnDebugMsg("Truncation failed\r\n");
                }
            }
        }
        break;
« Last Edit: October 02, 2013, 02:16:45 AM by mark »

Offline svl

  • Jr. Member
  • **
  • Posts: 52
    • View Profile
Re: How do I reduce the size of a file
« Reply #6 on: October 07, 2013, 03:00:14 PM »
Hi Mark

Thanks for the help the function is working perfect.;-)

Regards

Steen