Application using FAT Emulation

[mark]

I'm going to work on trying to get an emulated FAT working next, with the full speed mode, so that I can get the MSD requests into my code, and then I'll have to figure out what kind of backside transfer protocol I'm going to use, to transfer the data to the pi, and back.  I'm still willing to do more testing, if I can help you with the HS side of things, so just let me know if there's something you want me to try.  I've also got another idea on how to maybe help the process along, I'll shoot you a private message about it, in a bit :-)

There is a guide at http://www.utasker.com/docs/uTasker/uTaskerEmulatedFAT.pdf

Mark,

Well, I've spent tonight working my way through the emulated fat tutorial, and poking around in the code that supports it.  As a result, I have another question for you.  :-)

From what I can see in the code, it appears that the emulated drive is entirely setup in memory, as arrays, when the program is initialized, and then those data structures are used to answer the USB MSD requests from the host.

I've been looking in the code, trying to figure out how, if it's possible, I would go about doing things dynamically.  i.e. ideally I would want to be able to proxy requests for a specific file across my back side interface to the Raspberry Pi, so that I could then lookup the given path on the Pi's USB hard drive, and return the appropriate file.

I am not seeing a way to implement this within your current framework though, and I am wondering if I am on the wrong path here.  Would it maybe be better for me to try to proxy the actual MSD requests across the back interface, and then attempt to implement those on the Pi side of the fence?  Or is there something I'm missing in your emulated FAT layer, that would allow me to do what I want?  Or maybe I could write a replacement for the emulated FAT layer that would allow me to do what I want?

If you were trying to implement my project, knowing what you know about your codebase, how would you go about attacking it?

Thanks in advance for any advice you can give me,

Chris

Chris

The emulated FAT uses const FAT content but not const data content. The data content can be taken from anywhere - also dynamically.

The FAT itself can't be dynamic because it is read in by the host on USB connection (that is, the host then has a local copy of the FAT and so knows how many files are available, their file objects and such. If this were to dynamically changed the host would not know about it unless the USB were enumerated again to synchronise it.

This means that your files (location, size, names etc.) must be fixed at enumeration. The content can be defined/generated when the host requests it.

The restriction of USB-MSD is that only the host may write. If the device writes (or changes things) after enumeration the host knows nothing about it because this is not something expected with USB-MSD and it can result in corruption if both sides could change things. There is no 'synchronisation' mechanism where the device informs the host that it has changed something apart from doing a complete new enumeration (disconnect/connect).

Regards

Mark

Mark,

Quote from: mark on April 21, 2017, 09:02:54 AM

    The FAT itself can't be dynamic because it is read in by the host on USB connection (that is, the host then has a local copy of the FAT and so knows how many files are available, their file objects and such. If this were to dynamically changed the host would not know about it unless the USB were enumerated again to synchronise it.

    This means that your files (location, size, names etc.) must be fixed at enumeration. The content can be defined/generated when the host requests it.


I was aware of the writing restriction, and I think I wasn't clear in what I want to do before.  I understand the FAT must be static once the USB host has connected to the device, and that basically the host then owns the writing to the device.  What I meant by dynamic is this:

I'm going to be using a USB external hard drive, plugged in to my Raspberry Pi, as a media storage drive.  This drive's contents are read-only and fixed for any given running session of the system.  (The contents on the drive will change from time to time, as I add/remove media files, but this will be done while the drive is disconnected from the car computer system, and is connected to my PC.)  So, while the Pi, and the Teensy are connected to the drive, it is read-only, and the FAT is const.

The only other consumer of the drive, will be the Raspberry Pi, but it will also interact with it in a strictly read-only fashion (i.e. streaming the same media content via a wifi connection to my kids tablets/phones etc).

The problem I was trying to figure out a design to, within uTasker, is this:  currently your emulated FAT code, has a hard coded array, down in the mass storage code, stored in RAM, with hard coded file names and sizes.  I am going to have to build this FAT structure dynamically, when the device starts, by communicating with the Raspberry Pi, and enumerating the files on the USB drive.  I'm not yet sure if I can do this in RAM, or if I will have to store it on the SD Card, as the FAT will probably be a decent size, because the drive has hundreds of folders, and thousands of files (currently about 30 GB of files, but will expand in the future I'm sure), all using long filenames.

So, I am trying to determine within your code, what functions/callbacks I have to hook, to pull this off.  From what I've seen so far, I think I'm going to have to replace your const data, with function calls back to my code, that will communicate with the Pi, and return the results.  Maybe I just need an initial function to build the FAT structure from the Pi, and then store it in RAM, or on the SDCard, and then use that to serve the USB.  I'm really not sure yet, I was just asking if you had any pointers on which would be the easiest route to attempt, or how you would attempt to tackle a project like this, given your codes architecture.  :-)

Thanks,

Chris

Chris

I haven't used the FAT emulation for some time so took a quick look at it to refresh my memory.
In fact I don't think that it is fixed in any way, as long as the define MAXIMUM_DATA_FILES is set to be adequate for your maximum requirement. mass_storage.c is not responsible for the files, just for managing the interface between the user code and the FAT emulation.

It is the application code that defines how many files there are, how they should be named, their length, etc. and later for passing data to be read from them when the host reads them. The application code is free to do this in any way it wants - the simplest being a const method which is fixed for the product or else dynamically, based on whatever needs to control it.

In case of the reference it is the application that prepares the file list (see fnPrepareEmulatedFAT() in application.c). This has a fixed set in the reference but if you supply your own version of this routine (which is the idea) you can generate a list based on the present SD card content - you will just need to delay the start the USB task till after the SD card is ready and you have collected the list for insertion.

The functions that you need to support are:

•     fnGetDataFile() to inform the mass storage task of the files that it wants to be 'visible'
•     uDatacopy() to pass data from these files to the mass storage task when it is feeding the USB-MSD with file content.

These functions may be used as they are (probably possible in most cases) or can be modified to do whatever else they need - like generating random file content or encrypting it, etc.

Therefore I don't in fact think that there are any restrictions; you just need to insert your own file list.

Regards

Mark

P.S. If you haven't done it before, set a break point in fnPrepareEmulatedFAT() to see the application setting a fixed list of files - imagine now how you could insert a dynamic list instead.
Then set a break point in fnGetDataFile() to see how the application informs the mass storage task of the list to be used. See whether that is already adequate once you have your file list ready, or else imagine how you could do this call back in a different way to suite your requirement. This is called for each file, until you terminate the list by returning 0.
Of course, do this in the simulator since it is called on each start.

[creitzel]

Mark,

In fact I don't think that it is fixed in any way, as long as the define MAXIMUM_DATA_FILES is set to be adequate for your maximum requirement.

This define, and EMULATED_FAT_DISK_SIZE, were part of what was throwing me off I think, and was making me think that your code was setting up constant values. 

I've since figured out that EMULATED_FAT_DISK_SIZE just specifies the max size on the disk, so for my project, if I just hard code it to 32GB, I should be fine I think.  I tested this today, and it worked fine.

so MAXIMUM_DATA_FILES represents the maximum number of files that may be present in a given directory?  or on the disk, across all directories and subdirectories?

Obviously, I can't set this to 8, as I'm going to have more files than that, and the number of files I have will be unknown until I actually query the Raspberry Pi, and find out what the file count currently is for the given run session.  Could I convert this into a call back function, and have that then query the Pi, and return the count?  I know this value is used in a couple places to allocate array entries and I don't want to allocate a bunch of entries if they aren't going to be used either.  This is currently what I'm working on understanding in your code.  I set it down for a bit today, to sidebar into what I will do to expand the max disk size later.

P.S. If you haven't done it before, set a break point in fnPrepareEmulatedFAT() to see the application setting a fixed list of files - imagine now how you could insert a dynamic list instead.
Then set a break point in fnGetDataFile() to see how the application informs the mass storage task of the list to be used. See whether that is already adequate once you have your file list ready, or else imagine how you could do this call back in a different way to suite your requirement. This is called for each file, until you terminate the list by returning 0.
Of course, do this in the simulator since it is called on each start.

That is in fact what I have been doing today, debugging from those 2 places in the simulator, and playing with different file systems on thumb drives to determine exactly what my stereo recognizes.  Glad to hear I'm not totally nuts, and I was following your code correctly.  :-)

I'm also looking into what it might take to support the NTFS file system in an emulated fashion within your architecture.  I'm currently getting close to the 32GB volume size restriction of FAT 32 with my media library, and I'd eventually like to remove that restriction.  That's not for the current iteration of the project though, as I just want to get it working with my current library first.  It also may be a moot point, because I've been reading that my stereo starts to flake out once you cross the 32GB boundary anyways.


As usual, thanks a ton for the great info,

Chris

[creitzel]

I'm also looking into what it might take to support the NTFS file system

So, it appears that my stereo can't handle NTFS drives either.  So, it's either go with FAT32, or exFAT.  It can handle both of those.  So I'm gonna table this for now, and maybe later, I'll look at implementing exFAT within your framework.  Back to trying to define the file system dynamically.  :-)

[creitzel]

Mark,

Ok, I just finished debugging a bit more, and I think I've learned a few things, and I want to validate them with you.

1.  It appears that your emulated FAT is setup to publish a single directory, with MAXIMUM_DATA_FILES as the count of files in the directory.  Am I right that there is no support for sub directories?

This isn't a major deal if it doesn't support subdirectories, but it would be nice to represent the file structure as it is on the usb disk.  If I can't publish the real file structure, I can build some kind of file map to map from a filename in the MSD directory to an actual path on the Pi I guess.

2.  It also appears that for every file being represented in the emulated directory, your code in mass_storage.c is keeping a struct of size 28 bytes, in the dataFileList array, and another struct in the root_files array, of size 32 bytes.

I'm worried about these arrays growing too large, and consuming all of the teensy's memory.  I'm currently at about 6600 files across my entire disk, and that puts the memory requirements for the first array at 184,000 bytes, and the second array at 211,200 bytes.   

The Teensy 3.6 only has 256,000 bytes of RAM, obviously, these 2 arrays alone would blow memory.

It appears that these arrays are being used to calculate the sector/cluster information to return thru the MSD interface.  I'm thinking I may need to store this information on the local SD card instead of in RAM, for the amount of files I'm dealing with.  What do you think?

Am I on the right track here?  Or am I thinking about this wrong?  I have not actually located your memory allocation for these arrays, are they actually in ram?  Or are they in flash, like you were saying before?  If they're in flash, it might actually be better for me, as the Teensy 3.6 has 1 MB of flash.

It's been quite a few years since I've worked in C/C++, so I'm still getting used to the different memory allocations and such. (I've been working in Delphi Object Pascal for the past 20 years lol)

Thanks,

Chris

[creitzel]

ok, I reread the emulated FAT example, and came across the section stating that you can remove the ROOT_DIR_SECTORS define, to stop the code from creating one of the in memory arrays (root_files I think). 

When I try this though, it appears that not all of the code that should be wrapped with that define, is, because I get a couple errors.  Specifically, the following line:

Code: [Select]

        unsigned long ulFAT_cluster = (ROOT_DIR_SECTORS + 2);

found at line 7339 of mass_storage.c, and this line:

Code: [Select]

                                    ulFAT_cluster -= (ROOT_DIR_SECTORS + 2);

found at line 7381 of the same file.

both of these lines are in the function fnGenerateFAT.

not sure what they should be replaced with, if the define is turned off.

I also tried setting it to 0, which I didn't think was correct (the example says to remove it, not make it zero).  That leads to other errors though.

Chris

[mark]

Chris

The emulated FAT is flat in structure - its main idea is to allow files that are not really files (eg. dynamically generated or formatted data) to be presented to the USB-MSD host so that it can read it as if it were a file.

In order to do this each virtual file requires a bit of information so that its details can be returned and its content generated when requested. At some point there will not be enough memory to do this for a large number of files.

In your case, when you have an SD card attached, I don't see exactly why you need to emulate its content when it can be so large. Why doesn't the USB-MSD directly access it?

Regards

Mark

[creitzel]

The emulated FAT is flat in structure - its main idea is to allow files that are not really files (eg. dynamically generated or formatted data) to be presented to the USB-MSD host so that it can read it as if it were a file.

In order to do this each virtual file requires a bit of information so that its details can be returned and its content generated when requested. At some point there will not be enough memory to do this for a large number of files.

Yeah, the more I study the code, the more I was coming to this conclusion myself.  It's great for the purpose you intended it for, but I'm thinking it may not be the right fit for my use.  Which is fine, it just means I'll have to write a bit more code.  :-)

In your case, when you have an SD card attached, I don't see exactly why you need to emulate its content when it can be so large. Why doesn't the USB-MSD directly access it?

In my case, the SD card isn't really attached to the Teensy.  I've just been testing with a SD card in the Teensy, to get the USB MSD side of things ironed out, and working. 

The actual media content is stored on a USB hard drive, that is connected to my Raspberry Pi, it needs to be this way, to satisfy other requirements of my project (i.e. I am serving the same media content up via Plex media server, to my kids and their devices, directly from the Raspberry Pi). 

So, what I am aiming to do with the Teensy, is build a sort of media bridge, that can be connected to my car stereo on one end (via USB mass storage device, as that is the only input I have for the stereo, and I want to retain my stereo controls over the media, i.e. steering wheel controls to advance to the next song etc), and then on the other end, I was intending to use GPIO from the Teensy to the Raspberry Pi, to access the same media content that is already available over there, from the USB hard drive. 

At this point, I'm leaning towards modifying your code, and hooking into the calls between the USB MSD layer, and the FAT layer, and conveying those across the GPIO, to the Pi, where I can write a separate FAT emulation layer, to republish the hard drive's contents.  Or, if it's possible, maybe I can get low-level access to the drive, and just proxy the MSD requests back and forth.  I've got to do some more research on the Pi side of things, and see what Linux will let me do with the attached drive.  Either way, I'm pretty sure I'm going to do the FAT emulation on the Pi, because it has 1 GB of ram, and I have a bit more breathing room over there.  :-)

As usual, thanks for the great info,

Chris

[mark]

Chris

If you can get low level access to the SD card via the Pi you will "effectively" be connected directly to the SD card. This means that you can use USB-MSD directly - but rather than reading an SD card connected via SPI you read the same SD card via a Pi interface. This would remove any need for emulation - making the code much easier and removing the need to build up any tables of the SD card content in order for the emulation to be able to operate.

In fact I think that you need a method to read low level data from the files anyway since any emulation would still need to be able to request the SD card content to be able to return any file content. Since you must have this you can just as easily use it to also read the FAT content - no emulation complications at all.....

Therefore I think that what you actually need is a Teensy > Pi - SD card bridge which allows you to read a block of data from an SD card sector at the SD card connected to the Pi. This is called instead of a local SD card sector read (vai SPI) and you are already done.

Regards

Mark

[creitzel]

Mark,

If you can get low level access to the SD card via the Pi you will "effectively" be connected directly to the SD card. This means that you can use USB-MSD directly - but rather than reading an SD card connected via SPI you read the same SD card via a Pi interface. This would remove any need for emulation - making the code much easier and removing the need to build up any tables of the SD card content in order for the emulation to be able to operate.

In fact I think that you need a method to read low level data from the files anyway since any emulation would still need to be able to request the SD card content to be able to return any file content. Since you must have this you can just as easily use it to also read the FAT content - no emulation complications at all.....

Therefore I think that what you actually need is a Teensy > Pi - SD card bridge which allows you to read a block of data from an SD card sector at the SD card connected to the Pi. This is called instead of a local SD card sector read (vai SPI) and you are already done.

Great minds think alike lol.

This is basically where I landed as well.  I am planning on taking the calls coming into the uTasker MSD handling code (so basically raw sector read requests), and proxying those over to the Pi to be executed against the USB hard drive, in some code that I write.  I was leaning towards a custom byte-wide GPIO interface to communicate between the Teensy and the Pi. 

I've spent most of today, analyzing your code, and identifying where to plug in my calls to the Pi, and I believe I have a good handle on it.  So far, I've started out by emulating what you did with FAT_EMULATION, by providing a new define, and some new functions for reading sectors and partial sectors from the pi.  Oh, and I also copied your Prepare function, and made my own, as I think this is going to be needed when all is said and done.  I've also set it up, so that it basically registers itself as an emulated drive in your drive info structs, similar to what you did for the fat emulation.  Off hand, is there anything you can think of that I'm missing?

I also have to research whether Linux will let me access a mounted USB drive in that fashion (i.e. read raw blocks from it), and if it will, I should be all set I think.  :-)

Oh, and one more thing, do you have any suggestions as to how I could emulate the host calling in to read the MSD while in the simulator?  It would make this a lot easier to develop if I could do that.  Maybe I could somehow grab the calls being sent from Windows when it reads a MSD and then set up a script to send that binary data to the simulator or something?

Thanks again for all your help on this,

Chris

[creitzel]

Mark,

Ok, I have spent the time since my last post, researching on the Pi, about getting low level access to the usb hard drive, and it is quite doable.  In fact, I've already written a set of helper functions that will allow me to serve up the capacity of the drive (i.e. Sector count and Sector size), and will also allow me to read raw blocks from the drive, so they can be returned to the Teensy.

Now that that is done, I was planning on moving over to the Teensy side, and continuing work on my code there to plug into your USB MSD layer. 

In the process, I figured I would start by writing my functions to emulate the low level calls the pi is going to make, by reading a disk image file stored on the SD Card, and returning blocks from it to your MSD code, to ensure that my functions are hooked into your code correctly, before moving on to write my interface to the Pi. 

I wanted to do this in the simulator, so that it would be easier to develop, and I'm having a couple issues with it.  I have read through a few tutorials, and see references to connecting to the simulator via telnet to access the menu, and be able to format the simulated sd card, but it doesn't appear that the simulator is publishing anything on the ip I set up.  I looked at my config.h file, and noticed that ETH_INTERFACE define isn't being defined for the teensy 3.6.  I tried defining this my self, but when I do, the project won't build, I get some errors in kinetis_enet.h, about PHY_ADDRESS being an undeclared identifier.

Given that background, I've got a couple questions for ya.  :-)

Is it not possible to connect to the simulator via telnet while using the teensy 3.6? 

If it is possible, what all do I need to configure to make it work? 

If it isn't possible, how else can I connect to the simulator to get access to the menu interface?

Thanks for any help you can give me,

Chris

[mark]

Chris

The Teensy 3.6 HW doesn't support Ethernet and that is why there is no Ethernet configuration (requiring also setting up an external PHY) for its configuration.

However the same menu is available on one of its UARTS so there is no need to add Ethernet and Telnet.
It is also available on the FS USB's CDC interface on the actual HW if CDC is enabled (it can also be enabled as composite together with MSD). The CDC interface is however not practical for the simulation so UART is the method to use.

When the simulation runs you can see the UART used and its connection to COM ports on the PC (see attached screen-shot).
In app_hw_kinetis.h you can define the COM port to be mapped to each UART:
#define SERIAL_PORT_2    6                                           // if we open UART channel 2 we simulate using comx on the PC
I have COM0COM on the PC which creates a COM6 to COM7 loop-back so when I open a terminal emulator on COM 7 it talks to the Teensy 3.6's command line interface.

Search for com0com on the forum for some discussions about its use - it is perfect for such applications.

Regards

Mark

[creitzel]

Mark,

I eventually stumbled on a forum post that led me down the UART path, using com0com, and I was able to connect to the simulator, and use the menu to format the sd card.  Sorry for not posting sooner, to save you the time invested in that last post.  :-)

Now that I have that set up, I am trying to get a file onto the simulated SD card, so that I can use it in my debugging tests, but I am not seeing any way to transfer the file to the simulated SD card via the menu.  Is this something that is supported?  I initially was thinking of using telnet for the menu, and ftp to get the file uploaded to it, but I understand why that isn't possible.  if I can't copy the file up using the menu, is there some other way to transfer the file to the simulated SD card?  If not, I might look into modifying your simulator to add a function to copy it up.

Oh, and I've been meaning to tell you, but I kept forgetting, I might be able to do some high speed USB testing for you at some point in the future.  I have an EE friend who is going to use a Teensy 3.6 for a project he is working on, and he and I are going to build a debug/test board for the teensy 3.6, and we are going to wire up a usb connector to the high speed interface, so once I have that in hand, I can probably run whatever tests you want me to.  :-)  I'll let ya know when I've got my hands on it.   

Thanks,

Chris

[creitzel]

Oh, and I've been meaning to tell you, but I kept forgetting, I might be able to do some high speed USB testing for you at some point in the future.  I have an EE friend who is going to use a Teensy 3.6 for a project he is working on, and he and I are going to build a debug/test board for the teensy 3.6, and we are going to wire up a usb connector to the high speed interface, so once I have that in hand, I can probably run whatever tests you want me to.  :-)  I'll let ya know when I've got my hands on it.   

Ok, scratch that, we just finished figuring out how much it would cost us to build a development/debug rig for the teensy, and it's like $90 for each of us, so we decided to just buy the dev board you recommended (FRDM-K66F).  Got one ordered and on the way.  That board should still allow me to do some testing for you though, right? 

[mark]

Chris

What I usually do is use FTP to load files to the emulated card (you can use any target with Ethernet to do it). Then the SDCARD file in the simulator directly can also be copied to other projects (or one can keep a set of test SD cards like this which can be simply copied into the director when needed).

Regards

Mark

[creitzel]

Mark,

I attempted to build the code for a different device (one that supports Ethernet), in fact, I tried all the devices I could see that supported Ethernet, and I was unable to get them to work. 

I ran into one of 2 problems, the first problem I encountered with about half of the devices, was that the code wouldn't build.  Some code down in the ethernet files wouldn't compile. 

The 2nd problem I encountered was with the other half of the devices, and in that use case, it would build fine, but when I ran it, it would throw an access violation, when the simulator first started. 

Unfortunately, I was busy this weekend, and didn't get a chance to go after either of the two problems.

It may be a moot point now, as my development board should be here sometime tomorrow, or wednesday, and I will be able to debug on the chip at that point.

Chris