turning off FS

[twoerner]

platform: M5223x
app: uTasker V1.3 sample app with service packs 1-4

This might sound crazy, but I was wondering if it is possible to turn off (or disable) the uTasker file system in config.h yet still continue to use the HTTP server? In essence all I want is to use the HTTP-POST method to upload a new non-uTasker application to flash.

I replaced the 404 text #define (in config.h) with the POST section of 5admin.htm. Therefore I don't need "files" to serve up the HTML pages. This means I can diasable FTP (to save more space). I don't plan on saving the data that is uploaded with the POST as a "file". So I don't need "files" to serve, and I don't need to save uploaded data as a "file".

From the looks of it I'll need to hack the HTTP.c code since it seems to assume the filesystem is available, but if the above makes sense to you then you'll see that I don't really need files. Any "gotchas" I should consider?

[ps. I can't determine if this should be in this forum or the M5223x forum]

[mark]

Hi

I think that this is the correct place since the question is not that platform specific. It is just as valid for an ARM project for example.

I understand that you would like to use the web server just to allow posting software. In this case replacing the 404 error page with a post formula is reasonable trick - note that most chips can be used with the define "SUPPORT_INTERNAL_HTML_FILES" which enables the web pages to be positioned in code rather than be uploaded via FTP (this is not possible with the NE64 project since it uses banked memory and one bank especially for files where HTTP always wants to access in the bank rather than use linear memory).

You are correct that normally the HTTP server works quite closely with the file system, which is generally logical since most HTTP use is to do with file reads and writes (mainly reads...). However it is also possible to use the option "SUPPORT_HTTP_POST_TO_APPLICATION". In this case the data which is received by the post is not actually saved to a file but rather passed on to the application. Your fnHandleWeb() call back can handle the event "POSTING_DATA_TO_APP", which is passed with a pointer to the data in the frame and the length of the data.
Then you are free to do anything you want with the received data - typical application are to send it to another port or write it to a device like an FPGA, but of course if you have routines that write it to your required location in FLASH this is probably what you need.

The only possible catches I see are these:
1. the parameter system (managing application parameters and IP config etc.) is realised as part of the file system code so completely removing the file system may cause some difficulty there - depending on whether you need the parameter support or not.
2. You may find it easier to keep the file system but replace the routines (or some of them with dummy calls to ensure that all compiles and links). I am not sure though.
3. You may find that you need a certain amount of code from the file system simply for saving to FLASH (you can also look in the boot loader code since this has a minimum solution for FLASH operations) and I am wondering whether you may find it easier with only quite minimum code size requirement to still use the file system in the project. According to code size comparison in the M5223X tutorial the file system code occupies about 2,3k FLASH. This includes also the low level FLASH driver routines for the M5223X so potential savings will be somewhat less.

Take a look at SUPPORT_HTTP_POST_TO_APPLICATION support. I would run the uTasker simulator and check that your HTTP post data is really passed to call back - check with data sending several frames to ensure that the file system never kicks in somewhere. When the application detects the end of the file you may like to have a second small web page in the program code which it then serves. See how fnHandleWeb() handles the end of a normal post to the file system - it receives the event "INFORM_POST_SUCCESS" and then specifies the web side to be displayed out of program code by returning "DISPLAY_INTERNAL" .

If you are lucky you may find that the HTTP code itself doen't require any manipulation to achieve what you would like.

Good luck

Regards

Mark

[twoerner]

Hi Mark, thanks for your reply.

I've re-enabled the filesystem and am trying to get the SUPPORT_HTTP_POST_TO_APPLICATION / POSTING_DATA_TO_APP mechanism working. Unfortunately, every time the application comes to http.c -> fnPostFrame() -> case HTTP_STATE_POSTING_DATA, HTTP_session->ptrFile is always non-zero so the fnHandleWeb (POSTING_DATA_TO_APP...) function is never called and uFileWrite() is always called instead. As a very simple work-around I've just removed the if-statement which checks if HTTP_session->ptrFile is zero or not. But I'm guessing you probably had something in mind when you added that code, any idea what I'm doing wrong?

[mark]

Hi

Best continue with this change so that you can ensure that all otherwise works the way you want to use it.

I have used this for writing posted data to a BDM in the past and will check out the details about the pointer - I'll report back as soon as I have details.

Regards

Mark

[mark]

Hi

Now I have the details.

1. When posting, also make sure that you know what type of data is to be posted. Images, Text, binary etc. The single or multiple file types should each be defined (SUPPORT_POST_BINARY and/or SUPPORT_POST_TEXT and/or SUPPORT_POST_GIF). Unrecognised file types will be received without any errors but the contents will be dumped.
This is not a problem that you have since you already get further than this.

2. When the post begins, the call back event CAN_POST_BEGIN occurs, including details about the socket receiving the post. The details include a pointer to the file location where the file WILL be stored.
Your call back rountine (fnHandleWeb() in the demo SW) can check the file location if it wants and also change it - diverting the data elsewhere. In your case you can choose not to save it to a file by setting the file pointer to zero:
 

Code: [Select]

HTTP_session->ptrFile = 0;  // we decide not to save this to a file but handle it ourselves
Note that you can also decide to refuse the post by returning any non-zero value.

3. For each frame you receive you will need to handle the event POSTING_DATA_TO_APP (I think that you probably already have seen this).

4. After reception of all of the data the event INFORM_POST_SUCCESS or INFORM_POST_FAILED (only when serious error) will be received. You then know that the data has been completely received and you can then cause a different web page (or default) to be served - see the uTasker demo which already does this.

So I think that you only need to clear the ptrFile field in the call back and this will allow the post to operate as you require, thus without modifying the HTTP code.

Regards

Mark

[twoerner]

Hi Mark, awesome replies.

1. When posting, also make sure that you know what type of data is to be posted. Images, Text, binary etc. The single or multiple file types should each be defined (SUPPORT_POST_BINARY and/or SUPPORT_POST_TEXT and/or SUPPORT_POST_GIF). Unrecognised file types will be received without any errors but the contents will be dumped.
This is not a problem that you have since you already get further than this.

While trying to understand how the sample application works, I stumbled across this part of the code. I also discovered that the code is a bit too restrictive. For kicks I was fumbling around trying to upload an HTML file as part of the POST and it just wasn't working, when I dug around I noticed that your text MIME handling only accepts "text/plain" files and not, for example "text/html". So I edited the code to look like the following:

Code: [Select]

#ifdef SUPPORT_POST_TEXT
                    if (!(uMemcmp("text/", ucIp_Data, 5))) {
                        unsigned char *ptr;
                        unsigned myTmp, diff;
                       
                        ptr = ucIp_Data;

                        // find first '\n'
                        for (myTmp=0; myTmp<32; ++myTmp) {
                            ++ptr;
                            if (*ptr == '\n')
                                break;
                        }
                        if (myTmp == 32)
                            break;

                        // find second '\n'
                        for (myTmp=0; myTmp<32; ++myTmp) {
                            ++ptr;
                            if (*ptr == '\n')
                                break;
                        }
                        if (myTmp == 32)
                            break;

                        diff = (unsigned)ptr - (unsigned)ucIp_Data;
                        ucIp_Data += diff;
                        usPortLen -= diff;
                        HTTP_session->FileLength -= (diff + 2);
                        HTTP_session->FileLength -= HTTP_session->ucBoundaryLength;
                        HTTP_session->ucState = HTTP_STATE_POSTING_DATA;
                        break;
                    }
#endif

Let me be the first, of course, to admit that's not the most elegant code. But it was just a quickie doodle for education purposes. Now the "text" MIME handling will accept any "text/..." type.

Code: [Select]

HTTP_session->ptrFile = 0;  // we decide not to save this to a file but handle it ourselves

Your suggestion to add that one line to "webinterface.c -> fnHandleWeb() -> case CAN_POST_BEGIN" works perfectly! What I've slowly come to realize is when I've modified the code and want to test it, it seems that I need to use the flash tool to erase then program the board, THEN I can press debug in the IDE. If I don't re-flash the device before pressing "debug" there seems to be some sort of mixup between what I wrote and what's running. In other words, it turns out my first set of modifications work fine, but due to some flash mixup it didn't appear to be working. But I like your solution better.

For kicks I can upload html or plain text files using POST then view them by asking for the "H" file ("H" by itself gives a  binary, "H.htm" returns it in a way the browser can display). I assume something in the sample uTasker program is changing the sent MIME type based on the requested filename?

Is it correct to further assume that it is the browser that sets the MIME type for POST uploads?

[mark]

Yep - the post only handles predefined file types and to be honest I have never uploaded html files using the post method (that is the only reason why it is not there).

(although I could image using a default post page - like you use the 404 error page - to allow a user to upload all web pages in a project. This would allow the FTP server to be completely removed in favour of working purely with a browser. This would also remove some contraints when working with FTP whereas it often needs special firewall setting because people/companies are sometimes very restrictive with FTP ports.)

Generally it is possible to add further post types or, as you have done, make the text based type less restrictive. I will have a think and maybe add more text types or a general 'catch all' text type. In any case you seem to be having no real problem with the code and as you have seen it is not that difficult to add some more flexibilty if needed in an emergency.

For this type of work I never in fact work on the target - the simulator lets you test and develop much faster and comfortably. But I am assuming that you are using the Coldfire on an eval or your own board, and debugging is generally performed out fo FLASH (there is only 32k SRAM and this is usually too restrictive to share with a program and variables and decent LAN buffer space - RAM debugging is only possible with mini-programs. For Ethernet projects any spare RAM is best used for LAN buffers since here the rule is simply 'the bigger the better'). When working with Codewarrior, pressing the debug button will simply connect to the device in debug mode and doesn't mean that the FLASH content corresponds with the latest compiled project state. Therefore it is always necessary to first synchronise the FLASH contents using the FLASH utility tool.

Although this works well, you still have a limited amount of breakpoints (three I think) and the FLASHing time adds up throughout the day. This is another reason why it is best to do as much work with the simulator (no download times, very fast compiling, great debugging) as possible. Based on our own experiences over the last few years (we also had a forerunner of the simulator before the uTasker project) it is possible to cut many embedded project development times by around 60% by simply using the simulator consequently and then working on the target for verification and low level details.

As far as I understand, it is the browser which recognises the file type when performing the post. I don't actually understand what criteria they use and I have in the past simply looked at the content infomation to learn whether a PDF is for example of binary or other type.

I don't know whether you have already noticed but the port method used is a little special...
You will find that the uTasker project is a bit 'alternative' and sometimes does some little tricks to enable useful things to be performed without having to go a long way to do it.

Code: [Select]

<form action=HS.bin
enctype="multipart/form-data" method="post">
<p>
<input type="file" name="datafile" size="30" £DN0></p>
<p><input type="submit" value="Upload new software" £DN0></p>
</form>
You obviously know this extract from the post HTML.
If you have already learned a bit about the way the dynamic web serving is operating you will know that the £DN0 sequence means "disable the form field if the project doesn't have a large enough file system for a software upload". I am sure that much more detail of such things will be posted soon in a thread of this topic.

The form action=HS.bin is interpreted by the uTasker web server as "post the file which will follow to the file system and name it H.bin. Once the post has completed, serve the web page S.html". Of course the application can intervene and do it differently, but if it can also let the web server part do as it is told. This is practical because it allows the html file to define where posted data ends up in the file system and also what page to serve on success, and is very practical for the software upload in the demo project because the user can upload new software with any name he/she likes and it still ends up in exactly the place in FLASH that you want it to end up at. Afterwards, the web server serves a nice page thanking him/her for doing its and informing that the board will now be reset to complete the update. All will almost zero effort at the application layer!

Possibly this explains what you have been seeing with the different test uploads. If you post an html but set the post up to save it as a bin, it will display as a bin type in the file system (eg. look with FTP). Only files with HTM(L) file types will in fact be parsed by the the web server - this is because if every thing is parsed, including images, any occurance of a special sequence (as used for controling dynamic infomation) in the images would be interpreted and then cause corruption of the image...

Regards

Mark