Project

General

Profile

Real time SATA writes

Added by Mary Frantz about 6 years ago

I am having some trouble with a data acquisition system.

Using MDK_2012-08-10

There are two tasks (among others) that perform the data acquisition. The data is streaming in from the mcBSP and is loaded into a common buffer by the DSP.
Rough outline of code:

gSig_record is setup to generate a periodic signal every 10 msec.

void * taskOne(void *pArg)
{
    while(!quit)
    {
    sigwait(&gSig_record, &sig);            // recording timer event

        clock_gettime(1, &start);
        times[idxMark] = start.tv_nsec - last;          // measure interval of gSig_record event
        last = start.tv_nsec;

       // ....code omitted.... 
       // Collect 32Kbytes of data into global buffer

    // send message to taskTwo
    if (gfdThis > 0)
        mq_send(gfdThis, (char *) msg, msgLen, PRIORITY_NORMAL);

        clock_gettime(1, &end);
        samples[idxMark] = n;
        runTime[idxMark++] = end.tv_nsec - start.tv_nsec;   // measure run time of this task
        idxMark &= 31;

    } // endwhile: !quit

    return NULL;
} // end: taskOne()

void taskTwo()
{
    while(!quit)
    {
        // wait for instruction from taskOne
        ret = mq_receive(gfdThis, (char *)msg, sizeof(msg), NULL);

        clock_gettime(1, &start);
        timesRec[idxRec] = start.tv_nsec - last;          // measure the interval between messages
        last = start.tv_nsec;

        // write to SATA
        fwrite(globalBuffer, 1, 32 * 1024, file); 

        // measure how long that took
        clock_gettime(1, &end);
        runTimeRec[idxRec++] = end.tv_nsec - start.tv_nsec;
        idxRec &= 31;

    }

}

taskOne has a priority of 99.
taskTwo has a priority of 98.
taskOne causes taskTwo to run. So, they run sequentially.

taskOne runs in around 1 msec
taskTwo runs in around 5 msec

This works fine for a while, then occasionally (on the order of every 10ish seconds), taskOne gets held off. I calculate the interval at which taskOne runs, and it's 10 msec. But occasionally I see an interval of >30 msec.

With the priority of these tasks being so high, I am guessing it's the kernel that's holding it off. Could this be due to the buffering in fwrite? Should I call fflush() after each fwrite()? or use write() instead?

Which is more efficient for ongoing real time writes to the SATA, write() or fwrite()? I am writing block data 32Kbytes at a time. Does fwrite() involve an extra copy of data? Can the kernal accessing the SATA hold off my real time periodic tasks?

Is there a way to find out what the kernel was doing just before this long interval occurred?

Any insights would be helpful.
Thanks,
Mary


Replies (4)

RE: Real time SATA writes - Added by Mary Frantz over 5 years ago

After further testing and setting optimization, I have narrowed down the delay:

The call to fwrite() normally takes about 2.5 msec, but occasionally (on the order of minutes) this call can take 20 msec, 40 msec, 150 msec, or even 300 msec. This has a ripple effect causing messages to back up in the queue, and then taskOne blocks.

Why does fwrite() take the extra time? The SATA drive is a flash drive and the file system is ext2.

I found a mention of this problem on an ext4 file system: http://blog.empathybox.com/post/35088300798/why-does-fwrite-sometimes-block

Any suggestions?

Mary

RE: Real time SATA writes - Added by Dominic Giambo over 5 years ago

Hi Mary,

As you suspected in your original post, it is likely that when you are calling fwrite, in some cases the write cache is full and is causing full flush down to storage (NAND). In this case it will take longer than just writing to the in-memory filesystem cache. You should be able to test this by calling fflush with the file handle on every write.

If you have hard timing requirements, you may want to control the file write yourself by writing into an in-memory buffer and then periodically fwrite'ing that memory into file and calling fflush at that time. This should be more deterministic in terms of timing.

RE: Real time SATA writes - Added by Mary Frantz about 5 years ago

I have improved tolerance of this issue by increasing the interval in taskOne() to 20 msec, and increasing the message queue size to 128. This runs fine ... mostly. When running very long tests, I get a message queue backup after 48 hours, and 96 hours. That is, 48 and 96 hours after booting. It takes 2.5 seconds to fill the message queue. This means that the file write is delayed by more than 2.5 seconds.

Is there something scheduled in Linux at 48 hour intervals that has to do with the SATA drive? If so, can I disable it?

Could this be the wear leveling algorithm in the SATA? Even so, 2.5 seconds seems like a long time.

Mary

RE: Real time SATA writes - Added by Mary Frantz almost 5 years ago

This problem was resolved by using a SLC SSD drive instead of MLC. The write times on the SLC are much lower.

    (1-4/4)
    Add picture from clipboard (Maximum size: 500 MB)