Logging serial port data to file

[Cathey, Jim]

Normally if the baud rate needs setting, one would
set it together with igncr, and any other settings
that one needed to guarantee, in the same place.
Certainly not relying on those settings being preserved
across a close and re-open, which is what you're doing.
(If you haven't noticed the pattern, each redirection
your script hits represents closing and opening of files.
do...done is a compound statement, and is executed once,
and any I/O redirection on it is only done once, not once
per loop.)

You normally would NOT close and open the log file for
each line if there's any kind of fast rate of input.
That would really churn something like flash for
example.  OTOH, if it's infrequent data and you want
to protect against power failure induced loss, you'd
want to close it every time, or at least periodically.

Only you know the right answer to this, but with the
single redirection on the outside of the loop, the
file is never closed, and relies on the automatic
(once per minute?) system file sync.  If the daily log
file version I alluded to were in place it'd be closed
when the file name changed, and flushed through to the
storage medium on the system file sync timer.

If the echo statement has the output file log redirection
in it, then it's closed after each line of output.  That
would churn JFFS2 pretty hard, though.

Consider:  One approach is faster than the other.  The CPU's
speed isn't changing, so the slowdown represents OTHER WORK
that's avoided in the more efficient case.  What is that
other work, and what are its side-effects?  Case in point,
we had a data logger (to flash) that closed the files
every time.  It could log 13 records per second.  Keeping
the file open all the time moved up to 1000 records per
second possible, though we rarely needed even the 13 per
second unless there was a burst.  That 'lost time', however,
was consumed in flash activity anyway, which was getting
pounded at 13 per second just as hard as if it were the
1000 per second rate with the file kept open.  Flash has
a relatively limited lifetime, you have to be very careful
not to wear it out unnecessarily.  You can reduce the lifetime
of your hardware to a matter of months if you're not careful.
We have!  We only caught it before it was too late because
our flash driver was instrumented for system health purposes,
and we noticed the slowdown of the flash erases as the chips
wore out.

IMHO, you should ALWAYS worry about efficiency.  Then make
the correct decision based upon knowledge of the circumstances.
More efficient is not always better, but you need to know
the costs.  (For example, the 'more efficient' approach of
keeping the log file open is much easier on flash, but requires
more code if your log file varies on a daily basis, and requires
you to decide how much data loss vulnerability you can accept.)

-- Jim




-----Original Message-----
From: Michael Hagedorn [mailto:michael at hagedorn.co.nz] 
Sent: Monday, March 29, 2010 12:47 PM
To: Cathey, Jim
Cc: doug.clapp1970 at alumni.unc.edu; Michael Abbott; busybox at busybox.net
Subject: Re: Logging serial port data to file

Hi

Yes, you are right, I figured that out myself that there should be no
</dev/ttyX on the "done" line.

This is my current script:

# wait a bit (this is in case it is respawning fast)
sleep 10

# Make sure the boudrate is set correctly
stty -F /dev/ttyUSB0 57600

exec </dev/ttyUSB0
stty igncr


while read -r; do
    echo "[$(date)]: $REPLY"
>>/usb/BigDrive/CurrentCost/CurrentCost_$(date +"%Y_%m_%d").log
done



It works fine.

I like your approach as well. Which one do you think uses least resource?
In mine is the .log file opened for writing for every line or is it
kept open until the name changes?
Should I worry about that?

Regards

On 30 March 2010 05:50, Cathey, Jim <jcathey at ciena.com> wrote:
> You have missed the point of the empty exec,
> which is to change I/O assignment of the entire
> executing script.  You don't need (or want) the
> </dev/ttyX on the "done" line anymore.  The
> point of the exec was to switch everything
> to that device so that it wouldn't get closed
> and reopened around the stty.  Some tty settings
> are persistent across close, some are not.  Why
> ask for trouble?
>
> Exec is also your friend on the output side.
> Redirect once, and inside your read loop check
> the day.  When it changes do an exec to change
> the output log file, and continue on to log normally,
> and repeat the read loop.  Pseudo-shell-code:
>
> fileday = date today
> exec <tty >filedayfile
> stty
> while read ; do
>        if day != fileday
>                fileday = date today
>                exec >filedayfile
>        echo line
> done
>
> You're already calling date once inside the loop.
> Put its value in a shell variable and then extract
> the day-of-week from the beginning for the test,
> and use the whole thing for your log.  Won't really
> be any slower, if you're worried about that sort
> of thing.  (Inside the redirection you can call date
> again if that's easier, it'll only happen once per
> day!)
>
> -- Jim
>
>
>
> -----Original Message-----
> From: Michael Hagedorn [mailto:michael at hagedorn.co.nz]
> Sent: Saturday, March 27, 2010 12:01 AM
> To: Cathey, Jim
> Cc: doug.clapp1970 at alumni.unc.edu; Michael Abbott; busybox at busybox.net
> Subject: Re: Logging serial port data to file
>
> Ok, here we go:
>
> I didn't compile the dos2unix command into my busybox, therefore I
> didn't try it yet.
>
> tr great command good to know about, but I didn't get it to work
> correctly, I guess once a line has been read a line will be written.
>
> The
>
> exec </dev/ttyUSB0
> stty igncr
>
> while read -r; do
>    echo "[$(date)]: $REPLY"
> done </dev/ttyUSB0 >>/usb/BigDrive/CurrentCost/CurrentCost.log
>
> works perfect.
> Thanks for your help.
>
> As for the extra while loop to keep my logger alive if the usb2serial
> has been unplugged and wait to be plugged in again, that didn't work.
> I unplugged the usb2serial and plugged it back in. For some reason it
> then showed up on /dev/ttyUSB1 (normally the script would die and the
> usb would show up as /dev/ttyUSB0 again). I guess the script was
> blocking /dev/ttyUSB0 in some way. Do usb2serial devices have
> identifiers? If yes I could iterate through them to find the correct
> one (I want to log multiple usb2serial devices later on).
>
> I will leave that extra loop out for the moment and try the inittab
> option. How fast is the respawning happing?
>
>
> Ok here is one more question. I would like to have each day go into
> its own file, I tried:
>
> done </dev/ttyUSB0 >>/usb/BigDrive/CurrentCost/CurrentCost$(date
> +"%Y_%m_%d_%M").log
>
> That didn't work (I tried it with a %M in there so that I wouldn't
> have to wait for a day). I guessed that that wouldn't work since the
> output is probably once opened and then just continuously streamed to.
> What would be the best way to do this?
>
> Thanks again for your help
>
> Regards.
>
>
> On 26 March 2010 10:15, Cathey, Jim <jcathey at ciena.com> wrote:
>> Things in the /etc/inittab are automatically
>> respawned when they fail.  That's its main job.
>>
>> Your script closes the file between the stty and
>> the read loop.  You are counting on the settings
>> being persistent.  That's why I used the exec in
>> my last example.
>>
>> Alternately, the last "done" could be "done </dev/ttyUSB0",
>> and take out the other input redirections.  What you
>> want is a single program entity that's operating on
>> the intended device.
>>
>> I think your 10-second retry would work, but be sure
>> that the USB device doesn't end up as your controlling
>> terminal, giving your script a SIGHUP when it's disconnected.
>>
>> You need to read up on the TTY driver, a ^D will likely
>> also end the read loop.
>>
>> -- Jim
>>
>>
>>
>> -----Original Message-----
>> From: Michael Hagedorn [mailto:michael at hagedorn.co.nz]
>> Sent: Thursday, March 25, 2010 2:00 PM
>> To: doug.clapp1970 at alumni.unc.edu
>> Cc: Cathey, Jim; Michael Abbott; busybox at busybox.net
>> Subject: Re: Logging serial port data to file
>>
>> Thanks a lot for your input, I will try them tonight and report back
>> which one works the best.
>>
>> Another thing is that if somebody pulls out the usb 2 serial adapter
>> the script fails and exits.
>> I want it to continue or restart if the device has been put back.
>>
>> I have come up with this (but haven't tried it yet):
>>
>> while true; do
>>
>>    # Make sure the boudrate is set correctly
>>    stty -F /dev/ttyUSB0 57600
>>
>>    while read -r; do
>>        echo "[$(date)]: $REPLY"
>>    done </dev/ttyUSB0 >>/usb/BigDrive/CurrentCost/CurrentCost.log
>>
>>    # Try again in 10 seconds
>>    sleep 10
>> done
>>
>>
>> Rather than trying to read from /dev/ttyUSB0 is there a way to check whether
>> it is connected?
>>
>>
>> Regards
>>
>> On 26 March 2010 06:34, Doug Clapp <doug.clapp at triad.rr.com> wrote:
>>>
>>>
>>> Cathey, Jim wrote:
>>>>>
>>>>> Is there an easy way to disregard lines that contain only control
>>>>> characters or filter the text?
>>>>>
>>>>
>>>> Not trivially.  As you've been shown, "tr -d" is your
>>>> buddy.
>>>
>>> Another possibility is to pipe through dos2unix (rather than tr) to remove the CR characters. Ultimately you have to decide which method works best for you.
>>>
>>> Doug Clapp
>>
>>
>
>