OpenLog - Logging to card stopping at random times

Hi I’m using an OpenLog to log a data stream from a GNSS receiver. The system is running 24/7 for a month at a time before being powered down to retrieve the logged data.

I have six of these systems running and they are all displaying the same problem as follows:
Logging to the SD card is sometimes stopping at random times (typically days to weeks in to the log) - once it stops it does not start again. The data up until that point is good and the log file is not corrupt.
Occasionally the log will run for the entire month without issue, other times the fault will occur within a few days. Approximate file sizes for the logs range from 30 MB for an early-manifested fault up to 700 MB for a successful log of a complete month.

I’m looking for advice on where to start with troubleshooting this, given the time required to replicate the fault!
My hunch is that this fault is related to buffer overflow or that I haven’t got the baud rate set correctly, but I’d like to make informed changes to the system rather than guess.

The GNSS receiver sends multiple 30-90 byte packets per second and a 500-900 byte packet every 15 seconds. The logs appear to systematically stop part way through receiving one of the large packets, which is always truncated. Power cycling the system clears the fault and a new log file is created. Logging then resumes successfully for the complete month, or for a period of days to weeks, until the fault again manifests itself.

Pre-fault OpenLog LED behavior shows regular incoming data with a large packet of every 15 seconds (blue led, left hand side) and the data being written (green led, right hand side - it’s dimly lit):
Post-fault OpenLog LED behavior shows irregular, more frequent incoming data and no data being written:

The OpenLog is continually powered from the GNSS receiver which itself is powered from the mains via a UPS. There are no other components within the system.

The micro SD I’m using are 16 GB Sandisk Ultra Class 10. Each card is empty apart from the OpenLog config file and was freshly formatted using the SDcard formatter.

Baud rate is set to 9600 at the OpenLog and GNSS receiver.

The OpenLog units were purchased in November 2015. I’m currently waiting on an FTDI cable to confirm the firmware version.


Flashing the ‘minimal’ version of the firmware seems to have resolved the issue.