Back to the main page.
Bug 2076 - For some YOKOGAWA .con file, ft_read_data() could not read all samples
| Status | CLOSED FIXED |
| Reported | 2013-03-22 18:10:00 +0100 |
| Modified | 2019-08-10 12:03:35 +0200 |
| Product: | FieldTrip |
| Component: | yokogawa |
| Version: | unspecified |
| Hardware: | All |
| Operating System: | All |
| Importance: | P3 normal |
| Assigned to: | Tilmann Sander-Thommes |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: |
Paul Chang - 2013-03-22 18:10:33 +0100
dear all, We found an YOKOGAWA .con file that causes ft_read_data() to throw such exception and need you professionals to help. IMHO, it may require the vender YOKOGAWA to join in. The test file that produces this bug can be downloaded from http://xuyierjing.nccu.edu.tw/xuyi/tmp/sub14run29.con It is sized of 180 MB. We use the daily update version of fieldtrip running on Linux (CentOS). Here are the steps to re-produce the problem: -------------------------------------------------------------------- >> hdr = ft_read_header('sub14run29.con'); Warning: adding /usr/local/fieldtrip/daily/external/yokogawa_meg_reader toolbox to your Matlab path >> hdr hdr = orig: [1x1 struct] Fs: 1000 nChans: 256 nSamples: 400000 <=== Note the nSamples in the header nSamplesPre: 0 nTrials: 1 label: {256x1 cell} grad: [1x1 struct] chantype: {256x1 cell} chanunit: {256x1 cell} >> dat = ft_read_data('sub14run29.con'); Error using read_yokogawa_data_new (line 96) could not read all samples <=================== the bug Error in ft_read_data (line 1029) dat = read_yokogawa_data_new(filename, hdr, begsample, endsample, chanindx); -------------------------------------------------------------------- Then I started the Matlab debugging process: 1. set a breakpoint on Line #96 of read_yokogawa_data_new.m 2. when the problem occurs, the number of read samples is only 368402, instead of 400000 as described in the header There are other .con files for this subject, but only this one makes the problem. To ensure that this file is intact and not damaged, I use the Windows PC tool MEG160 (MegLaboratory V2.004A) provided by YOKOGAWA to read this file. At least, I can browse the samples and no problems are reported by MEG160. I suspect that the file sub14run29.con has some specific data patterns that causes read_yokogawa_data_new.m to stop. If it's true, please help us to inform the developers of the Yokogawa MEG Reader library. Many Thanks!
Tilmann Sander-Thommes - 2013-04-12 12:43:03 +0200
Dear Paul, I downloaded the file and got the same error on a SUSE linux machine. But calculating the expected file size and the size of the actual file gives a mismatch: You have 256 channels (that is correct ?) and you say the file has 400000 points. The Yokogawa file format stores 16 bits for each sample and therefore the expected file size is larger than 256*400000*2 = 204800000 Bytes (The 2 is for the 16 bit). The file I downloaded has only 186 MB. I cannot open the file with my MEG160 software as it is an earlier version. Maybe the following has happened: During the measurement the channels are stored on separate machines, the transfer procedure in the Yokogawa software puts the channels into a single file together. If a few channels did not get transferred you get 400000 points, but not all channels explaining the deviation in the file size. This is probably a very rare error and might be reported to Yokogawa, but your data will still be unusable. The file you attached was the only one with this problem ? Best, Till
Paul Chang - 2013-04-12 18:17:51 +0200
(In reply to comment #1) Till, thanks for your efforts and prompt reply. For the new MEG160 utility that can read the new format of .con files, please download from http://xuyierjing.nccu.edu.tw/xuyi/tmp/Meg160new.zip In the beginning, I also suspect whether the bug sample file remains intact during transferring. I have no knowledge about the format of a .con file, so this utility is the only way that I can take to check its integrity. In your comments, it seems that the size of the sample file is incorrect. Please use the new utility to check it again. Our team member has reported the same problem during another experiment. I will let you know when another sample is ready to download.
Tilmann Sander-Thommes - 2013-04-15 09:45:46 +0200
Created attachment 451 screen shot of Yokogawa file
Tilmann Sander-Thommes - 2013-04-15 09:46:41 +0200
Dear Paul, I am in a hurry and therefore quite brief: The file does not contain 400 s of data, see attached screen shot. Something is wrong with the recording mode or the data transfer. I am away until Thursday, best Till
Paul Chang - 2013-04-19 17:22:04 +0200
(In reply to comment #4) The file size of the latest reported sample, which may relate to this problem, is much less than the typical sample size in our experiment. I believe that this file is truncated and damaged. However, when I use MEG160 to read this file, it can still be opened and no problem is reported by MEG160. It seems that MEG160 only checks the header portion, so MEG160 is not a sufficient tool to check the integroty of a .con file. At this moment, I think that our problem is not related to Yokogawa MEG Reader library, so this bug report can be closed. Till, many thanks for your time to explore our problem. Your information about calculating the minimum size of a valid .con file is valuable to us. Since we still have not found the root cause, your experience and further suggestions will always be appreciated.
Tilmann Sander-Thommes - 2013-04-22 12:59:54 +0200
Dear Paul, I suggest you tell Yokogawa about the problem. It is really the first time, that I heard about that. If you want to use the corrupted files with the matlab toolbox then open the file in MEG160. Then use the "Extract" command in the "Edit" menue to extract the existing data. After this command save the data in a new file. The saved file has datapoints and header matching. Since one does not know which part of the data is missing (could be anywhere), check any derived results for consistency. Further check if you have enough space on your recording PCs to hold the data during the measurement. Leftover files might occur after multiple system crashes. But I do not know any details. I close the bug. Best, Till
Tilmann Sander-Thommes - 2013-04-22 13:02:05 +0200
The remaining problem is unrelated to FieldTrip and the supporting toolbox Yokogawa MEG reader. Hardware manufacturer should be informed.