Back to the main page.
Bug 1825 - read_plexon_nex and plexon_nex in ft_read_data
| Status | ASSIGNED |
| Reported | 2012-11-08 03:42:00 +0100 |
| Modified | 2016-09-13 12:26:40 +0200 |
| Product: | FieldTrip |
| Component: | fileio |
| Version: | unspecified |
| Hardware: | PC |
| Operating System: | All |
| Importance: | P3 normal |
| Assigned to: | Robert Oostenveld |
| URL: | |
| Tags: | |
| Depends on: | |
| Blocks: | |
| See also: | http://bugzilla.fieldtriptoolbox.org/show_bug.cgi?id=2093 |
Martin Vinck - 2012-11-08 03:42:05 +0100
line 912 dat(i,:) = nex.dat; can give an error when chanbegsmp<1. as the preallocation dat = zeros(length(chanindx), endsample-begsample+1); conflicts with the size of nex.dat given the existing code. I am reading in Pascal's old NIH data and according to read_plexon_nex sometimes the first ts of the LFP data lies about 100 ms after the onset of the recording according to read_plexon_nex. This is when the error arises.
Jan-Mathijs Schoffelen - 2012-11-08 09:36:41 +0100
What would be your solution?
Martin Vinck - 2012-11-08 09:40:56 +0100
I guess, nex.dat = [nan(1,offset) nex.dat]; could be simply changed where the length of the NaNs is set correctly. But let's hear Robert on this issue as he programmed this set of readout functions, I think.
Jan-Mathijs Schoffelen - 2012-11-08 09:49:39 +0100
but why is chanbegsmp<1 to begin with?
Martin Vinck - 2012-11-08 09:53:37 +0100
I am not quite sure, JM, what the meaning and function of this offset here is. offset = round(double(nex.ts-hdr.FirstTimeStamp)./hdr.TimeStampPerSample);
Robert Oostenveld - 2012-11-08 10:44:26 +0100
Hmm, have to dig deeply into my hippocampus.... I think that similar to the neuralynx system that the acquisition and the writing to disk are two separate aspects of the software. Consequently the file can already contain events prior to teh data starting. And also files can contain multiple channels that start at (slightly) different times. So is missing data is indicated with - and present data with +, then dat1 = - - - - - + + + + + + + + + + + dat2 = - - - + + + + + + + + + + + + + in representing these two channels in a single matrix, we can start at sample 1 (corresponding to the first timestamp that can be recognized in the file), at sample 3 (the first where a data sample is present) or sample 5 (where all channels have their first sample). If we start at sample 1 or 3, the missing samples can be represented as nans. But note that the challenge is that ft_read_header and especially ft_read_event have to be consistent! The cfg.trl that people have on disk would be invalidated if the sample counts change. Please note the other (old, but still open) bug 322 that relates to this.
Robert Oostenveld - 2012-11-08 10:45:20 +0100
(In reply to comment #0) > line 912 dat(i,:) = nex.dat; can give an error when chanbegsmp<1. > as the preallocation dat = zeros(length(chanindx), endsample-begsample+1); > conflicts with the size of nex.dat given the existing code. Could you give the precise code (i.e. call to ft_read_data with all arguments) that gives this error?
Martin Vinck - 2012-11-08 15:09:29 +0100
The handiest would be to write a proper bug file, but the data is pretty big (100 MB or so). Do you have access to the /DATA_pasfri/DATA_pasfri/V4forlong directory on the donders cluster or frankfurt cluster somehow?
Martin Vinck - 2012-11-08 15:11:42 +0100
Hi Robert, are you using that the cfg.trl must be made specifically for every channel given its nex.ts ? Because hdr.FirstTimeStamp is not the correct first timestamp for a given channel?
Martin Vinck - 2012-11-08 16:19:53 +0100
Where it goes wrong is for example if begsample = 4, endsample = 100 and offset = 5. In this case, we read 95 samples and pad with 5 nans. But the initiated dat would have endsample-begsample+1 = 97 datapoints, which gives a size discrepancy with the initiated vector. So we need something like: d = (endsample-begsample+1) - (chanendsmp) nex.dat = [nan(1,d) nex.dat];
Robert Oostenveld - 2012-11-08 16:40:02 +0100
(In reply to comment #7) that directory does not exist (any more) on the donders cluster storage. I don't have easy access to the ESI cluster. Please use one of the methods listed at http://fieldtrip.fcdonders.nl/bugzilla towards the bottom of the page and send me the URL for the download by email. (In reply to comment #8) hdr.FirstTimeStamp is (or should be) the timestamp corresponding to what ft_read_xxx considers the first sample (i.e. sample number 1). Any additional samples that are inconsistently present over channels are ignored. The sample specification in cfg.trl is derived from ft_read_event and passed on to ft_read_data. So whatever is hidden in those ft_read_xxx functions remains hidden to the user on purpose. I should note that this code is really old and I do not recall the details. This was well before starting to use SVN and bugzilla. So getting some test cases implemented in a test script would actually be very valuable. (In reply to comment #9) can you please write this in MATLAB style in fieldtrip/test/test_bug1825 with the assumption that the dataset is in /home/common/matlab/fieldtrip/data/test/bug1825/xxx.nex (where I'll put it when I receive it)? I'll then have a look at it.
Ali Bahramisharif - 2015-01-07 13:55:39 +0100
Hi Guys, I was on reporting a new bug, but I found it is very related to this one. I had some problems with reading some nex data and I noticed the 'header' becomes negative because of integer limits and sign flip at the boundary. I solved it by changing one if-statement (below). I was wondering if I have not changed anything important. Do you have any clue? In ~line 986 of ft_read_data, there is the following if: if chanbegsmp<1 which should be changed to : if (chanbegsmp<1 && chanendsmp>1) || (chanbegsmp>chanendsmp) Agree?
Teresa Madsen - 2016-03-08 21:58:20 +0100
I have another NEX file import error that I think may be related to these. It's not exactly the same, but I'm hoping my proposed fix might also solve yours...or if it accidentally makes yours worse, I'd like to know that too! The NEX file I'm trying to import has 2 fragments, formed by merging 2 PLX files of a discontinuous recording (automatic stop and restart to limit file sizes). Here are the exact messages I get: processing channel { 'AD01' 'AD02' 'AD03' 'AD04' 'AD05' 'AD06' 'AD07' 'AD08' 'AD09' 'AD10' 'AD11' 'AD12' 'AD13' 'AD14' 'AD15' 'AD16' } reading and preprocessing Index exceeds matrix dimensions. Error in ft_read_data (line 1111) dat(i,:) = cnt(begsample:endsample); Error in ft_preprocessing (line 576) dat = ft_read_data(cfg.datafile, 'header', hdr, 'begsample', begsample, 'endsample', endsample, 'chanindx', rawindx, 'checkboundary', strcmp(cfg.continuous, 'no'), 'dataformat', cfg.dataformat); Error in FT_preprocFear_chunks_TEM (line 176) data = ft_preprocessing(cfg); I traced this error back to line 1099: offset = round(double(nex.ts-hdr.FirstTimeStamp)./hdr.TimeStampPerSample); % of each fragment, in AD samples I don't understand why this doesn't just use the offsets provided in nex.indx ([0,4800007] in my file, as compared to offset = [0,4800006] calculated this way). There seems to have been a fraction of a ms of data lost in the stop/restart, which is causing this rounding error.