When loading > 30 stream lines I can't terminate matlab with ctrl+C

[giovannetti87]

Hello everybody,

I noticed that when I try to streamline more than about 30 lines together, matlab behaves weirdly, and I am left with no other option that terminating it abruptly.

Suppose I cross the 30 lines threshold and I run a small loop for Q from 1 to 30 (e.g., call bid/ask prices). After Q reaches 30, the command line appears again (just like when matlab is ready to take orders), but whatever order I give, is not operated. At the same time, every other element of the program is stuck. I thought this may due to an excess of buffer received in the background and bad indexing. Is that a possibility?
thanks

[Despair2000]

Do you have your buffers in a cell array? If so this might be the cause. The listeners you can have in a cell array but not the buffers. This causes very erratic errors. Don't ask me why.

[giovannetti87]

I see, I have a nested struct variable collecting the local buffer. I will try to take each sub-variable (i.e. circularfifobuffers) out and report back, thanks.

[giovannetti87]

I brought the buffers out of the struct variable and I confirm things massively improved, thanks. I can now hold a Q=1:2000 iterations of data collection of bid/ask spread for 54 streams with only 2 minutes of de-hanging time requires by matlab to return to be operational.

Also, when command line is rendered, I can actually use it (e.g. 2+2 returns 4 immediately).

So, the buffer structure was definitely a part of the problem, but there may be something additional to that, as for longer time series it seems matlab hangs for longer time. For example, for a Q=1:4000 series with 52 streams, matlab takes about 10 minutes to unhang.

1. Should I kill the stream requests as soon as I'm finished pulling the data from it?
2. Do I have to put some pause(xyz) within my code to ensure no punishment? I notice this used quite frequently in the example codes.

thanks,

[Despair2000]

I don't really understand what you mean by this 4000 iterations.

I stream data through IB4m the whole day long and nothing hangs.

And yes, you sometimes have to use pauses with the IB API to give it time to respond after a request for example.

[giovannetti87]

After setting up the streams, I want to test data collection by building a simple for loop which places the data inside a few vectors for a number of periods. What I mean is that after the loop Q=1,2,... ends, there is a variable time in which Matlab is frozen.

I double-checked when the issue arises, and I can see it is actually at the stream connection stage. If I connect more than 54 streams, it takes more than 3-4 minutes to matlab to return me variables and control.

All buffer sizes are set to "15" at the moment.

[Despair2000]

Sounds funny. The main part of my program is a while-loop. This loop starts with a time-condition when the market opens and runs all day until the markets close. All the time it is streaming market data and I can break matlab (go into debug-mode) at any time - no freezing.
Like I mentioned above the only time I saw a behaviour that was strange was when I had the buffers in a cell-array. Having them in a struct-array works fine.
Try clearing your buffers after you retrieved the data. This shouldn't be necessary but just to see what happens (or better if something happens).

[giovannetti87]

So far, no positive outcome. I am really puzzled by the behaviour. The problem is clearly in the buffer/stream part of the script as it engulfs well before the data-download. Probably I'm doing something wrong there. I start with a standard:

% get TWS session instance
session = TWS.Session.getInstance();

% create local buffer for market data events with size 10,000 (default size is 32)
[databuf,datalh] = TWS.initBufferForEvent(TWS.Events.MARKETDATA,32);

% create local buffer for market metadata events with size 1000
[metabuf,metalh] = TWS.initBufferForEvent(TWS.Events.MARKETMETADATA,32);

then, I have a small loop where all currency pairs are initiated:

       session.eClientSocket.reqMktData(reqId,contract,[],false,false,[]);
        [local_buf.([fx{i} fx{j}]), lh.([fx{i} fx{j}])] =  initMarketDataBufferWithReqId(loc_buf_sz, reqId);
        eval(sprintf('local_buf%d = local_buf.([fx{i} fx{j}])', reqId));
        clear local_buf

`
so I get many variables called local_buf11, local_buf12, local_buf13, etc. This used to be a much less messy structure variable earlier, but it turned out aggravating the speed problem in discussion now. Lastly, I clear all non-existent pairs

for k=1:numel(index_fx)
    if isempty(collection2cell(eval(['local_buf' num2str(k)])))
        index_fx{k} = {};
        clear(['local_buf' num2str(k)])
    
    else
        buffer_list = [buffer_list, ['local_buf' num2str(k)] ','];
    end
end

then, in my main code (corresponding to the data collection part), I run a simple
mktDataEvents = collection2cell(eval(buffer_list{k}));

Getting crazy..

[Despair2000]

First the obvious. The first two buffers are not used. Take them away.
Then I'd advise you to stop using eval. It is both very slow and makes the code hard to read.
You have there a cell-array "fx". It is not clear what it contains.
Then you must be careful not to get confused which functions you are using. For example "clear buffer" will deleted the whole buffer while clear(buffer) will only delete the content of the buffer and not the buffer itself. Another one that one can easily confuse is "isempty" (the matlab function) and "isEmpty" (the java-method).

Without having understood (and seen) your whole code I dare to say that the problem is how you arrange the buffers.

[giovannetti87]

Hi Despair,

Thanks for checking the code, I really appreciate it. I killed the two tautological processes and I am moving back into the struc. structure. I was wondering, how do you determine the pause coefficient, for your script? What coefficient do you use? I was wondering if part of my problem isn't due to a too short query interval

[Despair2000]

With the pauses it is a little bit trial and error. In some cases you can have a little loop checking if something dropped into your buffer every half second or so. In other situations you simply have to try what fits your set up since it also depends on the speed of your computer.
Some months ago I bought a faster computer and suddenly my program made strange problems. The reason was simply that the computer was too fast and so that the pauses I had in my code were too short. After increasing the pauses here and there all was fine again.

[giovannetti87]

Hi Despair,
Sorry for the delay, I wanted to pin down this thing once for all, so I run a few tests. I understand there were 2 compounding problems. The first problem is in the contract creation process, multiple unused listeners were created which I forgot to eliminate. Eliminating the listener removed the saturation problem I was experiencing.
The second problem is more interesting. It appears that operations outside the main loop require matlab much more time to execute. In my case, if I put the cancel stream code outside the main loop, the program hangs for several minutes after finishing the main loop. This is resolved by including the cancel stream code as a last iteration of the main loop