'py_io_signature' object does not support indexing

[devnulling]

I'm trying to replay samples using the Digital RF Source that have been stored with the Digital RF Sink block and running into the error:

Traceback (most recent call last):
  File "/home/user/top_block.py", line 215, in <module>
    main()
  File "/home/user/top_block.py", line 203, in main
    tb = top_block_cls()
  File "/home/user/top_block.py", line 163, in __init__
    min_chunksize=None if 0==0 else 0,
  File "/usr/local/lib/python2.7/dist-packages/gr_digital_rf/digital_rf_source.py", line 602, in __init__
    out_sig_dtypes = [src.out_sig()[0] for src in self._channels]
TypeError: 'py_io_signature' object does not support indexing

/home/user/data structure:

data
├── ch0
│   ├── 2018-12-03T05-00-00
│   │   ├── [email protected]
│   │   ├── [email protected]
│   │   ├── [email protected]
│   │   ├── [email protected]
│   │   ├── [email protected]
│   │   ├── [email protected]
│   │   ├── [email protected]
│   │   ├── [email protected]
│   │   ├── [email protected]
│   │   ├── [email protected]
│   │   └── [email protected]
│   ├── drf_properties.h5
│   └── metadata
│       ├── 2018-12-03T05-00-00
│       │   └── [email protected]
│       └── dmd_properties.h5
└── ch1
    ├── 2018-12-03T05-00-00
    │   ├── [email protected]
    │   ├── [email protected]
    │   ├── [email protected]
    │   ├── [email protected]
    │   ├── [email protected]
    │   ├── [email protected]
    │   ├── [email protected]
    │   ├── [email protected]
    │   ├── [email protected]
    │   ├── [email protected]
    │   └── [email protected]
    ├── drf_properties.h5
    └── metadata
        ├── 2018-12-03T05-00-00
        │   └── [email protected]
        └── dmd_properties.h5

8 directories, 28 files

This is using GR 3.7.13.4, numpy 1.15.4

I've tried saving/replaying both ints and complex floats, coming up with the same error.

Example flowgraphs:

Any suggestions are appreciated.

[ryanvolz]

This looks like a compatibility problem with GNU Radio 3.7.12+ that was fixed in 7baede2, which is in the 2.6.1 release. Upgrading digital_rf should fix your problem. 🙂

[devnulling]

@ryanvolz thanks for the quick response!

I had found that commit since posting the issue and figured I'd try rolling back to GR 3.7.12.0 to see if it would resolve it with the version pip installs (ubuntu 16.04.5). After this build is complete, I'll try updating digital_rf.

[ryanvolz]

I think you'd actually need to go back to 3.7.11 for it to work, if I'm remembering correctly.

…

On Mon, Dec 3, 2018, 12:57 AM devnulling ***@***.*** wrote: @ryanvolz <https://github.com/ryanvolz> thanks for the quick response! I had found that commit since posting the issue and figured I'd try rolling back to GR 3.7.12.0 to see if it would resolve it with the version pip installs (ubuntu 16.04.5). After this build is complete, I'll try updating digital_rf. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEMBJ3qQeTj4oCo6nCFCI990g34oPA4Eks5u1L1LgaJpZM4Y96BV> .

[devnulling]

It looks like pip is installing 2.6.1:

>>>drf.__version__
u'2.6.1'

I see the commit 7baede2 changed the sink block, my error was being thrown on the Source.

I updated this line linked below to match the list() call, which gets past the error, but it doesn't produce samples

https://github.com/MITHaystack/digital_rf/blob/master/python/gr_digital_rf/digital_rf_source.py#L602

[ryanvolz]

Ah, my mistake, there is clearly still a problem with the source block. I can look into it in more detail tomorrow, push a fix, and then see about making a new release. Does wrapping with list() make everything work, or are there still problems?

…

On Mon, Dec 3, 2018, 1:20 AM devnulling ***@***.*** wrote: It looks like pip is installing 2.6.1: >>>drf.__version__ u'2.6.1' I see the commit 7baede2 <7baede2> changed the sink block, my error was being thrown on the Source. I updated this line linked below to match the list() call, which gets past the error, but it produce samples https://github.com/MITHaystack/digital_rf/blob/master/python/gr_digital_rf/digital_rf_source.py#L602 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEMBJ0ggdr01HmgQjyM0nVloPVVpLSbhks5u1MLBgaJpZM4Y96BV> .

[devnulling]

Wrapping with list() will get rid of the original error, but the source block will not stream samples. This is running on GR 3.7.12.0.

[devnulling]

Digging into this more, I've found the Digital RF Source block is actually streaming samples, however, it is not stripping off some of the header information which GR doesn't like.

Here's my testing so far:

1. Noise source into a DRF Sink and regular GR file sink one.fc32
   

2. Read the DRF generated files with the DRF Source block and saved it to a new file using the GR File Sink two.fc32
   

In two.fc32 there is a header of the bytes 00 00 C0 7F repeating, which GR won't process as a stream.

Removing the offending bytes (00 00 C0 7F , 4424 in this capture) from two.fc32 and saving it to a new file, GR will accept / stream from a File Source block.

It seems the header is not being parsed correctly by the Digital RF Source block.

one.fc32.gz
two.fc32.gz
[email protected]

drf_convert.grc.gz
drf_gen.grc.gz

[ryanvolz]

Well, it looks like there are multiple factors at play here, and it's something I wouldn't anybody who's not familiar with the inner workings of Digital RF to anticipate.

First, Digital RF actually has two formats for writing files: continuous and chunked. In the default continuous mode, each file is a fixed size and stores a value for every sample that falls within its time window. If the write starts after the time where the file's time window starts, or if samples are skipped, it stores a default "missing" value. For floating point types, this is NaN. For integer types, this is the minimum (most negative) value. In chunked mode, missing samples are not included in the file, and each file can have multiple chunks of data. Continuous mode is the default because it's faster when reading the data to not have to glue together potential file chunks. We may need to revisit that, though.

Second, because you're using the default sink settings and setting the data start time using 'now', your test data is being written as 32-bit complex floats in continuous mode with a 1-second file cadence and the data is almost surely not starting at an integer second. So, the files that you're writing have some number of 'NaN' samples at the beginning.

Third, when the Digital RF Sink reads a channel, it starts by default at the beginning of the data present in the first file. So in this case, it's starting with some number of 'NaN' samples.

Fourth and finally, it appears that GNU Radio does not like 'NaN' values, so I guess it's refusing to move past them and then actually start streaming the real data.

Overall, this is a case where the documentation and default behavior are not clear enough about what's happening, and it leads to a non-functional setup. I haven't run into this before because we typically avoid it in one of two ways: 1) set a data start time that falls exactly on an integer second, so there are no "default" samples at the beginning of the first file in continuous mode, and 2) write data as an integer type, because this is usually what the radios provide at a base level (the default value for integer samples is still a valid integer, do GNU Radio does not choke on it).

The quickest fix for your setup is to change the Sink start time to 'nowish' from 'now', which rounds to an integer second. Then you can avoid storing 'NaN' and it should work fine.

But it's clear we should do something to fix this on our end, because I can easily see other people inadvertently running into this behavior. Options would include:

• Calling this out and making it more clear in the documentation
• Defaulting to chunked mode for writing (maybe if we test the performance again now, the difference is not so large?)
• Having Digital RF Source skip over default samples at the start of the stream, or somehow otherwise know at what point the Sink started writing
• Converting 'NaN' values to 0 or something valid when streaming in GNU Radio

We'll have to think about it and figure out the best way forward. Thanks for trying things out and discovering this problem!

[ryanvolz]

The fix for the digital_rf_source py_io_signature bug has been released with Digital RF 2.6.2, so I'm going to close this. I'll open a new bug for discussion about the Digital RF Source block and NaNs.