Proposal: Add an option to pilosa import command that can auto create new indexes/frames

[addos]

Description

Hi, I have been using pilosa for awhile now, and one of the things I've found would be really helpful would be if the pilosa import command could have some way to automatically create new indexes and new frames, perhaps with an extra parameter of some kind, or maybe it can just automatically detect if an index/frame already exists and if not create it? I personally feel this would make importing data into pilosa require fewer steps and would accomodate the majority of what I need for the data I am importing.

Success criteria (What criteria will consider this ticket closeable?)

pilosa import -i newIndexName -f newFrameName filename.csv automatically creates a new index with the default column name, and automatically creates a new frame with the default row name, and otherwise uses an existing index/frame if they already exist.

Thanks for your consideration and keep up the great work!

[travisturner]

This would be fairly easy to implement in ImportCommand.Run() by adding:

err := cmd.Client.CreateIndex(ctx, cmd.Index, pilosa.IndexOptions{})
err := cmd.Client.CreateFrame(ctx, cmd.Frame, pilosa.FrameOptions{})

(obviously handling the errors and likely ignoring ErrIndexExists and ErrFrameExists)

The only thing I question is whether this should be the default behavior (i.e. indexes/frames automatically get created when they don't exist), or whether we should add a boolean flag to the import command (that defaults to false) indicating that indexes/frames should be created when they don't exists: --auto-create-index or something like that.

[addos]

The only other thing I thought about, was also whether to create an inverse index. Not sure whether it is worth having that as another option.

[jaffee]

Not only the inverse index, but what cache type and size to use on the frame. I don't really see much issue with adding these as options as well because it seems like it will be pretty low effort, and this is really just a stopgap until better import/export support comes via #725

If we add a boolean flag as @travisturner suggested (--create?) which defaults to false, we can preserve backward compatibility with older versions of the command which seems worthwhile.

[addos]

Well, and from my perspective, anything that makes it quicker to import data into pilosa is a huge win. So all of this sounds good to me. 👍

[travisturner]

It makes sense to me to treat this as a stopgap—and not support index/frame options—for now.
And I am supportive of the --create flag for backward compatibility.

[jaffee]

@addos would this solve your problem with all defaults, or do you need to be able to specify cache type and size?

[addos]

So the primary goal was to reduce the friction to getting data into the index. The extra options would be nice. At a minimum having it use the defaults is better than having to create the index and frame manually as separate steps.

[travisturner]

Let's implement this and include command line flags for frame options.

    string RowLabel = 1;
    bool InverseEnabled = 2;
    string CacheType = 3;
    uint32 CacheSize = 4;
    string TimeQuantum = 5;
    bool RangeEnabled = 6;

[jaffee]

@addos if you have any interest in contributing this change, I'd be happy to walk you through it. If not, I'll go ahead and knock it out.

[addos]

Not sure how I would do that.