Fetch Dataset Versions #272
Conversation
cpp/src/lance/format/manifest.cc
Outdated
| @@ -85,4 +85,6 @@ void Manifest::AppendFragments(const std::vector<std::shared_ptr<DataFragment>>& | |||
| fragments_.insert(std::end(fragments_), std::begin(fragments), std::end(fragments)); | |||
| } | |||
|
|
|||
| arrow::DatasetVersion Manifest::ToDatasetVersion() const { return arrow::DatasetVersion{version_}; } | |||
There was a problem hiding this comment.
nit: not sure what's the right C++ convention but here's we're not turning a Manifest into a DatastVersion, instead we're getting it's version_ as a Datasetversion right? Isn't GetDatasetVersion more appropriate?
|
|
||
| ARROW_ASSIGN_OR_RAISE(auto file_infos, impl_->fs->GetFileInfo(selector)); | ||
| for (const auto& finfo : file_infos) { | ||
| ARROW_ASSIGN_OR_RAISE(auto manifest, OpenManifest(impl_->fs, finfo.path())); |
There was a problem hiding this comment.
Does this need an explicit close or are these automatically cleaned up?
There was a problem hiding this comment.
this opens as arrow::RandomInputFile which will be closed once it is out of scope.
| std::make_shared<lance::format::DataFragment>(lance::format::DataFile(path, field_ids))); | ||
| } | ||
| return fragments; | ||
| return paths | views::transform([&field_ids](auto& path) { |
There was a problem hiding this comment.
just confirming: these don't need any error handling because they're not actually opening the resources right?
There was a problem hiding this comment.
Yea, there is no system call in this transform.
We need |
Provide
LanceDataset::versions()andLanceDataset::latest_version()methods to fetch available versions.