Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce "use database" and update gaiac usage #633

Merged
merged 7 commits into from
Apr 26, 2021
Merged

Introduce "use database" and update gaiac usage #633

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b252542
draft change
chuan Apr 20, 2021
5c537c0
Implement use statement execution
chuan Apr 22, 2021
d35ef6f
Update gaiac command line usage for output path and db
chuan Apr 22, 2021
4161110
Update catalog README
chuan Apr 22, 2021
c462faa
Address CR comments
chuan Apr 24, 2021
8e00358
Minor doc and comment updates
chuan Apr 26, 2021
aed4ab1
Update indentation
chuan Apr 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 36 additions & 58 deletions production/catalog/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,62 +53,64 @@ By default without specifying any mode, `gaiac` will run under loading mode to
execute the DDL statements--translating them into catalog records--without
generating any output.

The interactive mode (`-i`) provides a REPL style command line interface to try
out the DDL. The DDL typed in will be executed, and fbs output if any will be
printed out to the console output.
The interactive mode (`--interactive` or `-i`) provides a REPL style command
line interface to try out the DDL. The DDL typed in will be executed, and fbs
output if any will be printed out to the console output.

Under generation mode (`-g`), the tool will generate the following two header
files either from an DDL file or a specified database.
Under generation mode (`--generate` or `-g`), the tool will generate the following two header
files from specified database(s) under the output path.

- The FlatBuffers header for field access, `<dbname>_generated.h`
- The EDC header file `gaia_<dbname>.h`

With the two headers, a direct access source file gains access to the database
as defined by the catalog.

Full command line usage can be shown with the `-h` option.
Full command line usage can be shown with the `--help` or `-h` option.

#### Examples

Enter interactive mode.

```
gaiac -i
gaiac --interactive

```

Execute DDL statements in `airport.ddl` file using the `airport` database, and
generate header files for tables in `airport` database.
Execute DDL statements in `airport.ddl` file, generate header files in the
`airport` directory for tables in the `airport` database.

```
gaiac -g airport.ddl
gaiac --generate --db-name airport airport.ddl
```

Generate catalog direct access APIs. This is the command used for bootstrapping.
Execute DDL statements in `incubator.ddl` file. Generate the headers in the
`incubator` directory for tables in the `barn_storage` and `lab`databases.

```
gaiac -d catalog -g
gaiac --db-name barn_storage --db-name lab --generate incubator.ddl
```

## Databases
Generate catalog direct access APIs in the `catalog` directory. This is the
command used for bootstrapping.

There are two ways to create a database and specifying a table in a database:
first, using DDL; second, using `gaiac` command. When both are specified, the
DDL definition will override the `gaiac` settings.
```
gaiac --database catalog --generate --output catalog
```

### Use DDL
## Databases

The DDL to create database is `create database`.
The DDL to create database is `create <database_name>`.

To specifying a table in a database, using the composite name of the format
`[database].[table]`.

#### Examples
### Examples

A database `addr_book` can be created using the following statement.

```
create database addr_book;
create database addr_book;
```

Use the following statement to create an `employee` table in `addr_book`
Expand All @@ -126,47 +128,23 @@ create table addr_book.employee (
);
```

As a syntactic sugar, the database name can be omitted when specifying a
reference to a table in the same database.

```
create table addr_book.address (
street: string,
apt_suite: string,
city: string,
state: string,
postal: string,
country: string,
current: bool,
addresses references employee
);
```

### Use `gaiac`

This is the way to create and specify a database before the introduction of the
database to DDL. The `gaiac` command line usage already documented how to do
this. See the following examples for more explanation.

#### Examples

In the following command line example, an `airport` database will be created
automatically. When no database prefix is specified for table names, the tables
will be created in the `airport` database. (The database prefix can still be
used to override the database name derived from the DDL file name.)

```
gaiac -g airport.ddl

```

The following command line will create all tables in `tmp_airport`. Because a
database name is specified via `-d`, the command will not create the database
`tmp_airport` automatically.
Switch to a database to make the DDL more succinct (by avoid the database name
when referring to a table) with the `use <database_name>` statement. In the
following example, the `address` table will be created in the `addr_book`
database.

```
gaiac -d tmp_airport -g airport.ddl
use addr_book;

create table address (
street: string,
apt_suite: string,
city: string,
state: string,
postal: string,
country: string,
current: bool,
);
```

## Catalog bootstrapping
Expand Down
131 changes: 68 additions & 63 deletions production/catalog/gaiac/src/main.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,14 @@

#include "flatbuffers/idl.h"

#include "gaia/common.hpp"
#include "gaia/db/db.hpp"

#include "gaia_internal/catalog/catalog.hpp"
#include "gaia_internal/catalog/ddl_execution.hpp"
#include "gaia_internal/common/gaia_version.hpp"
#include "gaia_internal/common/logger_internal.hpp"
#include "gaia_internal/common/scope_guard.hpp"
#include "gaia_internal/db/db_test_helpers.hpp"

#include "command.hpp"
Expand All @@ -42,9 +44,8 @@ enum class operate_mode_t
loading,
};

void start_repl(parser_t& parser, const string& db_name)
void start_repl(parser_t& parser)
{
gaia::db::begin_session();
initialize_catalog();

const auto prompt = "gaiac> ";
Expand Down Expand Up @@ -78,7 +79,7 @@ void start_repl(parser_t& parser, const string& db_name)
int parsing_result = parser.parse_line(line);
if (parsing_result == EXIT_SUCCESS)
{
execute(db_name, parser.statements);
execute(parser.statements);
}
else
{
Expand All @@ -90,8 +91,6 @@ void start_repl(parser_t& parser, const string& db_name)
cerr << c_error_prompt << e.what() << endl;
}
}

gaia::db::end_session();
}

// From the database name and catalog contents, generate the FlatBuffers C++ header file(s).
Expand Down Expand Up @@ -121,11 +120,10 @@ void generate_fbs_headers(const string& db_name, const string& output_path)
}

// From the database name and catalog contents, generate the Extended Data Class definition(s).
void generate_edc_headers(const string& db_name, const string& output_path)
void generate_edc_headers(const string& db_name, const filesystem::path& output_path)
{
std::string header_path = output_path + "gaia" + (db_name.empty() ? "" : "_" + db_name) + ".h";

cout << "Generating EDC headers in: '" << std::filesystem::absolute(header_path).string() << "'." << endl;
filesystem::path header_path = output_path;
header_path /= "gaia" + (db_name.empty() ? "" : "_" + db_name) + ".h";

ofstream edc(header_path);
try
Expand All @@ -140,10 +138,25 @@ void generate_edc_headers(const string& db_name, const string& output_path)
edc.close();
}

void generate_headers(const string& db_name, const string& output_path)
void generate_headers(const string& db_name, const filesystem::path& output_path)
{
generate_fbs_headers(db_name, output_path);
generate_edc_headers(db_name, output_path);
filesystem::path absolute_output_path = filesystem::absolute(output_path);
absolute_output_path += filesystem::path::preferred_separator;
absolute_output_path = absolute_output_path.lexically_normal();

if (!filesystem::exists(absolute_output_path))
{
filesystem::create_directory(output_path);
}
else if (!filesystem::is_directory(absolute_output_path))
{
throw std::invalid_argument("Invalid output path: '" + output_path.string() + "'.");
}

cout << "Generating headers in: " << absolute_output_path << "." << endl;

generate_fbs_headers(db_name, absolute_output_path);
generate_edc_headers(db_name, absolute_output_path);
}

// Check if a database name is valid.
Expand Down Expand Up @@ -172,15 +185,6 @@ bool valid_db_name(const string& db_name)
return true;
}

// Add a trailing '/' if not provided.
void terminate_path(string& path)
{
if (path.back() != '/')
{
path.append("/");
}
}

string usage()
{
std::stringstream ss;
Expand Down Expand Up @@ -236,11 +240,12 @@ int main(int argc, char* argv[])
int res = EXIT_SUCCESS;
server_t server;
string output_path;
string db_name;
vector<string> db_names;
string ddl_filename;
operate_mode_t mode = operate_mode_t::loading;
parser_t parser;
bool remove_persistent_store = false;
const char* path_to_db_server = nullptr;

// If no arguments are specified print the help.
if (argc == 1)
Expand Down Expand Up @@ -274,16 +279,7 @@ int main(int argc, char* argv[])
cerr << c_error_prompt << "Missing path to db server." << endl;
exit(EXIT_FAILURE);
}
const char* path_to_db_server = argv[i];
server.set_path(path_to_db_server);
if (remove_persistent_store)
{
server.start();
}
else
{
server.start_and_retain_persistent_store();
}
path_to_db_server = argv[i];
}
else if (argv[i] == string("-o") || argv[i] == string("--output"))
{
Expand All @@ -293,7 +289,6 @@ int main(int argc, char* argv[])
exit(EXIT_FAILURE);
}
output_path = argv[i];
terminate_path(output_path);
}
else if (argv[i] == string("-d") || argv[i] == string("--db-name"))
{
Expand All @@ -302,7 +297,15 @@ int main(int argc, char* argv[])
cerr << c_error_prompt << "Missing database name." << endl;
exit(EXIT_FAILURE);
}
db_name = argv[i];
string db_name = argv[i];
if (!valid_db_name(db_name))
{
cerr << c_error_prompt
<< "The database name '" + db_name + "' is invalid."
<< endl;
exit(EXIT_FAILURE);
}
db_names.push_back(db_name);
}
else if (argv[i] == string("-h") || argv[i] == string("--help"))
{
Expand All @@ -324,55 +327,57 @@ int main(int argc, char* argv[])
}
}

if (!valid_db_name(db_name))
if (path_to_db_server)
{
cerr << c_error_prompt
<< "The database name '" + db_name + "' supplied from the command line is incorrectly formatted."
<< endl;
exit(EXIT_FAILURE);
}

// This indicates if we should try to create the database automatically. If
// the database name is derived from the ddl file name, we will try to
// create the database for the user. This is to keep backward compatible
// with existing build scripts. Use '-d <db_name>' to avoid this behavior.
// GAIAPLAT-585 tracks the work to remove this behavior.
bool create_db = false;
if (!ddl_filename.empty() && db_name.empty())
{
db_name = get_db_name_from_filename(ddl_filename);
create_db = true;
server.set_path(path_to_db_server);
if (remove_persistent_store)
{
server.start();
}
else
{
server.start_and_retain_persistent_store();
}
}

if (!valid_db_name(db_name))
{
cerr << c_error_prompt
<< "The database name '" + db_name + "' derived from the filename is incorrectly formatted."
<< endl;
exit(EXIT_FAILURE);
}
gaia::db::begin_session();
auto close_db_session = scope_guard::make_scope_guard([&]() {
gaia::db::end_session();
});

if (mode == operate_mode_t::interactive)
{
start_repl(parser, db_name);
start_repl(parser);
}
else
{
try
{
gaia::db::begin_session();
initialize_catalog();

if (!ddl_filename.empty())
{
load_catalog(parser, ddl_filename, db_name, create_db);
load_catalog(parser, ddl_filename);

if (output_path.empty())
{
output_path = filesystem::path(ddl_filename).stem();
}
}

if (mode == operate_mode_t::generation)
{
generate_headers(db_name, output_path);
// Generate headers for the default global database if no database is given.
if (db_names.empty())
{
db_names.emplace_back("");
}

for (const auto& db_name : db_names)
{
generate_headers(db_name, output_path);
}
}
gaia::db::end_session();
}
catch (gaia::common::system_error& e)
{
Expand Down
Loading