Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: pixi create project scaffold generation. #2837

Open
zbowling opened this issue Jan 6, 2025 · 1 comment
Open

Feature: pixi create project scaffold generation. #2837

zbowling opened this issue Jan 6, 2025 · 1 comment

Comments

@zbowling
Copy link
Contributor

zbowling commented Jan 6, 2025

Problem description

Given pixi doesn't have an RFC process for feature discussion, I figured an issue here probably is the best place to discuss this feature idea. This feature is already in development as magic create in the magic CLI to solve problems with AI development use cases downstream where we are flushing out design issues, but we eventually want to share this feature upstream once it's a bit more flushed out and working.

pixi create

This is a proposal for a new feature called pixi create. This command enhances the project scaffolding capabilities of pixi, drawing inspiration from the npm init/npm create command by allowing for customizable "generator" packages.

Motivation

While pixi init provides a basic structure for new projects, there is a need for a more comprehensive solution to generate diverse project types, potentially spanning multiple programming languages. The idea is that feature can help setup full-fledged project scaffolds complete with best practices of developing with different frameworks. Some setups are difficult to get right out of the box for different frameworks and application types. The pixi create command aims to fulfill this requirement by leveraging a system of generator packages.

For example, a generator for a web framework could generate a Dockerfile to assist with deployment but also with local development that makes use of pixi-docker. A generator for a python library scafold could show the best patterns for using “pixi build” to show how to a build backend and publish to both Conda and PYPI. A generator for pytorch could show best practices for cross-platform building using different versions of pytorch (CPU, GPU, etc). A generator can also create .devcontainer configs around the project to assist with development.

Generators can be interactive, allowing for easy user customization of scaffold projects to pick perhaps different optional dependencies or integrations, database engines, tool support, etc. Generators can also be built to be non-interactive where you want to automate project generation with tooling.

Functionality

The pixi create command operates as follows:

Generator Packages:

Generator packages will be conda packages, identifiable by the "create-" prefix.
These packages will be responsible for the actual generation of project scaffolding.
They will be hosted on conda channels and installed using the same backend as pixi global except without directly exposed binaries in the path.

Differences from pixi init

pixi init is designed for lightweight project initialization that works even on existing projects.

pixi create is intended for more complex project generation, supporting a wider range of project types and languages. It’s generally not intended for use on an existing code base.

pixi init provides a basic structure where pixi create offers greater flexibility through the use of specialized generator packages.

Command Execution:

When pixi create <generator> <target_directory> is executed:

  • pixi will download and install the specified <generator> package into a new global environment.

  • A command within the <generator> package will be invoked, passing all arguments from pixi create to it.
    The generator package will then handle the creation of the project structure within the <target_directory>.

To assist with the development and debug of new generators, an env + task can be passed in an existing project on disk to invoke instead of using one installed from a package into a global env.

Example

pixi create python ./my_python_project -- --some-arg=hello

In this example:

  • pixi will fetch and install the "create-python" package.

  • The "create-python" command (or its designated entry point) will be executed by passing through any additional args to the generator.

  • The "create-python" package will generate the Python project scaffolding in the "./my_python_project" directory.
    Internal Implementation

Generator Package Structure:

Generator packages will adhere to a standardized structure to ensure compatibility with pixi create. Generator packages can be built using any language or framework as long as they can shipped as conda packages themselves. A simple generator could be written in Python and generate files using simple jinja templates, perhaps leaning on copier.

  • The generator MUST specify a script within the conda package that matches the package's name, so a generator named "create-python-project" should have a bin/create-python-project script within the package.

  • A generator package should contain the code and templates to generate the project structure but can fetch templates, assets, and scaffold files from the web.

Generator package environments:

Since generators are installed implicitly, they need an environment other than the project environment to execute. Before executing, generators will be installed in the temp env folder in a similar fashion to pixi global (except when an explicit project and task are passed in).

Special Considerations/TBD

Generators may want to download dependencies and initialize different envs in the new project directory while generating rather than defer to the first run of the project, perhaps to invoke tools from dependent packages on the newly generated project folder. We may have to consider re-entertrancy back into pixi in that case.

The exact arguments for pixi create are still in flux.

@baszalmstra
Copy link
Contributor

I like this idea a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants