DAG-based Rendering Pipelines

[emanuele3d]

Introduction

The current renderer is largely an hardcoded implementation. It features a number of features that can be toggled on/off, i.e. light shafts, or enabled to some pre-defined level, i.e. Depth of Field blur levels. But it is inflexible in the sense that new features must be added to the source code rather than, say, injected at runtime by a module.

Also, while the renderer might eventually support the concept of multiple cameras, each camera would have to render in pretty much the same visual style, dictated by the settings stored in the rendering config unless some kind of overriding machinery is added to enable/disable features separately from the rendering config. For example, CCTV-like cameras might not require the full range of visual effects currently available through the renderer, but might require some additional or alternative processing to provide their typical black&white, grainy/noisy look.

In this context I'd like to propose here the embryo of a new architecture for the renderer.

Rendering Tasks, Rendering Pipelines

The current renderer is mostly a collection of conceptually atomic rendering tasks. I.e, first the shadow map for the main light is rendered into a buffer. Then the scene reflected in the water surfaces is rendered into another buffer. Then the sky, the landscape and its inhabitants. Transparent objects are blended in and (skipping a number of steps for brevity) a number of post-processing effects are added eventually leading to (ta-dah!!!) the final image shown on screen or saved into a screenshot.

The key idea of the new architecture is to recognize that these tasks are organized into a Directed Acyclic Graph (DAG), by virtue of the fact that each task is potentially dependent on the output of other tasks being executed first and the fact that most tasks eventually provide the inputs for tasks that follow. This DAG is what is commonly referred to as the rendering pipeline, despite it not being quite a line but a number of branching and potentially overlapping paths.

In this context, adding new features should be as easy as adding tasks to the pipeline, or to better put it, nodes to the graph. Similarly, disabling features should be as easy as removing nodes from the graph. For example, at this stage OculusVR code is intertwined with the standard mono rendering mode. Enabling or disabling a given stereo mode (not just Oculus) would then be a matter of replacing a number of tasks in the pipeline with the stereo-only or mono-only version of them. All at runtime.

The new architecture should also allow for multiple pipelines to coexist.

• A CCTV monitor might display a grainy black&white image, while one nearby might display the same image in infrared. This would be handled by two separate rendering pipelines having world data and camera in common but applying different sets of rendering tasks to them.
• A portal nearby might trigger a whole pipeline to be attached to the primary pipeline (the one leading to the image on the user's screen) so that the world behind the portal is visible through it.

Advanced features

Also to keep in mind as potential, advanced, self-optimization features:

• automatic elimination of expensive, redundant state changes: some state changes in opengl are computationally expensive, i.e. binding or clearing a frame buffer. Ideally as soon as a node is added or removed from a rendering pipeline and well before the pipeline is executed, it should check for redundant state changes and eliminate all but the last one before a rendering. This would be done by wrapping expensive calls into their own rendering tasks.
• automatically estimate the fastest possible DAG configuration when alternatives are available: in some cases there is no unique DAG for the given set of rendering tasks. I.e. task B and C might need to be executed after task A, but executing task C first might reduce the number of expensive state changes and improve performance. Whenever a setting changes, a task is added or perhaps even when manually triggered by the user, the pipeline could do a few profiling runs with different DAG configurations to measure which one is the fastest. Frame rate would be expected to jitter during profiling, but it would happen only briefly, in the context of user-driven transitions, and would be probably acceptable.

[emanuele3d]

I propose for exploratory discussions to be held in the associated forum thread. Distilled ideas, features and considerations emerging there as important will be then added here as comments to this issue.

[tdgunes]

Can this issue be considered as GSOC ideas? I initially considered #319 for my GSOC proposal. Before working on that, I think it is better to focus on this one first.

[Cervator]

@tdgunes - I believe the scope might be suitable for GSOC, but I'm not sure if we'd have the mentoring capacity available. But then #319 and even #2111 share that concern :/

@emanuele3d probably isn't available for full time mentoring (I figure, anyway, please correct me if appropriate!), although maybe he can find enough time plus some occasional assistance from @flo and others to make it work.

[tdgunes]

@Cervator I think #319 and #2111 are dependent on this issue. Therefore I believe this must be done first, before starting on those. May be this one and #319 can be combined into one project for the summer. If there is additional time left, I can get an VR headset and focus on #2111.

It would be sad to not have anyone for 3d wizardy for the summer. Do GSOC allow more than one part-time mentors?

[Cervator]

We talked briefly on IRC but just to also put an update here we might be able to make do with several part time 3d wizards to make up one solid mentor. We still need to make one person official but maybe it is OK to do so with less of a time commitment if there are multiple extras standing behind them.

We should be getting a better feel for it over the next few days, but luckily there's plenty of time, still three weeks or so till proposals are due :-)

[Cervator]

Closing this as complete as part of Alpha 4 and GSOC 2016 :-)

@emanuele3d and @tdgunes feel free to open new smaller issues if it makes sense to get some going for future work.

[emanuele3d]

I'm sorry but I disagree. This is in progress. The dag is currently just a list, not a graph. We are going in the right direction but if it was up to me I'd keep it open. Perhaps we should turn it into the main item of an epic?

[Cervator]

@emanuele3d absolutely fair point :-) I wanted to close this out since it isn't a fresh new GSOC candidate issue anymore, but we could submit a new issue to be a non-GSOC container issue of sorts for the longer DAG pipeline implementation, make that an epic, add this completed issue to it, and submit a few other related ones to add to the same epic?

[emanuele3d]

Look, you seem to be decided about this, so I won't insist. But what this means is that I'll simply cut and paste the description of the issue into a container/epic issue and slowly add issues to it if I can. Ideally we'd leave this issue open and make the it starting point of an epic. I really don't know how epics work though, so it might be unfeasible that way.

[Cervator]

@emanuele3d yep the copy paste thing would be fine and expected. The main reason I'd like to close this rather than use it as the epic is that it has been "used" in relation to GSOC - but really we could just re-open it and reuse it, not a big deal :-)

To me one way or the other I think we should have a few issues, one of them could be upgraded to a ZenHub epic and serve as a container for the individual detail issues. This specific issue here could for instance be a detail issue renamed to "DAG pipeline - list phase" with another being the "graph phase", and a third a vague "Rendering structure overhaul" epic that only covers the goal at a high level but lets the individual issues cover the details.

In short I'm not worried which issues cover what topic. Issues are cheap to create. If you'd like to make a few related issues we can group them together any way you like :-)