Rust support

[TanJunKiat]

With the rise of the Rust Programming Language, is it possible for a native Rust IfcOpenShell API?

[Tsovak]

thump up this for adding support for Rust

[aothms]

100% in favour, but it's not going to come from me given limited time and experience with rust. We're always open to contributions :)

[gnuion]

Memory leakage issues are of big concern and could crash the app/library. The memory-safe feature of Rust makes the language a good fit for this project. I suggest we fork the project and rewrite it in Rust. If it is a success, we can move to the new library. I can only volunteer a few hours a week. So we will need human resources because AI can't do it by itself yet.

[aothms]

Maybe a better starting point would be to talk to guys at https://github.com/stepcode/stepcode. They have a narrower scope because they don't need to interface with the C++ geometry libraries. If that's a success we can swap out the C++ parser module in ifcopenshell with your rust port. That would also enable to most use of your effort because it can be more generally applied.

For me, highest priority would be reducing indirections and memory overhead. Currently in IfcOpenShell we have, at the parser level, object storage using inheritance and virtual methods. On top of that a schema layer is applied that coerces the tokens according to the schema. This is inefficient, because we have generic storage, but can only access the data through the typed layer that needs to do coercion. It would be interesting if the parser can already be informed on the structure of the entity so that it can at parse time already store the data in an efficiently packed tuple without indirections or virtual calls. The schema language (EXPRESS) doesn't support aggregates with non-uniform elements, so the type information does not need to be stored for every element, which we currently also do.

If we can get:

• A more efficient parser+storage
• Potentially offload to disk / memory mapping in high memory situations
• A modern API (also conveniently accessed from c/c++)
• Compilation to various backends such as wasm
• Memory safety

I'm sure people in the community will be eager to adopt it and contribute. I think rust also has some pretty good parser libraries. The P21 grammar that is used for the .ifc files is rather elegant.

[Krande]

Hey, I came across https://github.com/ricosjp/ruststep and think it might be relevant for this discussion.

They're building a rust cad kernel "truck" (see https://github.com/ricosjp/truck).

I havent tried it myself, but if anyone of you are keen on learning rust, this could be a way in?

[TanJunKiat]

Hmmm interesting.

Right now I am trying to create the IFC Openshell library in Rust, but am struggling to parse the IFC lines. I will look into using ruststep as suggested. Thanks @Krande

But most importantly, failing to find a suitable method to return the IFC element via get_by_id() because Rust does not allow function overloading like c++.

Looking at how we can better do the latter.

Any feedback or suggestion will be appreciated.

Update

Yes, it seems like ruststep is looking to parsing the IFC file and storing them in structs there. If that is possible, it will definitely be helpful for the ifcopenshell_rust development.

[civilx64]

Chiming in to register my interest as well.

Potentially offload to disk / memory mapping in high memory situations

such as deserializing face sets directly to USD prims?

I see two primary components:

• STEP parser
• Rust code generation from express schema

logos or nom are potential lower-level options for the first part. Ruststep appears to use nom so perhaps that component is already in hand.

Stepcode seems to be as the logical tool for the second part.

@TanJunKiat do you have a repo started already? Is ruststep complete enough to handle the parsing step?

[TanJunKiat]

@civilx64

Sorry but I haven't gotten the time to look in-depth into ruststep at the moment. I am trying to work on the Ifcopenshell-rust independently without other critical rust libraries to see if it is possible.

And yes, I have started a repository to test some concepts.

I will open the repository once I get it to the point that I think it's ready, hopefully some time around this month.

The current methodology that I am using is summarized as such:

Trait on IFC entities

All entities will be given a IfcDataOps trait that contains typical functions like parsing and getting ids. This also allows the model struct to contain a dynamic trait hashmap to store all the data.

Manual parsing

Right now the parsing of the data is pretty manual, with slicing and getting each element and storing them in the respective structs. Definitely would be helpful if we can have a dedicated serialiser/desarialiser for our purpose.

---

The biggest challenge right now for me is handling the large volume of IFC entities that needs to be programmed (e.g. to parse) and still figuring out the best way to approach the crate in general (e.g. proper use of traits, generics).

I will definitely be happy for the community to come together to do this together. With my limited knowledge on Rust and limited time to work on this, any help and advice is much appreciated.

[civilx64]

Sorry but I haven't gotten the time to look in-depth into ruststep at the moment.

No problem and no need to apologize! I was just wanting to document current status for the benefit of the community to minimize duplication of effort.

The biggest challenge right now for me is handling the large volume of IFC entities that needs to be programmed...

Indeed, generating the code for rust structs for all entities across multiple schema versions is near-impossible to do by hand. Some combination of stepcode and potentially LLM seems to be a promising option. In the meantime you might consider writing a generic EntityInstance struct that holds the IFC type as an attribute - similar to how things are exposed in the python wrapper.

With my limited knowledge on Rust and limited time to work on this...

I think this goes for most of us 😃.

[TanJunKiat]

Indeed, generating the code for rust structs for all entities across multiple schema versions is near-impossible to do by hand. Some combination of stepcode and potentially LLM seems to be a promising option. In the meantime you might consider writing a generic EntityInstance struct that holds the IFC type as an attribute - similar to how things are exposed in the python wrapper.

Yea, while tedious, I do find some incentive of writing the structs manually. Like embbed test cases for the differrent entities as well as documentation from web directly into the codebase.

We can definitely discuss and see what is the direction the community wants to go for.

[TanJunKiat]

Another note is that since we are writing the library in Rust, it might be worthwhile to gather feedback on what are the common issues in the existing libraries and see if we can solve them with the rust version.

[aothms]

In the meantime you might consider writing a generic EntityInstance struct that holds the IFC type as an attribute - similar to how things are exposed in the python wrapper.

Yes, this is the main design decision to make. In IfcOpenShell we have the schema types and data storage as separate things. I've recently rewrote quite some stuff. Data storage is a custom type called VariantArray. https://github.com/IfcOpenShell/IfcOpenShell/blob/v0.8.0/src/ifcparse/variantarray.h On top of the schema classes index into this array as functions with a return type. I imagined this would be best of both worlds: not too many code paths in the parser module and still type safety by means of the schema layer (that could potentially be inlined). This also allows to store invalid data types which we actually need for our validator.

[TanJunKiat]

Right now this is the approach that I am adopting for storing the entities, using just IfcDirection as example but can be applied for all entities.

Instead of a generic struct, I'm defining the data as a HashMap with the data index as the key and a Box<dyn trait> as the value. This allows us to store any kind of Struct that has this trait.

use log::debug;
use std::collections::HashMap;

#[derive(Debug, PartialEq, Eq, Clone)]
pub struct ParseError;

pub trait IfcDataOps {
    fn parse(s: &str) -> Result<Self, ParseError>
    where
        Self: Sized;
}

#[derive(Default)]
pub struct Model {
    data: HashMap<usize, Box<dyn IfcDataOps>>,
}

impl Model {
    pub fn open(file_path: &str) -> Model {
        let mut model = Model {
            data: HashMap::new(),
        };

        let contents = std::fs::read_to_string(file_path)
            .expect(format!("File path, {}, not available.", file_path).as_str());

        let lines: Vec<&str> = contents.split("\n").collect();

        for line in lines.iter() {
            if line.contains("=IFCDIRECTION(") {
                if let Ok(direction) = IfcDirection::parse(line) {
                    debug!("Parsed {:?}", direction);
                    model.data.insert(direction.id, Box::new(direction));
                }
            }
        }

        return model;
    }
}

#[derive(Debug)]
pub struct IfcDirection {
    id: usize,
    x: f32,
    y: f32,
    z: Option<f32>,
}

impl IfcDataOps for IfcDirection {
    fn parse(s: &str) -> Result<Self, ParseError> {
        let id = s[1..s.find("=").unwrap_or(s.len())]
            .parse::<usize>()
            .unwrap();

        let s = &s[s.find("((").unwrap_or(s.len()) + 2..s.len()];
        let s = &s[0..s.find("));").unwrap_or(s.len())];
        let mut coordinates = Vec::new();
        for (_, value) in s.split(",").enumerate() {
            coordinates.push(value.parse::<f32>().unwrap());
        }
        if coordinates.len() == 2 {
            return Ok(IfcDirection {
                id,
                x: coordinates[0],
                y: coordinates[1],
                z: None,
            });
        } else {
            return Ok(IfcDirection {
                id,
                x: coordinates[0],
                y: coordinates[1],
                z: Some(coordinates[2]),
            });
        }
    }
}

While programming, I want to make good use of some of the functions in Rust, such as using Option<> to define optional attributes and clever use of Results<> to control the workflow.

[aothms]

Have you investigated some of the parsing libraries out there. The step grammar is not that challenging, a little string manipulation goes a long way, but the problem is that newlines can exist anywhere in the file. And if you think about splitting on the semicolon, they can also exist within strings for example. Sorry to be that guy (but I don't know Rust well enough to do it myself) but I had generative AI generate the following for me, just to constrast with an other option:

use nom::{
    branch::alt,
    bytes::complete::{tag, take_while, take_until, is_not, take_till1},
    character::complete::{char, multispace0, multispace1, line_ending, anychar},
    combinator::{map, opt},
    multi::{many0, many1},
    sequence::{delimited, preceded, terminated, tuple},
    IResult,
};

#[derive(Debug, PartialEq)]
pub struct StepFile {
    pub header: HeaderSection,
    pub data: DataSection,
}

#[derive(Debug, PartialEq)]
pub struct HeaderSection {
    pub file_description: Option<String>,
    pub file_name: Option<String>,
    pub file_schema: Option<String>,
}

#[derive(Debug, PartialEq)]
pub struct DataSection {
    pub entities: Vec<EntityInstance>,
}

#[derive(Debug, PartialEq)]
pub struct EntityInstance {
    pub id: String,
    pub entity_type: String,
    pub parameters: String,
}

/// Utility parsers
fn ws(input: &str) -> IResult<&str, &str> {
    multispace0(input)
}

fn quoted_string(input: &str) -> IResult<&str, &str> {
    delimited(
        char('\''),
        take_while(|c| c != '\''),
        char('\'')
    )(input)
}

fn parenthesized(input: &str) -> IResult<&str, &str> {
    delimited(
        char('('),
        take_while(|c| c != ')'),
        char(')')
    )(input)
}

/// Parse the STEP file "envelope"
fn step_file(input: &str) -> IResult<&str, StepFile> {
    let (input, _) = ws(input)?;
    let (input, _) = tag("ISO-10303-21;")(input)?;
    let (input, header) = header_section(input)?;
    let (input, data) = data_section(input)?;
    let (input, _) = ws(input)?;
    let (input, _) = tag("END-ISO-10303-21;")(input)?;
    Ok((input, StepFile { header, data }))
}

/// Parse the HEADER section: HEADER; ... ENDSEC;
fn header_section(input: &str) -> IResult<&str, HeaderSection> {
    let (input, _) = ws(input)?;
    let (input, _) = tag("HEADER;")(input)?;
    let (input, _) = ws(input)?;
    let (input, file_desc) = opt(file_description)(input)?;
    let (input, file_name_val) = opt(file_name)(input)?;
    let (input, file_schema_val) = opt(file_schema)(input)?;
    let (input, _) = ws(input)?;
    let (input, _) = tag("ENDSEC;")(input)?;
    Ok((input, HeaderSection {
        file_description: file_desc,
        file_name: file_name_val,
        file_schema: file_schema_val,
    }))
}

fn file_description(input: &str) -> IResult<&str, String> {
    let (input, _) = ws(input)?;
    let (input, _) = tag("FILE_DESCRIPTION")(input)?;
    let (input, content) = parenthesized(input)?;
    let (input, _) = char(';')(input)?;
    Ok((input, content.to_string()))
}

fn file_name(input: &str) -> IResult<&str, String> {
    let (input, _) = ws(input)?;
    let (input, _) = tag("FILE_NAME")(input)?;
    let (input, content) = parenthesized(input)?;
    let (input, _) = char(';')(input)?;
    Ok((input, content.to_string()))
}

fn file_schema(input: &str) -> IResult<&str, String> {
    let (input, _) = ws(input)?;
    let (input, _) = tag("FILE_SCHEMA")(input)?;
    let (input, content) = parenthesized(input)?;
    let (input, _) = char(';')(input)?;
    Ok((input, content.to_string()))
}

/// Parse the DATA section: DATA; ... ENDSEC;
fn data_section(input: &str) -> IResult<&str, DataSection> {
    let (input, _) = ws(input)?;
    let (input, _) = tag("DATA;")(input)?;
    let (input, entities) = many0(entity_instance)(input)?;
    let (input, _) = ws(input)?;
    let (input, _) = tag("ENDSEC;")(input)?;
    Ok((input, DataSection { entities }))
}

/// Parse a single entity instance: something like:
/// #1=PRODUCT('ID','Name','Description',(#2),$);
fn entity_instance(input: &str) -> IResult<&str, EntityInstance> {
    let (input, _) = ws(input)?;
    let (input, _) = char('#')(input)?;
    let (input, id) = take_while(|c: char| c.is_digit(10))(input)?;
    let (input, _) = char('=')(input)?;
    // entity type name
    let (input, entity_type) = take_while(|c: char| c.is_alphabetic() || c == '_')(input)?;
    // parameters
    let (input, params) = parenthesized(input)?;
    let (input, _) = char(';')(input)?;

    Ok((input, EntityInstance {
        id: id.to_string(),
        entity_type: entity_type.to_string(),
        parameters: params.to_string(),
    }))
}

// For demonstration, let's run a test
fn main() {
    let example = r#"
ISO-10303-21;
HEADER;
FILE_DESCRIPTION(('Example STEP'), '1');
FILE_NAME(('example.stp'),('2024-12-11T12:00:00'),('Example'),('ExampleAuthor'),('Company'), 'Preprocessor', 'AuthoringTool');
FILE_SCHEMA(('AUTOMOTIVE_DESIGN'));
ENDSEC;
DATA;
#1=PRODUCT('ID','Name','Description',(#2),$);
#2=PRODUCT_CONTEXT('ContextID','ContextType',$);
ENDSEC;
END-ISO-10303-21;"#;

    let parsed = step_file(example);
    match parsed {
        Ok((_, step)) => {
            println!("Parsed STEP file: {:#?}", step);
        }
        Err(e) => eprintln!("Error parsing: {:?}", e),
    }
}

[TanJunKiat]

@aothms

No worries, appreciate the honesty.

Yea, I agree that a more rigorous and robust parser is needed. So, hopefully, someone can help and look into that once we get started.

Meanwhile I'll check if there are alternatives to improve the existing.

Yea, I did run into similar issues with parsing entities with IfcLabel and IfcText that contains commas

#17476=IFCSURFACESTYLE('Glass - Clear, Grey',.BOTH.,(#17475));

It screws up the parser logic.

[TanJunKiat]

Some findings for discussion.

Using nom

use nom::{
    bytes::complete::{tag, take_while},
    character::complete::{char, digit1},
    IResult,
};

#[derive(Debug, PartialEq)]
pub struct IfcCartesianPoint {
    id: usize,
    x: f64,
    y: f64,
    z: Option<f64>,
}

impl IfcCartesianPoint {
    fn parse(input: &str) -> IResult<&str, IfcCartesianPoint> {
        let (input, _) = tag("#")(input)?;
        let (input, id) = digit1(input)?;
        let id: usize = id.parse().unwrap();
        let (input, _) = tag("=IFCCARTESIANPOINT((")(input)?;
        let (input, x) = parse_float(input)?;
        let (input, _) = char(',')(input)?;
        let (input, y) = parse_float(input)?;
        let (input, z) = if let Ok((input, _)) = char::<_, nom::error::Error<_>>(',')(input) {
            let (input, z) = parse_float(input)?;
            (input, Some(z))
        } else {
            (input, None)
        };
        let (input, _) = tag("));")(input)?;
        Ok((input, IfcCartesianPoint { id, x, y, z }))
    }
}

fn parse_float(input: &str) -> IResult<&str, f64> {
    let (input, float_str) =
        take_while(|c: char| c.is_digit(10) || c == '.' || c == '-' || c == '+')(input)?;
    let float: f64 = float_str.parse().unwrap();
    Ok((input, float))
}

pub fn parse(file_path: &str) -> Vec<IfcCartesianPoint> {
    let mut points = Vec::new();
    let contents = std::fs::read_to_string(file_path).expect("File path not available.");
    let lines: Vec<&str> = contents.split('\n').collect();
    for line in lines.iter() {
        if line.contains("IFCCARTESIANPOINT") {
            match IfcCartesianPoint::parse(line) {
                Ok((_, point)) => {
                    points.push(point);
                }
                Err(e) => {
                    log::warn!("Error parsing line: {}", e);
                }
            }
        }
    }
    return points;
}

fn main() {
    env_logger::init();
    let start = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap();
    eprintln!("{:?}", start);
    let points = parse("assets/ifc2x3/clinic.ifc");
    eprintln!("{:?}", points.len());
    let end = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap();
    eprintln!("{:?}", end);
    eprintln!("{:?}", end - start);
}

Using regex

use regex::Regex;

pub struct ParseError;

#[derive(Debug, PartialEq)]
pub struct IfcCartesianPoint {
    id: usize,
    x: f64,
    y: f64,
    z: Option<f64>,
}

impl IfcCartesianPoint {
    fn parse(s: &str) -> Result<Self, ParseError> {
        let cartesian_point_re =
            Regex::new(r"#(\d+)=IFCCARTESIANPOINT\(\(([\d\.,\-]+)\)\);").unwrap();
        if let Some(captures) = cartesian_point_re.captures(&s) {
            let id = captures.get(1).unwrap().as_str().to_string();
            let coordinates: Vec<f64> = captures
                .get(2)
                .unwrap()
                .as_str()
                .split(',')
                .map(|s| s.parse::<f64>().unwrap())
                .collect();
            if coordinates.len() == 3 {
                return Ok(IfcCartesianPoint {
                    id: id.parse().unwrap(),
                    x: coordinates[0],
                    y: coordinates[1],
                    z: Some(coordinates[2]),
                });
            } else {
                return Ok(IfcCartesianPoint {
                    id: id.parse().unwrap(),
                    x: coordinates[0],
                    y: coordinates[1],
                    z: None,
                });
            }
        }
        return Err(ParseError);
    }
}

fn parse(file_path: &str) -> Vec<IfcCartesianPoint> {
    let mut points: Vec<IfcCartesianPoint> = Vec::new();
    let contents: String = std::fs::read_to_string(file_path).expect("File path not available.");
    let lines: Vec<&str> = contents.split("\n").collect();

    for line in lines.iter() {
        if line.contains("=IFCCARTESIANPOINT(") {
            match IfcCartesianPoint::parse(line) {
                Ok(point) => {
                    points.push(point);
                }
                Err(e) => {
                    log::warn!("Error parsing line");
                }
            }
        }
    }
    return points;
}

fn main() {
    env_logger::init();
    let start = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap();
    eprintln!("{:?}", start);
    let points = parse("assets/ifc2x3/clinic.ifc");
    eprintln!("{:?}", points.len());
    let end = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap();
    eprintln!("{:?}", end);
    eprintln!("{:?}", end - start);
}

Manual str parsing

pub struct ParseError;

#[derive(Debug)]
pub struct IfcCartesianPoint {
    id: usize,
    x: f32,
    y: f32,
    z: Option<f32>,
}

impl IfcCartesianPoint {
    fn parse(s: &str) -> Result<Self, ParseError> {
        let id = s[1..s.find("=").unwrap_or(s.len())]
            .parse::<usize>()
            .unwrap();

        let s = &s[s.find("((").unwrap_or(s.len()) + 2..s.len()];
        let s = &s[0..s.find("));").unwrap_or(s.len())];
        let mut coordinates = Vec::new();
        for (_, value) in s.split(",").enumerate() {
            coordinates.push(value.parse::<f32>().unwrap());
        }
        if coordinates.len() == 2 {
            return Ok(IfcCartesianPoint {
                id,
                x: coordinates[0],
                y: coordinates[1],
                z: None,
            });
        } else {
            return Ok(IfcCartesianPoint {
                id,
                x: coordinates[0],
                y: coordinates[1],
                z: Some(coordinates[2]),
            });
        }
    }
}

pub fn parse(file_path: &str) -> Vec<Box<IfcCartesianPoint>> {
    let mut points = Vec::new();
    let contents: String = std::fs::read_to_string(file_path).expect("File path not available.");
    let lines: Vec<&str> = contents.split("\n").collect();
    for line in lines.iter() {
        if line.contains("=IFCCARTESIANPOINT(") {
            if let Ok(direction) = IfcCartesianPoint::parse(line) {
                points.push(Box::new(direction));
            }
        }
    }
    return points;
}

fn main() {
    env_logger::init();
    let start = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap();
    eprintln!("{:?}", start);
    let points = parse("assets/ifc2x3/clinic.ifc");
    eprintln!("{:?}", points.len());
    let end = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap();
    eprintln!("{:?}", end);
    eprintln!("{:?}", end - start);
}

Findings

• Preliminary test with regex concludes that it is not suitable because it takes too long to parse a huge IFC file, though it offers the simplest and cleanest code implementation.
• Both manual and nom approaches work, both scoring less than 0.5 seconds for parsing a 300k+ lines IFC file with 80k+ matching entities. But it seems like 'nom' failed to parse some of the variables. (need to look into the cause).

Some data to share

	regex	manual
Speed	~400ms	~300ms
Entities parsed (counted 87278)	87152	87278

For now, I will stick with the manual method as it is the fastest method to get something working in hopes that we will attract more contributors and devise a better solution for parsing.

[aothms]

I would advise to move forward with a formal grammar as soon as possible. See https://www.steptools.com/stds/step/IS_final_p21e3.html#clause-5 for the definition. String splitting gets more and more unwieldy once you start to incorporate some of the string escaping rules. But I agree it's a good idea to assess performance, and maybe include some of the other libraries such as lalrpop.

Secondly, a schema is actually only a nice to have. As a first starting point I would start with an untyped parser. With maybe the simplest API possible as something like:

• pub get_instances_of_type(str entity_name) -> Vec<int>
• pub get_attribute_value_str(usize entity_instance_id, usize attribute_index) -> Maybe<str>
• pub get_attribute_value_int(usize entity_instance_id, usize attribute_index) -> Maybe<int>
• ...

That try to coerce the value found in the AST into the type requested.

That way you can already outline a minimal example application without any schema knowledge embedded into the program.

[TanJunKiat]

Hi everyone.

Thanks for all the active support and discussions.

I feel like we can really start pushing this and see if we can get it to a level where the community can come in to contribute.

I have taken the liberty to start a new GitHub organisation for now, ifcopenshell-rust, with the main repository being here.

I am not sure how to navigate this, but maybe eventually we can merge it to this organisation when the project matures.

But I believe having a proper channel for discussion and documentation will help push the project forward.

I have also created two discussions, which are the two most important tasks to iron out:

1. Designing and implementing a parser (click here)
2. Designing and implementing the data storage methodology/ architecture (click here)

Also, I would like to extend an invitation to keen members to take on shared responsibility for managing and overseeing the project. We could definitely get some help in terms of GitHub and project management. Please reach out to me if you are interested.

[civilx64]

Thanks for taking the initiative to set up a working location. On license you might consider LGPL or GPL for future potential compatibility with the overall IfcOpenShell project - but IANAL.

Also it will be a few days minimum before I can review or provide meaningful input.

[TanJunKiat]

@civilx64

Yes, we definitely need to set up a committee or at least invite a few of the current IfcOpenShell developers to comment and advise on the licensing.

Definitely not a decision to be made lightly or by one single individual.

But sure, we can stick to LGPL for now.

[aothms]

Just one thing to put out there. IFC5 appears to be rather fundamentally different. https://github.com/buildingSMART/IFC5-development We could also prioritize support for IFC5. It requires a simpler stack because it;s not a bespoke grammar, just JSON. Maybe the timing is also nice since it's new and since its integration into ifcopenshell might also be more complementary because we don't have anything for ifc5 yet.