Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] mc-apply script doesn't function #27612

Open
1 task done
CRCinAU opened this issue Dec 29, 2024 · 8 comments
Open
1 task done

[BUG] mc-apply script doesn't function #27612

CRCinAU opened this issue Dec 29, 2024 · 8 comments

Comments

@CRCinAU
Copy link
Contributor

CRCinAU commented Dec 29, 2024

Did you test the latest bugfix-2.1.x code?

Yes, and the problem still exists.

Bug Description

If CONFIGURATION_EMBEDDING is enabled, the M503 C command will write MC.ZIP to the printers SD card with the compile time settings used.

According to docs/ConfigEmbedding.md, the following should be used to then apply that config template back to the .h files:

$ git checkout -f
$ unzip mc.zip
$ python buildroot/share/PlatformIO/scripts/mc-apply.py

When attempting this, the script always returns:

$ python3 ../buildroot/share/PlatformIO/scripts/mc-apply.py 
No marlin_config.json found.

Removing the main try / except block, I get:

$ python3 buildroot/share/PlatformIO/scripts/mc-apply.py 
Traceback (most recent call last):
  File "/home/netwiz/git/Marlin/buildroot/share/PlatformIO/scripts/mc-apply.py", line 22, in <module>
    for k, v in sorted(conf[key].items()):
                       ^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'items'

When adding a debug to see what key contains, I see:

$ python3 buildroot/share/PlatformIO/scripts/mc-apply.py 
Key: Z_SAFE_HOMING_X_POINT
Traceback (most recent call last):
  File "/home/netwiz/git/Marlin/buildroot/share/PlatformIO/scripts/mc-apply.py", line 23, in <module>
    for k, v in sorted(conf[key].items()):
                       ^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'items'

As a result, no configuration files are modified and the marlin_config.json file cannot be folded back into the configuration.

Bug Timeline

Likely a while.

Expected behavior

The script should restore config settings from marlin_config.json to the related .h files.

Actual behavior

Error in output.

Steps to Reproduce

See main description.

Version of Marlin Firmware

bugfix-2.1.x

Printer model

No response

Electronics

No response

LCD/Controller

No response

Other add-ons

No response

Bed Leveling

None

Your Slicer

None

Host Software

None

Don't forget to include

  • A ZIP file containing your Configuration.h and Configuration_adv.h.

Additional information & file uploads

Added example marlin_config.json as per output of M503 C.
marlin_config.json

@ellensp
Copy link
Contributor

ellensp commented Dec 29, 2024

in older marlin CONFIG_EXPORT 1 the file marlin_config.json used to start with
'{"__INITIAL_HASH":"6c2dd0c6a928862c1c0c","Configuration.h":{"VALIDATE_HOMING_ENDSTOPS"

and the script got Configuration.h from here.

now CONFIG_EXPORT 1 marlin_config.json starts with
{"Z_SAFE_HOMING_X_POINT":"X_CENTER","VALIDATE_HOMING_ENDSTOPS":"","Y_DRIVER_TYPE":"TMC2209","PREHEAT_1_LABEL":""PLA"","EXTRUDE_MAXLENGTH":"600","MPC_STEADYSTATE":"0.5f","TEMP_WINDOW":"1","E

CONFIG_EXPORT 101 adds the filenames back in

@ellensp
Copy link
Contributor

ellensp commented Dec 29, 2024

Potential fix/rewrite Has a lot better error detection (doesn't say "No marlin_config.json found." on every single error) and is actually human readable.

options
--opt creates a .sh script with opt_ command
--bare-output creates a stripped down bare Configuration.h

Can others please re-test, my brain has shutdown for the week.

#!/usr/bin/env python
#
# mc-apply.py
# Create a Configuration from marlin_config.json
#
import json
import sys
import shutil
import os

def load_config(file_path):
    try:
        with open(file_path, 'r') as file:
            return json.load(file)
    except FileNotFoundError:
        print(f'Error: {file_path} not found.')
        sys.exit(1)
    except json.JSONDecodeError:
        print(f'Error: Failed to decode JSON from {file_path}.')
        sys.exit(1)

def write_output_file(file_path, content):
    try:
        with open(file_path, 'w') as outfile:
            outfile.write(content)
    except IOError as e:
        print(f'Error: Failed to write to {file_path}. {e}')
        sys.exit(1)

def move_file(src, dst):
    try:
        shutil.move(src, dst)
    except IOError as e:
        print(f'Error: Failed to move {src} to {dst}. {e}')
        sys.exit(1)

def process_configuration(conf, opt_output, output_suffix):
    for _, values in conf.items():
        for key in conf:
            if key == '__INITIAL_HASH':
                continue
            if key == 'VERSION':
                for k, v in sorted(values.items()):
                    print(f'{k}: {v}')
                continue

        output_file_path = os.path.join('Marlin', key + output_suffix)
        content = ''
        for k, v in sorted(values.items()):
            if opt_output:
                if v:
                    if '"' in v:
                        v = f"'{v}'"
                    elif ' ' in v:
                        v = f'"{v}"'
                    define = f'opt_set {k} {v}\n'
                else:
                    define = f'opt_enable {k}\n'
            else:
                define = f'#define {k} {v}\n'
            content += define

        write_output_file(output_file_path, content)

        if output_suffix:
            original_file_path = os.path.join('Marlin', key)
            backup_file_path = original_file_path + '.orig'
            move_file(original_file_path, backup_file_path)

            try:
                with open(backup_file_path, 'r') as file:
                    file_lines = file.read().split('\n')
            except IOError as e:
                print(f'Error: Failed to read from {backup_file_path}. {e}')
                sys.exit(1)

            content = ''
            for line in file_lines:
                sline = line.strip(" \t\n\r")
                if sline.startswith("#define"):
                    kv = sline[8:].strip().split(' ')
                    if kv[0] in values:
                        content += f'#define {kv[0]} {values[kv[0]]}\n'
                        del values[kv[0]]
                    else:
                        content += line + '\n'
                else:
                    content += line + '\n'

            for k, v in sorted(values.items()):
                content += f'#define {k} {v}\n'

            write_output_file(original_file_path, content)

        print(f'Output configuration written to: {output_file_path}')

def main():
    opt_output = '--opt' in sys.argv
    output_suffix = '.sh' if opt_output else '' if '--bare-output' in sys.argv else '.gen'
    config_file_path = 'marlin_config.json'

    conf = load_config(config_file_path)
    found_configuration_h = 'Configuration.h' in conf
    if not found_configuration_h:
        conf = {'Configuration.h': conf}
    process_configuration(conf, opt_output, output_suffix)

if __name__ == '__main__':
    main()

need to look into what is it meant to do with Configuration_adv.h ?
Seems to be just absent from the marlin_config.json so perhaps it is a creation issue... so this script not working may just be a symptom

@CRCinAU
Copy link
Contributor Author

CRCinAU commented Dec 29, 2024

Yeah, there's still a few cases that this doesn't cover (did it ever?)

  1. Configuration_adv.h as noted. Maybe its just better to load both into different string variables and search both for key?

Or am I misunderstanding and no values from Configuration_adv.h are actually making it into the embedded config? If this is the case, then we probably should look at fixing that too.

  1. Values that are added but are not mentioned in any .h file present

There are a bunch of opt_* files in buildroot/bin/ that seem to basically be the shell script equivalents to what mc-apply.py would probably need to have for proper functionality - however these are obviously designed to be run individually on a per config option basis.

@ellensp
Copy link
Contributor

ellensp commented Dec 29, 2024

--opt creates Configuration.h.sh which contains

eg
opt_enable ARC_SUPPORT
opt_enable AUTOTEMP
opt_set AUTOTEMP_FACTOR 0.1f
opt_set AUTOTEMP_MAX 250
opt_set AUTOTEMP_MIN 210
opt_set AUTOTEMP_OLDWEIGHT 0.98
etc

@ellensp
Copy link
Contributor

ellensp commented Dec 29, 2024

I see a light at the end of the tunnel...

In Configuration_adv.h is

 *  1 = marlin_config.json - Dictionary containing the configuration.
 *      This file is also generated for CONFIGURATION_EMBEDDING.
 *  2 = config.ini - File format for PlatformIO preprocessing.
 *  3 = schema.json - The entire configuration schema. (13 = pattern groups)
 *  4 = schema.yml - The entire configuration schema.
 *  5 = Config.h - Minimal configuration by popular demand.
 */
#define CONFIG_EXPORT 105 // :[1:'JSON', 2:'config.ini', 3:'schema.json', 4:'schema.yml', 5:'Config.h']

but the value is > 5
So what gives there ?

The answer is

    # Get the CONFIG_EXPORT value and do an extended dump if > 100
    # For example, CONFIG_EXPORT 102 will make a 'config.ini' with a [config:] group for each schema @section

101 processes both Configuration.h and Configuration_adv.h and puts the file name in as expected
(and Version.h, but thats a little broken)

@Ferrograph
Copy link

I tried the new scrip above but it still outputs not found.

PS D:\Working\PROJECTS\Marlin\Code> python buildroot/share/PlatformIO/scripts/mc-apply.py 
Error: marlin_config.json not found.
PS D:\Working\PROJECTS\Marlin\Code> 

@ellensp
Copy link
Contributor

ellensp commented Dec 31, 2024

it expects marlin_config.json in the directory you invoke the script from ie in this case
D:\Working\PROJECTS\Marlin\Code\marlin_config.json

But its not complete yet...

@Ferrograph
Copy link

Ferrograph commented Dec 31, 2024

Ok, Ive tried it in there (the root folder with configs h, and /src. I'll wait for an update.

Thanks for all your hard work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants