build: Add rules in Makefile.gen to generate a hex file #1447

jukkar · 2017-09-12T09:26:24Z

This commit is useful if there is a need to generate a file that can be included into the application at build time.
The file can also be compressed automatically when embedding it.

Files to be generated are listed in generate_inc_file and generate_inc_gz_file variables.

How to use this commit in your application:

Add this to your application Makefile

   SRC = $(ZEPHYR_BASE)/tests/lib/gen_inc_file/src
   include $(ZEPHYR_BASE)/Makefile.gen

Add needed binary/other embedded files into src/Makefile
to "generate_inc_file" or "generate_inc_gz_file" variables:

   # List of files that are used to generate a file that can be
   # included into .c file.
   generate_inc_file += \
           echo-apps-cert.der \
           echo-apps-key.der \
           file.bin

   generate_inc_gz_file += \
           index.html

   include $(ZEPHYR_BASE)/Makefile.gen

In the application, do something with the embedded file

   static const unsigned char inc_file[] = {
   #include "file.bin.inc"
   };

   static const unsigned char gz_inc_file[] = {
   #include "index.html_gz.inc"
   };

The generated files in src/*.inc are automatically removed when you do "make pristine"

pfalcon · 2017-09-12T10:29:48Z

I tried to apply this to #1378 using the patch below. Nothing is generated, what am I doing wrong?

Generally, this patch uses permutations of the same name with one-char difference: generate_inc_file, generate_inc_files, generated_inc_files, so it's hard to even spot if there can be a typo.

diff --git a/samples/net/sockets/socket_dumb_http/Makefile b/samples/net/sockets/socket_dumb_http/Makefile
index 58c183b5c..e804c6bf5 100644
--- a/samples/net/sockets/socket_dumb_http/Makefile
+++ b/samples/net/sockets/socket_dumb_http/Makefile
@@ -16,3 +16,10 @@ ifeq ($(CONFIG_NET_L2_BLUETOOTH), y)
 else
 	include $(ZEPHYR_BASE)/samples/net/common/Makefile.ipstack
 endif
+
+APP_SRC = $(ZEPHYR_BASE)/samples/net/sockets/socket_dumb_http/src
+
+all: generate_inc_files
+
+generate_inc_files:
+	$(MAKE) -C $(APP_SRC) generate_inc_files Q=$(Q)
diff --git a/samples/net/sockets/socket_dumb_http/src/Makefile b/samples/net/sockets/socket_dumb_http/src/Makefile
index 265f6807f..12958229a 100644
--- a/samples/net/sockets/socket_dumb_http/src/Makefile
+++ b/samples/net/sockets/socket_dumb_http/src/Makefile
@@ -1,3 +1,7 @@
 include $(ZEPHYR_BASE)/samples/net/common/Makefile.common
 
 obj-y += socket_dumb_http.o
+
+generate_inc_file += \
+	response_small.html \
+	response_big.html

nashif

Prefer to have this tool in python, this way we minimise dependency on host compilers

jukkar · 2017-09-12T11:59:57Z

I tried to apply this to #1378 using the patch below. Nothing is generated, what am I doing wrong?

I forgot one line you need to add to your src/Makefile, put that after setting generate_inc_file variable.

include $(ZEPHYR_BASE)/samples/net/common/Makefile.common

jukkar · 2017-09-12T12:01:18Z

Prefer to have this tool in python, this way we minimise dependency on host compilers

If you look the stdin2hex.c file, it is very minimalistic ANSI-C code which should compile everywhere.

nashif · 2017-09-12T12:13:59Z

If you look the stdin2hex.c file, it is very minimalistic ANSI-C code which should compile everywhere.

Thats not the point, it still requires a host compiler, something we are trying to avoid, especially on windows

pfalcon · 2017-09-12T12:33:13Z

@jukkar:

I forgot one line you need to add to your src/Makefile, put that after setting generate_inc_file variable.
include $(ZEPHYR_BASE)/samples/net/common/Makefile.common

There was such a line - in the very beginning of src/Makefile (as can be seen at the diff above). And expectedly, I added new content at the very end of it. It worked after I moved "generate_inc_file +=" stanza above the include, thanks. But all in all, it's pretty confusing, and I'd recommend describing that requirement explicitly in the commit message.

pfalcon · 2017-09-12T12:37:49Z

I'd personally want to merge this asap, but as @nashif raises an issue requiring refactoring anyway, can we also avoid black magicity like:

{
 #include "file_index.html_gz.h"

        NET_DBG("Sending index.html (%d bytes) to client",
                sizeof(file_index_html_gz));

The canonical way (as in: I kinda never saw it done differently) to do that is: an include file to contain only numeric initializer list, to be used as:

MODIFIERS_I_WANT_TO_USE char name_i_want_to_use[] = {
#include "data.inc"
};

pfalcon · 2017-09-12T12:40:10Z

And, @jukkar, did you really consider other options to have it done? Are we really can't have it e.g. like:

src/Makefile:

...

my.inc: some.bin
        $(BIN_2_H_RULE)

?

jukkar · 2017-09-12T12:45:20Z

Thats not the point, it still requires a host compiler, something we are trying to avoid, especially on windows

What python version we need, 3 or 2?

jukkar · 2017-09-12T12:47:02Z

And, @jukkar, did you really consider other options to have it done? Are we really can't have it e.g. like:

I tried that but did not manage it to work, perhaps with some makefile woodoo magic or something it could be done.

jukkar · 2017-09-12T12:48:05Z

The canonical way (as in: I kinda never saw it done differently) to do that is: an include file to contain only numeric initializer list, to be used as:

Sure, we can do it like that too. That is actually better as then application can define the variable name/type itself.

pfalcon · 2017-09-12T12:48:36Z

Also, commit message "samples: net: common: Add rules to generate a header file" doesn't do good to describe what actually happens here, and chance the "convert binary" key phrase can be fitted in there?

pfalcon · 2017-09-12T12:49:37Z

That is actually better as then application can define the variable name/type itself.

Yep, also easier for a reader, as one can grep where the variable is defined easily. Thanks.

nashif · 2017-09-12T14:50:33Z

What python version we need, 3 or 2?

python 3

jukkar · 2017-09-13T08:11:41Z

Sent a new version. I removed dependency to net subsystem as this can be made more generic. Also replaced the binary by python3 script that generates the hex value list.

pfalcon

This is definitely a good improvement over the initial version. Tested against #1378, works well. Thanks!

jukkar · 2017-09-13T15:09:05Z

I think this can be simplified even further. I will send a new version.

jukkar · 2017-09-15T09:25:38Z

@nashif Are you ok with this PR?

nashif · 2017-09-17T21:33:47Z

move Makefile.gen to scripts/ and move the sample to tests/application_developments, not sure why this is under lib/, otherwise ok.

This commit is useful if there is a need to generate a file that can be included into the application at build time. The file can also be compressed automatically when embedding it. Files to be generated are listed in generate_inc_file generate_inc_gz_file variables. How to use this commit in your application: 1. Add this to your application Makefile SRC = $(ZEPHYR_BASE)/<your-app-dir>/src include $(ZEPHYR_BASE)/scripts/Makefile.gen 2. Add needed binary/other embedded files into src/Makefile to "generate_inc_file" or "generate_inc_gz_file" variables: # List of files that are used to generate a file that can be # included into .c file. generate_inc_file += \ echo-apps-cert.der \ echo-apps-key.der \ file.bin generate_inc_gz_file += \ index.html include $(ZEPHYR_BASE)/scripts/Makefile.gen 3. In the application, do something with the embedded file static const unsigned char inc_file[] = { #include "file.bin.inc" }; static const unsigned char gz_inc_file[] = { #include "index.html.gz.inc" }; The generated files in ${SRC}/*.inc are automatically removed when you do "make pristine" Signed-off-by: Jukka Rissanen <[email protected]>

Add a simple test which generates a file that can be included into a .c file. Then verify in zephyr that the file contains the same bytes as the original file. Signed-off-by: Jukka Rissanen <[email protected]>

jukkar · 2017-09-18T07:53:58Z

New version changed according to comment and uploaded for review.

pfalcon

Retested with #1378

jukkar added the net label Sep 12, 2017

jukkar self-assigned this Sep 12, 2017

jukkar requested a review from pfalcon September 12, 2017 09:26

ghost added the In progress For PRs: is work in progress and should not be merged yet. For issues: Is being worked on label Sep 12, 2017

jukkar mentioned this pull request Sep 12, 2017

samples: sockets: Add a dumb HTTP server example #1378

Merged

jukkar requested a review from nashif September 12, 2017 09:27

nashif requested changes Sep 12, 2017

View reviewed changes

jukkar force-pushed the net-gen-inc-file branch from 187fa89 to 75dd24d Compare September 13, 2017 08:10

jukkar changed the title ~~samples: net: common: Add rules to generate a header file~~ build: Add rules to generate a header file Sep 13, 2017

jukkar changed the title ~~build: Add rules to generate a header file~~ build: Add rules in Makefile.gen to generate a hex file Sep 13, 2017

pfalcon previously approved these changes Sep 13, 2017

View reviewed changes

jukkar dismissed pfalcon’s stale review via 2dc9295 September 13, 2017 15:37

jukkar force-pushed the net-gen-inc-file branch from 75dd24d to 2dc9295 Compare September 13, 2017 15:37

jukkar removed the net label Sep 15, 2017

jukkar added 2 commits September 18, 2017 10:48

tests: Add testing for embedding a hex file

9107261

Add a simple test which generates a file that can be included into a .c file. Then verify in zephyr that the file contains the same bytes as the original file. Signed-off-by: Jukka Rissanen <[email protected]>

jukkar force-pushed the net-gen-inc-file branch from 2dc9295 to 9107261 Compare September 18, 2017 07:50

pfalcon approved these changes Sep 18, 2017

View reviewed changes

nashif approved these changes Sep 18, 2017

View reviewed changes

nashif merged commit 7830e60 into zephyrproject-rtos:master Sep 18, 2017

ghost removed the In progress For PRs: is work in progress and should not be merged yet. For issues: Is being worked on label Sep 18, 2017

jukkar deleted the net-gen-inc-file branch June 21, 2018 06:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: Add rules in Makefile.gen to generate a hex file #1447

build: Add rules in Makefile.gen to generate a hex file #1447

jukkar commented Sep 12, 2017 •

edited

Loading

pfalcon commented Sep 12, 2017

nashif left a comment

jukkar commented Sep 12, 2017

jukkar commented Sep 12, 2017

nashif commented Sep 12, 2017

pfalcon commented Sep 12, 2017

pfalcon commented Sep 12, 2017

pfalcon commented Sep 12, 2017

jukkar commented Sep 12, 2017

jukkar commented Sep 12, 2017

jukkar commented Sep 12, 2017

pfalcon commented Sep 12, 2017

pfalcon commented Sep 12, 2017

nashif commented Sep 12, 2017

jukkar commented Sep 13, 2017

pfalcon left a comment

jukkar commented Sep 13, 2017

jukkar commented Sep 15, 2017

nashif commented Sep 17, 2017

jukkar commented Sep 18, 2017

pfalcon left a comment

build: Add rules in Makefile.gen to generate a hex file #1447

build: Add rules in Makefile.gen to generate a hex file #1447

Conversation

jukkar commented Sep 12, 2017 • edited Loading

pfalcon commented Sep 12, 2017

nashif left a comment

Choose a reason for hiding this comment

jukkar commented Sep 12, 2017

jukkar commented Sep 12, 2017

nashif commented Sep 12, 2017

pfalcon commented Sep 12, 2017

pfalcon commented Sep 12, 2017

pfalcon commented Sep 12, 2017

jukkar commented Sep 12, 2017

jukkar commented Sep 12, 2017

jukkar commented Sep 12, 2017

pfalcon commented Sep 12, 2017

pfalcon commented Sep 12, 2017

nashif commented Sep 12, 2017

jukkar commented Sep 13, 2017

pfalcon left a comment

Choose a reason for hiding this comment

jukkar commented Sep 13, 2017

jukkar commented Sep 15, 2017

nashif commented Sep 17, 2017

jukkar commented Sep 18, 2017

pfalcon left a comment

Choose a reason for hiding this comment

jukkar commented Sep 12, 2017 •

edited

Loading