Methods for object resolves #1224

worksofliam · 2023-04-06T18:43:30Z

Changes

Adds new methods to help with resolving paths to source members and streamfiles.

Checklist

have tested my change
updated relevant documentation
Remove any/all console.logs I added
eslint is not complaining
have added myself to the contributors' list in CONTRIBUTING.md
for feature PRs: PR only includes one feature enhancement.

Signed-off-by: Liam Allan <[email protected]>

worksofliam · 2023-04-06T18:52:16Z

src/api/IBMiContent.ts

+    return undefined;
+  }
+
+  async streamfileResolve(name: string, directories: QsysPath[]): Promise<string|undefined> {


@edmundreinhardt @SanjulaGanepola Looks like there is no way to /usr/bin/find to take a list of file names.

Taking a look at the documentation, the find command supports the OR operator. I think we can handle a list of file names using this?

@SanjulaGanepola Thanks - that syntax works. Working on that now.

-bash-5.1$ /usr/bin/find /QSYS.LIB/LIAMA.LIB/QRPGLESRC.FILE/ /QSYS.LIB/LIAMA.LIB/QRPGLEREF.FILE -name 'BADEMPS.*' -o -name 'BANG.*'

Signed-off-by: Liam Allan <[email protected]>

chrjorgensen · 2023-04-07T21:51:34Z

@worksofliam Code looks good to me - but I have no way of testing this, since the new methods are not being called anywhere.

Btw, is this first step to make copy member resolution work like the RPG compiler? 🤞

worksofliam · 2023-04-07T23:16:16Z

@chrjorgensen

Btw, is this first step to make copy member resolution work like the RPG compiler?

That's exactly right!

I do hope in the future we can write tests for this also. (#1179)

Signed-off-by: Liam Allan <[email protected]>

worksofliam · 2023-04-10T01:35:15Z

I have added tests for these new APIs.

@SanjulaGanepola Note that find searches IFS directories recursively.

Signed-off-by: Liam Allan <[email protected]>

worksofliam · 2023-04-11T14:54:35Z

src/api/IBMiContent.ts

+    if (find) {
+      const command = [
+        find,
+        ...directories,


Still need to work out how we can get this to be non-recursive and only look in the directories provided.

https://unix.stackexchange.com/questions/372994/equivalent-maxdepth-for-find-in-aix

chrjorgensen

@worksofliam I think we're on the wrong track here...

You basically only want to check for whether some specific members exists.
This can easily be accomplished by a simple script:

for m in /QSYS.LIB/VSCODEIBMI.LIB/CCSID1147.FILE/MEMBER.MBR /QSYS.LIB/QGPL.LIB/CCSID1147.FILE/MEMBER.MBR;
do
  if [ -f $m ]
  then
    echo $m
  fi
done

This also works for streamfiles:

for m in /TMP/testing/subdir/test.SQL /tmp/testing/test.sql;
do
  if [ -f $m ]
  then
    echo $m
  fi
done

No need for find or any other utility...

Building the script can be done in a few lines - the list of files probably even in your favorite one-liners...! 😄

worksofliam · 2023-04-18T15:05:56Z

@chrjorgensen There is a reason we're standardising here with these APIs - I don't want to write the same script in every extension (this will be used in RPGLE, CL, etc!)

Though that is very cool that you have figured out how to keep it simple - how do think we could go about using your idea inside of our API without find?

chrjorgensen · 2023-04-18T15:07:47Z

@worksofliam Since copy members can be specified in different ways, it's probably a good idea to check the
/COPY documentation for how unspecified elements are handled.

IMHO Especially the following are important details:

The default sourcefile searched in the /QSYS.LIB file system - QRPGLESRC
When the library list are searched
default suffixes for IFS include files
Maybe also that the IFS is also searched when no member in /QSYS.LIB is found

worksofliam · 2023-04-18T15:10:31Z

@chrjorgensen The specifics of those rules will be handled in the right language server. The rules you mentioned are specific to RPGLE, whereas CL and COBOL have different rules. The way I implemented these APIs will allow each language server to be flexible and implement the rules separately, but still rely on the same resolver

worksofliam · 2023-05-04T15:40:59Z

@chrjorgensen Tried to use your suggestion, but it is not working as expected. Currently, find only returns the path if the file actually exists, and this suggestion returns the string nonetheless.

for f in /QSYS.LIB/QGPL.LIB/H.FILE/MATH.MBR,/QSYS.LIB/QSYSINC.LIB/MIH.FILE/MATH.MBR,/QSYS.LIB/QSYSINC.LIB/H.FILE/MATH.MBR; do echo $f; done

Signed-off-by: Liam Allan <[email protected]>

chrjorgensen · 2023-05-04T16:09:50Z

@worksofliam It returns the string nonetheless because you have not included the if condition... 😉

This should do it:

for f in ...a.lot.of.files...; do if [ -f $f ]; then echo $f; break; fi; done

Note I changed the exit 0 to a break which is more appropriate for a do...done loop.

Signed-off-by: Liam Allan <[email protected]>

worksofliam · 2023-05-04T16:44:02Z

@chrjorgensen Only just saw your comment.. let me try with your new string!

Signed-off-by: Liam Allan <[email protected]>

worksofliam · 2023-05-04T18:48:05Z

@chrjorgensen That is awesome! It will likely perform a lot better too! Thank you so much.

It is now ready for review.

Signed-off-by: Liam Allan <[email protected]>

chrjorgensen

@worksofliam I have tested your PR by running the test suite - not using the resolve methods in real life. I just made a little comment, which you may address or not...

When your have considered this, I will approve and merge.

chrjorgensen · 2023-05-06T10:00:56Z

src/api/IBMiContent.ts

+    });
+
+    if (result.code === 0) {
+      const [firstMost] = result.stdout.split(`,`);


Since you have a break in your for loop, you don't need to retrieve the "firstmost" - there will be only one...

chrjorgensen · 2023-05-06T10:01:15Z

src/api/IBMiContent.ts

+    });
+
+    if (result.code === 0 && result.stdout) {
+      const [firstMost] = result.stdout.split(`\n`);


Same here...

Signed-off-by: Liam Allan <[email protected]>

worksofliam

Thanks for your suggestion @chrjorgensen! I made the fix. Please review whenever you're ready.

worksofliam · 2023-05-16T14:11:30Z

@chrjorgensen Sorry to bug you on this. Can I get one more review?

chrjorgensen

@worksofliam LGTM - approved...

angelorpa · 2023-05-16T20:21:14Z

seems a bit late to ask, but here i go: this functions must necessarily be implemented using the /usr/bin/find command or coud be done with SQL? maybe using the QSYS2.SYSPARTITIONSTAT for member resolver and QSYS2.IFS_OBJECT_STATISTICS for the streamfile resolver ?

chrjorgensen · 2023-05-16T21:11:25Z

@angelorpa Why do you think it is a requirement to use /usr/bin/find?

The find command is used to search for files / directories in a directory and all subdirectories - but that it not what these functions are doing: They check for a member or streamfile in a list of source files / directories. And they only return the first member / streamfile found. Like the compilers do...

find is useful when you search for something in a directory tree - and I use it regularly.
But it is not suited for this use case. Here it is much faster to let the shell loop through the list and check for the member / file. find could potentially take a long time in a big directory tree and "choke" the resolve function...

chrjorgensen · 2023-05-16T21:11:39Z

@angelorpa Why do you think it is a requirement to use /usr/bin/find?

The find command is used to search for files / directories in a directory and all subdirectories - but that it not what these functions are doing: They check for a member or streamfile in a list of source files / directories. And they only return the first member / streamfile found. Like the compilers do...

find is useful when you search for something in a directory tree - and I use it regularly.
But it is not suited for this use case. Here it is much faster to let the shell loop through the list and check for the member / file. find could potentially take a long time in a big directory tree and "choke" the resolve function...

chrjorgensen · 2023-05-16T21:11:54Z

@angelorpa Why do you think it is a requirement to use /usr/bin/find?

The find command is used to search for files / directories in a directory and all subdirectories - but that it not what these functions are doing: They check for a member or streamfile in a list of source files / directories. And they only return the first member / streamfile found. Like the compilers do...

find is useful when you search for something in a directory tree - and I use it regularly.
But it is not suited for this use case. Here it is much faster to let the shell loop through the list and check for the member / file. find could potentially take a long time in a big directory tree and "choke" the resolve function...

chrjorgensen · 2023-05-16T21:12:12Z

@angelorpa Why do you think it is a requirement to use /usr/bin/find?

The find command is used to search for files / directories in a directory and all subdirectories - but that it not what these functions are doing: They check for a member or streamfile in a list of source files / directories. And they only return the first member / streamfile found. Like the compilers do...

find is useful when you search for something in a directory tree - and I use it regularly.
But it is not suited for this use case. Here it is much faster to let the shell loop through the list and check for the member / file. find could potentially take a long time in a big directory tree and "choke" the resolve function...

angelorpa · 2023-05-16T23:39:02Z

@angelorpa Why do you think it is a requirement to use /usr/bin/find?

The find command is used to search for files / directories in a directory and all subdirectories - but that it not what these functions are doing: They check for a member or streamfile in a list of source files / directories. And they only return the first member / streamfile found. Like the compilers do...

find is useful when you search for something in a directory tree - and I use it regularly. But it is not suited for this use case. Here it is much faster to let the shell loop through the list and check for the member / file. find could potentially take a long time in a big directory tree and "choke" the resolve function...

@chrjorgensen I think you misunderstood my question or may be i formulated in a wrong way. I understood what this functions have to do, but i want to know if is possible implement this functionality using SQL instead of using the find command. I only was suggesting another way to do it. Tables QSYS2.IFS_OBJECT_STATISTICS and QSYS2.SYSPARTITIONSTAT could be used to search for a streamfile or a member, may be using SQL could be faster, easy to understand and more flexible.

chrjorgensen · 2023-05-17T06:58:37Z

@angelorpa It sure is possible to create a search function using the SQL functions you mention - but that would only cover members, not streamfiles (unless you add the QSYS2.IFS_OBJECT_STATISTICS SQL function and then it get's tricky). Also running SQL has a cost when entering the SQL CLI, which runs in a separate job. SQL is great for retrieving information about something, but we just want to know the existence of a member or streamfile, not all its attributes.

Instead of SQL we went for shell scripting, which is widely used in the industry, especially in Linux, and is quick to check for files (does not start a new job for this). We use the for loop and if command, and you can google it easily searching for bash for loop and bash if test. So we actually never leave the shell program (which using SQL or find would do) and just let it do the lookups...

angelorpa · 2023-05-17T12:52:29Z

@angelorpa It sure is possible to create a search function using the SQL functions you mention - but that would only cover members, not streamfiles (unless you add the QSYS2.IFS_OBJECT_STATISTICS SQL function and then it get's tricky). Also running SQL has a cost when entering the SQL CLI, which runs in a separate job. SQL is great for retrieving information about something, but we just want to know the existence of a member or streamfile, not all its attributes.

Instead of SQL we went for shell scripting, which is widely used in the industry, especially in Linux, and is quick to check for files (does not start a new job for this). We use the for loop and if command, and you can google it easily searching for bash for loop and bash if test. So we actually never leave the shell program (which using SQL or find would do) and just let it do the lookups...

nice, thanks for the feedback.

Initial work for source member resolve

07e3ffc

Signed-off-by: Liam Allan <[email protected]>

worksofliam changed the title ~~Initial work for source member resolve~~ Initial work for object resolves Apr 6, 2023

Add streamfile resolve method

0f5c35d

Signed-off-by: Liam Allan <[email protected]>

worksofliam changed the title ~~Initial work for object resolves~~ Methods for object resolves Apr 6, 2023

worksofliam commented Apr 6, 2023

View reviewed changes

worksofliam self-assigned this Apr 6, 2023

Support for many names on streamfile lookup

cd3e392

Signed-off-by: Liam Allan <[email protected]>

worksofliam marked this pull request as ready for review April 7, 2023 15:28

worksofliam requested review from SanjulaGanepola, sebjulliand and chrjorgensen and removed request for SanjulaGanepola April 7, 2023 15:28

worksofliam removed their assignment Apr 7, 2023

worksofliam added 3 commits April 9, 2023 20:53

Merge branch 'master' into feature/source_resolve

c3396c8

Fixes for searching members

32f2243

Signed-off-by: Liam Allan <[email protected]>

Tests for memberResolve and streamfileResolve

894151b

Signed-off-by: Liam Allan <[email protected]>

worksofliam added 2 commits April 10, 2023 10:22

Merge branch 'master' into feature/source_resolve

18d1833

Signed-off-by: Liam Allan <[email protected]>

Content Suite using correct interface

03c9f7b

Signed-off-by: Liam Allan <[email protected]>

worksofliam linked an issue Apr 10, 2023 that may be closed by this pull request

API to do member/IFS resolves #1106

Closed

worksofliam requested a review from SanjulaGanepola April 10, 2023 17:35

worksofliam added 2 commits April 10, 2023 15:26

Tests for unqualifyPath

13061b9

Signed-off-by: Liam Allan <[email protected]>

Merge branch 'master' into feature/source_resolve

cc0b4f9

Signed-off-by: Liam Allan <[email protected]>

worksofliam commented Apr 11, 2023

View reviewed changes

chrjorgensen requested changes Apr 18, 2023

View reviewed changes

Change to streamfile resolve API

679bbf7

Signed-off-by: Liam Allan <[email protected]>

Merge branch 'master' into feature/source_resolve

11f0988

Signed-off-by: Liam Allan <[email protected]>

worksofliam marked this pull request as ready for review May 4, 2023 16:42

worksofliam requested a review from chrjorgensen May 4, 2023 16:43

worksofliam marked this pull request as draft May 4, 2023 16:43

worksofliam added 2 commits May 4, 2023 14:46

Use shell for to resolve files

c9e17fb

Signed-off-by: Liam Allan <[email protected]>

Remove find requirement

40e79d5

Signed-off-by: Liam Allan <[email protected]>

worksofliam marked this pull request as ready for review May 4, 2023 18:47

Bad name tests for resolves

86538aa

Signed-off-by: Liam Allan <[email protected]>

chrjorgensen requested changes May 6, 2023

View reviewed changes

worksofliam self-assigned this May 8, 2023

Simply resolve result handler

b743db3

Signed-off-by: Liam Allan <[email protected]>

worksofliam requested a review from chrjorgensen May 8, 2023 14:17

worksofliam commented May 8, 2023

View reviewed changes

worksofliam removed their assignment May 8, 2023

chrjorgensen approved these changes May 16, 2023

View reviewed changes

worksofliam merged commit e2be66a into master May 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Methods for object resolves #1224

Methods for object resolves #1224

worksofliam commented Apr 6, 2023

worksofliam Apr 6, 2023

SanjulaGanepola Apr 6, 2023

worksofliam Apr 7, 2023

chrjorgensen commented Apr 7, 2023

worksofliam commented Apr 7, 2023

worksofliam commented Apr 10, 2023

worksofliam Apr 11, 2023

chrjorgensen left a comment

worksofliam commented Apr 18, 2023

chrjorgensen commented Apr 18, 2023

worksofliam commented Apr 18, 2023

worksofliam commented May 4, 2023

chrjorgensen commented May 4, 2023

worksofliam commented May 4, 2023

worksofliam commented May 4, 2023

chrjorgensen left a comment

chrjorgensen May 6, 2023

chrjorgensen May 6, 2023

worksofliam left a comment

worksofliam commented May 16, 2023

chrjorgensen left a comment

angelorpa commented May 16, 2023

chrjorgensen commented May 16, 2023

chrjorgensen commented May 16, 2023

chrjorgensen commented May 16, 2023

chrjorgensen commented May 16, 2023

angelorpa commented May 16, 2023

chrjorgensen commented May 17, 2023

angelorpa commented May 17, 2023

Methods for object resolves #1224

Methods for object resolves #1224

Conversation

worksofliam commented Apr 6, 2023

Changes

Checklist

worksofliam Apr 6, 2023

Choose a reason for hiding this comment

SanjulaGanepola Apr 6, 2023

Choose a reason for hiding this comment

worksofliam Apr 7, 2023

Choose a reason for hiding this comment

chrjorgensen commented Apr 7, 2023

worksofliam commented Apr 7, 2023

worksofliam commented Apr 10, 2023

worksofliam Apr 11, 2023

Choose a reason for hiding this comment

chrjorgensen left a comment

Choose a reason for hiding this comment

worksofliam commented Apr 18, 2023

chrjorgensen commented Apr 18, 2023

worksofliam commented Apr 18, 2023

worksofliam commented May 4, 2023

chrjorgensen commented May 4, 2023

worksofliam commented May 4, 2023

worksofliam commented May 4, 2023

chrjorgensen left a comment

Choose a reason for hiding this comment

chrjorgensen May 6, 2023

Choose a reason for hiding this comment

chrjorgensen May 6, 2023

Choose a reason for hiding this comment

worksofliam left a comment

Choose a reason for hiding this comment

worksofliam commented May 16, 2023

chrjorgensen left a comment

Choose a reason for hiding this comment

angelorpa commented May 16, 2023

chrjorgensen commented May 16, 2023

chrjorgensen commented May 16, 2023

chrjorgensen commented May 16, 2023

chrjorgensen commented May 16, 2023

angelorpa commented May 16, 2023

chrjorgensen commented May 17, 2023

angelorpa commented May 17, 2023