Getting packages not listed in my jss

[charliwest]

When I run this I get packages in the output that aren't listed in my jss, then if I try and run it to remove them all I get errors.
Are there perhaps packages I'm not seeing for some reason?

The error is
./Spruce.py --remove output.txt
Traceback (most recent call last):
File "./Spruce.py", line 367, in
main()
File "./Spruce.py", line 363, in main
remove(j, removal_set)
File "./Spruce.py", line 220, in remove
j.distribution_points.delete(item)
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/distribution_points.py", line 246, in delete
repo.delete(filename)
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/distribution_points.py", line 870, in delete
self.connection['jss'].Package(filename).delete()
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/jss.py", line 421, in Package
return self.factory.get_object(Package, data)
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/jss.py", line 524, in get_object
xmldata = self.jss.get(url)
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/jss.py", line 231, in get
self._error_handler(JSSGetError, response)
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/jss.py", line 185, in _error_handler
raise exception
jss.exceptions.JSSGetError: Response Code: 404 Response: Not Found. The server has not found anything matching the request URI

[sheagcraig]

Whoa! That sounds wild. I imagine it's quite possible that fileshare DP's get out of sync with the DB/Casper server. I would think that whatever you get out of the Packages page or the API page's Packages endpoint would be what it thinks you have. So that's one thing to double-check. But I would think that whatever it is spitting out to spruce is what is actually present in the database, even if the files don't exist on the DP's.

I'm about to drop a big bad new Spruce that has WAY more reporting features, so look out for that soon too.

[charliwest]

OK will see what the new version brings, thanks.

I did do a db clean up and try again, still getting

jss.exceptions.JSSGetError: Response Code: 404 Response: Not Found. The server has not found anything matching the request URI

but will try again when the new version comes.

On 15 Aug 19, at 15:22, Shea Craig [email protected] wrote:

Whoa! That sounds wild. I imagine it's quite possible that fileshare DP's get out of sync with the DB/Casper server. I would think that whatever you get out of the Packages page or the API page's Packages endpoint would be what it thinks you have. So that's one thing to double-check. But I would think that whatever it is spitting out to spruce is what is actually present in the database, even if the files don't exist on the DP's.

I'm about to drop a big bad new Spruce that has WAY more reporting features, so look out for that soon too.

—
Reply to this email directly or view it on GitHub #6 (comment).

[sheagcraig]

This is from when you're trying to remove package objects that you just got spit out from the reports side of spruce?

I can imagine that the names don't transfer over correctly perhaps if they have spaces or something that I didn't account for-stuff that needs quoting/escaping, like unicode or accents or umlauts or whatnot? (This is why the new version will output in plist format and use that plist for removal.

For the time being, you could comment out the packages that are causing problems; you could probably even use that to determine whether it's ALL packages or just certain ones, and maybe we can narrow down why it isn't working.

Our school year is starting up, so my predications for a release are all messed up!

[charliwest]

Ok, so tested by taking out anything with a space or '-' in it, then got

./Spruce.py -v --remove output.txt
Traceback (most recent call last):
File "./Spruce.py", line 367, in
main()
File "./Spruce.py", line 363, in main
remove(j, removal_set)
File "./Spruce.py", line 220, in remove
j.distribution_points.delete(item)
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/distribution_points.py", line 246, in delete
repo.delete(filename)
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/distribution_points.py", line 870, in delete
self.connection['jss'].Package(filename).delete()
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/jss.py", line 421, in Package
return self.factory.get_object(Package, data)
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/jss.py", line 524, in get_object
xmldata = self.jss.get(url)
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/jss.py", line 231, in get
self._error_handler(JSSGetError, response)
File "/Library/Python/2.7/site-packages/python_jss-1.2.0-py2.7.egg/jss/jss.py", line 185, in _error_handler
raise exception
jss.exceptions.JSSGetError: Response Code: 404 Response: Not Found. The server has not found anything matching the request URI

Then tried individual removes

./Spruce.py -v --remove CommonServices.pkg
Traceback (most recent call last):
File "./Spruce.py", line 367, in
main()
File "./Spruce.py", line 362, in main
removal_set = load_removal_file(args.remove)
File "./Spruce.py", line 291, in load_removal_file
with open(os.path.expanduser(filename), "r") as ifile:
IOError: [Errno 2] No such file or directory: 'CommonServices.pkg'

And that package is definitely there

On 15 Aug 19, at 15:28, Shea Craig [email protected] wrote:

This is from when you're trying to remove package objects that you just got spit out from the reports side of spruce?

I can imagine that the names don't transfer over correctly perhaps if they have spaces or something that I didn't account for-stuff that needs quoting/escaping, like unicode or accents or umlauts or whatnot? (This is why the new version will output in plist format and use that plist for removal.

For the time being, you could comment out the packages that are causing problems; you could probably even use that to determine whether it's ALL packages or just certain ones, and maybe we can narrow down why it isn't working.

Our school year is starting up, so my predications for a release are all messed up!

—
Reply to this email directly or view it on GitHub #6 (comment).

[sheagcraig]

Yeha-you won't be able to remove anything directly with the --remove arg. It uses that argument to open a file named "CommonServices.pkg" which contains the list; not to remove a package named "CommonServices.pkg".

[charliwest]

Haha, ok.
Did some more tries, what I seem to get is it removes the first package, but not any more after it from the file. Then if I remove the top package from the output.txt file and run it again, it removes just the top one again and ignores the rest.

[sheagcraig]

That's super annoying> Hopefully the update will solve all of those issues then. (And no doubt create new ones ;)

[kernsb]

I'm seeing the exact same thing. List created with --report, edited, and when attempt to use it with --remove exact same error as charliwest.

[sheagcraig]

@kernsb (and @charliwest ) I'm finishing up the XML output of the testing version (actually it's in its own branch atm). After that, I'm onto writing the removal function.

This time around, it will log failures to remove things (like your mystery packages that don't exist) but it won't bring the program to a halt.

[kernsb]

All right so a new variant on this since 2.0 ... this may warrant a separate Issue.

So, similar to the previous issue, I'm still seeing issues with the removals, now specified in the XML file. It removes the first file specified in the list from the file share, but then chokes on removing it from the JSS database with a similar error as before.

./spruce.py --remove ./RemovePkgs.xml 
Are you sure you want to continue deleting objects from https://XXXX-XXXXXX.XXX.XXXX.XXX:8443? (Y or N): Y
Package object 590: CLA-AdobeColorCCPanel.dmg deleted.
Traceback (most recent call last):
  File "./spruce.py", line 1931, in <module>
    main()
  File "./spruce.py", line 1925, in main
    remove(removal_tree)
  File "./spruce.py", line 1875, in remove
    jss_connection.distribution_points.delete(item.text)
  File "/Library/Python/2.7/site-packages/jss/distribution_points.py", line 255, in delete
    repo.delete(filename)
  File "/Library/Python/2.7/site-packages/jss/distribution_points.py", line 908, in delete
    self.connection["jss"].Package(filename).delete()
  File "/Library/Python/2.7/site-packages/jss/jss.py", line 431, in Package
    return self.factory.get_object(Package, data)
  File "/Library/Python/2.7/site-packages/jss/jss.py", line 550, in get_object
    xmldata = self.jss.get(url)
  File "/Library/Python/2.7/site-packages/jss/jss.py", line 238, in get
    self._error_handler(JSSGetError, response)
  File "/Library/Python/2.7/site-packages/jss/jss.py", line 194, in _error_handler
    raise exception
jss.exceptions.JSSGetError: Response Code: 404  Response: Not Found. The server has not found anything matching the request URI

Again, the end result here is that referenced file is gone from the DP, but the entry for the "Package" still exists in the JSS, and no further items are processed.

[sheagcraig]

@kernsb Do you have a JDS?

Does the filename not match the display name for your packages?

[kernsb]

@sheagcraig We do have a JDS as well as a File Share DP. In this case (and a previous one I tried moments before) the Filename and Display Name do match. However, I have seen instances where they have not matched in this setup (I inherited this JSS and am trying to get it all in order).

[sheagcraig]

Aha! This all makes sense now.

Again, I don't have a test server, so the removal algorithm is my best guess.

Here's what happens:

1. Get the API object for the thing you want to delete (in this case, a package)
   a. If that package doesn't exist as an object in the DB, just go to the next one.
   b. If it does exist, delete the DB side of things.
2. If you have DP's configured, and it's an object type that has a file (packages and sometimes scripts), try to delete them.
   a. For file share DP's, mount the shares and do a delete.
   b. For JDS DP's, there is no (public-api anyway) way to delete packages; but you can just delete the db object. So it tries to do that. Except you delete the db object in 1.b, so it throws a JSSGetError.

AND I didn't catch JSSGetError's; just OSError (which is the exception you get when you try to delete a file that doesn't exist).

So, if you want to try adding this:

             except jss.JSSGetError:
                 # Tried to delete the db object again because user has a
                 # JDS or CDP. Just silently continue.
                 pass

to line 1880 (so it looks like this):

         if item.tag in needs_file_removal and file_type_removals:
             try:
                 jss_connection.distribution_points.delete(item.text)
                 print "%s file %s deleted." % (item.tag, obj.name)
             except OSError as error:
                 print ("Unable to delete %s: %s with error: %s" %
                        (item.tag, item.text, error.message))
             except jss.JSSGetError:
                 # Tried to delete the db object again because user has a
                 # JDS or CDP. Just silently continue.
                 pass

and then run again, let me know how it goes. If it works, I'll rush that out as the first 2.0 update.

[kernsb]

Ok, so that worked! It let it complete without error and the packages are gone from the DP.

Here's the interesting thing that happened next. I opened Casper Admin to confirm that they were gone, and at open entries for many of the things I just removed were re-created. And it looks like these are the ones, as you had mentioned, where the Display Name did not match the Filename.

[sheagcraig]

It's not super clear to me what the interaction is between Display Name and Filename, and honestly I wish it would just go away!

As a rule, we do NOT use different names, so this issue is a non-issue.

Hopefully you can just manually remove those divergent-named packages through the web interface and get them to "stay dead".

[kernsb]

Sounds good. I think that will work, and it will not be a problem for me moving forward, as I don't use the Display Name.

So, yes, your code handled it perfectly then!

[sheagcraig]

Long term it would be best for me to find a way to handle this.

Do you know, when you do an API GET for packages, using the name, does it use filename or display name (or both)?

(I would just try it through the https://yourjss.org:8443/api/ page. Do the Packages/GET/Get packages by name query and see if which of the names it works for (on a package that still exists, but has different file/display names).

[kernsb]

The get packages by name query on the API page works with the Display Name, but not with the Filename, when they differ.

[sheagcraig]

Indeed-looking at the xml for a package the "name" property must correspond to "display name", since there's also a "filename".

I'll have to fix that in python-jss and probably here too. It may be awhile before that change gets made, so I think you'll unfortunately be best off either getting the names to match or deleting them manually.

[kernsb]

No problem. Thanks for all of your help!

[sheagcraig]

Try the testing branch Spruce. I just added a thing to use filename for package and script file deletion instead of display name.

[kernsb]

Nope. Same behavior. If I open Casper Admin after it is removed, it adds it back like before. I did notice that the entry it creates uses the original filename for both the new Display Name and Filename.

I also ran Spruce against it a second time, now with the names matching, and it removed it again .... And it came back yet again. If I finally delete the recreated item using Casper Admin, it stays gone for good.

I'm thinking it may be doing this with every package that was deleted and not just the mismatched name ones. I'll investigate more and let you know what I find.

[kernsb]

Also, may need to do something similar for Scripts (and other objects) as we did for Packages. When I remove scripts I get responses such as

Script object cla-uninstall_mcafee.sh with ID 40 is not available or does not exist.
Status Code:404 Error: Response Code: 404   Response: Not Found. The server has not found anything matching the request URI

for each script in the list.

[sheagcraig]

That's a strange one-that's from the very first deletion, where it tries to get the db object from the JSS and then delete it. The problem is-it uses ID, so if that script actually exists, the GET should succeed.

[kernsb]

Interesting. Apart from the message for each item the operation was successful - the specified scripts were deleted.

[sheagcraig]

That's weird.

[grahampugh]

@kernsb @charliwest since this issue has sat here for 4 years and many things have changed since then, I'm going to close it, but reply if you are still experiencing the issues.