module grains.set not updating grains

[angeloudy]

I am trying to use grains.set to add some grains.
I can see that /usr/local/etc/salt/grains successfully updated, but I can not see it from grains.ls or grains.items. I also tried saltutil.refresh_grains with no avail.

To reproduce:

on minion:

salt-call grains.set carp master
salt-call saltutil.refresh_pillar

then, in /usr/local/etc/salt/grains

carp: master

salt-call grains.get carp

returns nothing

version report

Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: 1.12.3
       cherrypy: unknown
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.33.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: 1.4.2.post1
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, May 15 2019, 05:26:05)
   python-gnupg: Not Installed
         PyYAML: 5.1
          PyZMQ: 18.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist:
         locale: UTF-8
        machine: amd64
        release: 11.2-RELEASE-p10
         system: FreeBSD
        version: Not Installed

[Ch3LL]

im having a hard time replicating this. when you run grains.set the grains should get refreshed for you.

[root@14cb403a2ac6 /]# salt-call grains.set carp master
local:
    ----------
    changes:
        ----------
        carp:
            master
    comment:
    result:
        True
[root@14cb403a2ac6 /]# salt-call grains.get carp
local:
    master
[root@14cb403a2ac6 /]# salt --version
salt 2018.3.3 (Oxygen)
[root@14cb403a2ac6 /]# 

is 2018.3.3 the version for both your master and minion? Also anything of significance in your debug logs?

[Ch3LL]

looking at your report more closely, it looks like the only difference is the location of the file /usr/local/etc/salt/grains vs mine is at /etc/salt/grains. Mind creating that file at /etc/salt/grains and see if it makes a difference by chance?
but before that can you try restarting your minion and seeing if you see the grains then?

[angeloudy]

@Ch3LL
Yes, master and minion are running the same version.
I am running salt on FreeBSD, in which /usr/local/etc is the standard location. All my salt configurations are under /usr/local/etc/salt. In this case, the file /usr/local/etc/salt/grains is created only after I ran grains.set; it did not exists before. This means salt recognises the grains file.
I also tried create /etc/salt/grains by hand, not working either.
Also tried restarting the minion, didn't help.
How does the module grains.set work? According to the documentation, it simply alters /etc/salt/grains(in my case, /usr/local/etc/salt/grains). Does it also cause salt to reload the grains?

And how does saltutil.refresh_grains work? Does it force salt to read /etc/salt/grains?

Looking at my configuration, I do have grains_cache: True, will that affect this?

[angeloudy]

After disabling grains_cache, it's working. I can access grains immediately after I add them with grains.set cmd.

I think when grains_cache is set True, it just simply read grains from /var/cache/salt/minion/grains.cache.p until it reaches expiration set via grains_cache_expiration. I didn't find this documented anywhere.

Also, it would be nice if saltutil.refresh_grains should refresh grains and reset the timestamp.

[Ch3LL]

I did find the --refresh-grains-cache argument, but I think it would be good to add the ability to refresh grains cache with saltutil as well.

[angeloudy]

--refresh-grains-cache is an option of salt-call. It forces grains refresh before it runs any other command.
In the case of salt --refresh-grains-cache grains set carp_status backup, it updates the cache before it updates the grains and the newly added grains will still not be in the cache.

# salt-call --refresh-grains-cache grains.set carp_status master
local:
    ----------
    changes:
        ----------
        carp_status:
            master
    comment:
    result:
        True
 # salt-call grains.get carp_status
local:
    backup

[Ch3LL]

thanks for checking on that :) looks like we will need to get this fixed up even for that argument

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[ari]

Could the priority of this task be raised please. The fact that grains.set completely doesn't work is rather a big deal.

[Oloremo]

Hit something like that on 3003.2

I do a grains.set and refresh_grains after.
grains_cache is set to False.
Then I try to target minions based on just set grain and sometimes it doesn't work.

We have a CI that runs ~100 integration tests that all do more or less the same with grains.set and so on. And like 20 out of 100 failed with that error.

Feels like there is some kind of race.

@sagetherage Do you think someone can look at that issue?

[Oloremo]

Ok seems like if I do that like this:

salt_minion_management.refresh.grains:
  module.run:
    - saltutil.refresh_grains:
    - reload_grains: true

During the state execution and after I have to do a

salt \*  saltutil.refresh_grains

One more time to make it work as expected.