Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gh runners local #24

Closed
wants to merge 78 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
eeeec9a
Allow for secure-boot efivars directory having SecureBoot-xxx files, …
dmurphy18 Oct 10, 2024
76f22f0
Added changelog entry
dmurphy18 Oct 10, 2024
effada4
Initial test for testing secure-boot grain
dmurphy18 Oct 11, 2024
1b680c6
Added tests to excerise changes for efi-secure-boot grain
dmurphy18 Oct 14, 2024
061ca0d
Disable signing mac packages
twangboy Oct 15, 2024
28662c4
Fix slashes
twangboy Oct 16, 2024
e3f2596
Regularize yumpkg option separators
ben-grande Oct 15, 2024
30e0de3
Remove wmic usage from disk grains on Windows
twangboy Oct 14, 2024
6caf16d
Remove unused import
twangboy Oct 14, 2024
98f49eb
Remove wmic from iscsi grains
twangboy Oct 15, 2024
4e429ef
Remove perms from linux_acl.list_absent docstring
tacerus Sep 15, 2024
628c2d2
Add cryptography back as a dependency for Salt 3006 to base.txt requi…
dmurphy18 Oct 17, 2024
3b8c57e
Add test for saltclass nested classes expansion
baby-gnu Sep 26, 2024
9ef879a
fix(saltclass): don't lose nested classes and states in recursion
baby-gnu Sep 26, 2024
d1ca30c
Fix incorrect override of minion node definition
baby-gnu Sep 26, 2024
6e4f178
Remove expand_classes_in_order from missing_docstrings
baby-gnu Sep 26, 2024
983cfe7
fixes #66252 correct use of egrep to parse semanage output
ndptech Mar 21, 2024
eee8215
Skip tests unless on Linux
twangboy Jun 28, 2024
e4b4f57
Fix nightly builds and update psutil to 6.0.0 or greater
dmurphy18 Oct 22, 2024
a8a73c4
Revert psutil dependency back to greater than 5.0.0
dmurphy18 Oct 22, 2024
bbd061b
Forcing rebuild
dmurphy18 Oct 23, 2024
c61822f
Updates due to pytest-shell-utilities and psutil version updates
dmurphy18 Oct 23, 2024
d192585
Adjusted selinux and tests for egrep to grep -E
dmurphy18 Oct 23, 2024
ed89424
Updated selinux commands and tests after reviewer comments, need the …
dmurphy18 Oct 24, 2024
931e463
Add test for issue #66996
lkubb Oct 24, 2024
0e3d73e
Fix parallel states on spawning platforms in general
lkubb Oct 24, 2024
c4361cf
Ensure injected globals are defined on spawning platforms
lkubb Oct 24, 2024
a275498
Add test for issue #66999
lkubb Oct 24, 2024
fd7b0dc
Filter unpicklable objects from the context dict when necessary
lkubb Oct 24, 2024
6759191
Add some gh runners
dwoz Oct 28, 2024
badfa1b
Add runners to linter and disbale cache for salt onedir builds
dwoz Oct 28, 2024
685a9fd
Migrate build-deps-onedir step to gh runners
dwoz Oct 28, 2024
87d223e
Do not cache relenv for now
dwoz Oct 28, 2024
6578559
Disable more caches
dwoz Oct 28, 2024
417dada
Update constraints for python 3.12
dwoz Oct 28, 2024
7b1b009
Fix download location
dwoz Oct 29, 2024
680d77c
Set use s3 cache env to false
dwoz Oct 29, 2024
f9bbeb0
Build packages on gh runners
dwoz Oct 29, 2024
39fabf3
Migrate build deps ci action
dwoz Oct 29, 2024
ede2a3e
Add test-action-linux-ng.yml
dwoz Oct 29, 2024
04cae39
Fix workflow name
dwoz Oct 29, 2024
1521e93
Change template not generated files
dwoz Oct 29, 2024
6a5cce5
Disable s3 cache for build deps ci
dwoz Oct 29, 2024
c7b2cdc
Do not use tools vm for deps ci action
dwoz Oct 29, 2024
5ae3b6d
wean of aws bits
dwoz Oct 29, 2024
106bf9b
Setup python
dwoz Oct 30, 2024
6ae2d85
Migrate windows to gh runners
dwoz Oct 30, 2024
89a8e42
be more specific with python version
dwoz Oct 30, 2024
8411278
check python binary
dwoz Oct 30, 2024
4665b33
no container
dwoz Oct 30, 2024
c3fb1d7
revert cache change
dwoz Oct 30, 2024
85790c4
Disable more s3 caching
dwoz Oct 30, 2024
fd4767d
Do not install any system dependencies yet
dwoz Oct 30, 2024
1a2023e
Do not build 'source' packages for PRs
dwoz Oct 30, 2024
a1dc1a4
Fix nox tarball name
dwoz Oct 30, 2024
cfdc058
Build windows deps on github runner
dwoz Oct 30, 2024
ffbfd50
Fix decompress step
dwoz Oct 31, 2024
fea61da
Fix decompress nox step on windows
dwoz Oct 31, 2024
70fc72c
fetch correct nox tarball
dwoz Oct 31, 2024
392ec0a
bump relenv version
dwoz Oct 31, 2024
3153f61
Download from alt location
dwoz Oct 31, 2024
97242fd
Update download location
dwoz Oct 31, 2024
348a4b1
Disable most of linux and all of mac
dwoz Nov 1, 2024
92e1e21
Disable more OSes for package tests
dwoz Nov 1, 2024
8969237
use newer container for nwo
dwoz Nov 1, 2024
48575c4
No longer use sudo
dwoz Nov 2, 2024
26c718d
Less macos
dwoz Nov 2, 2024
6c35cbb
Less macos
dwoz Nov 2, 2024
0e7f3af
Fix up windows package test action
dwoz Nov 2, 2024
da4fe44
Show sys info fix
dwoz Nov 2, 2024
c8892fc
Meh
dwoz Nov 2, 2024
c5a5616
meh
dwoz Nov 2, 2024
921bd89
Skip reqs
dwoz Nov 2, 2024
1b4f4f2
No sudo for linux pkg tests
dwoz Nov 2, 2024
db88755
Attempt systemctl mock
dwoz Nov 2, 2024
5f6b8d1
Revert xlarge removal
dwoz Nov 2, 2024
65650cc
Try macos-13-arm64 instead of xlarge
dwoz Nov 2, 2024
8e9b24b
Stop using xlarge runners
dwoz Nov 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(saltclass): don't lose nested classes and states in recursion
baby-gnu authored and dwoz committed Oct 21, 2024
commit 9ef879aee4acf9c8dec8daa58beefa9ba8f7be76
4 changes: 4 additions & 0 deletions changelog/58969.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Issue 58969: Fixes an issue with `saltclass.expand_classes_in_order`
function where it was losing nested classes states during class
expansion. The logic now use `salt.utils.odict.OrderedDict` to keep
the inclusion ordering.
118 changes: 57 additions & 61 deletions salt/utils/saltclass.py
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@

from jinja2 import Environment, FileSystemLoader

import salt.utils.odict
import salt.utils.path
import salt.utils.yaml

@@ -277,9 +278,27 @@ def expand_classes_glob(classes, salt_data):
return expanded_classes


def expand_classes_in_order(
minion_dict, salt_data, seen_classes, expanded_classes, classes_to_expand
):
def expand_classes_in_order(minion_dict, salt_data, seen_classes, classes_to_expand):
"""
Expand the list of `classes_to_expand` and return them in the order found
The return order is `[C, B, A, M, B, L, MINION_ID]` when:
- minion node include classes `A` and `L`
- `A` include class `B`
- `B` include class `C`
- `L` include class `M` and `B`
:param dict minion_dict: definition of minion node
:param dict salt_data: configuration data
:param iterable(str) seen_classes: classes already processed
:param iterable(str) classes_to_expand: classes to recursivly expand
:return: Expanded classes in proper order
:rtype: salt.utils.odict.OrderedDict
"""

expanded_classes = salt.utils.odict.OrderedDict()

# Get classes to expand from minion dictionary
if not classes_to_expand and "classes" in minion_dict:
classes_to_expand = minion_dict["classes"]
@@ -290,71 +309,37 @@ def expand_classes_in_order(
for klass in classes_to_expand:
if klass not in seen_classes:
seen_classes.append(klass)
expanded_classes[klass] = get_class(klass, salt_data)
klass_dict = salt.utils.odict.OrderedDict(
{klass: get_class(klass, salt_data)}
)
# Fix corner case where class is loaded but doesn't contain anything
if expanded_classes[klass] is None:
expanded_classes[klass] = {}
if klass_dict[klass] is None:
klass_dict[klass] = {}

# Merge newly found pillars into existing ones
new_pillars = expanded_classes[klass].get("pillars", {})
new_pillars = klass_dict[klass].get("pillars", {})
if new_pillars:
dict_merge(salt_data["__pillar__"], new_pillars)

# Now replace class element in classes_to_expand by expansion
if expanded_classes[klass].get("classes"):
l_id = classes_to_expand.index(klass)
classes_to_expand[l_id:l_id] = expanded_classes[klass]["classes"]
expand_classes_in_order(
minion_dict,
if "classes" in klass_dict[klass]:
nested_classes = expand_classes_in_order(
{},
salt_data,
seen_classes,
expanded_classes,
classes_to_expand,
klass_dict[klass].get("classes", {}),
)
else:
expand_classes_in_order(
minion_dict,
salt_data,
seen_classes,
expanded_classes,
classes_to_expand,
)

# We may have duplicates here and we want to remove them
tmp = []
for t_element in classes_to_expand:
if t_element not in tmp:
tmp.append(t_element)

classes_to_expand = tmp

# Now that we've retrieved every class in order,
# let's return an ordered list of dicts
ord_expanded_classes = []
ord_expanded_states = []
for ord_klass in classes_to_expand:
ord_expanded_classes.append(expanded_classes[ord_klass])
# And be smart and sort out states list
# Address the corner case where states is empty in a class definition
if (
"states" in expanded_classes[ord_klass]
and expanded_classes[ord_klass]["states"] is None
):
expanded_classes[ord_klass]["states"] = {}

if "states" in expanded_classes[ord_klass]:
ord_expanded_states.extend(expanded_classes[ord_klass]["states"])
# Put current class after nested classes
klass_dict.update(nested_classes)
klass_dict.move_to_end(klass)

# Add our minion dict as final element but check if we have states to process
if "states" in minion_dict and minion_dict["states"] is None:
minion_dict["states"] = []
expanded_classes.update(klass_dict)

if "states" in minion_dict:
ord_expanded_states.extend(minion_dict["states"])
# Minion dict must be at the end
if minion_dict:
expanded_classes.update({salt_data["minion_id"]: minion_dict})

ord_expanded_classes.append(minion_dict)

return ord_expanded_classes, classes_to_expand, ord_expanded_states
return expanded_classes


def expanded_dict_from_minion(minion_id, salt_data):
@@ -382,17 +367,28 @@ def expanded_dict_from_minion(minion_id, salt_data):
# Get 2 ordered lists:
# expanded_classes: A list of all the dicts
# classes_list: List of all the classes
expanded_classes, classes_list, states_list = expand_classes_in_order(
node_dict[minion_id], salt_data, [], {}, []
)
expanded_classes = expand_classes_in_order(node_dict[minion_id], salt_data, [], [])

# Here merge the pillars together
pillars_dict = {}
for exp_dict in expanded_classes:
states_list = []
classes_list = list(expanded_classes.keys())[:-1]
classes_values = list(expanded_classes.values())
for exp_dict in classes_values:
if "pillars" in exp_dict:
dict_merge(pillars_dict, exp_dict)

return expanded_classes, pillars_dict, classes_list, states_list
if "states" in exp_dict:
states_list.extend(exp_dict["states"])

# Avoid duplicates, keep first
state_seen = set()
states_list = [
state
for state in states_list
if not (state in state_seen or state_seen.add(state))
]

return classes_values, pillars_dict, classes_list, states_list


def get_pillars(minion_id, salt_data):