[BUG] Backend Build Failure

[axelarator]

Describe the bug
I see this was an issue in the past as well and fixed in #186 but it appears again.

The command '/bin/sh -c echo '[+] Run build checks' && cd /strelka/strelka/ && python3 -m pytest -s tests/ && if $CONFIG_TESTS; then python3 -m pytest -s tests_configuration/; fi && echo '[+] Done'' returned a non-zero code: 1

ERROR: Service 'backend' failed to build : Build failed

Steps to reproduce
docker-compose -f build/docker-compose.yaml build

Expected behavior
Build without failure

Screenshots
N/A

Release

• Release: 0.23.01.07

Additional context
Recreated issue on both Ubuntu 20.04 and 22.04

[phutelmyer]

Thanks @axelarator - I'll review and keep you posted. Appreciate the issue posting.

[phutelmyer]

@axelarator Testing on 22.04 for 0.23.01.07

dev@dev-virtual-machine:~/Desktop/strelka-0.23.01.07$ docker-compose -f build/docker-compose.yaml build --no-cache --build-arg CONFIG_TESTS=true backend
Building backend
[+] Building 695.2s (24/24) FINISHED      
...
 => [19/19] RUN rm -rf /etc/strelka/                                                                                                                                0.5s
 => exporting to image                                                                                                                                             12.0s
 => => exporting layers                                                                                                                                            12.0s
 => => writing image sha256:c85c693abe0e0d27754e6c8294625abced8f6f84ad24b852057448cb4b50bce8                                                                        0.0s
 => => naming to docker.io/library/build_backend                                                                                                                    0.0s
dev@dev-virtual-machine:~/Desktop/strelka-0.23.01.07$ ^C

Built fine using no image cache. Do you mind building again with docker-compose -f build/docker-compose.yaml build --no-cache?

If the issue still persists, do you mind posting additional details such as architecture / additional build logs (looks to be failing on the tests, so maybe the above tests section?)

[axelarator]

No luck with --no-cache. This is what I'm seeing following the test session starts block. I'm testing this from a freshly updated Ubuntu 22.04 box that doesn't have any heavy configurations. Pretty bare bones.

uname -a
Linux beta-VirtualBox 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

=================================== FAILURES ===================================

_______________________________ test_distribute ________________________________



mocker = <pytest_mock.plugin.MockerFixture object at 0x7efedf81b220>



    def test_distribute(mocker):

        """

        Pass: Sample event matches output of scanner.

        Failure: Unable to load file or sample event fails to match.

        """

    

        expected = [

            {

                "file": {

                    "depth": 0,

                    "flavors": {"mime": ["text/html"], "yara": ["html_file"]},

                    "name": "",

                    "scanners": [

                        "ScanEntropy",

                        "ScanFooter",

                        "ScanHash",

                        "ScanHeader",

                        "ScanHtml",

                        "ScanTlsh",

                        "ScanYara",

                    ],

                    "size": 5875,

                    "source": "",

                    "tree": {

                        "node": mock.ANY,

                        "parent": "",

                        "root": mock.ANY,

                    },

                },

                "scan": {

                    "entropy": mock.ANY,

                    "footer": mock.ANY,

                    "hash": mock.ANY,

                    "header": mock.ANY,

                    "html": mock.ANY,

                    "tlsh": mock.ANY,

                    "yara": mock.ANY,

                },

            },

            {

                "file": {

                    "depth": 1,

                    "flavors": {

                        "external": ["", ""],

                        "mime": ["text/plain"],

                        "yara": ["javascript_file"],

                    },

                    "name": "script_1",

                    "scanners": [

                        "ScanEntropy",

                        "ScanFooter",

                        "ScanHash",

                        "ScanHeader",

                        "ScanJavascript",

                        "ScanTlsh",

                        "ScanYara",

                    ],

                    "size": 221,

                    "source": "ScanHtml",

                    "tree": {

                        "node": mock.ANY,

                        "parent": mock.ANY,

                        "root": mock.ANY,

                    },

                },

                "scan": {

                    "entropy": mock.ANY,

                    "footer": mock.ANY,

                    "hash": mock.ANY,

                    "header": mock.ANY,

                    "javascript": mock.ANY,

                    "tlsh": mock.ANY,

                    "yara": {

                        "elapsed": mock.ANY,

                        "flags": [],

                        "matches": ["test"],

                        "tags": [],

                        "meta": [],

                    },

                },

            },

        ]

    

        if os.path.exists("/etc/strelka/backend.yaml"):

            backend_cfg_path: str = "/etc/strelka/backend.yaml"

        else:

            backend_cfg_path: str = Path(

                Path(__file__).parent / "../../../../configs/python/backend/backend.yaml"

            )

    

        with open(backend_cfg_path, "r") as f:

            backend_cfg = yaml.safe_load(f.read())

    

            backend = strelka.Backend(backend_cfg, coordinator=None)

    

            with open(

                Path(Path(__file__).parent / f"../tests/fixtures/test.html"), "rb"

            ) as test_file:

                data = test_file.read()

                file = strelka.File(data=data)

    

                events = backend.distribute(str(uuid.uuid4()), file, int(time.time()) + 300)

    

                TestCase.maxDiff = None

>               TestCase().assertListEqual(expected, events)



tests/test_distribute.py:116: 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

/usr/lib/python3.10/unittest/case.py:1051: in assertListEqual

    self.assertSequenceEqual(list1, list2, msg, seq_type=list)

/usr/lib/python3.10/unittest/case.py:1033: in assertSequenceEqual

    self.fail(msg)

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 



self = <unittest.case.TestCase testMethod=runTest>

msg = "Lists differ: [{'fi[234 chars]de': <ANY>, 'parent': '', 'root': <ANY>}}, 'sc[648 chars]]}}}] != [{'fi[234 chars]de': ...-----\n\n+                     'matches': [],\n                      'meta': [],\n                      'tags': []}}}]"



    def fail(self, msg=None):

        """Fail immediately, with the given message."""

>       raise self.failureException(msg)

E       AssertionError: Lists differ: [{'fi[234 chars]de': <ANY>, 'parent': '', 'root': <ANY>}}, 'sc[648 chars]]}}}] != [{'fi[234 chars]de': 'e47f43d1-9097-463b-9a1c-86e9fe94426e', '[3716 chars]]}}}]

E       

E       First differing element 1:

E       {'fil[283 chars]de': <ANY>, 'parent': <ANY>, 'root': <ANY>}}, [193 chars][]}}}

E       {'fil[283 chars]de': 'e1bff311-b95f-4b20-95d5-23d787c5b50d', '[1424 chars][]}}}

E       

E         [{'file': {'depth': 0,

E                    'flavors': {'mime': ['text/html'], 'yara': ['html_file']},

E                    'name': '',

E                    'scanners': ['ScanEntropy',

E                                 'ScanFooter',

E                                 'ScanHash',

E                                 'ScanHeader',

E                                 'ScanHtml',

E                                 'ScanTlsh',

E                                 'ScanYara'],

E                    'size': 5875,

E                    'source': '',

E       -            'tree': {'node': <ANY>, 'parent': '', 'root': <ANY>}},

E       -   'scan': {'entropy': <ANY>,

E       -            'footer': <ANY>,

E       -            'hash': <ANY>,

E       -            'header': <ANY>,

E       -            'html': <ANY>,

E       -            'tlsh': <ANY>,

E       -            'yara': <ANY>}},

E       +            'tree': {'node': 'e47f43d1-9097-463b-9a1c-86e9fe94426e',

E       +                     'parent': '',

E       +                     'root': 'e47f43d1-9097-463b-9a1c-86e9fe94426e'}},

E       +   'scan': {'entropy': {'elapsed': 3.9e-05,

E       +                        'entropy': 4.847574566795829,

E       +                        'flags': []},

E       +            'footer': {'backslash': 'pan></span>\\n</p>\\n\\n\\n<p>&nbsp;</p>\\n\\n\\n</body>\\n</html>',

E       +                       'elapsed': 1.8e-05,

E       +                       'flags': [],

E       +                       'footer': b'pan></span>\n</p>\n\n\n<p>&nbsp;</p>\n\n\n<'

E       +                                 b'/body>\n</html>'},

E       +            'hash': {'elapsed': 0.002884,

E       +                     'flags': [],

E       +                     'md5': 'ba4ffdba7f62b2333a23a97d3ba5f1f6',

E       +                     'sha1': 'a1f900c64ed49bc111462c6fd91546640b5ac20c',

E       +                     'sha256': '38e2d4d56acf228fcebbbf5a60a16bb36ffcee490299ea52c9b1ffbcbeb62db8',

E       +                     'ssdeep': '96:qWJQC5siJJ+tH6STSTSTSTkvsAw2gF3BgwQWhhSTSTSTSTS/sItklIy7STSTSTSh:qOQGsiJJO3eeeIvspjJPyeeeefklCeew',

E       +                     'tlsh': 'T14AC16713EF67021152BDA0E9E0BF4A64D494560CA3465BF4B2AE477ABFCD93136122CC'},

E       +            'header': {'backslash': '<!DOCTYPE html>\\n<html '

E       +                                    'lang="en">\\n<head>\\n    <title',

E       +                       'elapsed': 8.1e-05,

E       +                       'flags': [],

E       +                       'header': b'<!DOCTYPE html>\n<html lang="en">\n<head>\n'

E       +                                 b'    <title'},

E       +            'html': {'elapsed': 0.005203,

E       +                     'flags': [],

E       +                     'forms': [],

E       +                     'frames': [],

E       +                     'hyperlinks': [],

E       +                     'inputs': [],

E       +                     'scripts': [{'language': None,

E       +                                  'src': 'https://example.com/example.js',

E       +                                  'type': 'text/javascript'},

E       +                                 {'language': None, 'src': None, 'type': None}],

E       +                     'spans': [{'class': None, 'style': 'font-size:11pt'},

E       +                               {'class': None,

E       +                                'style': 'background-color:white'},

E       +                               {'class': None,

E       +                                'style': 'font-family:Calibri,sans-serif'},

E       +                               {'class': None, 'style': 'font-size:52.5pt'},

E       +                               {'class': None, 'style': 'color:black'},

E       +                               {'class': None, 'style': 'font-size:12pt'},

E       +                               {'class': None,

E       +                                'style': 'font-family:"Times New Roman",serif'},

E       +                               {'class': None, 'style': 'font-size:10.5pt'},

E       +                               {'class': None,

E       +                                'style': 'font-family:"Arial",sans-serif'}],

E       +                     'title': 'Lorem Ipsum',

E       +                     'total': {'extracted': 1,

E       +                               'forms': 0,

E       +                               'frames': 0,

E       +                               'inputs': 0,

E       +                               'scripts': 2,

E       +                               'spans': 35}},

E       +            'tlsh': {'elapsed': 0.000782,

E       +                     'flags': [],

E       +                     'match': {'family': None, 'score': None}},

E       +            'yara': {'elapsed': 0.654638,

E       +                     'flags': [],

E       +                     'matches': [],

E       +                     'meta': [],

E       +                     'tags': []}}},

E          {'file': {'depth': 1,

E                    'flavors': {'external': ['', ''],

E                                'mime': ['text/plain'],

E                                'yara': ['javascript_file']},

E                    'name': 'script_1',

E                    'scanners': ['ScanEntropy',

E                                 'ScanFooter',

E                                 'ScanHash',

E                                 'ScanHeader',

E                                 'ScanJavascript',

E                                 'ScanTlsh',

E                                 'ScanYara'],

E                    'size': 221,

E                    'source': 'ScanHtml',

E       -            'tree': {'node': <ANY>, 'parent': <ANY>, 'root': <ANY>}},

E       -   'scan': {'entropy': <ANY>,

E       -            'footer': <ANY>,

E       -            'hash': <ANY>,

E       -            'header': <ANY>,

E       -            'javascript': <ANY>,

E       -            'tlsh': <ANY>,

E       +            'tree': {'node': 'e1bff311-b95f-4b20-95d5-23d787c5b50d',

E       +                     'parent': 'e47f43d1-9097-463b-9a1c-86e9fe94426e',

E       +                     'root': 'e47f43d1-9097-463b-9a1c-86e9fe94426e'}},

E       +   'scan': {'entropy': {'elapsed': 2.8e-05,

E       +                        'entropy': 4.620200029985679,

E       +                        'flags': []},

E       +            'footer': {'backslash': '   '

E       +                                    'document.body.appendChild(newParagraphElement)\\n',

E       +                       'elapsed': 2.8e-05,

E       +                       'flags': [],

E       +                       'footer': b'   document.body.appendChild(newParagraphEle'

E       +                                 b'ment)\n'},

E       -            'yara': {'elapsed': <ANY>,

E       ?             ^ ^^               ^^^^^

E       

E       +            'hash': {'elapsed': 0.000103,

E       ?             ^ ^^               ^^^^^^^^

E       

E                             'flags': [],

E       +                     'md5': 'ed2a6dffc68bcbe361f4539b5f423d66',

E       +                     'sha1': '172771134de76ede1df66cfa95a839237e485c40',

E       +                     'sha256': '8c3e97cc7103eec2f8959b0f27e2011f09f26386131b075a59f2423c791917ff',

E       +                     'ssdeep': '6:8/tuR78mgO9lV3K0Ji8mOFf0/tuRhBeJY1lLB/etuRMv:8/tuRYu80J17F8/tuRhBein2tuRu',

E       +                     'tlsh': 'T159D0A715143A07E4A34AB04F24344394F870045A30173115545F4CCF6F20E922485494'},

E       +            'header': {'backslash': '\\n    newParagraphElement = '

E       +                                    'document.createElement(',

E       +                       'elapsed': 1.3e-05,

E       +                       'flags': [],

E       +                       'header': b'\n    newParagraphElement = document.createEl'

E       +                                 b'ement('},

E       +            'javascript': {'beautified': True,

E       +                           'elapsed': 0.020824,

E       +                           'flags': [],

E       +                           'identifiers': ['createElement',

E       +                                           'textLoremIpsum',

E       +                                           'appendChild',

E       +                                           'body',

E       +                                           'document',

E       +                                           'createTextNode',

E       +                                           'newParagraphElement'],

E       +                           'keywords': [],

E       +                           'regular_expressions': [],

E       +                           'strings': ['span', 'Lorem Ipsum'],

E       +                           'tokens': ['Identifier', 'String', 'Punctuator']},

E       +            'tlsh': {'elapsed': 5.3e-05,

E       +                     'flags': [],

E       +                     'match': {'family': None, 'score': None}},

E       +            'yara': {'elapsed': 0.004409,

E       +                     'flags': [],

E       -                     'matches': ['test'],

E       ?                                 ------

E       

E       +                     'matches': [],

E                             'meta': [],

E                             'tags': []}}}]
/usr/lib/python3.10/unittest/case.py:675: AssertionError

FAILED tests/test_distribute.py::test_distribute - AssertionError: Lists diff...

[sl0wz3r]

Same issue building here, I also tried the recommendation above

[ryanohoro]

The test output @axelarator shared shows scan_yara is missing the test match, indicating that the supplied rules.yara file is missing the default test rule. This would happen when following the Quick Start, which overwrites rules.yara

E       +            'yara': {'elapsed': 0.004409,
E       +                     'flags': [],
E       -                     'matches': ['test'],
E       ?                                 ------
E       
E       +                     'matches': [],
E                             'meta': [],
E                             'tags': []}}}]

It's likely a mistake to assume the rules.yara file is unchanged specifically for tests/test_distribute.py::test_distribute which is made sensitive to quite a few config changes. I'll submit a PR that moves test_distribute.py to tests_configuration so it only runs when testing against the default backend config.

Workaround 1: Move tests/test_distribute.py to tests_configuration/test_distribute.py, build without CONFIG_TESTS

Workaround 2: Include the default test yara in the rules.yara file, and have that be the only yara rule that matches on src/python/strelka/tests/fixtures/test.html

rule test
{
  condition:
    true
}

[phutelmyer]

Thank you @ryanohoro, good catch - that would indeed cause it to fail.

@axelarator + @sl0wz3r, do you mind testing the latest commit at https://github.com/target/strelka/tree/master? If successful, I'll push out a new release. Appreciate the issue posts.

[axelarator]

Working great. Tested in a conda environment running Python 3.9.13 and could follow the install steps without error.