Add Prolog Support

[javatlacati]

Some good edimburg prolog will be nice!

What about some open-source SWI-Prolog?

[javatlacati]

Swi prolog has its own testing framework package known as plunit

[kazk]

Does it support custom output?

[javatlacati]

You mean like writing to the console?

[kazk]

No, we need test results in Codewars format:

<PASSED::>
<FAILED::>
<ERROR::>
<DESCRIBE::>
<IT::>

Some test frameworks provides a way to add custom reporter.

[javatlacati]

I think that you should write that format as output like in:

error:-write('<ERROR::>'),nl,fail.
passed:-write('<PASSED::>'),nl.
it(Itdescription):-write('<IT::>'),write(Itdescription),nl.
describe(Description):-write('<DESCRIBE::>'),write(Description),nl.
describe(Description,It):-write('<DESCRIBE::>'),it(It),write(Description),nl.

where nl, prints a new line, and write provides a string output to standard output.

write is pretty much the equivalent of console.log in javascript except that it accepts only one term, so that's why you see a call to write for every single term type

[javatlacati]

Regarding that the test unit framework says

2.2 Writing the test body

The test-body is ordinary Prolog code. 

[kazk]

I think you're misunderstanding.
Is it possible for plunit to output <FAILED::> whenever the test fail?

[kazk]

So this example from the documentation

test(add) :-
        A is 1 + 2,
        A =:= 3.

needs to output <PASSED::>.

[kazk]

Another options is to derive the Codewars output from its test results. For PowerShell, we generate Codewars format from NUnit XML report.

The last resort is to parse its output and try our best to generate compatible result. We currently do this for Rust.

[javatlacati]

rules are evaluated sequentially, so in

error:-write('<ERROR::>'),nl,fail.

it will print

ERROR::
fail.

in the fail may have some error messages according the error.

And your example translated will be:

error:-write('<ERROR::>'),nl,fail.
passed:-write('<PASSED::>'),nl.
test(add) :-
        A is 1 + 2,
        A =:= 3,
        passed .
test(add) :-
        A \= 3,
        error.

That as you may think resembles current erlang code style, if one rule doesn't match the input it will try the next allowing recursive definitions and rewriting of simple rules.

[kazk]

I tried your suggestion

:- begin_tests(example).

error:-write('<ERROR::>'),nl,fail.
passed:-write('<PASSED::>'),nl.

test(add) :-
        A is 1 + 2,
        A =:= 3,
        passed .

test(add) :-
        A \= 3,
        error .

:- end_tests(example).

and got

Warning: /runner/output.pl:11:
	Singleton variables: [A]
% PL-Unit: example <PASSED::>
.
ERROR: /runner/output.pl:11:
	test add: failed

 done
% 1 test failed
% 1 tests passed
ERROR: -g run_tests: false

Everything except <PASSED::> is written to stderr so I can redirect them to /dev/null to get <PASSED::> only, but it doesn't work for failed tests. It also gets really repetitive having to write opposite rules for all the test cases.

[javatlacati]

I've asked in the English StackOverflow and Spanish StackOverflow sites in order to integrate it with the default unit test framework. I'll update later.

[kazk]

How does the regular output of plunit look like? Maybe we can do post processing (like Rust, Coq, etc) instead of letting it output what we need.

[javatlacati]

Sucessfull:

?- run_tests.
% PL-Unit: my_add .. done
% All 2 tests passed
true.

Unsucessfull:

?- run_tests.
% PL-Unit: my_add 
ERROR: c:/users/ruslanlopez/documents/prolog/addition_ut.pl:22:
        test my_add: failed

. done
% 1 test failed
% 1 tests passed
false.

[javatlacati]

What people said in the StackOverflow was that I could try to run a query with something like

Successful

my_add(2,4,X),(is(X,6)->writeln('<PASSED::>');writeln('<ERROR::>')).
<PASSED::>
X = 6.

Unsuccessful

my_add(2,4,X),(is(X,6)->writeln('<PASSED::>');writeln('<ERROR::>')).
<ERROR::>
X = 8.

But my concern is that this could implicate not using the standard test framework.

If you think it's the more viable option I can try creating a rule doing queries like that, something like

prolog_test(Predicate, Params, ExpectedLastParam, Value):- call(Predicate,Params,ExpectedLastParam), (is(ExpectedLastParam,Value)->writeln('<PASSED::>');writeln('<ERROR::>')).

so the unit test could be written something like:

prolog_test(my_add,[2,4],X,6).

[kazk]

I'd like to avoid that if possible. I think we should look for another option if plunit cannot be used normally. It'll be just confusing for users who are familiar with plunit so there's no real benefit of using the existing tool.

I was reading their docs and found the section for assertion that might produce enough output for post-processing.

Failing Example:

% test.pl
:- begin_tests(test).

test(a) :-
        A is 2^3,
        assertion(float(A)),
        assertion(A == 9).

:- end_tests(test).

% PL-Unit: test 
ERROR: /test.pl:17:
	test a: assertion failed
	Assertion: float(8)
ERROR: /test.pl:17:
	test a: assertion failed
	Assertion: 8==9
A done
% 2 assertions failed
% 1 test failed
% 0 tests passed

Passing Example:

% test.pl
:- begin_tests(test).

test(a) :-
        A is 2^3,
        assertion(A == 8).

:- end_tests(test).

% PL-Unit: test . done
% test passed
% halt

Running Tests:
To run the tests, do:

# `-g true` to suppress welcome message.
swipl -g true -t 'run_tests.' -s test.pl

For tests in separate file, do:

# tests in example.plt
swipl -g true -t "load_test_files([]), run_tests." -s example.pl

[kazk]

I think I can do the following (assuming the failure messages are always well formatted):

1. Extract test names from the test file so we can report passed tests.
2. From the failure message, detect failed tests:
   • When assertions are used, the output will include the failure info based on them.
     Each assertion failures will be grouped by the test name.
   • When just a goal is used, the output will be just "Test Failed".
3. Mark all tests not found in the failure message as passed.

---

Files:

% solution.pl
add_3_and_double(X,Y) :- Y is (X+3)*3.

% solution.plt
:- begin_tests(solution).
:- include(solution).

test(test_add_3_and_double_1) :-
        add_3_and_double(1, X),
        assertion(float(X)),
        assertion(X == 8).

test(test_add_3_and_double_2) :-
        add_3_and_double(2, X),
        X == 10.

test(test_add_3_and_double_3) :-
        add_3_and_double(-3, X),
        X == 0.

:- end_tests(solution).

Run tests:

swipl -g true -t 'load_test_files([]), run_tests.' -s solution.pl

plunit output:

% PL-Unit: solution 
ERROR: /solution.plt:5:
	test test_add_3_and_double_1: assertion failed
	Assertion: float(12)
ERROR: /solution.plt:5:
	test test_add_3_and_double_1: assertion failed
	Assertion: 12==8
A
ERROR: /solution.plt:10:
	test test_add_3_and_double_2: failed

. done
% 2 assertions failed
% 2 tests failed
% 1 tests passed

(don't know where "A. done" comes from and seems inconsistent)

Post-processed:

<IT::>test_add_3_and_double_1

<FAILED::>Assertion Failed: float(12)

<FAILED::>Assertion Failed: 12==8

<COMPLETEDIN::>

<IT::>test_add_3_and_double_2

<FAILED::>Test Failed

<COMPLETEDIN::>

<IT::>test_add_3_and_double_3

<PASSED::>Test Passed

<COMPLETEDIN::>

Rendered:

@javatlacati, @flip111, @razvan-flavius-panda, @Radivarig, @Kinrany

What do you think? Can any of you provide some example challenge for me to play with?

[Kinrany]

(don't know where "A. done" comes from and seems inconsistent)

Could it be a concurrency issue, with multiple messages being printed at the same time?

What do you think? Can any of you provide some example challenge for me to play with?

Would the recursive Euclid algorithm be a good enough example?

https://rosettacode.org/wiki/Greatest_common_divisor#Prolog

[kazk]

I think A is printed after the assertion failures when a test fails with assertion, nothing is printed on other failure, and . is printed after the test pass.

When I change the second test to use assertion, the output changes to:

% PL-Unit: solution 
ERROR: /solution.plt:5:
	test test_add_3_and_double_1: assertion failed
	Assertion: float(12)
ERROR: /solution.plt:5:
	test test_add_3_and_double_1: assertion failed
	Assertion: 12==8
A
ERROR: /solution.plt:10:
	test test_add_3_and_double_2: assertion failed
	Assertion: 15==10
A. done
% 3 assertions failed
% 2 tests failed
% 1 tests passed

When all 3 tests pass, the output is:

% PL-Unit: solution ... done

It looks weird, but it makes sense now.

---

Would the recursive Euclid algorithm be a good enough example?

I'm looking for different ways to fail the test to observe the outputs and adjust the post processing as needed. So it'll be really helpful if you can come up with some tests and different examples of failures for them.

[Kinrany]

It looks weird, but it makes sense now.

It looks like it's printing both "A" and the error message in parallel, and "done" is just the end of enumeration. Maybe the error messages can be suppressed or redirected, so you can get an easy to parse "solution AA. done"?

I'm looking for different ways to fail the test to observe the outputs and adjust the post processing as needed. So it'll be really helpful if you can come up with some tests and different examples of failures for them.

Like this?

test(test_gcd_1) :-
    gcd(2, 3, 1).

test(test_gcd_2) :- 
    gcd(4, 6, 2).

test(test_gcd_3) :- 
    forall(gcd(4, 6, X), X is 2).

Disclaimer: I barely know anything about Prolog, I upvoted because I'd like to learn it :)

[kazk]

Got it working. plunit can be used normally and results will be shown like other languages we support. When assertion is used, its failure message is shown. Otherwise, the failure message is just "Test Failed".
It'll look something like this:

The name of the solution module will be inferred from the test file (:- begin_tests(example).). Preloaded code is a separate module (preloaded.pl) that can be optionally included.

There might be some cases that breaks the post processing, but I'll let you guys play with it once it's on Codewars and fix them as it gets reported.

I'll close this issue when it's deployed on Codewars.

[kazk]

Deployed Prolog support.
Please let me know if you find any outputs that looks weird by opening new issues.

[javatlacati]

Hope this could help for the error messages:

addition.pl

my_add(A,B,Result):-
    number(A),
    number(B),
    is(Result,+(A,B+2)).

addition_ut.pl

:-['C:/Users/RuslanLopez/Documents/Prolog/adittion.pl']. %load file

:- begin_tests(my_add). %unit test block start
:- include(adittion). %load rules form file


% Codewars output format
error:-nl,write('<ERROR::>'),nl.
passed:-nl,write('<PASSED::>'),nl.
failed(Error):-nl,write('<FAILED::>'),write(Error) ,error.

unpacking2(Lists,L1,I):-
   findall(I,member([I],Lists),L1).


codewars_test(Rule, Parameters,Result, Condition, Message):-
        Parameters=[],
        call(Rule, Result), %evaluate predicate
        assertion(Condition) % Assertion
       ,(
            call(Condition) % test condition
           ->
              passed %sucess case
            ;
              failed(Message) %failed case

       )
      .

codewars_test(Rule, Parameters,Result, Condition, Message):-
        Parameters=[H],
        call(Rule, H, Result), %evaluate predicate
        assertion(Condition) % Assertion
       ,(
            call(Condition) % test condition
           ->
              passed %sucess case
            ;
              failed(Message) %failed case

       )
      .


codewars_test(Rule, Parameters,Result, Condition, Message):-
        Parameters=[H|T],
        T=[],
        call(Rule, H, Result), %evaluate predicate
        assertion(Condition) % Assertion
       ,(
            call(Condition) % test condition
           ->
              passed %sucess case
            ;
              failed(Message) %failed case

       )
      .


codewars_test(Rule, Parameters,Result, Condition, Message):-
        Parameters=[H|T],
        T=[S|_],
        call(Rule,H,S, Result), %evaluate predicate
        assertion(Condition) % Assertion
       ,(
            call(Condition) % test condition
           ->
              passed %sucess case
            ;
              failed(Message) %failed case

       )
      .


%unwraps up to 3 parameters
codewars_test(Rule, Parameters,Result, Condition, Message):-
        Parameters=[H|T],
        T=[S|W],
        W=[X,_],
        call(Rule,H,S,X, Result), %evaluate predicate
        assertion(Condition) % Assertion
       ,(
            call(Condition) % test condition
           ->
              passed %sucess case
            ;
              failed(Message) %failed case

       )
      .


test(my_add):- call(codewars_test(my_add, [1,2],Result, Result is 3,'1+2 should be 3')).
test(my_add):- call(codewars_test(my_add, [1,2],Result, Result \= 5,'1+2 should not be 5')).

:- end_tests(my_add).%unit test block end

Interactive run

?- run_tests.

% PL-Unit: my_add 
ERROR: c:/users/ruslanlopez/documents/prolog/addition_ut.pl:94:
        test my_add: assertion failed
        Assertion: 5 is 3

<FAILED::>1+2 should be 3
<ERROR::>
A
ERROR: c:/users/ruslanlopez/documents/prolog/addition_ut.pl:95:
        test my_add: assertion failed
        Assertion: 5\=5

<FAILED::>1+2 should not be 5
<ERROR::>
A done
% 2 assertions failed
% 2 tests failed
% 0 tests passed
false.