Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JRuby updates and fixes #481

Merged
merged 7 commits into from
Jun 4, 2021
Merged

JRuby updates and fixes #481

merged 7 commits into from
Jun 4, 2021

Conversation

headius
Copy link
Contributor

@headius headius commented Feb 25, 2021

Trying to get the suite green(er) again.

@headius
Negotiate to utf-8 in scalar
d2f5e1d
@headius
Flush writer after each emit
5d734aa 
OutputStreamWriter from JDK buffers outgoing bytes with a 8k
buffer, which causes some small document emits to never make it
into the outgoing stream unless that stream gets flushed or
closed.
@headius
Copy link
Contributor Author

headius commented Feb 25, 2021 •
edited
Loading

Back to 13F4E with commits so far, and we are back to most failures being related to minor behavior differences in SnakeYAML. Here is a complete list of failures now; only the two TestStream failures are new to me, but they appear to also be caused by the same sort of issue.

I would really like to get this library green somehow so if it fails we know something new is broken, but I am not sure how to rectify these differences.

Failure:
Psych::TestEncoding#test_io_shiftjis [/Users/headius/projects/psych/test/psych/test_encoding.rb:66]:
Psych::SyntaxError expected but nothing was raised.

Failure:
Psych::TestEncoding#test_emit_alias [/Users/headius/projects/psych/test/psych/test_encoding.rb:124]:
RuntimeError expected but nothing was raised.

Error:
Psych::TestEncoding#test_dump_non_ascii_string_to_file:
Encoding::UndefinedConversionError: "\xE3" from ASCII-8BIT to UTF-8
    org/jruby/RubyIO.java:1474:in `write'
    org/jruby/RubyIO.java:1430:in `write'
    org/jruby/ext/psych/PsychEmitter.java:284:in `end_mapping'
    /Users/headius/projects/psych/lib/psych/visitors/emitter.rb:44:in `visit_Psych_Nodes_Mapping'
    /Users/headius/projects/psych/lib/psych/visitors/visitor.rb:30:in `visit'
    /Users/headius/projects/psych/lib/psych/visitors/visitor.rb:6:in `accept'
    /Users/headius/projects/psych/lib/psych/visitors/emitter.rb:27:in `block in visit_Psych_Nodes_Document'
    org/jruby/RubyArray.java:1809:in `each'
    /Users/headius/projects/psych/lib/psych/visitors/emitter.rb:27:in `visit_Psych_Nodes_Document'
    /Users/headius/projects/psych/lib/psych/visitors/visitor.rb:30:in `visit'
    /Users/headius/projects/psych/lib/psych/visitors/visitor.rb:6:in `accept'
    /Users/headius/projects/psych/lib/psych/visitors/emitter.rb:21:in `block in visit_Psych_Nodes_Stream'
    org/jruby/RubyArray.java:1809:in `each'
    /Users/headius/projects/psych/lib/psych/visitors/emitter.rb:21:in `visit_Psych_Nodes_Stream'
    /Users/headius/projects/psych/lib/psych/visitors/visitor.rb:30:in `visit'
    /Users/headius/projects/psych/lib/psych/visitors/visitor.rb:6:in `accept'
    /Users/headius/projects/psych/lib/psych/nodes/node.rb:61:in `yaml'
    /Users/headius/projects/psych/lib/psych.rb:514:in `dump'
    /Users/headius/projects/psych/test/psych/test_encoding.rb:268:in `block in test_dump_non_ascii_string_to_file'
    /Users/headius/projects/jruby-9.2/lib/ruby/stdlib/tempfile.rb:45:in `create'
    /Users/headius/projects/psych/test/psych/test_encoding.rb:266:in `test_dump_non_ascii_string_to_file'

Error:
Psych::TestEncoding#test_start_mapping:
Psych::SyntaxError: (<unknown>): expected URI, but found バ(12496) while scanning a tag at line 1 column 8
    org/jruby/ext/psych/PsychParser.java:246:in `parse'
    org/jruby/ext/psych/PsychParser.java:110:in `parse'
    /Users/headius/projects/psych/test/psych/test_encoding.rb:155:in `test_start_mapping'

Error:
Psych::TestEncoding#test_start_sequence:
Psych::SyntaxError: (<unknown>): expected URI, but found バ(12496) while scanning a tag at line 1 column 8
    org/jruby/ext/psych/PsychParser.java:246:in `parse'
    org/jruby/ext/psych/PsychParser.java:110:in `parse'
    /Users/headius/projects/psych/test/psych/test_encoding.rb:174:in `test_start_sequence'

Error:
Psych::TestEncoding#test_doc_tag_encoding:
Psych::SyntaxError: (<unknown>): expected URI, but found 鍵(37749) while scanning a directive at line 2 column 8
    org/jruby/ext/psych/PsychParser.java:246:in `parse'
    org/jruby/ext/psych/PsychParser.java:110:in `parse'
    /Users/headius/projects/psych/test/psych/test_encoding.rb:191:in `test_doc_tag_encoding'

Failure:
Psych::JSON::TestStream#test_time [/Users/headius/projects/psych/test/psych/json/test_stream.rb:99]:
Expected /\{"a":\ "2010\-10\-10\ 00:00:00\.000000000\ Z"\}\n/ to match "--- {\"a\": \"2010-10-10 00:00:00.000000000 Z\"}".

Failure:
Psych::JSON::TestStream#test_datetime [/Users/headius/projects/psych/test/psych/json/test_stream.rb:106]:
Expected /\{"a":\ "2010\-10\-10\ 00:00:00\.000000000\ \-05:00"\}\n/ to match "--- {\"a\": \"2010-10-10 00:00:00.000000000 -05:00\"}".

Failure:
Psych::TestParser#test_start_document_tag [/Users/headius/projects/psych/test/psych/test_parser.rb:351]:
[nil, [[], [["!yaml!", "tag:yaml.org,2002"]], false]] not in [[:event_location, [0, 0, 0, 0]], [:start_stream, [0]], [:event_location, [0, 0, 1, 3]], [:start_document, [[], [["!!", "tag:yaml.org,2002:"], ["!", "!"], ["!yaml!", "tag:yaml.org,2002"]], false]], [:event_location, [2, 0, 2, 15]], [:scalar, ["foo", nil, "tag:yaml.org,2002str", false, false, 3]], [:event_location, [3, 0, 3, 0]], [:end_document, [true]], [:event_location, [3, 0, 3, 0]], [:end_stream, []]]

Failure:
Psych::TestParser#test_line_numbers [/Users/headius/projects/psych/test/psych/test_parser.rb:90]:
--- expected
+++ actual
@@ -1 +1 @@
-[[0, :event_location], [0, :start_stream], [0, :event_location], [0, :start_document], [1, :event_location], [1, :start_sequence], [2, :event_location], [2, :scalar], [3, :event_location], [3, :scalar], [3, :event_location], [3, :end_sequence], [3, :event_location], [3, :end_document], [3, :event_location], [3, :end_stream]]
+[[0, :event_location], [0, :start_stream], [1, :event_location], [1, :start_document], [1, :event_location], [1, :start_sequence], [2, :event_location], [2, :scalar], [2, :event_location], [2, :scalar], [2, :event_location], [2, :end_sequence], [2, :event_location], [2, :end_document], [2, :event_location], [2, :end_stream]]

Failure:
Psych::TestParser#test_column_numbers [/Users/headius/projects/psych/test/psych/test_parser.rb:115]:
--- expected
+++ actual
@@ -1 +1 @@
-[[0, :event_location], [0, :start_stream], [3, :event_location], [3, :start_document], [1, :event_location], [1, :start_sequence], [0, :event_location], [0, :scalar], [0, :event_location], [0, :scalar], [0, :event_location], [0, :end_sequence], [0, :event_location], [0, :end_document], [0, :event_location], [0, :end_stream]]
+[[0, :event_location], [0, :start_stream], [0, :event_location], [0, :start_document], [2, :event_location], [2, :start_sequence], [2, :event_location], [2, :scalar], [7, :event_location], [7, :scalar], [7, :event_location], [7, :end_sequence], [7, :event_location], [7, :end_document], [7, :event_location], [7, :end_stream]]

Failure:
Psych::TestParser#test_start_document_version [/Users/headius/projects/psych/test/psych/test_parser.rb:346]:
[nil, [[1, 1], [], false]] not in [[:event_location, [0, 0, 0, 0]], [:start_stream, [0]], [:event_location, [0, 0, 1, 3]], [:start_document, [[1, 1], [["!!", "tag:yaml.org,2002:"], ["!", "!"]], false]], [:event_location, [2, 0, 2, 5]], [:scalar, ["foo", nil, nil, false, true, 3]], [:event_location, [3, 0, 3, 0]], [:end_document, [true]], [:event_location, [3, 0, 3, 0]], [:end_stream, []]]

Failure:
Psych::TestParser#test_index_numbers [/Users/headius/projects/psych/test/psych/test_parser.rb:140]:
--- expected
+++ actual
@@ -1 +1 @@
-[[0, :event_location], [0, :start_stream], [3, :event_location], [3, :start_document], [5, :event_location], [5, :start_sequence], [12, :event_location], [12, :scalar], [19, :event_location], [19, :scalar], [19, :event_location], [19, :end_sequence], [19, :event_location], [19, :end_document], [19, :event_location], [19, :end_stream]]
+[[0, :event_location], [0, :start_stream], [0, :event_location], [0, :start_document], [0, :event_location], [0, :start_sequence], [0, :event_location], [0, :scalar], [0, :event_location], [0, :scalar], [0, :event_location], [0, :end_sequence], [0, :event_location], [0, :end_document], [0, :event_location], [0, :end_stream]]

Failure:
Psych::TestParser#test_event_location [/Users/headius/projects/psych/test/psych/test_parser.rb:362]:
--- expected
+++ actual
@@ -1 +1 @@
-[[:start_stream, [0, 0, 0, 0]], [:start_document, [0, 0, 0, 0]], [:start_mapping, [0, 0, 0, 0]], [:scalar, [0, 0, 0, 3]], [:start_mapping, [1, 2, 1, 2]], [:scalar, [1, 2, 1, 8]], [:start_sequence, [1, 10, 1, 11]], [:scalar, [1, 11, 1, 12]], [:scalar, [1, 14, 1, 15]], [:end_sequence, [1, 15, 1, 16]], [:end_mapping, [2, 0, 2, 0]], [:end_mapping, [2, 0, 2, 0]], [:end_document, [2, 0, 2, 0]], [:end_stream, [2, 0, 2, 0]]]
+[[:start_stream, [0, 0, 0, 0]], [:start_document, [0, 0, 0, 0]], [:start_mapping, [0, 0, 0, 0]], [:scalar, [0, 0, 0, 3]], [:start_mapping, [1, 2, 1, 2]], [:scalar, [1, 2, 1, 8]], [:start_sequence, [1, 10, 1, 11]], [:scalar, [1, 11, 1, 12]], [:scalar, [1, 14, 1, 15]], [:end_sequence, [1, 15, 1, 16]], [:end_mapping, [1, 16, 1, 16]], [:end_mapping, [1, 16, 1, 16]], [:end_document, [1, 16, 1, 16]], [:end_stream, [1, 16, 1, 16]]]

Failure:
Psych::Handlers::TestRecorder#test_replay [/Users/headius/projects/psych/test/psych/handlers/test_recorder.rb:22]:
--- expected
+++ actual
@@ -1,3 +1,5 @@
-"--- foo
+"%TAG ! !
+%TAG !! tag:yaml.org,2002:
+--- foo
 ...
 "

Failure:
Psych::TestTreeBuilder#test_documents [/Users/headius/projects/psych/test/psych/test_tree_builder.rb:33]:
--- expected
+++ actual
@@ -1 +1 @@
-[]
+[["!!", "tag:yaml.org,2002:"], ["!", "!"]]

Failure:
Psych::TestDocument#test_parse_tag [/Users/headius/projects/psych/test/psych/test_document.rb:17]:
--- expected
+++ actual
@@ -1 +1 @@
-[["!", "tag:tenderlovemaking.com,2009:"]]
+[["!!", "tag:yaml.org,2002:"], ["!", "tag:tenderlovemaking.com,2009:"]]

@headius
Skip deprecated require form
b4b5b49
@headius
Make malformed input noisy
dce46b1 
parse should reject input encoded incorrectly. The default
behavior for InputStreamReader is to replace malformed characters,
which causes one exception-expecting test to fail. This patch
changes the behavior to report malformed input, which can then be
reraised as a syntax error.
@headius
Clean up type checks and trancodes
ca1c154
@headius
Minor optimization
bce2752
@headius
Minor optimization: defer runtime access
d0165b6
@headius
Copy link
Contributor Author

headius commented Mar 1, 2021

Link with jruby/jruby#6578.

headius added a commit to headius/jruby that referenced this pull request Mar 1, 2021
@headius
Call write directly rather than through stream
ac272b9 
The OutputStream returned by RubyIO.getOutputStream encodes all
strings as ASCII-8BIT which makes it incompatible with any IO that
has an incompatible external encoding. That is a separate bug, but
it affects uses of IOOutputStream that should be ok.

IOOutputStream aggregates an encoding to use, so it should be able
to pass properly-encoded strings through to the IO. This change
avoids using the stream from getOutputStream and instead calls
RubyIO.write directly.

This fixes the ASCII-8BIT failure noted in ruby/psych#481.
This was referenced Mar 1, 2021
Merged
Open
@headius
Copy link
Contributor Author

headius commented Mar 1, 2021

jruby/jruby#6587 brings this to 12F3E.

Most of these are in the following categories, which I have brought up with @asomov in https://bitbucket.org/asomov/snakeyaml/issues/498/clarifying-differences-from-libyaml:

  • Position information differing for some tokens
  • YAML version propagating as an unexpected tag
  • Subtle formatting differences in emitted YAML (newlines, --)

I'm still trying to figure out the two URI failures and the missing error in test_emit_alias.

@headius headius merged commit 68f8145 into ruby:master Jun 4, 2021
@headius headius deleted the jruby_updates branch June 4, 2021 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@headius