Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicode escape #354

Merged
merged 8 commits into from
Dec 27, 2020
Merged

Unicode escape #354

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f922d9a
[#334] parse and unparse tests
dariodsa Nov 14, 2020
d1b48b7
removed parsing and unparing tests
dariodsa Nov 18, 2020
0e81ac8
[#334] showUnicodeText
dariodsa Nov 18, 2020
7587a21
[#334] escaping unicode character as well as regular characters
dariodsa Dec 8, 2020
0349ff4
[#334] resolved issue with escaping regular unescaped chars
dariodsa Dec 9, 2020
2aace95
added tests, but they are not in use
dariodsa Dec 9, 2020
52d68c5
examples.hs revert to original content
dariodsa Dec 9, 2020
fb68968
[#334] changes requested by chshersh
dariodsa Dec 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion src/Toml/Type/Printer.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ module Toml.Type.Printer
) where

import Data.Bifunctor (first)
import Data.Char (isAscii, ord)
import Data.Coerce (coerce)
import Data.Function (on)
import Data.HashMap.Strict (HashMap)
Expand All @@ -29,6 +30,8 @@ import Data.Semigroup (stimes)
import Data.Text (Text)
import Data.Time (ZonedTime, defaultTimeLocale, formatTime)

import Text.Printf (printf)

import Toml.Type.AnyValue (AnyValue (..))
import Toml.Type.Key (Key (..), Piece (..))
import Toml.Type.PrefixTree (PrefixMap, PrefixTree (..))
Expand All @@ -40,6 +43,7 @@ import qualified Data.List.NonEmpty as NonEmpty
import qualified Data.Text as Text



{- | Configures the pretty printer.

@since 0.5.0
Expand Down Expand Up @@ -161,7 +165,7 @@ prettyKeyValue options i = mapOrdered (\kv -> [kvText kv]) options . HashMap.toL
valText (Bool b) = Text.toLower $ showText b
valText (Integer n) = showText n
valText (Double d) = showDouble d
valText (Text s) = showText s
valText (Text s) = showTextUnicode s
valText (Zoned z) = showZonedTime z
valText (Local l) = showText l
valText (Day d) = showText d
Expand All @@ -171,6 +175,19 @@ prettyKeyValue options i = mapOrdered (\kv -> [kvText kv]) options . HashMap.toL
showText :: Show a => a -> Text
showText = Text.pack . show

showTextUnicode :: Text -> Text
dariodsa marked this conversation as resolved.
Show resolved Hide resolved
showTextUnicode text = Text.pack quotedText
where
quotedText = show finalText
finalText = foldl (\acc (ch, asciiCh) -> acc ++ getCh ch asciiCh) "" asciiArr
dariodsa marked this conversation as resolved.
Show resolved Hide resolved
xss = Text.unpack text
asciiArr = zip xss $ asciiStatus xss
getCh ch True = [ch]
getCh ch False = printf "\\U%08x" ordChr :: String
dariodsa marked this conversation as resolved.
Show resolved Hide resolved
where
ordChr = ord ch
dariodsa marked this conversation as resolved.
Show resolved Hide resolved
asciiStatus = map isAscii
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add types to all functions and values inside the where block? Otherwise, it's a bit hard to understand the whole context and what is going on here.


showDouble :: Double -> Text
showDouble d | isInfinite d && d < 0 = "-inf"
| isInfinite d = "inf"
Expand Down
9 changes: 9 additions & 0 deletions test/Test/Toml/Gen.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,13 @@ genUniHex8Color = do
hex <- genDiffHex 8
pure . Text.pack $ "\\U" ++ hex

-- | Generates some unescaped unicode string
genUnicodeChar :: Gen Text
genUnicodeChar = Gen.element
[ "č", "ć", "š", "đ", "ž", "Ö", "ё"
, "в", "ь", "ж", "ю", "ч", "ü", "я"
]

-- | Generates text from different symbols.
genText :: Gen Text
genText = genNotEscape $ fmap Text.concat $ Gen.list (Range.constant 0 256) $ Gen.choice
Expand All @@ -307,8 +314,10 @@ genText = genNotEscape $ fmap Text.concat $ Gen.list (Range.constant 0 256) $ Ge
, genPunctuation
, genUniHex4Color
, genUniHex8Color
--, genUnicodeChar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be uncommented back?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

genUnicodeChar is the function that I wrote myself thinking that I might be helpful to add it into the testing phase. But if I enable it, the test will fail.
The reason why it fails is mentioned above but I can repeat it. Character č is transformed into \U0000010d but decoding it back will give \U0000010d again, not č so the encode . decode tests will fail.
Currently, tests are only include escaped unicode characters in encode . decode phase.

]


genString :: Gen String
genString = Text.unpack <$> genText

Expand Down