Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Codable records: JSON encoding and decoding of codable record properties #397

Merged
merged 38 commits into from
Aug 15, 2018
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
feeedcb
Added nested Codable support by flattening into/from JSON
Aug 7, 2018
74aae3d
implemented requested changes
Aug 8, 2018
7def60c
fixed checking for nested codable
Aug 8, 2018
2826472
sorted encoder outputFormatting for ios11+ etcl
Aug 9, 2018
6262d10
added catch
Aug 14, 2018
7f16e35
extended Codable documentation and added test for the example used in…
Aug 14, 2018
8c06e7b
Detached rows string to data and tests
gusrota Aug 14, 2018
a08974f
Added tests for arrays
gusrota Aug 14, 2018
d81475a
6262d10 continued
Aug 14, 2018
76f5adf
moved the extension to Swift.Data to follow the extension to Swift.St…
Aug 14, 2018
c52b59f
Switching to JSON encoding for nested containers.
gusrota Aug 14, 2018
4b7f17d
Merge remote-tracking branch 'origin/development' into rotacloud
groue Aug 15, 2018
05fda18
Introduce RowSingleValueDecoder
groue Aug 15, 2018
2750980
Introduce JSONDecodingRequiredError
groue Aug 15, 2018
aa8bc0f
Don't let JSONDecodingRequiredError go
groue Aug 15, 2018
942b32e
RowSingleValueDecodingContainer performance improvements
groue Aug 15, 2018
0cdc7d8
Just some cleanup, nothing new
groue Aug 15, 2018
5f41c57
amended the documentation and tests to include Dictionary support to …
Aug 15, 2018
1c66ad4
Restore GRDB/Core/Support/Foundation/Data.swift
groue Aug 15, 2018
820673b
PersistableRecord+Encodable: make it as robust as FetchableRecord+Dec…
groue Aug 15, 2018
358225b
Cleanup, documentation, and preserve coding paths
groue Aug 15, 2018
b2234b4
5f41c57c addition
Aug 15, 2018
2d6140a
Refactor value conversion tests
groue Aug 15, 2018
de4f57c
Test decoding of non-UTF8 data
groue Aug 15, 2018
0b77d0d
Better feedback for conversion errors from StatementColumnConvertible
groue Aug 15, 2018
e8dacb3
Tests for failed value conversions
groue Aug 15, 2018
b47f586
Even better feedback for conversion errors from StatementColumnConver…
groue Aug 15, 2018
eba21d3
Oh men of little faith!
groue Aug 15, 2018
4d343e8
More tests for Data -> String conversion: play with JPEG :-)
groue Aug 15, 2018
d00421a
Restore SPM: add missing Foundation import
groue Aug 15, 2018
659db4f
Safer default implementation for RowImpl.copiedRow(_:)
groue Aug 15, 2018
2eb0955
JSON encoding: choose strategies
groue Aug 15, 2018
9bb4ff8
Tests for JSON date strategy
groue Aug 15, 2018
02db2a8
Tests for JSON data strategy
groue Aug 15, 2018
dfb9421
Documentation for JSON support
groue Aug 15, 2018
2b8c838
Fix typo
groue Aug 15, 2018
44b998c
CHANGELOG.md
groue Aug 15, 2018
7ea15dc
Welcome @valexa and @gusrota to the thank you notice
groue Aug 15, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 50 additions & 2 deletions GRDB/Record/FetchableRecord+Decodable.swift
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import Foundation

private struct RowKeyedDecodingContainer<Key: CodingKey>: KeyedDecodingContainerProtocol {
let decoder: RowDecoder

Expand Down Expand Up @@ -79,7 +81,30 @@ private struct RowKeyedDecodingContainer<Key: CodingKey>: KeyedDecodingContainer
} else if dbValue.isNull {
return nil
} else {
return try T(from: RowDecoder(row: row, codingPath: codingPath + [key]))
do {
// This decoding will fail for types that need a keyed container,
// because we're decoding a database value here (string, int, double, data, null)
// if we find a nested Decodable type then pass the string to JSON decoder
let value = try T(from: RowDecoder(row: row, codingPath: codingPath + [key]))
if let _ = value as? Codable {
// Support for nested ( Codable )
return try JSONDecoder().decode(type.self, from: row.dataNoCopy(named: key.stringValue)!)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the reason for if let _ = value as? Codable. The comment Support for nested ( Codable ) doesn't help me understand.

  • we are decoding twice here, which looks like a big waste of time.

  • why do we check for Codable, since we are only decoding? Isn't Decodable enough?

  • how on Earth this condition could be false, since we are decoding a Decodable type?

All this code is pretty obscure to me. What do I miss?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure this is the best but the way the current code is basically considering the Encodable compliance a such 4 tests hit this code the following way :
if let _ = value as? Codable {
//testOptionalNestedStruct
} else {
//testDecodableRawRepresentableProperty
//testNonTrivialSingleValueDecodableProperty
//testTrivialSingleValueDecodableProperty
}

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the details. Please provide an answer to other questions as well.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want you to understand, @valexa, that the if let _ = value as? Codable test has to go.

That's because it would have GRDB behave differently if I start decoding this type:

struct MyStruct: Decodable { ... }

And then I decide that MyStruct should be Encodable as well:

struct MyStruct: Codable { ... }

In the first version of MyStruct, if let _ = value as? Codable doesn't pass. In the second version, if let _ = value as? Codable passes. Decoding will change. Just because I added a conformance to MyStruct, totally unrelated to decoding.

This is an important bug that has to be fixed.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've downloaded your code, and run the tests. I see the problem you are facing. Please hold on until I find the proper way out.

Copy link
Owner

@groue groue Aug 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All right, the problem is that we use a RowDecoder (dedicated to decoding a row) when we decode a column value. That decoder happily builds keyed and unkeyed containers. But we don't want to build keyed and unkeyed containers when we decode a column value! It is quite the contrary: we want a decoding error, so that we can switch to JSON. Conclusion: RowDecoder is not the decoder we need here. We need another decoder instead. I'll take care of fixing this one. Please make sure I can push to your branch.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

} else {
return value
}
} catch {
switch error as! DecodingError {
case .typeMismatch(_, let context):
if context.debugDescription == "unkeyed decoding is not supported" {
// Support for nested keyed containers ( [Codable] )
return try JSONDecoder().decode(type.self, from: row.dataNoCopy(named: key.stringValue)!)
} else {
throw(error)
}
default:
throw(error)
}
}
}
}

Expand Down Expand Up @@ -116,7 +141,30 @@ private struct RowKeyedDecodingContainer<Key: CodingKey>: KeyedDecodingContainer
// This allows decoding Date from String, or DatabaseValue from NULL.
return type.fromDatabaseValue(dbValue) as! T
} else {
return try T(from: RowDecoder(row: row, codingPath: codingPath + [key]))
do {
// This decoding will fail for types that need a keyed container,
// because we're decoding a database value here (string, int, double, data, null)
// if we find a nested Decodable type then pass the string to JSON decoder
let value = try T(from: RowDecoder(row: row, codingPath: codingPath + [key]))
if let _ = value as? Codable {
// Support for nested ( Codable )
return try JSONDecoder().decode(type.self, from: row.dataNoCopy(named: key.stringValue)!)
} else {
return value
}
} catch {
switch error as! DecodingError {
case .typeMismatch(_, let context):
if context.debugDescription == "unkeyed decoding is not supported" {
// Support for nested keyed containers ( [Codable] )
return try JSONDecoder().decode(type.self, from: row.dataNoCopy(named: key.stringValue)!)
} else {
throw(error)
}
default:
throw(error)
}
}
}
}

Expand Down
112 changes: 109 additions & 3 deletions GRDB/Record/PersistableRecord+Encodable.swift
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import Foundation

private struct PersistableRecordKeyedEncodingContainer<Key: CodingKey> : KeyedEncodingContainerProtocol {
let encode: (_ value: DatabaseValueConvertible?, _ key: String) -> Void

Expand Down Expand Up @@ -40,7 +42,23 @@ private struct PersistableRecordKeyedEncodingContainer<Key: CodingKey> : KeyedEn
// This allows us to encode Date as String, for example.
encode((value as! DatabaseValueConvertible), key.stringValue)
} else {
try value.encode(to: PersistableRecordEncoder(codingPath: [key], encode: encode))
do {
try value.encode(to: PersistableRecordEncoder(codingPath: [key], encode: encode))
} catch {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we carefully check for the error type before we switch to JSON decoding, but not when we switch to JSON encoding? I'd like some consistency, and the same level of care to be displayed in both places.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a oversight, fixed now that you pointed it out.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean in 6262d10? It still needs a little extra effort: just like we did for decoding, we have to check for a particular encoding error before switching to JSON.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in d81475a

// If value.encode does not work e.g. "unkeyed encoding is not supported" then see if model can be stored as JSON
do {
let encoder = JSONEncoder()
if #available(watchOS 4.0, OSX 10.13, iOS 11.0, *) {
encoder.outputFormatting = .sortedKeys
}
let json = try encoder.encode(value)
//the Data from the encoder is guaranteed to convert to String
let modelAsString = String(data: json, encoding: .utf8)!
return encode(modelAsString, key.stringValue)
} catch {
throw(error)
}
}
}
}

Expand Down Expand Up @@ -169,7 +187,8 @@ private struct PersistableRecordEncoder : Encoder {
func container<Key>(keyedBy type: Key.Type) -> KeyedEncodingContainer<Key> {
// Asked for a keyed type: top level required
guard codingPath.isEmpty else {
fatalError("unkeyed encoding is not supported")
let error = EncodingError.invalidValue(encode, EncodingError.Context(codingPath: codingPath, debugDescription: "unkeyed encoding is not supported"))
return KeyedEncodingContainer(ThrowingKeyedContainer(error: error))
}
return KeyedEncodingContainer(PersistableRecordKeyedEncodingContainer<Key>(encode: encode))
}
Expand All @@ -180,7 +199,8 @@ private struct PersistableRecordEncoder : Encoder {
/// - precondition: May not be called after a prior `self.container(keyedBy:)` call.
/// - precondition: May not be called after a value has been encoded through a previous `self.singleValueContainer()` call.
func unkeyedContainer() -> UnkeyedEncodingContainer {
fatalError("unkeyed encoding is not supported")
let error = EncodingError.invalidValue(encode, EncodingError.Context(codingPath: [], debugDescription: "unkeyed encoding is not supported"))
return ThrowingUnkeyedContainer(error: error)
}

/// Returns an encoding container appropriate for holding a single primitive value.
Expand All @@ -194,6 +214,92 @@ private struct PersistableRecordEncoder : Encoder {
}
}

class ThrowingKeyedContainer<KeyType: CodingKey>: KeyedEncodingContainerProtocol {
let errorMessage: Error
var codingPath: [CodingKey] = []

init(error: Error) {
errorMessage = error
}

func encodeNil(forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: Bool, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: Int, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: Int8, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: Int16, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: Int32, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: Int64, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: UInt, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: UInt8, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: UInt16, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: UInt32, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: UInt64, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: Float, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: Double, forKey key: KeyType) throws { throw errorMessage }
func encode(_ value: String, forKey key: KeyType) throws { throw errorMessage }
func encode<T>(_ value: T, forKey key: KeyType) throws where T : Encodable { throw errorMessage }

func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type, forKey key: KeyType) -> KeyedEncodingContainer<NestedKey> where NestedKey : CodingKey {
fatalError("Not implemented")
Copy link
Owner

@groue groue Aug 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should return another throwing container here. Imagine a codable type that tries to access a nested container before it stores its properties and triggers one of our throwing encode methods above. Program will crash instead of switching to JSON encoding. This type should never fatalError, so that we can always switch to JSON. That's the whole point :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @groue good point :-), commit c52b59f now returns another throwing container.

}
func nestedUnkeyedContainer(forKey key: KeyType) -> UnkeyedEncodingContainer {
fatalError("Not implemented")
}
func superEncoder() -> Encoder {
fatalError("Not implemented")
}

func superEncoder(forKey key: KeyType) -> Encoder {
fatalError("Not implemented")
}
}

class ThrowingUnkeyedContainer: UnkeyedEncodingContainer {
let errorMessage: Error
var codingPath: [CodingKey] = []
var count: Int = 0

init(error: Error) {
errorMessage = error
}

func encode(_ value: Int) throws { throw errorMessage }
func encode(_ value: Int8) throws { throw errorMessage }
func encode(_ value: Int16) throws { throw errorMessage }
func encode(_ value: Int32) throws { throw errorMessage }
func encode(_ value: Int64) throws { throw errorMessage }
func encode(_ value: UInt) throws { throw errorMessage }
func encode(_ value: UInt8) throws { throw errorMessage }
func encode(_ value: UInt16) throws { throw errorMessage }
func encode(_ value: UInt32) throws { throw errorMessage }
func encode(_ value: UInt64) throws { throw errorMessage }
func encode(_ value: Float) throws { throw errorMessage }
func encode(_ value: Double) throws { throw errorMessage }
func encode(_ value: String) throws { throw errorMessage }
func encode<T>(_ value: T) throws where T : Encodable { throw errorMessage }
func encode(_ value: Bool) throws { throw errorMessage }
func encodeNil() throws { throw errorMessage }

func nestedContainer<NestedKey>(keyedBy keyType: NestedKey.Type) -> KeyedEncodingContainer<NestedKey> where NestedKey : CodingKey {
fatalError("Not implemented")
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above: there should be no fatalError in this type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commit c52b59f also returns another throwing container here


}

func nestedUnkeyedContainer() -> UnkeyedEncodingContainer {
fatalError("Not implemented")

}

func superEncoder() -> Encoder {
fatalError("Not implemented")

}
}

enum JsonStringError: Error {
case covertStringError(String)
}

extension MutablePersistableRecord where Self: Encodable {
public func encode(to container: inout PersistenceContainer) {
// The inout container parameter won't enter an escaping closure since
Expand Down
Loading