more flexible JSON serialization #65
Comments
|
Hi @Byron! Allowing the indent character is a great idea. Filtering out the Hm. If we added a
It's a little ugly, and would probably slow down the common case. I wonder if there's a better way to do this... |
|
Hi @erickt ! Judging from my inability to follow your statements starting with In any case, I am looking forward to the progress on this issue, as it will be very relevant to me when the advanced CLIs for Google Services are due for release. |
|
@Byron: Hehe :) Yeah sometimes my thinking out loud isn't particularly clear. The real challenge here is how to enable the serializer to do some lookahead. I had thought that this was only necessary for deserializers. |
This unfortunately loses the simd-ish whitespace printer, but since pretty printing shouldn't be on a hot path, this shouldn't really matter. Partially addresses #65.
|
@erickt I have updated the issue to be more general, and added a more prominent example to show the importance of the 'ignore empty' feature. Interesting enough, some remote servers like the ones for Google Drive, seem to ignore |
Byron
changed the title
|
@erickt Thanks for your help ! This is a major issue for me and I can help if you give me some hints on how to solve this. It's about time I do something for serde :). |
|
@erickt There has been no feedback on this matter for 8 days now, and with the 15th of May approaching, I feel like I have to step in and try to implement what I need myself and do so in an unguided fashion. This worries me as I am not at all familiar with the code-base. Additionally I am not comfortable submitting PRs anymore as these were usually ignored for quite a while until they have been superseded by code coming in from the maintainers. |
|
During lunch I cleared my mind and came to realize that attempting the implement this feature in serde has low chances of success, and that it will be more economic to just implement a client-side filter that operates on the generated string. That way, I can have my CLI remove the unwanted |
|
Have a look at the relevant function for json: http://serde-rs.github.io/serde/src/serde/json/ser.rs.html#248-258 One could simply call |
|
Thanks for the information ! |
|
@Byron: hey there! I've been very distracted with trying to get serde_macros working with stable, so I haven't had much time for serde proper. Another option too which I thought of earlier is that you could serialize into a use serde::json::value::{self, Value};
fn remove_nulls(value: &mut Value) {
match value {
Value::Object(ref mut map) => {
let to_remove = Vec::new();
for (key, value) in map {
if value.is_null() {
to_remove.push(key.clone());
} else {
remove_nulls(value);
}
}
for key in to_remove {
map.delete(key);
}
}
_ => {}
}
}
#[derive(Serialize, Deserialize)]
struct Point { x: u32, y: u32, name: Option<String> }
let point = Point { x: 0, y: 0, name: None };
let mut value = serde::json::value::to_value(&point);
remove_nulls(&mut value);
println!("{}", serde::json::to_string(&value));I haven't tried running it, but something like that should probably work. |
|
@erickt Thanks so much for this hint ! It solved my problem right away. Doing that could be a general workaround any kind of filtering one would wish, or could be used to implement custom Serializers which offer more options when pretty-printing. |
|
This can be worked around today using serde's new attributes. Define a trait and blanket impl for trait ShouldSkip {
fn should_skip(&self) -> bool;
}
impl<T> ShouldSkip for Option<T> {
fn should_skip(&self) -> bool {
self.is_none()
}
}Then in the struct definition struct Thing {
/// Optional string where null is undesirable
#[serde(skip_serializing_if="ShouldSkip::should_skip")]
field: Option<String>
}Even so, it would be nice to have some better default behavior for the format. |
|
You don't need the ShouldSkip trait: struct Thing {
#[serde(skip_serializing_if="Option::is_none")]
field: Option<String>
} |
|
Closing this because both solutions are sufficiently simple and I think the current behavior makes sense as the default. |
|
Why is the current behavior a sensible default, if I may ask? I haven't seen many consumers expecting null. Most I've used are happy with skipping the nulled fields (as it actually saves bandwidth). |
|
I think the current default is at least not confusing. Pretty sure if nulls were skipped by default it would confuse quite a few people. |
|
Pretty sure it wouldn't. Which framework does output null by default, and doesn't allow a global overriding? That is short to say that there should be some sort of API with this form: let encoder = serde_json::Encoder::new_with_some_options(x,y,z);
let str = try!(encoder.to_string(&item));where in this case options would look like ``IgnoreNone` or something like it. At the moment it's very unreadable. This is a small snippet out of a large, hierarchical structure I'm using. ...
#[serde(skip_serializing_if="Option::is_none")]
pub DayHoursStart: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub DayHours: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub NightHoursStart: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub NightHours: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub SpeedOfTime: Option<Vec<Float>>,
#[serde(skip_serializing_if="Option::is_none")]
pub TrackInfrastructure: Option<Vec<Int>>,
#[serde(skip_serializing_if="Option::is_none")]
pub DryCoverage: Option<Vec<String>>,
#[serde(skip_serializing_if="Option::is_none")]
pub WetCoverage: Option<Vec<String>>,
#[serde(skip_serializing_if="Option::is_none")]
pub MinimumPlayers: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub MaximumPlayers: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub Interval: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub NextSlotOffsetSeconds: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub LockoutTime: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub MatchedIntoTimeout: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub EndOfRace: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub VehicleSelect: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub TimeTrialDuration: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")]
pub DryToWet: Option<Bool>,
#[serde(skip_serializing_if="Option::is_none")]
pub Events: Option<Vec<Event>>,
... |
|
@norru i am not sure what serialization frameworks do but a naive serialization of a data structure to JSON in all environments I have seen emit a null. That includes dynamic languages as well as JSON libraries in C and C++. Especially because of web apis that accept arbitrary JSON without schema it's important that null is emitted as otherwise the entire key is missing which then would require the consumer of that data blob to be aware of the schema. |
|
It's the other way round. I have never used an API which requires null, omitting the field seems to be the common case. On the contrary, some APIs I've used in the past won't accept "null". I generally remove nulls whenever possible for 1) readability 2) message size 3) Any Other Business I don't feel the need to justify here. I don't know if those APIs do it by design or by accident, and you may well dismiss those APIs as "broken" (which is not necessarily so), but, hey! It's a wild wild world out there and that's life. If "send null" is a default for |
|
+1 for more configurability |
|
@norru: |
|
@norru: #[allow(non_snake_case)]
#[derive(Clone, Serialize)]
pub struct LooseJson {
#[serde(skip_serializing_if="Option::is_none")] pub DayHoursStart: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")] pub DayHours: Option<Int>,
#[serde(skip_serializing_if="Option::is_none")] pub NightHoursStart: Option<Int>,
}With the quality of syntax highlighting in IntelliJ IDEA this is even more readable. |
|
Thanks for the discussion everybody. Our hands are sort of tied here by non-self-describing binary formats. If Serde were for JSON only, I would agree that skip none may make sense by default. However, this is overruled by the desire to have the default generated code work in all data formats supported by Serde. How would JSON folks feel about a container attribute to enable this behavior for all Option fields of a particular struct? #[derive(Serialize)]
#[serde(skip_serializing_none)]
struct Norru {
pub DayHoursStart: Option<Int>,
pub DayHours: Option<Int>,
pub NightHoursStart: Option<Int>,
pub NightHours: Option<Int>,
pub SpeedOfTime: Option<Vec<Float>>,
pub TrackInfrastructure: Option<Vec<Int>>,
pub DryCoverage: Option<Vec<String>>,
pub WetCoverage: Option<Vec<String>>,
pub MinimumPlayers: Option<Int>,
pub MaximumPlayers: Option<Int>,
pub Interval: Option<Int>,
pub NextSlotOffsetSeconds: Option<Int>,
pub LockoutTime: Option<Int>,
pub MatchedIntoTimeout: Option<Int>,
pub EndOfRace: Option<Int>,
pub VehicleSelect: Option<Int>,
pub TimeTrialDuration: Option<Int>,
pub DryToWet: Option<Bool>,
pub Events: Option<Vec<Event>>,
} |
|
What about making it possible to apply |
|
@Mark-Simulacrum: what would happen when not all fields have |
|
@sanmai-NL formatting-only solution doesn't work for me, looks very cluttered. Yes, it's marginally better, but: how did you set up On the other hand, I like the per-struct attribute, which can be overridden per-field (specifying |
If not all fields have the |
pow: avoid unnecessary overflows
|
So there is still NOT a macro to cover a whole struct with non-option and option attributes? Do I really have to add #[serde(skip_serializing_if = "Option::is_none")] before every option key? |
Serialization for machines
When structures are serialized to JSON to be transferred over the wire, there may be servers that expect
nullvalues to be omitted, possibly to save bandwidth. Right now, there is no way inserdeto omit null values.As a practical example, imagine someone trying to upload a video to youtube:
Which yields the following error:
As well as the following dialogue between client and server:
As you can see, the request contains
nullvalues which are not allowed.To further stress the importance of this feature, have a look at the respective Go implementation ...
... where the marker
omitemptywill prevent it to be serialized if unset.You can try it yourself using the
youtube3program, which can be downloaded here.Serialization for human consumption
Right now there is exactly one method to get 'pretty', i.e. more human-friendly json output. It provides no option to specify how exactly that could be done.
The most prominent one to me would be a setting for whether or not to ignore
nullvalues. Other options could be the indentation string to use, e.g.\torMotivation
When printing the server response of any of the various google apis using a generated command-line interface, simple invocation yield results like this:
The above should look like this:
The text was updated successfully, but these errors were encountered: