The Go Blog
A new experimental Go API for JSON
Introduction
JavaScript Object Notation (JSON) is a simple data interchange format. Almost 15 years ago, we wrote about support for JSON in Go, which introduced the ability to serialize and deserialize Go types to and from JSON data. Since then, JSON has become the most popular data format used on the Internet. It is widely read and written by Go programs, and encoding/json now ranks as the 5th most imported Go package.
Over time, packages evolve with the needs of their users,
and encoding/json is no exception. This blog post is about Go 1.25’s new
experimental encoding/json/v2 and encoding/json/jsontext packages,
which bring long-awaited improvements and fixes.
This post argues for a new major API version,
provides an overview of the new packages,
and explains how you can make use of it.
The experimental packages are not visible by default and
may undergo future API changes.
Problems with encoding/json
Overall, encoding/json has held up well.
The idea of marshaling and unmarshaling arbitrary Go types
with some default representation in JSON, combined with the ability to
customize the representation, has proven to be highly flexible.
However, in the years since its introduction,
various users have identified numerous shortcomings.
Behavior flaws
There are various behavioral flaws in encoding/json:
-
Imprecise handling of JSON syntax: Over the years, JSON has seen increased standardization in order for programs to properly communicate. Generally, decoders have become stricter at rejecting ambiguous inputs, to reduce the chance that two implementations will have different (successful) interpretations of a particular JSON value.
-
encoding/jsoncurrently accepts invalid UTF-8, whereas the latest Internet Standard (RFC 8259) requires valid UTF-8. The default behavior should report an error in the presence of invalid UTF-8, instead of introducing silent data corruption, which may cause problems downstream. -
encoding/jsoncurrently accepts objects with duplicate member names. RFC 8259 does not specify how to handle duplicate names, so an implementation is free to choose an arbitrary value, merge the values, discard the values, or report an error. The presence of a duplicate name results in a JSON value without a universally agreed upon meaning. This could be exploited by attackers in security applications and has been exploited before (as in CVE-2017-12635). The default behavior should err on the side of safety and reject duplicate names.
-
-
Leaking nilness of slices and maps: JSON is often used to communicate with programs using JSON implementations that do not allow
nullto be unmarshaled into a data type expected to be a JSON array or object. Sinceencoding/jsonmarshals a nil slice or map as a JSONnull, this may lead to errors when unmarshaling by other implementations. A survey indicated that most Go users prefer that nil slices and maps are marshaled as an empty JSON array or object by default. -
Case-insensitive unmarshaling: When unmarshaling, a JSON object member name is resolved to a Go struct field name using a case-insensitive match. This is a surprising default, a potential security vulnerability, and a performance limitation.
-
Inconsistent calling of methods: Due to an implementation detail,
MarshalJSONmethods declared on a pointer receiver are inconsistently called byencoding/json. While regarded as a bug, this cannot be fixed as too many applications depend on the current behavior.
API deficiencies
The API of encoding/json can be tricky or restrictive:
-
It is difficult to correctly unmarshal from an
io.Reader. Users often writejson.NewDecoder(r).Decode(v), which fails to reject trailing junk at the end of the input. -
Options can be set on the
EncoderandDecodertypes, but cannot be used with theMarshalandUnmarshalfunctions. Similarly, types implementing theMarshalerandUnmarshalerinterfaces cannot make use of the options and there is no way to plumb options down the call stack. For example, theDecoder.DisallowUnknownFieldsoption loses its effect when calling a customUnmarshalJSONmethod. -
The
Compact,Indent, andHTMLEscapefunctions write to abytes.Bufferinstead of something more flexible like a[]byteorio.Writer. This limits the usability of those functions.
Performance limitations
Setting aside internal implementation details, the public API commits it to certain performance limitations:
-
MarshalJSON: The
MarshalJSONinterface method forces the implementation to allocate the returned[]byte. Also, the semantics require thatencoding/jsonverify that the result is valid JSON and also to reformat it to match the specified indentation. -
UnmarshalJSON: The
UnmarshalJSONinterface method requires that a complete JSON value be provided (without any trailing data). This forcesencoding/jsonto parse the JSON value to be unmarshaled in its entirety to determine where it ends before it can callUnmarshalJSON. Afterwards, theUnmarshalJSONmethod itself must parse the provided JSON value again. -
Lack of streaming: Even though the
EncoderandDecodertypes operate on anio.Writerorio.Reader, they buffer the entire JSON value in memory. TheDecoder.Tokenmethod for reading individual tokens is allocation-heavy and there is no corresponding API for writing tokens.
Furthermore, if the implementation of a MarshalJSON or UnmarshalJSON method
recursively calls the Marshal or Unmarshal function,
then the performance becomes quadratic.
Trying to fix encoding/json directly
Introducing a new, incompatible major version of a package is a heavy consideration. If possible, we should try to fix the existing package.
While it is relatively easy to add new features, it is difficult to change existing features. Unfortunately, these problems are inherent consequences of the existing API, making them practically impossible to fix within the Go 1 compatibility promise.
We could in principle declare separate names, such as MarshalV2 or UnmarshalV2,
but that is tantamount to creating a parallel namespace within the same package.
This leads us to encoding/json/v2 (henceforth called v2),
where we can make these changes within a separate v2 namespace
in contrast to encoding/json (henceforth called v1).
Planning for encoding/json/v2
The planning for a new major version of encoding/json spanned years.
In late 2020, spurred on by the inability to fix issues in the current package,
Daniel Martí (one of the maintainers of encoding/json) first drafted his
thoughts on what a hypothetical v2 package should look like.
Separately, after previous work on the Go API for Protocol Buffers,
Joe Tsai was disapppointed that the protojson package
needed to use a custom JSON implementation because encoding/json was
neither capable of adhering to the stricter JSON standard that the
Protocol Buffer specification required,
nor of efficiently serializing JSON in a streaming manner.
Believing a brighter future for JSON was both beneficial and achievable,
Daniel and Joe joined forces to brainstorm on v2 and
started to build a prototype
(with the initial code being a polished version of the JSON serialization logic from the Go protobuf module).
Over time, a few others (Roger Peppe, Chris Hines, Johan Brandhorst-Satzkorn, and Damien Neil)
joined the effort by providing design review, code review, and regression testing.
Many of the early discussions are publicly available in our
recorded meetings and
meeting notes.
This work has been public since the beginning,
and we increasingly involved the wider Go community,
first with a
GopherCon talk and
discussion posted in late 2023,
formal proposal posted in early 2025,
and most recently adopting encoding/json/v2 as a Go experiment
(available in Go 1.25) for wider-scale testing by all Go users.
The v2 effort has been going on for 5 years,
incorporating feedback from many contributors and also gaining valuable
empirical experience from use in production settings.
It’s worth noting that it’s largely been developed and promoted by people not employed by Google, demonstrating that the Go project is a collaborative endeavor with a thriving global community dedicated to improving the Go ecosystem.
Building on encoding/json/jsontext
Before discussing the v2 API, we first introduce the experimental
encoding/json/jsontext package
that lays the foundation for future improvements to JSON in Go.
JSON serialization in Go can be broken down into two primary components:
- syntactic functionality that is concerned with processing JSON based on its grammar, and
- semantic functionality that defines the relationship between JSON values and Go values.
We use the terms “encode” and “decode” to describe syntactic functionality and the terms “marshal” and “unmarshal” to describe semantic functionality. We aim to provide a clear distinction between functionality that is purely concerned with encoding versus that of marshaling.
This diagram provides an overview of this separation.
Purple blocks represent types, while blue blocks represent functions or methods.
The direction of the arrows approximately represents the flow of data.
The bottom half of the diagram, implemented by the jsontext package,
contains functionality that is only concerned with syntax,
while the upper half, implemented by the json/v2 package,
contains functionality that assigns semantic meaning to syntactic data
handled by the bottom half.
The basic API of jsontext is the following:
package jsontext
type Encoder struct { ... }
func NewEncoder(io.Writer, ...Options) *Encoder
func (*Encoder) WriteValue(Value) error
func (*Encoder) WriteToken(Token) error
type Decoder struct { ... }
func NewDecoder(io.Reader, ...Options) *Decoder
func (*Decoder) ReadValue() (Value, error)
func (*Decoder) ReadToken() (Token, error)
type Kind byte
type Value []byte
func (Value) Kind() Kind
type Token struct { ... }
func (Token) Kind() Kind
The jsontext package provides functionality for interacting with JSON
at a syntactic level and derives its name from
RFC 8259, section 2
where the grammar for JSON data is literally called JSON-text.
Since it only interacts with JSON at a syntactic level,
it does not depend on Go reflection.
The Encoder and
Decoder
provide support for encoding and decoding JSON values and tokens.
The constructors
accept variadic options
that affect the particular behavior of encoding and decoding.
Unlike the Encoder and Decoder types declared in v1,
the new types in jsontext avoid muddling the distinction between syntax and
semantics and operate in a truly streaming manner.
A JSON value is a complete unit of data and is represented in Go as
a named []byte.
It is identical to RawMessage in v1.
A JSON value is syntactically composed of one or more JSON tokens.
A JSON token is represented in Go as the opaque Token type
with constructors and accessor methods.
It is analogous to Token in v1
but is designed represent arbitrary JSON tokens without allocation.
To resolve the fundamental performance problems with
the MarshalJSON and UnmarshalJSON interface methods,
we need an efficient way of encoding and decoding JSON
as a streaming sequence of tokens and values.
In v2, we introduce the MarshalJSONTo and UnmarshalJSONFrom interface methods
that operate on an Encoder or Decoder, allowing the methods’ implementations
to process JSON in a purely streaming manner. Thus, the json package need not
be responsible for validating or formatting a JSON value returned by MarshalJSON,
nor would it need to be responsible for determining the boundaries of a JSON value
provided to UnmarshalJSON. These responsibilities belong to the Encoder and Decoder.
Introducing encoding/json/v2
Building on the jsontext package, we now introduce the experimental
encoding/json/v2 package.
It is designed to fix the aforementioned problems,
while remaining familiar to users of the v1 package.
Our goal is that usages of v1 will operate mostly the same if directly migrated to v2.
In this article, we will primarily cover the high-level API of v2.
For examples on how to use it, we encourage readers to
study the examples in the v2 package or
read Anton Zhiyanov’s blog covering the topic.
The basic API of v2 is the following:
package json
func Marshal(in any, opts ...Options) (out []byte, err error)
func MarshalWrite(out io.Writer, in any, opts ...Options) error
func MarshalEncode(out *jsontext.Encoder, in any, opts ...Options) error
func Unmarshal(in []byte, out any, opts ...Options) error
func UnmarshalRead(in io.Reader, out any, opts ...Options) error
func UnmarshalDecode(in *jsontext.Decoder, out any, opts ...Options) error
The Marshal
and Unmarshal functions
have a signature similar to v1, but accept options to configure their behavior.
The MarshalWrite
and UnmarshalRead functions
directly operate on an io.Writer or io.Reader,
avoiding the need to temporarily construct an Encoder or Decoder
just to write or read from such types.
The MarshalEncode
and UnmarshalDecode functions
operate on a jsontext.Encoder and jsontext.Decoder and
is actually the underlying implementation of the previously mentioned functions.
Unlike v1, options are a first-class argument to each of the marshal and unmarshal functions,
greatly extending the flexibility and configurability of v2.
There are several options available
in v2 which are not covered by this article.
Type-specified customization
Similar to v1, v2 allows types to define their own JSON representation
by satisfying particular interfaces.
type Marshaler interface {
MarshalJSON() ([]byte, error)
}
type MarshalerTo interface {
MarshalJSONTo(*jsontext.Encoder) error
}
type Unmarshaler interface {
UnmarshalJSON([]byte) error
}
type UnmarshalerFrom interface {
UnmarshalJSONFrom(*jsontext.Decoder) error
}
The Marshaler
and Unmarshaler interfaces
are identical to those in v1.
The new MarshalerTo
and UnmarshalerFrom interfaces
allow a type to represent itself as JSON using a jsontext.Encoder or jsontext.Decoder.
This allows options to be forwarded down the call stack, since options
can be retrieved via the Options accessor method on the Encoder or Decoder.
See the OrderedObject example
for how to implement a custom type that maintains the ordering of JSON object members.
Caller-specified customization
In v2, the caller of Marshal and Unmarshal can also specify
a custom JSON representation for any arbitrary type,
where caller-specified functions take precedence over type-defined methods
or the default representation for a particular type.
func WithMarshalers(*Marshalers) Options
type Marshalers struct { ... }
func MarshalFunc[T any](fn func(T) ([]byte, error)) *Marshalers
func MarshalToFunc[T any](fn func(*jsontext.Encoder, T) error) *Marshalers
func WithUnmarshalers(*Unmarshalers) Options
type Unmarshalers struct { ... }
func UnmarshalFunc[T any](fn func([]byte, T) error) *Unmarshalers
func UnmarshalFromFunc[T any](fn func(*jsontext.Decoder, T) error) *Unmarshalers
MarshalFunc and
MarshalToFunc
construct a custom marshaler that can be passed to a Marshal call
using WithMarshalers to override the marshaling of particular types.
Similarly,
UnmarshalFunc and
UnmarshalFromFunc
support similar customization for Unmarshal.
The ProtoJSON example
demonstrates how this feature allows serialization of all
proto.Message types
to be handled by the protojson package.
Behavior differences
While v2 aims to behave mostly the same as v1,
its behavior has changed in some ways
to address problems in v1, most notably:
-
v2reports an error in the presence of invalid UTF-8. -
v2reports an error if a JSON object contains a duplicate name. -
v2marshals a nil Go slice or Go map as an empty JSON array or JSON object, respectively. -
v2unmarshals a JSON object into a Go struct using a case-sensitive match from the JSON member name to the Go field name. -
v2redefines theomitemptytag option to omit a field if it would have encoded as an “empty” JSON value (which arenull,"",[], and{}). -
v2reports an error when trying to serialize atime.Duration, which currently has no default representation, but provides options to let the caller decide.
For most behavior changes, there is a struct tag option or caller-specified option
that can configure the behavior to operate under v1 or v2 semantics,
or even other caller-determined behavior.
See “Migrating to v2” for more information.
Performance optimizations
The Marshal performance of v2 is roughly at parity with v1.
Sometimes it is slightly faster, but other times it is slightly slower.
The Unmarshal performance of v2 is significantly faster than v1,
with benchmarks demonstrating improvements of up to 10x.
In order to obtain greater performance gains,
existing implementations of
Marshaler and
Unmarshaler should
migrate to also implement
MarshalerTo and
UnmarshalerFrom,
so that they can benefit from processing JSON in a purely streaming manner.
For example, recursive parsing of OpenAPI specifications in UnmarshalJSON methods
significantly hurt performance in a particular service of Kubernetes
(see kubernetes/kube-openapi#315),
while switching to UnmarshalJSONFrom improved performance by orders of magnitude.
For more information, see the
go-json-experiment/jsonbench
repository.
Retroactively improving encoding/json
We want to avoid two separate JSON implementations in the Go standard library,
so it is critical that, under the hood, v1 is implemented in terms of v2.
There are several benefits to this approach:
-
Gradual migration: The
MarshalandUnmarshalfunctions inv1orv2represent a set of default behaviors that operate according tov1orv2semantics. Options can be specified that configureMarshalorUnmarshalto operate with entirelyv1, mostlyv1with a somev2, a mix ofv1orv2, mostlyv2with somev1, or entirelyv2semantics. This allows for gradual migration between the default behaviors of the two versions. -
Feature inheritance: As backward-compatible features are added to
v2, they will inherently be made available inv1. For example,v2adds support for several new struct tag options such asinlineorformatand also support for theMarshalJSONToandUnmarshalJSONFrominterface methods, which are both more performant and flexible. Whenv1is implemented in terms ofv2, it will inherit support for these features. -
Reduced maintenance: Maintenance of a widely used package demands significant effort. By having
v1andv2use the same implementation, the maintenance burden is reduced. In general, a single change will fix bugs, improve performance, or add functionality to both versions. There is no need to backport av2change with an equivalentv1change.
While select parts of v1 may be deprecated over time (supposing v2 graduates from being an experiment),
the package as a whole will never be deprecated.
Migrating to v2 will be encouraged, but not required.
The Go project will not drop support for v1.
Experimenting with jsonv2
The newer API in the encoding/json/jsontext and encoding/json/v2 packages are not visible by default.
To use them, build your code with GOEXPERIMENT=jsonv2 set in your environment or with the goexperiment.jsonv2 build tag.
The nature of an experiment is that the API is unstable and may change in the future.
Though the API is unstable, the implementation is of a high quality and
has been successfully used in production by several major projects.
The fact that v1 is implemented in terms of v2 means that the underlying implementation of v1
is completely different when building under the jsonv2 experiment.
Without changing any code, you should be able to run your tests
under jsonv2 and theoretically nothing new should fail:
GOEXPERIMENT=jsonv2 go test ./...
The re-implementation of v1 in terms of v2 aims to provide identical behavior
within the bounds of the Go 1 compatibility promise,
though some differences might be observable such as the exact wording of error messages.
We encourage you to run your tests under jsonv2 and
report any regressions on the issue tracker.
Becoming an experiment in Go 1.25 is a significant milestone on the road to
formally adopting encoding/json/jsontext and encoding/json/v2 into the standard library.
However, the purpose of the jsonv2 experiment is to gain broader experience.
Your feedback will determine our next steps, and the outcome of this experiment,
which may result in anything from abandonment of the effort, to adoption as stable packages of Go 1.26.
Please share your experience on go.dev/issue/71497, and help determine the future of Go.
Next article: It's survey time! How has Go has been working out for you?
Previous article: Testing Time (and other asynchronicities)
Blog Index
