The performance of these commonly used official Go libraries is not as good as that of the third-party open source libraries

The official library of the Go language has always given us the impression that it is powerful and easy to use. However, no gold is enough, and no one is perfect. In some performance-critical scenarios such as json parsing, the performance of some official libraries is not satisfactory.

This article introduces several open source libraries that are benchmarked against official libraries, and their performance far exceeds the former. When readers encounter performance problems using these official libraries, they can try to replace optimizations.

net/http -> fasthttp

Address: https://github.com/valyala/fasthttp

fasthttpso-called rationet/httpIt is ten times faster, and the core idea of ​​its optimization is very simple: resource reuse.

  • Reuse goroutines to reduce runtime scheduling pressure;

  • Object reuse, a lot of use sync.Pool to reduce GC pressure.

In addition to multiplexing, there are other optimization methods, such as avoiding the conversion overhead of string and []byte as much as possible.

These optimization techniques and best practices have been intimately given on its Github homepage: https://github.com/valyala/fasthttp#fasthttp-best-practices , it is recommended to read and learn.

becausefasthttpThere is a big gap between the implementation of and the standard library, so it is the same asnet/httpThe API interface is different, which leads fromnet/httprefactored tofasthttpSome learning costs are required.

usefasthttpWell-known projects: Fiber, Gearbox, atreugo, etc.

encoding/json -> jsoniter

Address: https://github.com/json-iterator/go

jsoniter(json-iterator) is a fast and flexible JSON parser, available in both Java and Go. Officially, the Golang version is better than the standard library (encoding/json) is as much as 6 times faster.

65e76b8b1665318d0c060c537d838222.png

Most importantly, it is compatible with the standard libraryencoding/jsonFully compatible.

  • Marshal()

  1. # encoding/json 
  2. import "encoding/json"
  3. json.Marshal(&data)
  4. # jsoniter
  5. import jsoniter "github.com/json-iterator/go"
  6. var json = jsoniter.ConfigCompatibleWithStandardLibrary
  7. json.Marshal(&data)
  • Unmarshal()

  1. # encoding/json
  2. import "encoding/json"
  3. json.Unmarshal(input, &data)
  4. # jsoniter
  5. import jsoniter "github.com/json-iterator/go"
  6. var json = jsoniter.ConfigCompatibleWithStandardLibrary
  7. json.Unmarshal(input, &data)

Readers interested in its optimization principle can look here: http://jsoniter.com/benchmark.html#optimization-used

golang/protobuf -> gogo/protobuf

Address: https://github.com/gogo/protobuf

The full name of ProtoBuf is Protocol Buffers, which is a protocol format similar to XML and JSON developed and defined by Google for efficient storage and reading of structured data. It is binary based, so using ProtoBuf can compress the data smaller.

gogo/protobufis based on the official librarygolang/protobufEnhanced implementation of:

  • Comparegolang/protobufFaster serialization and deserialization;

  • More canonical Go structure;

  • compatiblegolang/protobuf;

  • Optionally generate additional help code, reducing code typing;

  • Can generate test code and benchmark code;

  • other serialization formats;

There are many well-known projects using this library, such as etcd, k8s, docker swarmkit, tidb, nakama, etc.

html / template -> valyala / quicktemplate

Address: https://github.com/valyala/quicktemplate

quicktemplateInspired by Python, the Mako project is a fast, powerful and easy-to-use template rendering engine for Go. Its main features are as follows:

  • quicktemplateThe template code written will be converted into Go language code first, and then compiled and rendered. Therefore, it is better than the standard libraryhtml/template20 times faster.

  • quicktemplateThe syntax is very similar to the Go syntax, with almost no learning cost.

  • Almost all bugs can be caught at template compilation time, so in real projects, there are very few template-related bugs affected.

  • Arbitrary Go code can be embedded in templates.

AlthoughquicktemplateThe main purpose is to generate HTML, but it can also be used to generate other data.

For example, usequicktemplateJSON and XML serialization can be easily achieved, and viaquicktemplateSerialization ofencoding/jsonandencoding/xmlfaster.

Summarize

The list of these libraries in this article does not make you immediately replace the official library. E.gnet/httppackage, in fact, it can already meet most usage scenarios.

We encountered a problem when using the official library, we can easily find the solution through the search engine, or file an issue directly to the Go official. When switching to an open source library, if you encounter a problem, it may not be solved in time.

The API of the official library is almost guaranteed to be compatible with the iteration of the Go version, and the third-party library may have potential version compatibility problems, which is also a problem that needs to be considered when switching.

Several open source libraries listed in this article focus on optimizing the performance of the corresponding official libraries. We can learn a lot of practical Go code optimization techniques from these open source libraries.

Of course, if you have performance issues with these official libraries in your project, give it a try.

Thank you for your likes and watching ~

a87cfe6d39f54dfe167ab7d55bcfc269.gif

Related: The performance of these commonly used official Go libraries is not as good as that of the third-party open source libraries