Protobuf Name Conflict 分析与解决

问题背景

最近，在对老项目进行重构工作。在重构过程中发现需要通过 grpc 调用若干远端微服务，远端微服务都有提前定义好的 proto，在运行时（编译通过，运行则panic）出现了 name conflict 冲突问题。也就是说，在运行时报错提示存在相同名称的 message 消息体。

具体报错信息，如下所示：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
➜  app > go mod tidy      
➜  app > go build main.go 
➜  app > ./main api start

panic: proto: file "usermgt.proto" has a name conflict over pb.Response
        previously from: "eslServer/pkg/microgrpc/merchant/pb"
        currently from:  "eslServer/pkg/microgrpc/usermgt/pb"
See https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

goroutine 1 [running]:
google.golang.org/protobuf/reflect/protoregistry.glob..func1({0x1fcfb20?, 0xc00046c350?}, {0x1fcfb20?, 0xc00046c390})
        /Users/ivanli/go/pkg/mod/google.golang.org/protobuf@v1.27.1/reflect/protoregistry/registry.go:54 +0x1ee
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile.func1({0x1fdca28, 0xc0001f51c0})
        /Users/ivanli/go/pkg/mod/google.golang.org/protobuf@v1.27.1/reflect/protoregistry/registry.go:152 +0x279
google.golang.org/protobuf/reflect/protoregistry.rangeTopLevelDescriptors({0x1fdfa10, 0xc0005ae000}, 0xc0005453b8)
        /Users/ivanli/go/pkg/mod/google.golang.org/protobuf@v1.27.1/reflect/protoregistry/registry.go:415 +0x174
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc000594048, {0x1fdfa10?, 0xc0005ae000?})
        /Users/ivanli/go/pkg/mod/google.golang.org/protobuf@v1.27.1/reflect/protoregistry/registry.go:147 +0x739
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0x1bcc89c, 0x22}, {0x284a380, 0x6ef, 0x6ef}, 0x0, 0xd, 0x0, 0x1, {0x1fd3e50, ...}, ...})
        /Users/ivanli/go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/filedesc/build.go:113 +0x1d6
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x1bcc89c, 0x22}, {0x284a380, 0x6ef, 0x6ef}, 0x0, 0xd, 0x0, 0x1, {0x0, ...}, ...}, ...})
        /Users/ivanli/go/pkg/mod/google.golang.org/protobuf@v1.27.1/internal/filetype/build.go:139 +0x19d
eslServer/pkg/microgrpc/usermgt/pb.file_usermgt_proto_init()
        /Users/ivanli/code/xxxx/esl-server/app/pkg/microgrpc/usermgt/pb/usermgt.pb.go:1179 +0x19a
eslServer/pkg/microgrpc/usermgt/pb.init.0()
        /Users/ivanli/code/xxxx/esl-server/app/pkg/microgrpc/usermgt/pb/usermgt.pb.go:1003 +0x17

微服务merchant与微服务usermgt的proto中定义的 package 名称都叫 pb，并且都存在一个同名的 Response message 消息体。由于是老的项目，并且已经按照现有 proto 定义的格式与接口对外服务了很长时间。所以，这个时候再修改包名称、message名称则不可行。

分析了一下，既然老的项目可以正常运行，则意味着可能是由于所依赖的的某些包的版本不同导致以前的项目可运行，现在重构时却出现panic。

由于项目久远，没有文档，没有注释，没人交接。对于这种三无项目，只能靠自己摸索。

分析过程

通过报错信息显示的调用栈可以看出，最终是在 /Users/ivanli/go/pkg/mod/google.golang.org/protobuf@v1.27.1/reflect/protoregistry/registry.go:54 这行代码处抛出的panic，其源码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// ignoreConflict reports whether to ignore a registration conflict
// given the descriptor being registered and the error.
// It is a variable so that the behavior is easily overridden in another file.
var ignoreConflict = func(d protoreflect.Descriptor, err error) bool {
	const env = "GOLANG_PROTOBUF_REGISTRATION_CONFLICT"
	const faq = "https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict"
	policy := conflictPolicy
	if v := os.Getenv(env); v != "" {
		policy = v
	}
	switch policy {
	case "panic":
		panic(fmt.Sprintf("%v\nSee %v\n", err, faq))  // 54行 就是这里
	case "warn":
		fmt.Fprintf(os.Stderr, "WARNING: %v\nSee %v\n\n", err, faq)
		return true
	case "ignore":
		return true
	default:
		panic("invalid " + env + " value: " + os.Getenv(env))
	}
}

找到源码之后，可以看出对于这种冲突其实存在几种处理策略：

  1. 如果存在环境变量 GOLANG_PROTOBUF_REGISTRATION_CONFLICT，则会根据环境变量具体值进行不同处理
  • 环境变量值为：panic，则会在运行时 panic
  • 环境变量值为：warn，则仅仅会输出一个提示信息
  • 环境变量值为：ignore，则会直接忽略（提示信息也没有）
  1. 默认使用 panic

同时也发现，上面逻辑是 google.golang.org/protobuf 包的 v1.27.1 版本。于是，就分别找出版本v1.26.0、v1.25.0 中对应的逻辑。

v1.26.0版本：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// ignoreConflict reports whether to ignore a registration conflict
// given the descriptor being registered and the error.
// It is a variable so that the behavior is easily overridden in another file.
var ignoreConflict = func(d protoreflect.Descriptor, err error) bool {
	const env = "GOLANG_PROTOBUF_REGISTRATION_CONFLICT"
	const faq = "https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict"
	policy := conflictPolicy
	if v := os.Getenv(env); v != "" {
		policy = v
	}
	switch policy {
	case "panic":
		panic(fmt.Sprintf("%v\nSee %v\n", err, faq))
	case "warn":
		fmt.Fprintf(os.Stderr, "WARNING: %v\nSee %v\n\n", err, faq)
		return true
	case "ignore":
		return true
	default:
		panic("invalid " + env + " value: " + os.Getenv(env))
	}
}

v1.25.0版本：

1
2
3
4
5
6
7
8
9
10
11
// ignoreConflict reports whether to ignore a registration conflict
// given the descriptor being registered and the error.
// It is a variable so that the behavior is easily overridden in another file.
var ignoreConflict = func(d protoreflect.Descriptor, err error) bool {
	log.Printf(""+
		"WARNING: %v\n"+
		"A future release will panic on registration conflicts. See:\n"+
		"https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict\n"+
		"\n", err)
	return true
}

可以明显的看出，在不同的版本中处理冲突的方式大有不同。在 v1.25.0 中还只是输出一段提示信息。在 v1.26.0以及之后的版本就会根据不同的策略进行处理。

最近几个版本的发布时间如下图所示：

https://github.com/protocolbuffers/protobuf-go/tags

针对此问题，官方也有一个说明文档：

https://developers.google.cn/protocol-buffers/docs/reference/go/faq
https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict

How do I fix a protocol buffer namespace conflict?

The way to best fix a namespace conflict depends on the reason why a conflict is occurring.
Common ways that namespace conflicts occur:

  • Vendored .proto files. When a single .proto file is generated into two or more Go packages and linked into the same Go binary, it conflicts on every protobuf declaration in the generated Go packages. This typically occurs when a .proto file is vendored and a Go package is generated from it, or the generated Go package itself is vendored. Users should avoid vendoring and instead depend on a centralized Go package for that .proto file.

    • If a .proto file is owned by an external party and is lacking a go_package option, then you should coordinate with the owner of that .proto file to specify a centralized Go package that a plurality of users can all depend on.

  • Missing or generic proto package names. If a .proto file does not specify a package name or uses an overly generic package name (for example, “my_service”), then there is a high probability that declarations within that file will conflict with other declarations elsewhere in the universe. We recommend that every .proto file have a package name that is deliberately chosen to be universally unique (for example, prefixed with the name of a company).

    • Warning: Retroactively changing the package name on a .proto file can potentially cause the use of extension fields or messages stored in google.protobuf.Any to stop working properly.

Starting with v1.26.0 of the google.golang.org/protobuf module, a hard error will be reported when a Go program starts up that has multiple conflicting protobuf names linked into it. While it is preferable that the source of the conflict be fixed, the fatal error can be immediately worked around in one of two ways:

  1. At compile time. The default behavior for handling conflicts can be specified at compile time with a linker-initialized variable: go build -ldflags "-X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn"

  2. At program execution. The behavior for handling conflicts when executing a particular Go binary can be set with an environment variable: GOLANG_PROTOBUF_REGISTRATION_CONFLICT=warn ./main

解决办法

通过上面分析可知，在不同版本的包中处理策略也不同。所以，可以通过设置环境变量、降低版本来解决。

  1. 设置环境变量
1
2
3
4
5
# 编译时
go build -ldflags "-X google.golang.org/protobuf/reflect/protoregistry.conflictPolicy=warn"

# 运行时
GOLANG_PROTOBUF_REGISTRATION_CONFLICT=warn ./main
  1. 降低版本

在 go.mod 中添加 replace 降低包的版本

1
2
3
replace (
	google.golang.org/protobuf => google.golang.org/protobuf v1.25.0
)

添加之后执行 go mod tidy，然后重新编译、运行。

目前，笔者这里使用的方式2，通过降低版本来解决问题。

扩展知识

关于 protobuf 的 github.com/golang/protobuf 与 google.golang.org/protobuf 区别

The github.com/golang/protobuf module is the original Go protocol buffer API.

The google.golang.org/protobuf module is an updated version of this API designed for simplicity, ease of use, and safety. The flagship features of the updated API are support for reflection and a separation of the user-facing API from the underlying implementation.

We recommend that you use google.golang.org/protobuf in new code.

Version v1.4.0 and higher of github.com/golang/protobuf wrap the new implementation and permit programs to adopt the new API incrementally. For example, the well-known types defined in github.com/golang/protobuf/ptypes are simply aliases of those defined in the newer module. Thus, google.golang.org/protobuf/types/known/emptypb and github.com/golang/protobuf/ptypes/empty may be used interchangeably.

github.com/golang/protobuf 模块是原始的Go协议缓冲区API。
google.golang.org/protobuf 模块是此API的更新版本，其设计宗旨是简单、易于使用和安全。更新后的API的主要特性是支持反射，并将面向用户的API与底层实现分离。

建议在新代码中使用 google.golang.org/protobuf

github.com/golang/protobuf 的v1.4.0及更高版本包装了新的实现，并允许程序以增量方式采用新的API。例如，在 github.com/golang/protobuf/ptypes 中定义的知名类型只是在新模块中定义的那些类型的别名。因此，google.golang.org/protobuf/types/known/emptypb 和github.com/golang/protobuf/ptypes/empty 可以互换使用。