plugin: build path contributes to types not equal

[zhangxu19830126]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.8

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/opt/lib/go-path"
GORACE=""
GOROOT="/opt/lib/go"
GOTOOLDIR="/opt/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build171911883=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

I want to use custom interface based on go plugin, but i found it's not support.

func newExternalFilter(filterSpec *conf.FilterSpec) (filter.Filter, error) {
        p, err := plugin.Open(filterSpec.ExternalPluginFile)
        if err != nil {
            return nil, err
        }

        s, err := p.Lookup("NewExternalFilter")
        if err != nil {
            return nil, err
        }

        sf := s.(func() (filter.Filter, error))
        return sf()
    }

    // NewExternalFilter create a External filter
    func NewExternalFilter() (filter.Filter, error) {
        return &AccessFilter{}, nil
    }

What did you expect to see?

I expect this operation will success.

What did you see instead?

panic: interface conversion: plugin.Symbol is func() (filter.Filter, error), not func() (filter.Filter, error)

[crawshaw]

That's an interesting failure. Can you provide a functioning reproduction? (In particular, including the definition of filter.Filter.)

[zhangxu19830126]

Thanks for your reply, :-)

definition of filter.Filter

package filter

import (
	"net/http"

	"github.com/valyala/fasthttp"
)

// Context filter context
type Context interface {
	SetStartAt(startAt int64)
	SetEndAt(endAt int64)
	GetStartAt() int64
	GetEndAt() int64

	GetProxyServerAddr() string
	GetProxyOuterRequest() *fasthttp.Request
	GetProxyResponse() *fasthttp.Response
	NeedMerge() bool

	GetOriginRequestCtx() *fasthttp.RequestCtx

	GetMaxQPS() int

	ValidateProxyOuterRequest() bool

	InBlacklist(ip string) bool
	InWhitelist(ip string) bool

	IsCircuitOpen() bool
	IsCircuitHalf() bool

	GetOpenToCloseFailureRate() int
	GetHalfTrafficRate() int
	GetHalfToOpenSucceedRate() int
	GetOpenToCloseCollectSeconds() int

	ChangeCircuitStatusToClose()
	ChangeCircuitStatusToOpen()

	RecordMetricsForRequest()
	RecordMetricsForResponse()
	RecordMetricsForFailure()
	RecordMetricsForReject()

	GetRecentlyRequestSuccessedCount(sec int) int
	GetRecentlyRequestCount(sec int) int
	GetRecentlyRequestFailureCount(sec int) int
}

// Filter filter interface
type Filter interface {
	Name() string

	Pre(c Context) (statusCode int, err error)
	Post(c Context) (statusCode int, err error)
	PostErr(c Context)
}

// BaseFilter base filter support default implemention
type BaseFilter struct{}

// Pre execute before proxy
func (f BaseFilter) Pre(c Context) (statusCode int, err error) {
	return http.StatusOK, nil
}

// Post execute after proxy
func (f BaseFilter) Post(c Context) (statusCode int, err error) {
	return http.StatusOK, nil
}

// PostErr execute proxy has errors
func (f BaseFilter) PostErr(c Context) {

}

plugin go file

package main

import (
	"C"
	"strings"
	"time"

	"github.com/CodisLabs/codis/pkg/utils/log"
	"github.com/fagongzi/gateway/pkg/filter"
	"github.com/valyala/fasthttp"
)

// AccessFilter record the http access log
// log format: $remoteip "$method $path" $code "$agent" $svr $cost
type AccessFilter struct {
}

// NewExternalFilter create a External filter
func NewExternalFilter() (filter.Filter, error) {
	return &AccessFilter{}, nil
}

// Name return name of this filter
func (f *AccessFilter) Name() string {
	return "HTTP-ACCESS"
}

// Pre pre process
func (f *AccessFilter) Pre(c filter.Context) (statusCode int, err error) {
	return 200, nil
}

// Post execute after proxy
func (f *AccessFilter) Post(c filter.Context) (statusCode int, err error) {
	cost := (c.GetStartAt() - c.GetEndAt())

	log.Infof("%s %s \"%s\" %d \"%s\" %s %s",
		GetRealClientIP(c.GetOriginRequestCtx()),
		c.GetOriginRequestCtx().Method(),
		c.GetProxyOuterRequest().RequestURI(),
		c.GetProxyResponse().StatusCode(),
		c.GetOriginRequestCtx().UserAgent(),
		c.GetProxyServerAddr(),
		time.Duration(cost))

	return 200, nil
}

// PostErr post error process
func (f *AccessFilter) PostErr(c filter.Context) {

}

// GetRealClientIP get read client ip
func GetRealClientIP(ctx *fasthttp.RequestCtx) string {
	xforward := ctx.Request.Header.Peek("X-Forwarded-For")
	if nil == xforward {
		return strings.SplitN(ctx.RemoteAddr().String(), ":", 2)[0]
	}

	return strings.SplitN(string(xforward), ",", 2)[0]
}

load plugin file

package proxy

import (
	"errors"
	"plugin"
	"strings"

	"github.com/fagongzi/gateway/pkg/conf"
	"github.com/fagongzi/gateway/pkg/filter"
)

var (
	// ErrKnownFilter known filter error
	ErrKnownFilter = errors.New("unknow filter")
)

const (
	// FilterHTTPAccess access log filter
	FilterHTTPAccess = "HTTP-ACCESS"
	// FilterHeader header filter
	FilterHeader = "HEAD" // process header fiter
	// FilterXForward xforward fiter
	FilterXForward = "XFORWARD"
	// FilterBlackList blacklist filter
	FilterBlackList = "BLACKLIST"
	// FilterWhiteList whitelist filter
	FilterWhiteList = "WHITELIST"
	// FilterAnalysis analysis filter
	FilterAnalysis = "ANALYSIS"
	// FilterRateLimiting limit filter
	FilterRateLimiting = "RATE-LIMITING"
	// FilterCircuitBreake circuit breake filter
	FilterCircuitBreake = "CIRCUIT-BREAKE"
	// FilterValidation validation request filter
	FilterValidation = "VALIDATION"
)

func newFilter(filterSpec *conf.FilterSpec) (filter.Filter, error) {
	if filterSpec.External {
		return newExternalFilter(filterSpec)
	}

	input := strings.ToUpper(filterSpec.Name)

	switch input {
	case FilterHTTPAccess:
		return newAccessFilter(), nil
	case FilterHeader:
		return newHeadersFilter(), nil
	case FilterXForward:
		return newXForwardForFilter(), nil
	case FilterAnalysis:
		return newAnalysisFilter(), nil
	case FilterBlackList:
		return newBlackListFilter(), nil
	case FilterWhiteList:
		return newWhiteListFilter(), nil
	case FilterRateLimiting:
		return newRateLimitingFilter(), nil
	case FilterCircuitBreake:
		return newCircuitBreakeFilter(), nil
	case FilterValidation:
		return newValidationFilter(), nil
	default:
		return nil, ErrKnownFilter
	}
}

func newExternalFilter(filterSpec *conf.FilterSpec) (filter.Filter, error) {
	p, err := plugin.Open(filterSpec.ExternalPluginFile)
	if err != nil {
		return nil, err
	}

	s, err := p.Lookup("NewExternalFilter")
	if err != nil {
		return nil, err
	}

	sf := s.(func() (filter.Filter, error))
	return sf()
}

[zhangxu19830126]

You can find code in this project https://github.com/fagongzi/gateway/tree/go18-plugin-support.

1. filter.Filter is in pkg/filter package.
2. load plugin file in proxy/factory.go
3. plugin go file is in another project.

[edaniels]

Wouldn't this be similar to #18827? Technically the filter package lives at two distinct paths (gateway project and other project containing plugin) and go doesn't necessarily know the difference. This problem exists for non plugin code where two libraries try to interact with each other using the same type from two different versions of a package (at different paths but equivalent contents). See #12432

[zhangxu19830126]

Yes, @edaniels . You are right. Thanks very much

[sparrc]

I am having this same issue, though it manifests a little differently.

We have a project: github.com/influxdata/telegraf, which is meant to run our plugins. Each plugin has one dependency: github.com/influxdata/telegraf

This means that the plugins do not work unless they are built in the exact same GOPATH as the telegraf binary was built in.

ie, if I built the telegraf binary on my workstation from this GOPATH:

/home/sparrc/go/src/github.com/influxdata/telegraf

then build the plugins from a slightly different GOPATH (different home directory) on our CI system:

# telegraf dependency in /home/ubuntu/go/src/github.com/influxdata/telegraf
/home/ubuntu/go/src/github.com/influxdata/telegraf-plugins

I will get the following error:

2017/03/09 12:44:46 error loading [main/main.so]: plugin.Open: plugin was built with a different version of package github.com/influxdata/telegraf

This despite the fact that both telegraf and our plugins were built with the exact same commit hash of the github.com/influxdata/telegraf project. The only difference is the directory that github.com/influxdata/telegraf was located in at build time.

[easeway]

The cause is the logic in func genhash (src/cmd/link/internal/ld/lib.go) is problematic: it simply calculate SHA1 of first line of __.PKGDEF plus the export data (between first \n$$ and second \n$$), however, the export data contains strings of full source paths. If a package is shared by the host executable and plugin, but they are built in different environment with different source paths, without change of any code, the hash calculated of this shared package is different in host executable and plugin. That causes the error plugin.Open: plugin was built with a different version of package ...

[crawshaw]

I believe this was fixed by http://golang.org/cl/63693. Please comment if you still see this at HEAD. (See also, #21373.)

[alperkose]

Late note: I found that the issue is addressed at #18827

A similar issue exists for builds with vendoring support. I've created a sample repository which contains the examples and my observations: https://github.com/alperkose/golangplugins

I've run the example with go version go 1.10.2 darwin/amd64

The issue goes away if vendor folder is not used.

[juhwany]

@crawshaw

This issue is reproducible in Go 1.11 with simple plugin usage scenario like the below.

Directory structure

- gos (not GOAPTH)
       \_ sample_plugin (plugin module)
          \_ main.go
          \_ go.mod
       \_ server (server module, main program)
           \_ main.go
           \_ apis
                  \_ do.go

1. go.mod for plugin

module github.com/xxx/sample_plugin

require (
	github.com/xxx/server v2.0.0
)

2. plugin source (sample.go)

package main

import (
	"github.com/xxx/server/apis"
)

func Register(t apis.Test) {
       t.DoSomething()
...
}
...

3. server source (do.go)

package apis 

type Test interface { // interface api which server exposes
        DoSomething()
}

2. build plugin with go build -buildmode=plugin (not in GOPATH)
3. build main program (of course, after checking out git tag v2.0.0, and not in GOPATH)
4. run main program and when it try to load plugin, it failed with plugin was built with a different version of package github.com/x/server/apismessage

There's only one common dependency package(github.com/xxx/server/apis) between plugin and main program and the package belongs to server module as you can see the above directory structure.
I could not understand why it fails even though the referenced package version is same.