proxy.golang.org, cmd/go: 403 Forbidden on go get #71094

EliRibble · 2025-01-02T13:10:16Z

Go version

go version go1.23.4 linux/amd64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/eliribble/.cache/go-build'
GOENV='/home/eliribble/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/eliribble/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/eliribble/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/nix/store/jfv85qbj4vb1dafcg6kncg4vrbq2bbxv-go-1.23.4/share/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/nix/store/jfv85qbj4vb1dafcg6kncg4vrbq2bbxv-go-1.23.4/share/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.4'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/eliribble/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/eliribble/src/sovr.cloud/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/nix-shell-107044-0/go-build3527668268=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I'm working on NixOS, building my own flake of my own software. Ran nix develop which does a bunch of Nix stuff you don't care about. Within that command is a go get. The command worked fine on my laptop, fails on a cloud VPS with a 403 Forbidden.

What did you see happen?

$ nix develop
error: builder for '/nix/store/kiyqr9llfcxxjnri7n3ifh4ar71mbxwj-sovr-server-1.0.0-go-modules.drv' failed with exit code 1;
       last 25 log lines:
       > go: downloading google.golang.org/protobuf v1.34.1
       > go: downloading gopkg.in/yaml.v3 v3.0.1
       > go: downloading github.com/bytedance/sonic v1.11.6
       > go: downloading github.com/goccy/go-json v0.10.2
       > go: downloading github.com/json-iterator/go v1.1.12
       > go: downloading golang.org/x/sys v0.23.0
       > go: downloading github.com/x448/float16 v0.8.4
       > go: downloading github.com/gabriel-vasile/mimetype v1.4.3
       > go: downloading github.com/go-playground/universal-translator v0.18.1
       > go: downloading github.com/leodido/go-urn v1.4.0
       > go: downloading golang.org/x/text v0.17.0
       > go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
       > go: downloading github.com/modern-go/reflect2 v1.0.2
       > go: downloading github.com/go-playground/locales v0.14.1
       > go: downloading github.com/cloudwego/base64x v0.1.4
       > go: downloading golang.org/x/arch v0.8.0
       > go: downloading github.com/bytedance/sonic/loader v0.1.1
       > go: downloading github.com/klauspost/cpuid/v2 v2.2.7
       > go: downloading github.com/twitchyliquid64/golang-asm v0.15.1
       > go: downloading github.com/cloudwego/iasm v0.2.0
       > go: sovr.io imports
       >         github.com/gin-gonic/gin imports
       >       github.com/gin-gonic/gin/binding imports
       >       github.com/go-playground/validator/v10 imports
       >         github.com/gabriel-vasile/mimetype: github.com/gabriel-vasile/[email protected]: reading https://proxy.golang.org/github.com/gabriel-vasile/mimetype/@v/v1.4.3.zip: 403 Forbidden
       For full logs, run 'nix log /nix/store/kiyqr9llfcxxjnri7n3ifh4ar71mbxwj-sovr-server-1.0.0-go-modules.drv'.
error: 1 dependencies of derivation '/nix/store/ba086j7fnga3zh2c7sp27pzv9x27c8p7-sovr-server-1.0.0-env.drv' failed to build

I can reproduce the specific failure directly with go get:

GODEBUG=http2debug=1 GOPROXY=direct go get github.com/gabriel-vasile/[email protected] 2>&1 | tee go-mod-download-mimetype.txt
http2: Transport failed to get client conn for golang.org:443: http2: no cached connection was available
http2: Transport creating client conn 0xc000004000 to [2607:f8b0:4003:c0c::8d]:443
http2: Transport encoding header ":authority" = "golang.org"
http2: Transport encoding header ":method" = "GET"
http2: Transport encoding header ":path" = "/x/net?go-get=1"
http2: Transport encoding header ":scheme" = "https"
http2: Transport encoding header "accept-encoding" = "gzip"
http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
http2: Transport received SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=65536
http2: Transport received WINDOW_UPDATE len=4 (conn) incr=983041
http2: Transport received SETTINGS flags=ACK len=0
http2: Transport received HEADERS flags=END_HEADERS stream=1 len=96
http2: Transport received DATA flags=END_STREAM stream=1 len=1584 data="<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n  <title>Error 403 (Forbidden)!!1</title>\n  <style>\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-ser" (1328 bytes omitted)
http2: Transport received PING len=8 ping="\x00\x00\x00\x00\x00\x00\x01c"

My IP addresses are:

    inet 194.135.104.85/24 brd 194.135.104.255 scope global eth0
    inet6 2605:e440::2:3e/64 scope global

What did you expect to see?

I expected the mimetype module to download and be available to build software.

The text was updated successfully, but these errors were encountered:

gabyhelp · 2025-01-02T13:10:29Z

Related Issues

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

EliRibble · 2025-01-02T13:17:57Z

Working through some error triangulation steps from other related issues:

Specify a proxy directly, get 403 Forbidden

$ GOPROXY=https://proxy.golang.org go get github.com/gabriel-vasile/[email protected]
go: downloading github.com/gabriel-vasile/mimetype v1.4.3
go: github.com/gabriel-vasile/[email protected]: reading https://proxy.golang.org/github.com/gabriel-vasile/mimetype/@v/v1.4.3.zip: 403 Forbidden

Specify direct (without a proxy) and get a different 403 Forbidden

$ GOPROXY=direct go get github.com/gabriel-vasile/[email protected]
go: golang.org/x/[email protected]: unrecognized import path "golang.org/x/arch": reading https://golang.org/x/arch?go-get=1: 403 Forbidden

Specify a third party proxy, don't get a 403 Forbidden

$ GOPROXY=https://goproxy.cn go get github.com/gabriel-vasile/[email protected]
go: downloading github.com/gabriel-vasile/mimetype v1.4.3
go: downloading golang.org/x/net v0.25.0
$

Another third party

$ GOPROXY=https://goproxy.io go get github.com/gabriel-vasile/[email protected]
go: downloading github.com/gabriel-vasile/mimetype v1.4.3
go: downloading golang.org/x/net v0.25.0
$

seankhliao · 2025-01-02T13:21:29Z

can you use curl to download the zip file?

See golang/go#71094 which I filed with the go project. It seems that just.hosting has an IP address that is likely getting geolocated incorrectly leading to being forbidden from downloading go modules from go's default proxy. This uses another proxy (which I haven't vetted in any way) instead. Does that sound like a vector for a supply chain attack to you? It does to me. Hat tip to https://discourse.nixos.org/t/git-buildgomodule-private-repositories/5167/8 for the magical incantation to get Nix to set a GOPROXY variable.

EliRibble · 2025-01-02T13:58:44Z

Maybe? I'm not sure how exactly to translate the go get command to a comparable curl command. Do you know how?

EliRibble · 2025-01-02T14:06:56Z

Assuming this is valid:

curl -Lvo /tmp/mimetype https://proxy.golang.org/github.com/gabriel-vasile/mimetype/@v/v1.4.3.zip

I end up with /tmp/mimetype:

$ file /tmp/mimetype 
/tmp/mimetype: Zip archive data, at least v2.0 to extract, compression method=deflate

It appears to unzip to a valid archive.

hyangah · 2025-01-02T18:35:20Z

Looks like you got 403 not only while interacting with https://proxy.golang.org, but also when reading https://golang.org/x/arch?go-get=1 (in #71094 (comment)) and https://golang.org/x/net?go-get=1 (in the original report) in GOPROXY=direct mode. Given that the only shared part between proxy.golang.org and golang.org is the Google network (they are on different servers, and on different corners in Google), it may be an issue between Google frontend and the client. OTOH the last curl (on proxy.golang.org) succeeded, so that's strange. Does the problem persist?

EliRibble · 2025-01-02T18:38:17Z

Yeah, I can repro at will with:

$ go clean -modcache
$ go get github.com/gabriel-vasile/[email protected]

hyangah · 2025-01-02T19:13:12Z

Is it possible to try the similar setup from a machine with a different IP address?

EliRibble · 2025-01-03T16:37:00Z

Sure, tested from my laptop, works fine.

$ curl -4 icanhazip.com
98.171.80.66

$ go clean -modcache

$ go get github.com/gabriel-vasile/[email protected]
go: downloading github.com/gabriel-vasile/mimetype v1.4.3
go: downloading golang.org/x/net v0.25.0

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/eliribble/.cache/go-build'
GOENV='/home/eliribble/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/eliribble/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/eliribble/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/nix/store/zhq8dnjbhwspzdglyq28j74axvqyk86q-go-1.23.3/share/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/nix/store/zhq8dnjbhwspzdglyq28j74axvqyk86q-go-1.23.3/share/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.3'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/eliribble/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/eliribble/src/sovr.cloud/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/nix-shell-19324-0/go-build1867468048=/tmp/go-build -gno-record-gcc-switches'

gopherbot added the proxy.golang.org label Jan 2, 2025

dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 3, 2025

dmitshur modified the milestones: Unreleased, proxy.golang.org/backlog Jan 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

proxy.golang.org, cmd/go: 403 Forbidden on go get #71094

proxy.golang.org, cmd/go: 403 Forbidden on go get #71094

EliRibble commented Jan 2, 2025

gabyhelp commented Jan 2, 2025

EliRibble commented Jan 2, 2025

seankhliao commented Jan 2, 2025

EliRibble commented Jan 2, 2025

EliRibble commented Jan 2, 2025

hyangah commented Jan 2, 2025

EliRibble commented Jan 2, 2025

hyangah commented Jan 2, 2025

EliRibble commented Jan 3, 2025

proxy.golang.org, cmd/go: 403 Forbidden on go get #71094

proxy.golang.org, cmd/go: 403 Forbidden on go get #71094

Comments

EliRibble commented Jan 2, 2025

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

gabyhelp commented Jan 2, 2025

EliRibble commented Jan 2, 2025

seankhliao commented Jan 2, 2025

EliRibble commented Jan 2, 2025

EliRibble commented Jan 2, 2025

hyangah commented Jan 2, 2025

EliRibble commented Jan 2, 2025

hyangah commented Jan 2, 2025

EliRibble commented Jan 3, 2025

Output of `go env` in your module/workspace: