internal: fix GO_AWAY deadlock
A deadlock can occur when a GO_AWAY is followed by a connection closure. This
happens because onClose needlessly closes the current ac.transport: if a
GO_AWAY already occured, and the transport was already reset, then the later
closure (of the original address) sets ac.transport - which is now healthy -
to nil.
The manifestation of this problem is that picker_wrapper spins forever trying
to use a READY connection whose ac.transport is nil.
Method logger will be picked by each RPC at the beginning to log binary entries. It also handles truncating.
This PR also adds script to download proto from grpc/grpc-proto
internal: fix client send preface problems
This CL fixes three problems:
- In clientconn_state_transitions_test.go, sometimes tests would flake because there's not enough buffer to send client side settings, causing the connection to unpredictably enter TRANSIENT FAILURE. Each time we set up a server to send SETTINGS, we should also set up the server to read. This allows the client to successfully send its SETTINGS, unflaking the test.
- In clientconn.go, we incorrectly transitioned into TRANSIENT FAILURE when creating an http2client returned an error. This should be handled in the outer resetTransport main reset loop. The reason this became a problem is that the outer resetTransport has very specific conditions around when to transition into TRANSIENT FAILURE that the egregious transition did not have. So, it could transition into TRANSIENT FAILURE after failing to dial, even if it was trying to connect to a non-final address in the list of addresses.
- In clientconn.go, we incorrectly stay in CONNECTING after `createTransport` when a server sends its connection preface but the client is not able to send its connection preface. This CL causes the addrconn to correctly enter TRANSIENT FAILURE when `createTransport` fails, even if a server preface was received. It does so by making ac.successfulHandshake to consider both server preface received as well as client preface sent.
internal: clean up and unflake state transitions test
Switches state transitions test to using a notification from a custom load
balancer, instead of relying on waiting for laggy balancer state updates.
Also generally adds more coverage around state transitions and a framework
for easily adding more of these kinds of tests.
Fixes#2348
Closing `ClientConn` sets `balancerWrapper` to nil.
If service config switches balancer, the new balancer will be notified of the existing addresses.
When these two happens together, there's a chance that a method will be called on the nil `balancerWrapper`. This change adds a check to make sure that never happens.
fixes#2367
internal: fix onClose state transitions
When onClose occurs during WaitForHandshake, it should immediately
exit createTransport instead of leaving CONNECTING and entering READY.
Furthermore, when any onClose happens, the state should change to
TRANSIENT FAILURE.
Fixes#2340Fixes#2341
Also fixes an unreported bug in which entering READY causes a
Dial call to end prematurely, instead of blocking until a READY
transport is found.
Google default creds is a combo of ALTS, TLS and OAuth2. The right set of creds will be picked to use based on environment.
This PR contains:
- A new `creds.Bundle` type
- changes to use it in ClientConn and transport
- dial option to set the bundle for a ClientConn
- balancer options and NewSubConnOption to set it for SubConn
- Google default creds implementation by @cesarghali
- grpclb changes to use different creds mode for different servers
- interop client changes for google default creds testing
Commit 35c3afad17f705e725ca04bff658932689920154 added a `defer cancel()` line
to transport_test. While this line is generally good, there happens to be a
Go 1.6 bug that when a context that gets passed into Dial is canceled, it
has a 50% chance to incorrectly causes all future Dial writes to fail.
This commit removes that defer and adds a comment explaining the situation.
https://github.com/golang/go/issues/15078https://github.com/golang/go/issues/15035
This commit also fixes a perceived (by the race detector) racy use of
server.conn by making conn.Close happen inside the goroutine that
uses conn.
This commit also fixes a SIGSEV caused by an incorrect typed nil
check.
This fixes a race in ac.tearDown and ac.resetTransport. If ac.resetTransport
is in backoff when ac.tearDown occurs, there's a race between the state
changing to Shutdown and ac.resetTransport calling ac.createTransport.
This fixes it by returning when ac.resetTransport encounters an error
during ac.nextAddr (specifically ac.ctx.Error()). It also fixes it by
making sure that ac.tearDown changes state to Shutdown before canceling
the context.
Both fixes were implemented because they both seem to be valuable
standalone additions: the former makes ac.resetTransport more
understandable and less dependent on behavior happening elsewhere,
and the latter makes ac.tearDown more correct.
Finally, TestDialParseTargetUnknownScheme had its buffer removed; the
buffer was likely added a while ago to assuage this issue. It should
not be necessary anymore.
internal: remove transportMonitor, replace with callbacks
This refactors the internal http2 transport to use callbacks instead
of continuously monitoring the transport in a separate goroutine. This
has several advantages:
- Less goroutines.
- Less complexity: synchronous callbacks are much easier to reason to
reason about than asynchronous monitoring goroutines.
- Callbacks: these provide definitive locations for monitoring the
creation and closure of a transport, paving the way for GracefulStop.
This CL also consolidates all the logic about backoff and iterating
through the list of addresses into a single method.
tests: fix goroutine leak
If TestResetConnectBackoff fails, the resetTransport goroutine will be
stuck dialing and subsequently the goroutine will be leaked. This is
all despite the test including `defer cc.Close()`:
- defer cc.Close() will cause ac.cancel to be called
- ac.context will be appropriately cancelled
- ac.context is correctly the context that gets passed to the dialer
- However, the WithDialer throws away the context and only passes its
deadline, which is for `backoffForever{}` is math.MaxInt64. So, even
though teardown occurs, the resetTransport goroutine will still be
stuck dialing.
This CL adds a small amendment: before performing leakcheck, attempt
to take an item off the synchronous `dials` channel. Either the tests
passed and there is no item, or the tests failed and there is one.
