'Hyperledger Blockchain Explorer-Fail to connect before the deadline on Endorser, fail to connect to remote gRPC server
I am trying to set up a Hyperledger Fabric Network with Hyperledger Explorer. I spin up a VM on the digital ocean cloud with ubuntu OS. From there, I spin up 3 orderers node, and 2 peers node. Which result in total of 5 nodes. (I am using RAFT setup).
However, I encounter the error as below when trying to start the hyperledger fabric explorer docker-container images.
Error:
> [email protected] app-start /opt/explorer
> ./start.sh
[2021-02-18T07:32:21.828] [INFO] PgService - SSL to Postgresql disabled
[2021-02-18T07:32:21.829] [INFO] PgService - connecting to Postgresql postgres://hppoc:******@explorerdb.mynetwork.com:5432/fabricexplorer
[2021-02-18T07:32:21.898] [INFO] Platform - network_config.id test-network network_config.profile ./connection-profile/test-network.json
[2021-02-18T07:32:22.013] [INFO] FabricConfig - config.client.tlsEnable true
[2021-02-18T07:32:22.013] [INFO] FabricConfig - FabricConfig, this.config.channels airlinechannel
[2021-02-18T07:32:22.016] [INFO] FabricGateway - enrollUserIdentity: userName : exploreradmin
2021-02-18T07:32:25.221Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:25.222Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.acme.com url:grpcs://peer1.acme.com:7051 timeout:3000
2021-02-18T07:32:25.223Z - info: [NetworkConfig]: buildPeer - Unable to connect to the endorser peer1.acme.com due to Error: Failed to connect before the deadline on Endorser- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:28.250Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Discoverer- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:28.250Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.acme.com url:grpcs://peer1.acme.com:7051 timeout:3000
2021-02-18T07:32:28.250Z - error: [ServiceEndpoint]: ServiceEndpoint grpcs://peer1.acme.com:7051 reset connection failed :: Error: Failed to connect before the deadline on Discoverer- name: peer1.acme.com, url:grpcs://peer1.acme.com:7051, connected:false, connectAttempted:true
2021-02-18T07:32:28.251Z - error: [DiscoveryService]: send[airlinechannel] - no discovery results
[2021-02-18T07:32:28.251] [ERROR] FabricClient - Error: DiscoveryService has failed to return results
at DiscoveryService.send (/opt/explorer/node_modules/fabric-network/node_modules/fabric-common/lib/DiscoveryService.js:370:10)
at process._tickCallback (internal/process/next_tick.js:68:7)
[2021-02-18T07:32:28.252] [INFO] FabricClient - ********* call to initializeDetachClient **********
[2021-02-18T07:32:28.253] [INFO] FabricClient - initializeDetachClient, network config) { name: 'test-network',
version: '1.0.0',
client:
{ tlsEnable: true,
adminCredential: { id: 'exploreradmin', password: 'exploreradminpw' },
enableAuthentication: true,
organization: 'AcmeMSP',
connection: { timeout: [Object] } },
channels: { airlinechannel: { peers: [Object] } },
organizations:
{ AcmeMSP:
{ mspid: 'AcmeMSP',
adminPrivateKey: [Object],
peers: [Array],
signedCert: [Object] } },
peers:
{ 'peer1.acme.com': { tlsCACerts: [Object], url: 'grpcs://peer1.acme.com:7051' } } }
[2021-02-18T07:32:28.253] [INFO] FabricClient - ************************************* initializeDetachClient *************************************************
[2021-02-18T07:32:28.254] [INFO] FabricClient - Error : Failed to connect client peer, please check the configuration and peer status
[2021-02-18T07:32:28.254] [INFO] FabricClient - Info : Explorer will continue working with only DB data
[2021-02-18T07:32:28.254] [INFO] FabricClient - ************************************** initializeDetachClient ************************************************
[2021-02-18T07:32:28.259] [INFO] Platform - initializeListener, network_id, network_client test-network { name: 'test-network',
version: '1.0.0',
client:
{ tlsEnable: true,
adminCredential: { id: 'exploreradmin', password: 'exploreradminpw' },
enableAuthentication: true,
organization: 'AcmeMSP',
connection: { timeout: [Object] } },
channels: { airlinechannel: { peers: [Object] } },
organizations:
{ AcmeMSP:
{ mspid: 'AcmeMSP',
adminPrivateKey: [Object],
peers: [Array],
signedCert: [Object] } },
peers:
{ 'peer1.acme.com': { tlsCACerts: [Object], url: 'grpcs://peer1.acme.com:7051' } } }
[2021-02-18T07:32:28.260] [INFO] main - Please open web browser to access ：http://localhost:8080/
[2021-02-18T07:32:28.261] [INFO] main - pid is 20
[2021-02-18T07:32:28.263] [ERROR] main - <<<<<<<<<<<<<<<<<<<<<<<<<< Explorer Error >>>>>>>>>>>>>>>>>>>>>
[2021-02-18T07:32:28.263] [ERROR] main - Error : [ 'Default client peer is down and no channel details available database' ]
[2021-02-18T07:32:30.264] [INFO] main - Received kill signal, shutting down gracefully
[2021-02-18T07:32:30.266] [INFO] Platform - <<<<<<<<<<<<<<<<<<<<<<<<<< Closing explorer >>>>>>>>>>>>>>>>>>>>>
[2021-02-18T07:32:30.266] [INFO] main - Closed out connections
Version Detail
- Hyperledger Fabric: 2.3.1
- Hyperledger Explorer: v1.1.1 (latest tag)
Part 1: Docker Container Setup
a) Docker PS
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9e8768914140 hyperledger/explorer:latest "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 0.0.0.0:8080->8080/tcp explorer.mynetwork.com
903c8c4a4326 hyperledger/explorer-db:latest "docker-entrypoint.s…" 3 minutes ago Up 3 minutes (healthy) 5432/tcp explorerdb.mynetwork.com
9fed267ae9b1 dev-peer1.budget.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86-e014c6d933da036b6c79b53d29a800d8a6886e374ceb435a30642b885735f8f5 "chaincode -peer.add…" 21 minutes ago Up 21 minutes dev-peer1.budget.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86
12407497fa4c dev-peer1.acme.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86-a3b6caa9293bb826d231b1a31cc47437e58764abe5169a20ed0ee43f25c16b24 "chaincode -peer.add…" 21 minutes ago Up 21 minutes dev-peer1.acme.com-gocc1.1.0-1.0-2593f1a95def85c64fdfed35e3d3b1051b92ed91549cfe789292ec5475d7db86
0448e71f68e1 hyperledger/fabric-peer:latest "peer node start" 22 minutes ago Up 22 minutes 0.0.0.0:8051->7051/tcp, 0.0.0.0:8052->7052/tcp peer1.budget.com
3266ec37b360 hyperledger/fabric-peer:latest "peer node start" 22 minutes ago Up 22 minutes 0.0.0.0:7051-7052->7051-7052/tcp peer1.acme.com
47ebe9ad79d1 hyperledger/fabric-orderer:latest "orderer" 22 minutes ago Up 22 minutes 0.0.0.0:8050->7050/tcp orderer2.acme.com
09a5f771f47f hyperledger/fabric-tools:latest "/bin/bash" 22 minutes ago Up 22 minutes tools
e132bb01ce22 hyperledger/fabric-orderer:latest "orderer" 22 minutes ago Up 22 minutes 0.0.0.0:9050->7050/tcp orderer3.acme.com
3c61b0316385 hyperledger/fabric-orderer:latest "orderer" 22 minutes ago Up 22 minutes
b) I use 3 docker-compose files for my configuration setting.
$ docker-compose -f ./config/docker-compose-base.yaml -f ./tls/docker-compose-tls.yaml -f ./raft/docker-compose-raft.yaml up -d
- docker-compose-base.yaml
https://gist.github.com/Skyquek/03d1ffad5643d67d8da5b268a4814a7d
- docker-compose-tls.yaml
https://gist.github.com/Skyquek/b3b314cb2152ab541e822f72c60a2cbd
- docker-compose-raft.yaml
https://gist.github.com/Skyquek/7f8ec2d4d1876283f4a9444675971be8
c) Core.yaml
- acme core.yaml
https://gist.github.com/Skyquek/8cdcbc4ee3d53a2277b1c34bb2fca704
Part 2: Blockchain Explorer Setting
1. connection-profile.json
{
"name": "test-network",
"version": "1.0.0",
"client": {
"tlsEnable": true,
"adminCredential": {
"id": "exploreradmin",
"password": "exploreradminpw"
},
"enableAuthentication": true,
"organization": "AcmeMSP",
"connection": {
"timeout": {
"peer": {
"endorser": "300"
},
"orderer": "300"
}
}
},
"channels": {
"airlinechannel": {
"peers": {
"peer1.acme.com": {}
}
}
},
"organizations": {
"AcmeMSP": {
"mspid": "AcmeMSP",
"adminPrivateKey": {
"path": "/tmp/crypto/peerOrganizations/acme.com/users/[email protected]/msp/keystore/priv_sk"
},
"peers": ["peer1.acme.com"],
"signedCert": {
"path": "/tmp/crypto/peerOrganizations/acme.com/users/[email protected]/msp/signcerts/[email protected]"
}
}
},
"peers": {
"peer1.acme.com": {
"tlsCACerts": {
"path": "/tmp/crypto/peerOrganizations/acme.com/tlsca/tlsca.acme.com-cert.pem"
},
"url": "grpcs://peer1.acme.com:7051"
}
}
}
2. docker-compose.yaml
# SPDX-License-Identifier: Apache-2.0
version: '2.1'
volumes:
pgdata:
walletstore:
networks:
mynetwork.com:
external:
name: acloudfan_airline
services:
explorerdb.mynetwork.com:
image: hyperledger/explorer-db:latest
container_name: explorerdb.mynetwork.com
hostname: explorerdb.mynetwork.com
environment:
- DATABASE_DATABASE=fabricexplorer
- DATABASE_USERNAME=hppoc
- DATABASE_PASSWORD=password
healthcheck:
test: "pg_isready -h localhost -p 5432 -q -U postgres"
interval: 30s
timeout: 10s
retries: 5
volumes:
- pgdata:/var/lib/postgresql/data
networks:
- mynetwork.com
explorer.mynetwork.com:
image: hyperledger/explorer:latest
container_name: explorer.mynetwork.com
hostname: explorer.mynetwork.com
environment:
- DATABASE_HOST=explorerdb.mynetwork.com
- DATABASE_DATABASE=fabricexplorer
- DATABASE_USERNAME=hppoc
- DATABASE_PASSWD=password
- LOG_LEVEL_APP=debug
- LOG_LEVEL_DB=debug
- LOG_LEVEL_CONSOLE=info
- LOG_CONSOLE_STDOUT=true
- DISCOVERY_AS_LOCALHOST=true
volumes:
- ./config.json:/opt/explorer/app/platform/fabric/config.json
- ./connection-profile:/opt/explorer/app/platform/fabric/connection-profile
- ./organizations:/tmp/crypto
- walletstore:/opt/explorer/wallet
ports:
- 8080:8080
depends_on:
explorerdb.mynetwork.com:
condition: service_healthy
networks:
- mynetwork.com
Solution that I tried
1. Change the tlscs certs path
As stated in Hyperledger Fabric 2.0.1: Error: Failed to connect before the deadline on Discoverer- name:
The problem most likely is due to the error from the path. But the problem still persists.
2. Change the env variable DISCOVERY_AS_LOCALHOST=true to false
Some of them mention that this will fixed the problem. But I can't seem to fix it this way.
3. Tried with hyperledger fabric 2.0 test-network
I tried to run with fabric sample test-network and its run perfectly fine.
4. docker exec -it sh to the explorer.mynetwork.com to ping the peer
The ping can run perfectly fine.
/opt/explorer # ping peer1.acme.com:7051
PING peer1.acme.com:7051 (172.23.0.6): 56 data bytes
64 bytes from 172.23.0.6: seq=0 ttl=64 time=0.138 ms
64 bytes from 172.23.0.6: seq=1 ttl=64 time=0.087 ms
64 bytes from 172.23.0.6: seq=2 ttl=64 time=0.090 ms
64 bytes from 172.23.0.6: seq=3 ttl=64 time=0.089 ms
64 bytes from 172.23.0.6: seq=4 ttl=64 time=0.101 ms
64 bytes from 172.23.0.6: seq=5 ttl=64 time=0.088 ms
^C
--- peer1.acme.com:7051 ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 0.087/0.098/0.138 ms
Its keep on showing that the peer is down or the connection to the peer is fail
I am struggling with this error for few days now. Hope someone can help me to identify the problem. Thank you very much.
Solution 1:[1]
I think you should double-check your network. Explorer should be spun up on the same network with Fabric, so that the peers and other nodes can be connected. To check the Fabric network name, you should check in the docker-compose file which set it up and look for CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE or you can navigate to you CLI and run docker network ls. You should be able to find your fabric network name with the DRIVER which should be bridge
Also, you should make sure that your Fabric network is up and running properly before bringing Explorer up.
Solution 2:[2]
With a similar error, the problem I had was that the keys and certificates for the network had gotten out of sync with the keys and certs for the blockchain explorer. This is a particularly significant problem when running a test network, because the keys and certs will (AFAICT) be regenerated whenever you restart a network, and a test network restarts much more often than a production network.
Copy over everything into the organizations directory, or whatever you call it; repeat this every time you bring up the network you want to use the explorer on. Symlinks also work, though that is probably more brittle and I wouldn't want to rely on it in production.
Solution 3:[3]
All configurations seems good, however you have to upgrade explorer version to be compatible with hyperledger fabric version.
So please use v1.1.4 instead of v1.1.1
Also make sure that you have mounted crypto config correctly, try to access this path inside the container /tmp/crypto/peerOrganizations/acme.com/tlsca/tlsca.acme.com-cert.pem
Try to change tlsCACerts path to use peer tls ca.crt
/tmp/crypto/peerOrganizations/acme.com/peers/peer1.acme.com/tls/ca.crt
You have mentioned that the same configurations works with hyperledger fabric v2, if you have tried it locally not on the same server so I please disable the firewall on the server and give it a try
To check if you can reach domain and port please try this
cat > /dev/tcp/peer1.acme.com/7051
Check this https://support.bluemedora.com/s/article/Using-Bash-to-test-if-a-TCP-port-on-a-remote-system-is-open
Solution 4:[4]
I've had this error
[INFO] FabricClient - Error : Failed to connect client peer, please check the configuration and peer status
...
Error : [ 'Default client peer is down and no channel details available database' ]
What I found was that when we use 256-bit vs 384-bit key length, the error occurs.
Setting local/host made no difference.
I'm using v1.1.4. Will likely test the same with v1.1.5 soon enough and suspect it'll have the same issue.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Dharman |
| Solution 2 | |
| Solution 3 | Yasser Mas |
| Solution 4 | Rob Evans |