To get started running a Bitcoin Verde full node and block explorer on a private server, run the following commands:
Linux / Mac OS X
- git clone https://github.com/softwareverde/bitcoin-verde
- cd bitcoin-verde
- ./scripts/make.sh
- ./scripts/run-node.sh > logs/node.log &
- ./scripts/run-explorer.sh > logs/explorer.log &
- Visit http://localhost:8080/status/
Windows
Bitcoin Verde has not been well tested on Windows; please report any issues to bitcoin-verde@softwareverde.com. To run and install Bitcoin Verde on Windows, we recommend installing git-bash and following the Linux / Max OS X steps above within the git-bash terminal.
Bitcoin (Core) (BTC) is the sole group responsible for development on BTC's client. In the past, lack of a diversified development team and node implementation, have caused bugs to become a part of the protocol. BCH currently has one mining full-node implementation (Bitcoin Cash Node). This implementation is a forked version of the Bitcoin (Core) reference client, which means it may share the same (undiscovered) bugs. With a diverse network of nodes, bugs in the implementation of the protocol will result in incompatible blocks, causing a temporary fork. This situation is healthy for the network in the long term, as the temporary forks will resolve over time, with the intended implementation becoming the consensus.
Bitcoin Verde is a ground-up implementation of the Bitcoin (Cash) protocol. This primary function of this project is to be a unique java implementaion of a BCH full node to diversify the BCH network's implementation.
Bitcoin Verde is a full node, blockchain explorer, library, SPV wallet, electrum server, and stratum server, with a mining pool in active development.
Unlike the reference client, Bitcoin Verde stores most of the blockchain within a traditional SQL database. The SQL database allows for tracking multiple BCH forks/orphans and enables insightful queries to be run within the entire history of Bitcoin--for instance, via a query, Bitcoin Verde can report the ratio of P2PKH vs P2SH outputs, or easily report the number of transactions added to the network per period of time.
Bitcoin Verde tracks and syncs forks; it determines the head blockchain by respecting the chain with the most Proof of Work (with a compatible ruleset). Forked/Orphaned blockchains may be visualized within the explorer. Forked blocks are stored as BlockchainSegments. These segments are stored as a Nested Set, allowing efficient queries to determine relationship among one another.
Unconfirmed Transactions are stored with referential integrity to their spent inputs and are inserted when they are received (assuming validity and its dependant outputs have been stored). In this sense, Bitcoin Verde has an unlimited mempool size, since transactions are written to disk. Transactions attempting to spend outputs that have not yet been received are stored in a separate table until their dependent transactions have been received; in theory, this is also of unlimited size but is limited in order to prevent resource exhaustion attacks. Stored transactions respect the first-seen principle and not inserted into the database unless provided within a mined block.
TransactionOutput addresses are indexed as transactions are received. This functionality is required for the Explorer to be able to query the balance of addresses. With addresses indexed, it is possible to instantly sync SPV wallets in the future, once privacy concerns have been addressed and the protocol is expanded to support this functionality. As of Bitcoin Verde v2.2.0, an the node is compatible with the Electrum protocol, allowing Bitcoin Verde nodes to serve as Electrum Cash Wallet servers.
Reorgs are handled slightly differently than the reference client. Since all forked blocks are tracked and validated, and a full ledger is maintained (not just the UTXO set as in the reference client), switching to a new fork is a trivial process and does not require a reorg at all in order to validate new transactions. However, when mining, the existing mempool must be cleaned up during a reorg. This mempool reorg is handled by undoing the abandoned blocks and reapplying the blocks within the new head chain.
Nodes are also tracked within the database. In this way, Bitcoin Verde can report on the distribution of node versions and supported features.
Blocks and Transactions may be trimmed from Bitcoin Verde once they are spent. With trimming enabled, spent Blocks, Transactions, and TransactionOutputs are deleted from the database and disk once they are sufficiently old. With this feature enabled, the Node is no longer considered a Full Node as it loses the ability to serve historic blocks. Additionally, the Explorer fails to display blocks that have been pruned.
All functionality, including networking, within Bitcoin Verde is multi-threaded. The multithreaded nature of Bitcoin Verde should allow the node process, validate, and relay more transactions per second than the reference client; furthermore, Bitcoin Verde's unlimited mempool size should help to mitigate some problems encountered during past stress-tests on the network. These claims are hypothetical, and will be proven (or disproven) during the next network stress-test.
The project's scripts are located within scripts, its java source code is located within src, its html assets are located within www, its jni-libraries are located within jni, its default configuration is located within conf, and the built project is located within out. When deploying the application to a server, the out is intended to contain everything necessary for the application to run.
In general, all scripts located within any scripts directory are expected to be ran from the project root-directory.
At a minimum, the build process consists of a single script: scripts/make.sh . The make script creates an out directory, builds the application, copies dependencies, loads assets, creates a default configuration file, and creates scripts to run the application.
To update the Explorer HTML/CSS/JS, run scripts/copy-www-data.sh. Changes made to the out/www directory can be made, but excluded from commits. The Explorer may not always find new resources once the server has been started, so it is recommended to restart the Explorer in between modifications.
libsecp256k1 & NativeUnspentTransactionOutputCache
By default, libsecp256k1 and NativeUnspentTransactionOutputCache prebuilt binaries are included in this project. The prebuilt libraries were built for 64-bit Linux, 64-bit OSX, and 64-bit Windows.
libsecp256k1
libsecp256k1 is an open-source C library for validating secp256k1 signatures. It is heavily optimized and provides a substantial speedup for transaction signature verification. libsecp256k1 was written by Bitcoin Core developers, Pieter Wuille and Gregory Maxwell.
If preferred, libsecp256k1 may be built from source (located at https://github.com/bitcoin-core/secp256k1) and included in Bitcoin Verde by copying the new binaries to src/main/resources/lib/libsecp256k1.[so, dylib, dll], then rebuilding the application jar.
It is not necessary to use libsecp256k1, and if the binaries are removed before building, then the application will fallback to using Bouncy Castle's implementation of secp256k1.
Enabling SSL/TLS
Enabling SSL/TLS for the Block Explorer can be done by providing a certificate key and certificate to the explorer configuration. To generate a key, consider using the scripts/ssl/generate-certificate.sh script, or use any normal SSL script generation/signature process. In order for HttpServlet to read the certificate key, the keyfile must be in PKCS format. To convert a PEM file to a PKCS format, use the scripts/ssl/pem2pkcs.sh script. A typical location for the SSL certificates is out/ssl.
The Bitcoin Verde project has modules for varying purposes--ranging from the block-explorer module (allowing viewing and tracing transfers through the blockchain) to a simple command-line utility to generate an private-key/address.
Below are the available modules to run within Bitcoin Verde. Each module may be run either by executing their associated script or by invoking run.sh <COMMAND> <PARAMETERS> from the out directory.
| Module | Command | Parameters | Description |
|---|---|---|---|
| Full Node | NODE | Configuration File | Syncs with the Bitcoin network and validates blocks. Once fully synchronized, the node will relay transactions and new blocks to its peers. |
| Block Explorer | EXPLORER | Configuration File | Starts a web-server to search blocks, transactions, and addresses. The explorer also monitors synchronization progress, server load, and peers. Additionally, the Block Explorer visualizes forks and orphaned blocks. Currently, the Block Explorer also displays this documentation. |
| Electrum Server | ELECTRUM | Configuration File | Starts an Electrum server. This server is compatible with Electron-Cash-like wallets, and requires the Node module be running, with indexBlocks enabled and its synchronization/indexing completed. |
| Stratum Server | STRATUM | Configuration File | Starts a Stratum server. Stratum servers are used to mine and coordinate ASIC miners for generating new blocks. This module also includes a simplistic pool framework. The Node module must be running and its synchronization must be complete before starting the stratum server. |
| Chain (Re)Validation | VALIDATE | Configuration File | Iterates through the already-synced blockchain to check for any data corruption. This tool is primarily a debug module, but can improve confidence in the history of the chain in case a bug is found in the future to determine if it's been exploited without having to re-download and store the whole chain. |
| Database-Only | DATABASE | Configuration File | Starts the internal database; this module does not connect to any nodes nor does it start any of the blockchain services. The DATABASE module is primarily a debug module, but can be used to run manual queries against the state of the blockchain via ./scripts/connect-to-database.sh. |
| Address Generator | ADDRESS | Generates a private key from secure-random and outputs the hex-encoded private key, its public key, and compressed and uncompressed addresses. | |
| Test-Block Miner | MINER | Previous Block Hash Bitcoin Address CPU Thread Count GPU Thread Count | Creates a block based off the provided previous-block-hash, with a single coinbase transaction to the address provided. The block created will satisfy the initial Bitcoin difficulty. This mode is intended to be used to generate test-blocks. |
Hosting a Block Explorer
By default, the Block Explorer runs on port 8080. When hosting the Block Explorer for the public, either a reverse proxy routing port 80 to internal 8080 can be used, or the server can be configured to route 80 to 8080 via iptables. On linux, the forward-ports.sh script in the scripts directory can be run as root to configure iptables to handle this routing. Changes made to iptables are not permanent and will be lost on restart, so it may be useful to configure this command to run on startup.
Security
Security is always a vital consideration when hosting any application. Security is a cost-benefit scale between ease of use and risk. Risk is a combination of the consequence of compromise, the scale of the vulnerability, and the value incentive to an attacker. The Bitcoin Verde node and explorer do not currently hold or manage private keys which reduces the consequence of an improperly configured server, however the value any server has to an attacked is never zero. Exposing any server to the public internet always creates a target for attacks. For this reason, Bitcoin Verde should never be run as root. Never expose bitcoin private keys (or other sensitive information) to a publicly exposed server running the Bitcoin Verde Block Explorer.
The comitted node configuration is located at conf/server.conf and it copied to out/conf/server.conf at build-time. Users are expected to modify the conf file located at out/conf/server.conf.
Below is a table with each configuration property and its purpose.
| Module | Property | Default Value | Description |
|---|---|---|---|
| database | rootPassword | d3d4a3d0533e3e83bc16db93414afd96 | The root password configured for the database. This field is not necessary to exist after the initial start of the node. |
| database | hostname | localhost | The host used to connect to the database. This value is ignored unless useEmbeddedDatabase is enabled. |
| database | port | 8336 | The port used to connect to the database. |
| database | dataDirectory | data | The name of the directory that contains the database data within the out directory. |
| database | schema | bitcoin | The name of the database schema used to hold all tables. |
| database | maxMemoryByteCount | 2GB | The maximum byte count for the InnoDB buffer pool. The buffer pool size is one of the most important configurations for performance as it configures the space used to store/cache table indexes. |
| database | username | bitcoin | The username used when connecting to the database. This value may only be changed before the initial start of the node. |
| database | password | 81b797117e8e0233ea8fd1d46923df54 | The password used when connecting to the database. This value may only be changed before the initial start of the node. |
| database | useEmbeddedDatabase | 1 | When set to 1, the node will manage its own MariaDB instance. Otherwise, the node will attempt to connect to the hostname/port and manage its database at that location. In each scenario, the node will create and manage its own schema, including creating a maintenance account and user account. |
| bitcoin | port | 8333 | The port used to listen for incoming connections from other nodes on the network. |
| bitcoin | rpcPort | 8334 | The port used to listen for RPC commands sent to this node. |
| bitcoin | seedNodes | ["btc.softwareverde.com", "bitcoinverde.org"] | The seed nodes which are initially connected to on start. Currently at least one seed node is required since DNS seeding is not currently supported. Should not exceed maxPeerCount. May optionally contain a port (eg: "bitcoinverde.org:8333"). |
| bitcoin | whitelistedNodes | [] | The list of nodes that are prevented from being banned under any circumstance. |
| bitcoin | enableBanFilter | 1 | If set to zero or false, then nodes will not be banned under any circumstances. Additionally, any previously banned nodes will be unbanned while disabled. |
| bitcoin | enableBlockPruning | 0 | If set to 1 or true, then the node will only keep the most recent blocks on disk once they are processed, reducing disk footprint. |
| bitcoin | enableBlockchainCache | 1 | If set to 1 or true, then the node will allocate enough memory to store all blockchain headers and metadata within the application (roughly 500mb), increasing block/header processing. |
| bitcoin | enableBlockCompression | 1 | If set to 1 or true, then the node will gzip all blocks on disk. If the blockchain cache (enableBlockchainCache) is not enabled, then header lookups will be significantly less performant when using this feature. |
| bitcoin | maxPeerCount | 32 | The maximum number of peers that the node will accept. |
| bitcoin | maxThreadCount | 4 | The max number of threads used to validate a block. Currently, the server will create max(maxPeerCount * 8, 256) threads for network communication; in the future this property will likely claim this label. |
| bitcoin | trustedBlockHeight | 391180 | Blocks prior to this height will not have their transactions validated. |
| bitcoin | skipNetworking | 0 | When set, this property disables networking with other nodes. |
| bitcoin | prioritizeNewPeers | 0 | Specifies the behaviour when a new connection is received but the node is already at its maxPeerCount. If unset (the default), the node will prioritize its existing connections and reject the new one. If set, the node will disconnect from a non-preferred peer to make room for the new connection. Note that enabling this setting may increase the risk of a Sybil attack. |
| bitcoin | maxUtxoCacheByteCount | 536870912 | The max memory used to store Unspent Transaction Outputs in memory. This properties does nothing if the NativeUnspentTransactionOutputCache library is not loaded. |
| bitcoin | useTransactionBloomFilter | 1 | When set, uses an internal bloom filter to maintain a filter of transactions that have already been seen, greatly reducing the time required to store a block. When enabled, this setting requires about 1GB of memory. This property is safe to be toggled on/off between restarts. |
| bitcoin | trimBlocks | 0 | Experimental feature. When enabled, spent TransactionOutputs that are older than 144 blocks will be deleted from the database. Enabling this feature disables full-node functionality and will prevent the block explorer from functioning. |
| bitcoin | cacheBlocks | 1 | Caches the serialized block on disk within the data directory. Normally blocks are reinflated from the database whenever accessed; caching the inflated block on disk greatly improves performance of large blocks when serving blocks and merkle blocks to peers. This cache increases disk-usage. |
| bitcoin | maxMessagesPerSecondPerNode | 250 | May be used to prevent flooding the node; requests exceeding this throttle setting are queued. 250 msg/s supports a little over 32MB blocks. |
| stratum | port | 3333 | The external stratum protocol port. |
| explorer | port | 8080 | The port the explorer will listen on for incoming HTTP requests. Setting to a value lower than 1024 would require the application be run as root; this is highly discouraged. Instead, consider running on a higher port and running the forward-ports script. |
| explorer | rootDirectory | www | The directory where the explorer will serve its HTTP content from (eg. out/www). |
| explorer | bitcoinRpcUrl | localhost | The host the explorer will connect to the Bitvoin Verde node on. |
| explorer | bitcoinRpcPort | 8334 | The port the explorer will connect to the Bitcoin Verde node on. |
| explorer | tlsPort | 4443 | The port the explorer will listen on for incoming HTTPS requests. Setting to a value lower than 1024 would require the application be run as root; this is highly discouraged. Instead, consider running on a higher port and running the forward-ports script. |
| explorer | tlsKeyFile | The SSL/TLS certificate private key filename. A typical value would be something like "out/ssl/certificate.key". The default value is empty; when empty, SSL/TLS is disabled on the explorer. To generate a certificate, consider looking at scripts/ssl/generate-certificate.sh. The private key file must be in PKCS format; to convert a PEM private key to a PKCS private key, use scripts/ssl/pem2pkcs.sh. | |
| explorer | tlsCertificateFile | The SSL/TLS certificate filename. A typical value would be something like "out/ssl/certificate.crt". The default value is empty; when empty, SSL/TLS is disabled on the explorer. |
Bitcoin Verde has an RPC protocol to control the node while its running. The protocol listens on port 8334 by default and is configurable via the bitcoin.rpcPort configuration parameter. The RPC protocol is JSON-based and loosely mimics HTTP methods, using both GET and POST to indicate mutating/write and immutable/read operations.
An example RPC invocation to shutdown the node looks like:
{
"method": "POST",
"query": "SHUTDOWN"
}
An example RPC invocation to retrieve information about a transaction looks like:
{
"method": "GET",
"query": "TRANSACTION",
"parameters": {
"hash": "917A95AADA7B17A10661CAA2EE379E05CD816E9AB9E5C7651A371909B7222812"
}
}
The procedures Bitcoin Verde currently support are listed below:
| Method | Query | Parameters | Description |
|---|---|---|---|
| GET | BLOCK_HEADERS | blockHeight maxBlockCount=10 rawFormat=0 | List block headers starting after blockHeight, or the head block if not provided. Returns at most maxBlockCount headers. |
| GET | BLOCK | blockHeight | hash rawFormat=0 | Returns the block of matching hash or at blockHeight of the head chain. blockHeight is ignored if hash is provided. When rawFormat is set to 1 the returned block will be as hex-encoded bytes; otherwise, it is a detailed breakdown of the block as JSON. |
| GET | BLOCK_HEADER | blockHeight | hash | Returns the blockHeader of matching hash or at blockHeight of the head chain. blockHeight is ignored if hash is provided. |
| GET | BLOCK_TRANSACTIONS | blockHashpageNumber=0 pageSize=128 | Returns the set of Transactions contained within the provided blockHash. Results are returned in order and paginated, starting at pageNumber 0. The result will return at most pageSize Transactions. To return all Transactions for the provided Block, set pageSize to 2147483647. |
| GET | TRANSACTION | hash rawFormat=0 | Returns the transaction by its hash. When rawFormat is set to 1 the returned transaction will be as hex-encoded bytes; otherwise, it is a detailed breakdown of the transaction as JSON. |
| GET | BLOCK_HEIGHT | Returns the current head block height and the block header height. These values can be used to determine synchronization progress, assuming block headers have finished processing. | |
| GET | DIFFICULTY | Returns the Target Difficulty for the next Block. | |
| GET | UTXO_CACHE | Returns the superset of cached UTXOs in-memory. | |
| GET | BLOCK_REWARD | Returns the maximum reward for the next Block. | |
| GET | UNCONFIRMED_TRANSACTIONS | Returns the superset of Transactions that are valid for the next Block. | |
| GET | STATUS | Returns the node's status, including block/header height, services' statuses, memory usage, thread-pool utiliztion, and synchronization statistics. | |
| GET | NODES | Returns a list of connected peers. | |
| GET | BALANCE | address | Returns the current balance of the provided address. Keep in mind that nodes that have not completed the initial block download may not have an up-to-date balance. |
| GET | ADDRESS | address | Returns transactions that have either sent-to or were sent-from the provided address. Keep in mind that nodes that have not completed the initial block download may not have an up-to-date list. |
| GET | BLOCKCHAIN | Returns forked and orphaned blockchains that the node has witnessed. Forks and orphans are fairly rare until after the initial block download has completed. | |
| GET | IS_SLP_TRANSACTION | hash | Returns whether the Transaction matches the format of an SLP Transaction, regardless of its validity. |
| GET | IS_VALID_SLP_TRANSACTION | hash | Returns whether the transaction is a valid SLP transaction. This method includes validating the entire DAG for this transaction. |
| GET | SLP_TOKEN_ID | hash | Returns the Transaction's SLP Token ID, regardless of its validity. |
| POST | SHUTDOWN | Gracefully shuts down the node. | |
| POST | COMMIT_UTXO_CACHE | Synchronizes the current in-memory UTXO state to disk. | |
| POST | ADD_NODE | host port | Forces the node to connect to the specified peer. Once maxPeerCount has been reached, adding a peer will eject a node from the currently connected set. |
| POST | BAN_NODE | host | Disconnects from and bans all nodes connected at the provided ip/hostname. |
| POST | UNBAN_NODE | host | Clears all nodes resolving to the provided ip/hostname of their banned status. |
| POST | WHITELIST_NODE | host | Prevents all nodes resolving to the provided ip/hostname from being banned. |
| POST | REMOVE_WHITELIST_NODE | host | Removes any nodes from the whitelist. This function does not automatically ban the provided ip/hostname. |
| POST | ADD_HOOK | events | Upgrades the RPC connection to a HookConnection. The HookConnection remains open until closed by the client. The node will relay live-updates to the HookConnection based on the events it has subscribed to. The mandatory events parameter is an array of HookEvents. Currently supported HookEvent values are: NEW_BLOCK, NEW_TRANSACTION. |
| POST | TRANSACTION | transactionData | Queues the Transaction for processing into the mempool. Transactions submitted via RPC are not treated differently than transactions received from peer nodes and will be validated and accepted in the order they are received. transactionData should be the deflated bytes of the Transaction in its entirety, hex-encoded. |
| POST | BLOCK | blockData | Queues the Block for processing. Blocks submitted via RPC are not treated differently than transactions received from peer nodes and will be validated and accepted in the order they are received. transactionData should be the deflated bytes of the Block in its entirety, hex-encoded. |
| POST | VALIDATE_PROTOTYPE_BLOCK | blockData | Validates the Block as a prototype block for mining. The prototype block is validated against the current head blockchain in its entirety except for its hash. A blockValidation object is returned in the response; the blockValidation object contains the validation error and a list of invalid transactions, if any. The invalid transactions set may only be a subset of the invalid transactions. |
| POST | VALIDATE_TRANSACTION | transactionData enableSlpValidation=1 | Validates the Transaction for the mempool, but does not add the transaction to the mempool regardless of its validity. SLP validation can be disabled via the enableSlpValidation parameter. |
| POST | SET_LOG_LEVEL | packageNamelogLevel | Sets the minimum required log statement severity in order to be written to the logfile for the provided Java packageName. Valid logLevels include OFF, TRACE, DEBUG, INFO, WARN, ERROR. |
An example VALIDATE_PROTOTYPE_BLOCK response looks like:
{
"wasSuccess": 1,
"errorMessage": null,
"blockValidation": {
"isValid": false,
"errorMessage": "Block coinbase does not contain block height.",
"invalidTransactions": [
"7E7337FDEA7B55BD2105AE611EAC478889AA5D321B18B5E9389666CFB85619B6"
]
}
}
Architecture
Most of the node's operations are asychronous. The primary asynchronous services are the BlockDownloader, BlockHeaderDownloader, BlockchainBuilder, AddressProcessor, TransactionDownloader, and TransactionProcessor. These various services alongside the NodeManager and the JsonRpcSocketServerHandler comprise the majority of the node's functionality. Events from one service often trigger other services to wake up; for instance, completing a block download from within the BlockDownloader triggers the BlockchainBuilder to check for available blocks to process.
Three steps are taken when a block hash or header is received: the hash is queued for download within the pending_blocks table, the node_block_inventory table is updated to keep a record of which node(s) can be used to download the block, and the BlockDownloader is woken up. The BlockDownloader queries pending_blocks for any blocks that have not been downloaded; when a match is found, the BlockDownloader checks the NodeManager for nodes that have the block inventory, and requests the block. If there are blocks that can't be downloaded because there are no peers connected that have announced that block, then a BlockFinder message is broadcasted and the pending block hashes are deleted from pending_blocks.
When a block is downloaded, the BlockchainDownloader is woken up. If there is a block that has been downloaded that has its parent block processed, the BlockchainBuilder stores and validates the block. The BlockchainBuilder continues to process blocks until it runs out of available blocks; once complete, a BlockFinder message is broadcast to all peers to ensure there are no new blocks available before the service goes back to sleep.
The AddressProcessor is woken up when a new block has been stored and validated. The AddressProcessor iterates through each TransactionOutput to index Addresses for the Block Explorer. When blockTrimming is enabled, the AddressProcessor is disabled.
Similarly to how blocks are queued for download, when a new transaction is announced its hash is queued into pending_transactions. The TransactionDownloader, TransactionProcessor share the same relationship and life-cycles as the BlockDownloader and BlockchainBuilder.
Bitcoin Verde's database is a self contained MariaDB database. The database is started when the node starts and is shut down when the node shuts down. If the node crashes, the database is not always shut down with it and lives as a zombie process; in this state, the node will fail to restart until the database is killed manually (usually via a kill command).