https://github.com/gridcoin-community/Gridcoin-Research/releases/tag/184.108.40.206 submitted by
Finally! After over ten months of development and testing, "Fern" has arrived! This is a whopper. 240 pull requests merged. Essentially a complete rewrite that was started with the scraper (the "neural net" rewrite) in "Denise" has now been completed. Practically the ENTIRE Gridcoin specific codebase resting on top of the vanilla Bitcoin/Peercoin/Blackcoin vanilla PoS code has been rewritten. This removes the team requirement at last (see below), although there are many other important improvements besides that.
Fern was a monumental undertaking. We had to encode all of the old rules active for the v10 block protocol in new code and ensure that the new code was 100% compatible. This had to be done in such a way as to clear out all of the old spaghetti and ring-fence it with tightly controlled class implementations. We then wrote an entirely new, simplified ruleset for research rewards and reengineered contracts (which includes beacon management, polls, and voting) using properly classed code. The fundamentals of Gridcoin with this release are now on a very sound and maintainable footing, and the developers believe the codebase as updated here will serve as the fundamental basis for Gridcoin's future roadmap.
We have been testing this for MONTHS on testnet in various stages. The v10 (legacy) compatibility code has been running on testnet continuously as it was developed to ensure compatibility with existing nodes. During the last few months, we have done two private testnet forks and then the full public testnet testing for v11 code (the new protocol which is what Fern implements). The developers have also been running non-staking "sentinel" nodes on mainnet with this code to verify that the consensus rules are problem-free for the legacy compatibility code on the broader mainnet. We believe this amount of testing is going to result in a smooth rollout.
Given the amount of changes in Fern, I am presenting TWO changelogs below. One is high level, which summarizes the most significant changes in the protocol. The second changelog is the detailed one in the usual format, and gives you an inkling of the size of this release.
Note that the protocol changes will not become active until we cross the hard-fork transition height to v11, which has been set at 2053000. Given current average block spacing, this should happen around October 4, about one month from now.
Note that to get all of the beacons in the network on the new protocol, we are requiring ALL beacons to be validated. A two week (14 day) grace period is provided by the code, starting at the time of the transition height, for people currently holding a beacon to validate the beacon and prevent it from expiring. That means that EVERY CRUNCHER must advertise and validate their beacon AFTER
the v11 transition (around Oct 4th) and BEFORE
October 18th (or more precisely, 14 days from the actual date of the v11 transition). If you do not advertise and validate your beacon by this time, your beacon will expire and you will stop earning research rewards until you advertise and validate a new beacon. This process has been made much easier by a brand new beacon "wizard" that helps manage beacon advertisements and renewals. Once a beacon has been validated and is a v11 protocol beacon, the normal 180 day expiration rules apply. Note, however, that the 180 day expiration on research rewards has been removed with the Fern update. This means that while your beacon might expire after 180 days, your earned research rewards will be retained and can be claimed by advertising a beacon with the same CPID and going through the validation process again. In other words, you do not lose any earned research rewards if you do not stake a block within 180 days and keep your beacon up-to-date.
The transition height is also when the team requirement will be relaxed for the network.
Besides the beacon wizard, there are a number of improvements to the GUI, including new UI transaction types (and icons) for staking the superblock, sidestake sends, beacon advertisement, voting, poll creation, and transactions with a message. The main screen has been revamped with a better summary section, and better status icons. Several changes under the hood have improved GUI performance. And finally, the diagnostics have been revamped.
The wallet sync speed has been DRASTICALLY improved. A decent machine with a good network connection should be able to sync the entire mainnet blockchain in less than 4 hours. A fast machine with a really fast network connection and a good SSD can do it in about 2.5 hours. One of our goals was to reduce or eliminate the reliance on snapshots for mainnet, and I think we have accomplished that goal with the new sync speed. We have also streamlined the in-memory structures for the blockchain which shaves some memory use.
There are so many goodies here it is hard to summarize them all.
I would like to thank all of the contributors to this release, but especially thank @cyrossignol, whose incredible contributions formed the backbone of this release. I would also like to pay special thanks to @barton2526, @caraka, and @Quezacoatl1, who tirelessly helped during the testing and polishing phase on testnet with testing and repeated builds for all architectures.
The developers are proud to present this release to the community and we believe this represents the starting point for a true renaissance for Gridcoin!
Most significantly, nodes calculate research rewards directly from the magnitudes in EACH superblock between stakes instead of using a two- or three- point average based on a CPID's current magnitude and the magnitude for the CPID when it last staked. For those long-timers in the community, this has been referred to as "Superblock Windows," and was first done in proof-of-concept form by @denravonska.
- Network magnitude unit pinned to a static value of 0.25
- Max research reward allowed per block raised to 16384 GRC (from 12750 GRC)
- New CPIDs begin accruing research rewards from the first superblock that contains the CPID instead of from the time of the beacon advertisement
- 500 GRC research reward limit for a CPID's first stake
- 6-month expiration for unclaimed rewards
- 10-block spacing requirement between research reward claims
- Rolling 5-day payment-per-day limit
- Legacy tolerances for floating-point error and time drift
- The need to include a valid copy of a CPID's magnitude in a claim
- 10-block emission adjustment interval for the magnitude unit
- One-time beacon activation requires that participants temporarily change their usernames to a verification code at one whitelisted BOINC project
- Verification codes of pending beacons expire after 3 days
- Self-service beacon removal
- Burn fee for beacon advertisement increased from 0.00001 GRC to 0.5 GRC
- Rain addresses derived from beacon keys instead of a default wallet address
- Beacon expiration determined as of the current block instead of the previous block
- The ability for developers to remove beacons
- The ability to sign research reward claims with non-current but unexpired beacons
As a reminder:
- Beacons expire after 6 months pass (180 days)
- Beacons can be renewed after 5 months pass (150 days)
- Renewed beacons must be signed with the same key as the original beacon
- Magnitudes less than 1 include two fractional places
- Magnitudes greater than or equal to 1 but less than 10 include one fractional place
- A valid superblock must match a scraper convergence
- Superblock popularity election mechanics
- Yes/no/abstain and single-choice response types (no user-facing support yet)
- To create a poll, a maximum of 250 UTXOs for a single address must add up to 100000 GRC. These are selected from the largest downwards.
- Burn fee for creating polls scaled by the number of UTXOs claimed
- 50 GRC for a poll contract
- 0.001 GRC per claimed UTXO
- Burn fee for casting votes scaled by the number of UTXOs claimed
- 0.01 GRC for a vote contract
- 0.01 GRC to claim magnitude
- 0.01 GRC per claimed address
- 0.001 GRC per claimed UTXO
- Maximum length of a poll title: 80 characters
- Maximum length of a poll question: 100 characters
- Maximum length of a poll discussion website URL: 100 characters
- Maximum number of poll choices: 20
- Maximum length of a poll choice label: 100 characters
- Magnitude, CPID count, and participant count poll weight types
- The ability for developers to remove polls and votes
[220.127.116.11] 2020-09-03, mandatory, "Fern"
- Backport newer uint256 types from Bitcoin #1570 (@cyrossignol)
- Implement project level rain for rainbymagnitude #1580 (@jamescowens)
- Upgrade utilities (Update checker and snapshot downloadeapplication) #1576 (@iFoggz)
- Provide fees collected in the block by the miner #1601 (@iFoggz)
- Add support for generating legacy superblocks from scraper stats #1603 (@cyrossignol)
- Port of the Bitcoin Logger to Gridcoin #1600 (@jamescowens)
- Implement zapwallettxes #1605 (@jamescowens)
- Implements a global event filter to suppress help question mark #1609 (@jamescowens)
- Add next target difficulty to RPC output #1615 (@cyrossignol)
- Add caching for block hashes to CBlock #1624 (@cyrossignol)
- Make toolbars and tray icon red for testnet #1637 (@jamescowens)
- Add an rpc call convergencereport #1643 (@jamescowens)
- Implement newline filter on config file read in #1645 (@jamescowens)
- Implement beacon status icon/button #1646 (@jamescowens)
- Add gridcointestnet.png #1649 (@caraka)
- Add precision to support magnitudes less than 1 #1651 (@cyrossignol)
- Replace research accrual calculations with superblock snapshots #1657 (@cyrossignol)
- Publish example gridcoinresearch.conf as a md document to the doc directory #1662 (@jamescowens)
- Add options checkbox to disable transaction notifications #1666 (@jamescowens)
- Add support for self-service beacon deletion #1695 (@cyrossignol)
- Add support for type-specific contract fee amounts #1698 (@cyrossignol)
- Add verifiedbeaconreport and pendingbeaconreport #1696 (@jamescowens)
- Add preliminary testing option for block v11 height on testnet #1706 (@cyrossignol)
- Add verified beacons manifest part to superblock validator #1711 (@cyrossignol)
- Implement beacon, vote, and superblock display categories/icons in UI transaction model #1717 (@jamescowens)
- neuralnet: Add integrity checking to researcher accrual snapshot registry #1727 (@jamescowens)
- Add workaround for scrypt assembly on macOS #1740 (@cyrossignol)
- gui: Build onboarding/beacon wizard #1739 (@cyrossignol)
- doc: Add CONTRIBUTING.md from bitcoin #1723 (@div72)
- rpc: Implement inspectaccrualsnapshot and parseaccrualsnapshotfile #1744 (@jamescowens)
- scraper: Add disk based state backing for verified beacon list in scraper #1751 (@jamescowens)
- Add ability to recover beacon in block version 11+ #1768 (@cyrossignol)
- refactor: Add transaction context to contract handlers #1777 (@cyrossignol)
- gui: Add context for when BOINC is attached to a pool #1775 (@cyrossignol)
- doc: Clarify what to do if PR in multiple categories (for CONTRIBUTING.md) #1798 (@RoboticMind)
- qt: Add option to choose not to start the wallet minimized #1804 (@jamescowens)
- superblock: Add check for OutOfSyncByAge to SuperblockValidator::Validate #1806 (@jamescowens)
- contract: Standardize contract validation and add block context #1808 (@cyrossignol)
- add seed.gridcoin.pl to default config #1812 (@wilkart)
- gui: Implement sidestake send display #1813 (@jamescowens)
- gui: Add pool/investor pages to researcher wizard #1819 (@cyrossignol)
- ci: Port lint scripts from Bitcoin #1823 (@div72)
- doc: Create basic readme in contrib #1826 (@RoboticMind)
- gui: Implement TransactionRecord::Message #1829 (@jamescowens)
- rpc: Add private_key_available to beaconstatus #1833 (@a123b)
- gui: Validate email address in researcher wizard #1840 (@a123b)
- rpc: Add "getrawwallettransaction" RPC function #1842 (@cyrossignol)
- consensus: Set block version 11 threshold height for mainnet #1862 (@cyrossignol)
- Upgrade LevelDB from v1.17 to v1.20 #1562 (@cyrossignol)
- Re-enable scrypt optimizations #1450 (@denravonska)
- Derive CScript from prevector type (optimization) #1554 (@cyrossignol)
- Disable quorum for grandfathered blocks to speed up sync #1568 (@cyrossignol)
- Refactor hashBoinc for binary claim contexts #1558 (@cyrossignol)
- integrated_scraper_2 branch tracking PR #1559 (@jamescowens)
- Upgrade depends - OpenSSL to 1.1.1d #1581 (@jamescowens)
- Ubuntu 19.10 fixes #1590 (@denravonska)
- Force a re-parse of legacy claims in generated blocks #1592 (@cyrossignol)
- Improve the "versionreport" RPC output #1595 (@cyrossignol)
- Overhaul the core tally and accrual system #1583 (@cyrossignol)
- Overhaul the superblock quorum system #1597 (@cyrossignol)
- Add more data to the "superblocks" RPC output #1599 (@cyrossignol)
- Update Windows Build doc #1606 (@barton2526)
- Change the order of calls in gridcoinresearchd.cpp to optimize rpc shunt path #1610 (@jamescowens)
- Change staking tooltip to display frequency #1611 (@jamescowens)
- Enhancements to ETTS #1442 (@jamescowens)
- Standardize money values as integers #1614 (@cyrossignol)
- Clean up and optimize legacy coin age code #1616 (@cyrossignol)
- Some scraper cleanups #1620 (@jamescowens)
- Reorganize accrual code and fix 6-month cutoff #1630 (@cyrossignol)
- Update Copyright years #1633 (@barton2526)
- Change team whitelist delimiter to <> for CPID detection #1634 (@cyrossignol)
- Change team whitelist separator to <> to accomodate more team names #1632 (@jamescowens)
- Change Curl download speed type to support older environments #1640 (@cyrossignol)
- Optimize logo SVGs used for tray icons #1638 (@cyrossignol)
- Tweak consolidateunspent rpc function #1644 (@jamescowens)
- ETTS and staking icon enhancements #1650 (@jamescowens)
- Implement new transaction fees for block version 11 #1652 (@jamescowens)
- Optimize in-memory storage of superblock data #1653 (@cyrossignol)
- Miscellaneous superblock API improvements and housekeeping #1654 (@cyrossignol)
- Update openssl to 1.1.1f compatibility #1660 (@jamescowens)
- Optimize bdb to avoid synchronous flush of database #1659 (@jamescowens)
- Add support for CPID input to "lifetime" RPC function #1668 (@cyrossignol)
- Overhaul the contract handling system #1669 (@cyrossignol)
- Make the autostart mainnet/testnet aware #1671 (@jamescowens)
- Remove slashes from User Agent in peers tab #1674 (@div72)
- Refactor contracts for polymorphic binary payloads #1676 (@cyrossignol)
- Overhaul the beacon system #1678 (@cyrossignol)
- Replace boost::optional with non-owning pointers #1680 (@cyrossignol)
- Optimize proof-of-stake validation #1681 (@cyrossignol)
- Updated Slack link #1683 (@NeuralMiner)
- Update build-unix.md #1686 (@Quezacoatl1)
- Replace deprecated QT methods #1693 (@Pythonix)
- Made protocol.h more similar to bitcoin #1688 (@Pythonix)
- Touch up some details for block version 11 #1697 (@cyrossignol)
- More tweaks for block version 11 #1700 (@cyrossignol)
- Finish the conversion to the BCLog class based logger #1699 (@jamescowens)
- Move claim version transitional code in miner for proper signature #1712 (@cyrossignol)
- doc: Update threads in coding.txt #1730 (@div72)
- qt: Include QPainterPath in trafficgraphwidget.cpp #1733 (@div72)
- doc: Update doc/build-unix.md #1731 (@div72)
- gui: Show peers tab on connections icon click #1734 (@div72)
- refactor: Change return type of IsMine to isminetype && move wallet files to wallet directory #1722 (@div72)
- build: Updates boost to 1.73.0 for depends #1673 (@jamescowens)
- doc: Update Unit Test Readme #1743 (@RoboticMind)
- wallet: Change Assert To Error Message In kernel.cpp #1748 (@RoboticMind)
- scraper: Shorten display representation of verification codes #1754 (@cyrossignol)
- log: Change ".B." to Clear Message #1758 (@RoboticMind)
- util: Fix braindamage in GetDefaultDataDir() #1737 (@jamescowens)
- scraper: Improve scraper processing of beacon verifications #1760 (@jamescowens)
- scraper: Add instrumentation to convergencereport #1763 (@jamescowens)
- rpc: Improve rpc stress test script #1767 (@tunisiano187)
- Generalize enum serialization #1770 (@cyrossignol)
- scraper: Improve handling of ETags in http class and tweak verified beacon logic #1776 (@jamescowens)
- scraper: Improve ProcessNetworkWideFromProjectStats and other tweaks #1778 (@jamescowens)
- researcher: Automate beacon advertisement for renewals only #1781 (@cyrossignol)
- gui: Tweak behavior of beacon page in researcher wizard #1784 (@cyrossignol)
- Prepare for block version 11 hard-fork on testnet #1787 (@cyrossignol)
- scraper: Modify UpdateVerifiedBeaconsFromConsensus #1791 (@jamescowens)
- gui: Optimize OverviewPage::updateTransactions() #1794 (@jamescowens)
- ci: Adopt ci changes from Bitcoin #1795 (@div72)
- consensus: switch snapshot accrual calculation to integer arithmetic #1799 (@cyrossignol)
- voting: Overhaul the voting system #1809 (@cyrossignol)
- contract: Optimize contract replay after chain reorganization #1815 (@cyrossignol)
- contract: Reimplement transaction messages as contracts #1816 (@cyrossignol)
- staking: Sign claim contracts with coinstake transaction #1817 (@cyrossignol)
- gui: Change research wizard text #1820 (@div72)
- net: Update protocol version and clean up net messaging #1824 (@cyrossignol)
- rpc, wallet: Corrections to GetAmounts #1825 (@jamescowens)
- gui: Tweak some minor researcher wizard details #1830 (@cyrossignol)
- gui: Change GetEstimatedStakingFrequency text #1836 (@jamescowens)
- scraper: Scraper global statistics cache optimization #1837 (@jamescowens)
- doc: Update Vulnerability Response Process #1843 (@RoboticMind)
- scraper: Optimization of manifest and parts sharing between ConvergedScraperStatsCache, mapManifest, and mapParts #1851 (@jamescowens)
- consensus: Update Checkpoints #1855 (@barton2526)
- docs: Update docs to build off master #1856 (@barton2526)
- gui: Fix and improve GUI combo box styles #1858 (@cyrossignol)
- build: Tweak Gridcoin installer for Fern release #1863 (@jamescowens)
- Remove old research age checks (rebase #1365) #1572 (@cyrossignol)
- Remove PrimaryCPID check from diagnostics dialog #1586 (@cyrossignol)
- Remove missed label for PrimaryCPID from diagnostics #1588 (@cyrossignol)
- Remove legacy quorum messaging system (@neural network) #1589 (@cyrossignol)
- Remove old remnants of legacy smart contract experiments #1594 (@cyrossignol)
- Remove block nonce for version 11 #1622 (@cyrossignol)
- Delete obsolete contrib/Installer and Upgrader directories #1623 (@jamescowens)
- Remove redundant LoadAdminMessages() calls #1625 (@cyrossignol)
- Remove some legacy informational RPC commands #1658 (@cyrossignol)
- Remove informational magnitude field from binary claims #1661 (@cyrossignol)
- Remove fDebug3,4, and net and convert to BCLog::LogFlags #1663 (@jamescowens)
- Remove qt5.7.1 depends support build System #1665 (@iFoggz)
- Remove unused jQuery library #1679 (@cyrossignol)
- Remove unused NetworkTimer() function and global state #1701 (@cyrossignol)
- Refactor claim context objects into contracts #1704 (@cyrossignol)
- Clean old assets up #1718 (@div72)
- Remove legacy "rain" RPC (not by-project rain) #1742 (@cyrossignol)
- Temporarily disable voting system on testnet #1769 (@cyrossignol)
- gui: Remove legacy GUI transaction description for contracts #1772 (@cyrossignol)
- gui: Remove transaction fee setting #1780 (@cyrossignol)
- trivial: Cleanup unused legacy functions #1793 (@cyrossignol)
- mining, rpc: Remove kernel-diff-best and kernel-diff-sum #1796 (@jamescowens)
- refactor: Remove libs subdirectory #1802 (@div72)
- scraper: cleanup unused/unnecessary functions #1803 (@jamescowens)
- gui: Remove useless "Detach databases at shutdown" #1810 (@jamescowens)
- test: Remove testnet condition for standard transactions #1814 (@cyrossignol)
- consensus: Remove transitional testnet code #1854 (@cyrossignol)
- Fix "Owed" amount in output of "magnitude" RPC method #1569 (@cyrossignol)
- Add support for paths with special characters on Windows #1571 (@cyrossignol)
- Fix lingering peers.dat temp files and clean up remaining paths #1582 (@cyrossignol)
- Fix incorrect beacon length warning in GUI transaction list #1585 (@cyrossignol)
- Fix default config file line endings on Windows #1587 (@cyrossignol)
- Reenable Travis builds for MacOS #1591 (@jamescowens)
- Correct peer detail info background color #1593 (@jamescowens)
- Fix exception in debug3 mode #1598 (@cyrossignol)
- Fix deadlock in "getmininginfo" RPC function #1596 (@cyrossignol)
- Fix accuracy of statistics in "network" RPC output #1602 (@cyrossignol)
- Fix heights for quorum vote weight calculations #1604 (@cyrossignol)
- Fix deadlock in log archiver when rename fails #1607 (@cyrossignol)
- Fix a spurious segmentation fault during client load on Windows with fast CPUs #1608 (@jamescowens)
- Fix lock order debugging and potential deadlocks #1612 (@jamescowens)
- Add dependencies #1613 (@Scalextrix)
- Fix std namespace pollution #1617 (@denravonska)
- Add missing condition for newbie accrual computer #1618 (@cyrossignol)
- Track first reward blocks in research accounts #1619 (@cyrossignol)
- Fix lingering beacon warning after advertisement #1627 (@cyrossignol)
- Fix accrual calculation for new, zero-magnitude CPIDs #1636 (@cyrossignol)
- Fix diagnostics, add ETTS test, fix tooltipcolor, add missing lock, and add email=investor check #1647 (@jamescowens)
- Fix help message of two RPC methods #1656 (@div72)
- Fix legacy accrual for newbie with non-zero past reward #1667 (@cyrossignol)
- Fix GUI autostart on Windows for paths with wide characters #1670 (@cyrossignol)
- Qualify boost bind placeholders with their full namespace #1672 (@Ponce)
- Fix suffix when copying txids #1677 (@div72)
- Unnecessary if-statement removed #1685 (@Pythonix)
- Fix consolidatemsunspent Help Message #1687 (@Pythonix)
- Fix gettransaction help message #1691 (@Pythonix)
- Fix GetNewMint To Look for Stakes #1692 (@RoboticMind)
- Suppress deprecated copy warnings for Qt with GCC 9+ #1702 (@cyrossignol)
- Fix exclusion error on stats processing and misplaced ENDLOCK logging entry #1710 (@jamescowens)
- Removed unnecessary comparison #1708 (@Pythonix)
- Fixed typo #1707 (@Pythonix)
- Fix out-of-bounds exception for peers tab version slashes #1713 (@cyrossignol)
- Fix transition for v1 superblocks when reorganizing #1714 (@cyrossignol)
- Touch up transition to version 2 transactions #1715 (@cyrossignol)
- Avoid mutating transactions in ConnectBlock() #1716 (@cyrossignol)
- Skip beacon advertisement when already pending #1726 (@cyrossignol)
- Fix Windows cross-compilation in newer environments #1728 (@cyrossignol)
- Fix out-of-bounds access in IsMineInner() #1736 (@cyrossignol)
- Fix a couple of block version 11 issues #1738 (@cyrossignol)
- Fix null pointer dereference in GUI researcher model #1741 (@cyrossignol)
- accrual: Reset research accounts when rebuilding accrual snapshots #1745 (@cyrossignol)
- scraper: Correct update for verified beacons #1747 (@jamescowens)
- accrual: Refactor tally initialization for snapshot rebuild #1749 (@cyrossignol)
- rpc: Fix "cpid" field in "beaconconvergence" RPC output #1750 (@cyrossignol)
- accrual: Fix snapshot accrual superblock state transitions #1752 (@cyrossignol)
- scraper: Correct stale verified beacon logic #1753 (@jamescowens)
- rpc: Correct possible divide by zero in getblockstats #1755 (@jamescowens)
- gui: Fix issues with researcher wizard flow #1756 (@cyrossignol)
- wallet: Stop Error When Starting From Zero #1759 (@RoboticMind)
- Don't count empty email as explicit investor #1761 (@cyrossignol)
- accrual: Fix snapshot accrual superblock state transitions #1764 (@cyrossignol)
- rpc: Cleanup Help Message and Fix Typo #1771 (@RoboticMind)
- scraper: Fix scraper etag header case sensitivity #1773 (@cyrossignol)
- consensus: Use explicit time to check if superblock needed #1774 (@cyrossignol)
- gui: Fix scroll area dark theme styles #1785 (@cyrossignol)
- rpc, gui: Fix three divide by zero possibilities #1789 (@jamescowens)
- rpc: Fix balance pre-check in "rainbymagnitude" RPC #1792 (@cyrossignol)
- accrual: Fix outdated comment and correct grammar #1800 (@RoboticMind)
- gui: Fix stuck cursor on labels #1801 (@div72)
- beacon: Fix research wizard beacon renewal status #1805 (@cyrossignol)
- gui: Fix translations for port numbers #1818 (@cyrossignol)
- util: Create parent directory #1821 (@div72)
- mining: Fix coinstake/claim signature order #1828 (@cyrossignol)
- voting: Remove double increment in loop #1831 (@cyrossignol)
- neuralnet, scraper: Fix compilation with gcc5 and older libcurl #1832 (@a123b)
- wallet: Fix smallest coin selection for contracts #1841 (@cyrossignol)
- gui: Fix display of polls with no votes yet #1844 (@cyrossignol)
- gui: add indentation to diagnostic status bar labels #1849 (@jamescowens)
- voting, gui: Fix formatting and alignment of vote shares and percent #1850 (@jamescowens)
- wallet, rpc: Fix for self-transactions in listtransactions #1852 (@jamescowens)
- accrual: Clear any accrual snapshots when syncing from pre-v11 #1853 (@cyrossignol)
- accrual: Fix reset of accrual directory if starting sync below research age height #1857 (@jamescowens)
- gui: Fix researcher wizard layout on macOS with native theme #1860 (@cyrossignol)
I am going to post in this thread my experiences in getting my Desktop (Debian) machine running Armory in watch-only mode, and coupling that with an offline Raspberry Pi (which holds my private keys) for signing the transactions previously made in watch-only mode.
I actually compiled Armory from source directly on my Pi. This guide is probably more for the bitcoin 'power user', as to run Armory online, and broadcast the signed transactions, you need to have a bitcoin full node running (bitcoind).
- Online machine - running a full node (bitcoind)
- Raspberry Pi - I used an old Pi 1 Model B with just 512Mb memory, and 2 USB slots.
- 2x USB thumb-drives. One for wallet backups, the other for transferring unsigned tx's to the rPi, and signed tx's back to the Desktop.
- Armory 0.96.4 for the Raspberry Pi 1, Model B (512Mb RAM, 2xUSB) (compiled from github sourcecode on the Pi itself!)
- Using the Pi as an offline complement to a Debian Desktop "watch-only" Armory install.
- Desktop Debian Armory watch-only talks to my full node, bitcoind, which is also on the Debian desktop.
I'll post the guide in digestible sections...
I should begin by saying I installed source code from git, and got Armory to build the DB on my desktop initially, WITHOUT creating a wallet.. (This allowed me to debug what was going on a little!)
- I am using Armory 0.96.4
- Desktop: Debian 9, Dual-Core, 2Gb Memory, 2Gb Swap.
- Pi : Pi 1, Model B (512Mb RAM, 2x USB, Ethernet)
Go to Bitcoin.org, select Armory..
It leads to a Download from Git: https://github.com/goatpig/BitcoinArmory/releases
Followed the procedure for Linux Debian verify code, compile, install, all straight-forward..
Began by running bitcoind, and telling Armory where to find it. This is the command I used, obviously it was all on one line and didn't include the arrows/explanations!:
python ArmoryQt.py \ --satoshi-datadir=/BlockChain/chain20180414/blocks \ # <-----(where my bitcoind blocks live) --datadir=/ArmoryDataDi \ # <-----(this is instead of ~/.armory) --dbdir=/ArmoryDataDidatabases # <-------(again, non std. place used for Armory's databases.. my choice.)
So, on the Desktop, after the initial "build databases"
(NB the initial "Build Databases" took about 1.5h and my two CPUs were maxed the whole time, Temps up to 62C. Not ideal; Im not in a rush!)
I then wanted to import a watch-only wallet.
Before I did this, I took a full backup of the Armory data dir:
(or ~/.armory in a default installation).
I'd hate to have to make Armory do another full sync with the bitcoind node!
Next step: offline wallet (with Private Keys) is on a Raspberry Pi.
I downloaded the source and managed to compile it on the pi itself! :)
Though there were some gymnastics needed to setup the Pi.
My Pi is running Raspbian based on Wheezy.. quite old!
I did the following on the Pi:
apt-get update apt-get upgrade (<---took about an hour!) apt-get install autotools-dev apt-get install autoconf
Then I followed the instructions exactly as I had done for my Debian Desktop machine, EXCEPT:
I had to increase the Pi's swap space. I upped it from 100Mb to 400Mb.
The compilation took 7 hours, and my poor SD card got a thrashing.
But after compilation, I put the Swap back to 100Mb and Armory runs ok with about 150Mb of memory (no swap needed).
Swap increase on the Pi:
use your favourite editor, and open the file /etc/dphys-swapfile
add/change the following line:
Then, REBOOT the Pi:
sudo shutdown -h -P now
Once the compilation was done on the Pi, put the swap back, rebooted and created an Armory wallet.
I added manual entropy and upped the encryption 'time' from 250ms to 2500ms - since the Pi is slow, but I'll be happy to wait for more iterations in the Key Derivation Function.
Once the wallet was created, it obviously prompts you for backup.
I want to add a private key of my own (i.e. import), so don't do the backup until this is over.
I import my Private Key, and Armory checks that this corresponds to a Public Key, which I check is correct.
This is the point now where the Pi storage medium (e.g an SD card) has to be properly destroyed if you ever get rid of it.
I had thought that now would be a good time to decide if your new wallet will generate Segwit receiving addresses, and also addresses used to receive 'change' after a transaction..
But it seems Armory WON'T let you switch to P2SH-P2WPKH unless your Armory is connected to a node offering "WITNESS" service.
Obviously, my Pi is offline and will never connect to a node, so the following will not work on the Pi:
- x Use File Settings Fee and address types.
- x Set the "Preferred Receive Address Type" to P2SH-P2WPKH
- x Also Set the "Change Address" to P2SH-P2WPKH for left-over loose change!
NB: I thought about setting this on the Debian "watch-only" wallet, but that would surely mean doom, as the Pi would not know about those addresses and backups might not keep them.. who knows...
So, end result:- no segwit for me just yet in my offline funds.
--If anyone can offer a solution to this, I'd be very grateful--
Ok, now this is a good point to back up your wallet on the Pi. It has your imported keys. I choose a Digital Backup - and put it on a USB key, which will never touch the internet and will be stored off-site. I also chose to encrypt it, because I'm good with passwords..
NB: The Armory paper backup will NOT back up your imported private keys, so keep those somewhere if you're not sweeping them. It would be prudent to have an Armory paper backup anyway, but remember it will likely NOT help you with that imported key.
Now for the watch-only copy of the wallet. I want to get the "watch-only" version onto my Desktop Debian machine.
On the Pi, I created (exported to a USB key) a "watching-only" copy of my wallet.
I would use the RECOMMENDED approach, export the "Entire Wallet File".
As you will see below, I initially exported only the ROOT data, which will NOT capture the watching-only part of the Private Key I entered manually above (i.e. the public Key!).
Now, back on the Debian Desktop machine...
I stopped all my crontab jobs; just give Armory uninterrupted CPU/memory/disk...
I also stopped bitcoind and made a backup prior to any watch-only wallet being imported.
I already made a backup of Armory on my Desktop, before any wallet import.
(this was needed, as I made a mistake.. see below)
So on the Debian Desktop machine, I begin by firing up bitcoind.
my command for this is:
./bitcoind -daemon -datadir=/BlockChain/chain20180414 -dbcache=400 -maxmempool=400
I try running Armory like this:
(I'm actually starting Armory from a script - StartArm.sh)
Inside the script StartArm.sh, it has the line:
python ArmoryQt.py --ram-usage=4 --satoshi-datadir=/BlockChain/chain20180414/blocks --datadir=/ArmoryDataDi --dbdir=/ArmoryDataDidatabases
I know from bitter experience that doing a scan over the blockchain for a new wallet takes a looong time and a lot of CPU, and I'd like it to play nicely; not gobble all the memory and swap and run my 2xCPUs both at 100% for four hours...
So... I aim to run with --ram-usage=X and --thread-count=X
(For me in the end, X=1 but I began with X=4)
I began with --ram-usage=4 (<--- = 4x128Mb)
The result is below...
TypeError: cannot concatenate 'str' and 'int' objects
It didn't recognise the ram-usage and carried on, crippling my Debian desktop PC.
This is where it gets dangerous; Armory can gobble so much memory and CPU that the windowing environment can cease up, and it can take over 30 minutes just to exit nicely from bitcoind and ArmoryDB.
So, I ssh to the machine from another computer, and keep an eye on it with the command
I'd also be able to do a "sudo reboot now" if needed from here.
So, trying to get my --ram-usage command recognised, I tried this line (added quotes):
python ArmoryQt.py --ram-usage="4" --satoshi-datadir=/BlockChain/chain20180414/blocks --datadir=/ArmoryDataDi --dbdir=/ArmoryDataDidatabases
But no, same error...
Loading Armory Engine: Armory Version: 0.96.4 Armory Build: None PyBtcWallet Version: 1.35 Detected Operating system: Linux OS Variant : ('debian', '9.4', '') User home-directory : /home/ Satoshi BTC directory : /BlockChain/chain20180414 Armory home dir : /ArmoryDataDi ArmoryDB directory : /ArmoryDataDidatabases Armory settings file : /ArmoryDataDiArmorySettings.txt Armory log file : /ArmoryDataDiarmorylog.txt Do wallet checking : True (ERROR) ArmoryUtils.py:3723 - Unsupported language specified. Defaulting to English (en) (ERROR) ArmoryQt.py:1833 - Failed to start Armory database: cannot concatenate 'str' and 'int' objects Traceback (most recent call last): File "ArmoryQt.py", line 1808, in startArmoryDBIfNecessary TheSDM.spawnDB(str(ARMORY_HOME_DIR), TheBDM.armoryDBDir) File "/BitcoinArmory/SDM.py", line 387, in spawnDB pargs.append('--ram-usage=' + ARMORY_RAM_USAGE) TypeError: cannot concatenate 'str' and 'int' objects
So, I edit the Armory python file SDM.py:
if ARMORY_RAM_USAGE != -1: pargs.append('--ram-usage=4') #COMMENTED THIS, SO I CAN HARDCODE =4 # ' + ARMORY_RAM_USAGE)
Running it, I now have acknowledgement of the --ram-usage=4:
(WARNING) SDM.py:400 - Spawning DB with command: /BitcoinArmory/ArmoryDB --db-type="DB_FULL" --cookie --satoshi-datadir="/BlockChain/chain20180414/blocks" --datadir="/ArmoryDataDi" --dbdir="/ArmoryDataDidatabases" --ram-usage=4
Also, even with ram-usage=4, it used too much memory, so I told it to quit.
It took over 30 minutes to stop semi-nicely. The last thing it reported was:
ERROR - 00:25:21: (StringSockets.cpp:351) FcgiSocket::writeAndRead FcgiError: unexpected fcgi header version
But that didn't seem to matter or corrupt the Armory Database, so I think it's ok.
So, I get brave and change SDM.py as below, and I make sure my script has a command line for --ram-usage="ABCDE" and --thread-count="FGHIJ"; the logic being that these strings "ABCDE" will pass the IF criteria below, and my hardcoded values will be used...
if ARMORY_RAM_USAGE != -1: pargs.append('--ram-usage=1') #COMMENTED THIS, SO I CAN HARDCODE =1 # ' + ARMORY_RAM_USAGE) if ARMORY_THREAD_COUNT != -1 pargs.append('--thread-count=1') #COMMENTED THIS, SO I CAN HARDCODE =1 #' + ARMORY_THREAD_COUNT)
So, as usual, I use my script and start this with: ./StartArm.sh
(which uses command line:)
python ArmoryQt.py --ram-usage="ABCDE" --thread-count="FGHIJ" --satoshi-datadir=/BlockChain/chain20180414/blocks --datadir=/ArmoryDataDi --dbdir=/ArmoryDataDidatabases
(this forces it to use my hard-coded values in SDM.py...)
So, this is the command which it reports that it starts with:
(WARNING) SDM.py:400 - Spawning DB with command: /BitcoinArmory/ArmoryDB --db-type="DB_FULL" --cookie --satoshi-datadir="/BlockChain/chain20180414/blocks" --datadir="/ArmoryDataDi" --dbdir="/ArmoryDataDidatabases" --ram-usage=1 --thread-count=1
Again, this is where it gets dangerous; Armory can gobble so much memory and CPU that the windowing environment can cease up. So I ssh to the machine and keep an eye on it with:
So, on the Debian Desktop PC, I inserted the USB stick with the watch-only wallet I exported from the Pi.
Import "Entire Wallet File" watch-only copy.
Wait 4 hours..
After running Armory for about 30m, the memory usage dropped by 400m... wierd...
It took ~2 hours to get 40% completion.
After 3.5 hours it's almost there...
The memory went up to about 1.7Gb in use and 900Mb of Swap, but the machine remained fairly responsive throughout, apart from a few (10?) periods at the start, where it appeared to freeze for 10-30s at a time.
(That's where my ssh session came in handy - I could check the machine was still ok with a "free -h" command)
Now, I can:
Create an unsigned transaction on my Desktop,
Save the tx to USB stick,
Move to the Pi,
Sign the tx,
Move back to the Desktop,
Broadcast the signed tx.
My initial Mistake:
This caused me to have to roll-back my Armory database, using the backup. so you should try to avoid doing this..
On the Pi, I exported only the ROOT data, which will NOT capture the watching-only part of the Private Key
It is RECOMMENDED to use the Digital Export of Entire Wallet File from the Pi when making a watch-only copy. If you just export just the "ROOT data", not the "Entire Wallet File", you'll have problems if you used an imported Private Key in the offline wallet, like I did.
Using the ROOT data text import, after it finished... my balance was zero. So,. I tried a Help->Rescan Balance (Restart Armory, takes 1minute to get back up and running) No Luck. Still zero balance.
So, I try Rescan Databases.. This will take longer. Nah.. no luck.
So, I tried again, thinking it might be to do with the fact that I imported the text "root data" stuff, instead of following the (Recommended) export of watching-wallet file.
So, I used my Armory backup, and wound back the ArmoryDataDi to the point before the install of the (zero balance) wallet. (you should not need to do this, as you will hopefully use the RECOMMENDED approach of exporting the "Entire Wallet File"!)
fa64b947bb3075ff8737656706b941af691908ab util: No translation of `Bitcoin Core` in the copyright (MarcoFalke) fab85208f678ba1be53bdb73a73ce3c5c937d448 qt: Run «make ... If you want to be able to downgrade smoothly, make a backup of your entire data directory. Without this your node will need start syncing (or importing from bootstrap.dat) anew afterwards. It is possible that the data from a completely synchronised 0.10 node may be usable in older versions as-is, but this is not supported and may break as soon as the older version attempts to reindex. Model must emit signals to notify views when its data changed. Choose appropriate signals depending on how exactly data is changed: dataChanged signal forces view to update specific cells, but not to create or remove cells. layoutAboutToBeChanged and layoutChanged signals forces view to update everything. Now you will see all the files are created in the new data directory. Apr 10, 2018 - If you need to change the data directory after the first launch, there is no The Bitcoin Core GUI wallet is based on QT, and it uses QSettings Nov 12, 2017 - Bitcoin Core is probably already running.We regularly publish content about Bitcoin, Ethereum, Altcoins, wallet guides, mining tutorials I moved my ... Using data recovery tools, Alice may be able to salvage the Bitcoin-Qt wallet from the faulty hard drive, and with it her lost funds. Prevention: Count the number of manually-created addresses and spending transactions since your last backup. If this number is greater than about 80, back up again. Weekly backups might be enough for most users. Set a very high value (e.g., 10,000) for the ...
Learn how to configure / restore the datadirectory of your bitcoin-core 19.0.1 wallet the cool way!! Steal private key with Data-Oriented Programming ... No Bitcoin Bank Accounts + Directory.io + Tax Implications Morning Bit Ep 17 - Duration: 3:22. MorningBit Recommended for you. 3:22. Export ... For Bitcoin Core Client updates: https://bitcoincore.org @bitcoincoreorg in Twitter Linux terminal commands: sha256sum, tar, cp, sudo, rm, exit, less * All other commands have been covered in ... Date: Friday, 10th April Project: Blockstream Role: Director of Research Bitcoin is very conservative. Unlike the typical tech industry ethos of move fast and break things, it is paramount that ... Dash: QT Wallet TestNet Guide This tutorial will help you run the latest Dash-qt client on Test-net. This video is mean't for Linux but can be used for Windows/Mac as well. You will just need to ...