aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* disabled resilver_defer feature leads to looping resilversvendor/illumosAndriy Gapon2019-11-211-8/+10
| | | | | | | | | | | | illumos/illumos-gate@233f6c49954dadfb21fa0809febd15e2160e0ff5 https://github.com/illumos/illumos-gate/commit/233f6c49954dadfb21fa0809febd15e2160e0ff5 Note: unusually for illumos this commit does not have a bug ID. Author: Kody Kantor <kody@kkantor.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354962
* 10952 defer new resilvers and misc. resilver-related fixesAndriy Gapon2019-11-2119-33/+336
| | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@e4c795beb33bf59dd4ad2e3f88f493111484b890 https://github.com/illumos/illumos-gate/commit/e4c795beb33bf59dd4ad2e3f88f493111484b890 https://www.illumos.org/issues/10952 From ZoL 612c4930dd2 Fix the spelling of deferred ??? cef48f14da6 Remove races from scrub / resilver tests 4021ba4cfaa Make vdev_set_deferred_resilver() recursive 8cb119e3dc0 Fix 2 small bugs with cached dsl_scan_phys_t 5e0bd0ae056 Fix issue with scanning dedup blocks as scan ends b3d7725c943 Remove zfs_gitrev.h (this shouldn't be part of 80a91e74696) 80a91e74696 Defer new resilvers until the current one ends Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Portions contributed by: Brian Behlendorf <behlendorf1@llnl.gov> Portions contributed by: Arkadiusz Bubała <arkadiusz.bubala@open-e.com> Author: Tom Caputi <tcaputi@datto.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354961
* 10829 zpool-features.5 add missing .REAndriy Gapon2019-11-211-3/+3
| | | | | | | | | | | | | illumos/illumos-gate@1d775d5906fd85b9ab1ac2251af8d38b5567d9a1 https://github.com/illumos/illumos-gate/commit/1d775d5906fd85b9ab1ac2251af8d38b5567d9a1 https://www.illumos.org/issues/10829 zpool-features.5 is missing .RE before allocation_classes. Author: Toomas Soome <tsoome@me.com> Notes: svn path=/vendor/illumos/dist/; revision=354960
* 11056 vdev_disk_io_start will panic the system if ldi_strategy returns an error.Andriy Gapon2019-11-211-3/+10
| | | | | | | | | | | | illumos/illumos-gate@fa88c70fa54c01be933cc49fd271e5c4468f7eb8 https://github.com/illumos/illumos-gate/commit/fa88c70fa54c01be933cc49fd271e5c4468f7eb8 https://www.illumos.org/issues/11056 Author: Jerry Jelinek <jerry.jelinek@joyent.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354959
* 11051 zfs miscounts BP_IS_EMBEDDED blocks during scan.Andriy Gapon2019-11-211-0/+7
| | | | | | | | | | | | illumos/illumos-gate@ee2f9ca4ea24f72b05598c92aad7f42fb77b1345 https://github.com/illumos/illumos-gate/commit/ee2f9ca4ea24f72b05598c92aad7f42fb77b1345 https://www.illumos.org/issues/11051 Author: Bill Sommerfeld <sommerfeld@alum.mit.edu> Notes: svn path=/vendor-sys/illumos/dist/; revision=354958
* 10853 spa_sync, vs_alloc can underflow and checkpoint test fixesAndriy Gapon2019-11-214-171/+214
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@9740f25f0360eb7d9131fa15fabebf958bf19126 https://github.com/illumos/illumos-gate/commit/9740f25f0360eb7d9131fa15fabebf958bf19126 https://www.illumos.org/issues/10853 From ZoL 7558997 vs_alloc can underflow in L2ARC vdevs 8dc2197 Simplify spa_sync by breaking it up to smaller functions db58794 Make zdb results for checkpoint tests consistent Here are the commit msg summaries from ZoL: vs_alloc can underflow in L2ARC vdevs The current L2 ARC device code consistently uses psize to increment vs_alloc but varies between psize and lsize when decrementing it. The result of this behavior is that vs_alloc can be decremented more that it is incremented and underflow. This patch changes the code so asize is used anywhere. In addition, it ensures that vs_alloc gets incremented by the L2 ARC device code as buffers are written and not at the end of the l2arc_write_buffers() routine. The latter (and old) way would temporarily underflow vs_alloc as buffers that were just written, would be destroyed while l2arc_write_buffers() was still looping. Simplify spa_sync by breaking it up to smaller functions The point of this refactoring is to break the high-level conceptual steps of spa_sync() to their own helper functions. In general large functions can enhance readability if structured well, but in this case the amount of conceptual steps taken could use the help of helper functions. Make zdb results for checkpoint tests consistent This patch exports and re-imports the pool when these tests are analyzed with zdb to get consistent results. Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Author: Serapheim Dimitropoulos <serapheim@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354957
* 10900 Fix estimated scrub completion timeAndriy Gapon2019-11-212-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@3db6627c6730f7ec4426ac0be0e9338235aed2d8 https://github.com/illumos/illumos-gate/commit/3db6627c6730f7ec4426ac0be0e9338235aed2d8 https://www.illumos.org/issues/10900 ZoL update: Author: Tom Caputi <tcaputi@datto.com> Date: Wed May 1 20:34:24 2019 -0400 Fix estimated scrub completion time Currently, it is possible for the 'zpool scrub' command to progress slightly beyond 100% due to concurrent changes happening on the live pool. This behavior is expected, but the userspace code for 'zpool status' would subtract the expected amount of data from the amount of data already scrubbed, resulting in a negative integer being casted to a large positive one. This number was then used to calculate the estimated completion time, resulting in wildly wrong results. This code changes the behavior so that 'zpool status' does not attempt to report an estimate during this period. Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Igor Kozhukhov <igor@dilos.org> Reviewed-by: George Melikov <mail@gmelikov.ru> Signed-off-by: Tom Caputi <tcaputi@datto.com> Closes #8611 Closes #8687 After this fix, the zpool status output does display more reasonable output. Author: Tom Caputi <tcaputi@datto.com> Notes: svn path=/vendor/illumos/dist/; revision=354955
* 10566 Multiple DVA Scrubbing FixAndriy Gapon2019-11-2114-184/+682
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@12a8814c13fbb1d6d58616cf090ea5815dc107f9 https://github.com/illumos/illumos-gate/commit/12a8814c13fbb1d6d58616cf090ea5815dc107f9 https://www.illumos.org/issues/10566 ZoL PR_8453 Author: Tom Caputi <tcaputi@datto.com> Date: Fri Mar 15 17:14:31 2019 -0400 Multiple DVA Scrubbing Fix Currently, there is an issue in the sequential scrub code which prevents self healing from working in some cases. The scrub code will split up all DVA copies of a bp and issue each of them separately. The problem is that, since each of the DVAs is no longer associated with the others, the self healing code doesn't have the opportunity to repair problems that show up in one of the DVAs with the data from the others. This patch fixes this issue by ensuring that all IOs issued by the sequential scrub code include all DVAs. Initially, only the first DVA of each is attempted. If an issue arises, the IO is retried with all available copies, giving the self healing code a chance to correct the issue. To test this change, this patch also adds the ability for zinject to specify individual DVAs to inject read errors into. We then add a new test case that utilizes this functionality to ensure scrubs and self-healing reads can handle and transparently fix issues with individual copies of blocks. This update is followup on #10405 While attempting to port this update, the following ZoL updates are included: 551905dd4 vdev_mirror: kstat observables for preferred vdev d6c6590c5 vdev_mirror: load balancing fixes 9f500936c FreeBSD r256956: Improve ZFS N-way mirror read performance by Portions contributed by: Toomas Soome <tsoome@me.com> Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Author: Tom Caputi <tcaputi@datto.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354954
* 10573 define TASKQID_INVALID as (taskq_id)0Andriy Gapon2019-11-217-7/+10
| | | | | | | | | | | | | | | illumos/illumos-gate@fc8ae2ec4282de7ec96f48e11078345f3dc0ac3d https://github.com/illumos/illumos-gate/commit/fc8ae2ec4282de7ec96f48e11078345f3dc0ac3d https://www.illumos.org/issues/10573 We do have taskqid_t taskq_dispatch() and the result of this function is compared with 0, NULL and combined with type casts. Define TASKQID_INVALID (taskq_id)0 and use TASKQID_INVALID everywhere. Author: Toomas Soome <tsoome@me.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354953
* 10405 Implement ZFS sorted scansAndriy Gapon2019-11-2134-905/+3208
| | | | | | | | | | | | | illumos/illumos-gate@a3874b8b1fe5103fc1f961609557c0587435fec0 https://github.com/illumos/illumos-gate/commit/a3874b8b1fe5103fc1f961609557c0587435fec0 https://www.illumos.org/issues/10405 The original implementation is: https://github.com/zfsonlinux/zfs/commit/d4a72f23863382bdf6d0ae33196f5b5decbc48fd Author: Toomas Soome <tsoome@me.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354949
* 11770 additional mmp fixesAndriy Gapon2019-11-068-126/+445
| | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@4348eb901228d2f8fa50bb132a34248e8662074e https://github.com/illumos/illumos-gate/commit/4348eb901228d2f8fa50bb132a34248e8662074e https://www.illumos.org/issues/11770 Port a few additional MMP fixes from ZoL that came in after our initial MMP port. 4ca457b065 ZTS: Fix mmp_interval failure ca95f70dff zpool import progress kstat (only minimal changes from above can be pulled in right now) 060f0226e6 MMP interval and fail_intervals in uberblock Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Portions contributed by: Brian Behlendorf <behlendorf1@llnl.gov> Author: Olaf Faaland <faaland1@llnl.gov> Notes: svn path=/vendor-sys/illumos/dist/; revision=354386
* 11541 allocation_classes feature must be enabled to add log deviceAndriy Gapon2019-11-061-2/+3
| | | | | | | | | | | | | | | | illumos/illumos-gate@c1064fd7ce62fe763a4475e9988ffea3b22137de https://github.com/illumos/illumos-gate/commit/c1064fd7ce62fe763a4475e9988ffea3b22137de https://www.illumos.org/issues/11541 After the allocation_classes feature was integrated, one can no longer add a log device to a pool unless that feature is enabled. There is an explicit check for this, but it is unnecessary in the case of log devices, so we should handle this better instead of forcing the feature to be enabled. Author: Jerry Jelinek <jerry.jelinek@joyent.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354385
* 10570 Need workaround to EFI boot on AMI BIOSAndriy Gapon2019-11-061-2/+3
| | | | | | | | | | | | | illumos/illumos-gate@fd7977362aae2eaa5dcb89671159f4fd82f22ca1 https://github.com/illumos/illumos-gate/commit/fd7977362aae2eaa5dcb89671159f4fd82f22ca1 https://www.illumos.org/issues/10570 Some BIOS types are very pick about the GPT header size. Author: John Levon <john.levon@joyent.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354384
* 10592 misc. metaslab and vdev related ZoL bug fixesAndriy Gapon2019-11-0616-436/+761
| | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@555d674d5d4b8191dc83723188349d28278b2431 https://github.com/illumos/illumos-gate/commit/555d674d5d4b8191dc83723188349d28278b2431 https://www.illumos.org/issues/10592 This is a collection of recent fixes from ZoL: 8eef997679b Error path in metaslab_load_impl() forgets to drop ms_sync_lock 928e8ad47d3 Introduce auxiliary metaslab histograms 425d3237ee8 Get rid of space_map_update() for ms_synced_length 6c926f426a2 Simplify log vdev removal code 21e7cf5da89 zdb -L should skip leak detection altogether df72b8bebe0 Rename range_tree_verify to range_tree_verify_not_present 75058f33034 Remove unused vdev_t fields Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Author: Serapheim Dimitropoulos <serapheim@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354383
* 10601 10757 Pool allocation classesAndriy Gapon2019-11-0633-404/+1820
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@663207adb1669640c01c5ec6949ce78fd806efae https://github.com/illumos/illumos-gate/commit/663207adb1669640c01c5ec6949ce78fd806efae 10601 Pool allocation classes https://www.illumos.org/issues/10601 illumos port of ZoL Pool allocation classes. Includes at least these two commits: 441709695 Pool allocation classes misplacing small file blocks cc99f275a Pool allocation classes 10757 Add -gLp to zpool subcommands for alt vdev names https://www.illumos.org/issues/10757 Port from ZoL of d2f3e292d Add -gLp to zpool subcommands for alt vdev names Note that a subsequent ZoL commit changed -p to -P a77f29f93 Change full path subcommand flag from -p to -P Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Portions contributed by: Håkan Johansson <f96hajo@chalmers.se> Portions contributed by: Richard Yao <ryao@gentoo.org> Portions contributed by: Chunwei Chen <david.chen@nutanix.com> Portions contributed by: loli10K <ezomori.nozomu@gmail.com> Author: Don Brady <don.brady@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354382
* 9521 Add checkpoint field in the default list of the zpool-list man pageAndriy Gapon2019-11-061-2/+2
| | | | | | | | | | | | | illumos/illumos-gate@f41179bd376293096297cdc1f32e610d44f65c8b https://github.com/illumos/illumos-gate/commit/f41179bd376293096297cdc1f32e610d44f65c8b https://www.illumos.org/issues/9521 The default list of fields in the zpool list man page is missing checkpoint. Author: Eitan Adler <lists@eitanadler.com> (eadler) Notes: svn path=/vendor/illumos/dist/; revision=354381
* 8899 zpool list property documentation doesn't match actual behaviourAndriy Gapon2019-11-061-9/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@b0e142e57dfda6f02bf1fa66973c610f5fefe8cb https://github.com/illumos/illumos-gate/commit/b0e142e57dfda6f02bf1fa66973c610f5fefe8cb https://www.illumos.org/issues/8899 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221795: The `zpool list` section of the zpool(8) manpage says that the default list of displayed properties is as below: The default list is name, size, used, available, fragmentation, expandsize, capacity, health, altroot. This does not seem to be the case. In fact: # zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT tank 10.9T 6.05T 4.83T - 29% 55% 1.00x ONLINE - The Properties section of the same manpage includes the `used` property, but this seems not to be recognised: # zpool list -o used bad property list: invalid property 'used' usage: [...] # The usage message produced here indeed doesn't include `used`. Author: Yuri Pankov <yuri.pankov@nexenta.com> (yuripv) Notes: svn path=/vendor/illumos/dist/; revision=354380
* 10701 Correct lock ASSERTs in vdev_label_read/writeAndriy Gapon2019-11-061-13/+6
| | | | | | | | | | | | | | | | | illumos/illumos-gate@58447f688d5e308373ab16a3b129bc0ba0fbc154 https://github.com/illumos/illumos-gate/commit/58447f688d5e308373ab16a3b129bc0ba0fbc154 https://www.illumos.org/issues/10701 Port of ZoL commit: 0091d66f4e Correct lock ASSERTs in vdev_label_read/write At a minimum, this fixes a blown assert during an MMP test run when running on a DEBUG build. Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Author: Olaf Faaland <faaland1@llnl.gov> Notes: svn path=/vendor-sys/illumos/dist/; revision=354379
* 10499 Multi-modifier protection (MMP)Andriy Gapon2019-11-0636-266/+1709
| | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@e0f1c0afa46cc84d4b1e40124032a9a87310386e https://github.com/illumos/illumos-gate/commit/e0f1c0afa46cc84d4b1e40124032a9a87310386e https://www.illumos.org/issues/10499 Port the following ZFS commits from ZoL to illumos. 379ca9cf2 Multi-modifier protection (MMP) bbffb59ef Fix multihost stale cache file import 0d398b256 Do not initiate MMP writes while pool is suspended Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Portions contributed by: Tim Chase <tim@chase2k.com> Portions contributed by: sanjeevbagewadi <sanjeev.bagewadi@gmail.com> Portions contributed by: John L. Hammond <john.hammond@intel.com> Portions contributed by: Giuseppe Di Natale <dinatale2@llnl.gov> Portions contributed by: Prakash Surya <surya1@llnl.gov> Author: Olaf Faaland <faaland1@llnl.gov> Notes: svn path=/vendor-sys/illumos/dist/; revision=354378
* 10554 Implemented zpool sync commandAndriy Gapon2019-11-068-11/+154
| | | | | | | | | | | | | | | | | | | illumos/illumos-gate@9c2acf00e275b6b2125a306f33cdddcc58393220 https://github.com/illumos/illumos-gate/commit/9c2acf00e275b6b2125a306f33cdddcc58393220 https://www.illumos.org/issues/10554 During the port of MMP (illumos bug 10499) from ZoL, I found this earlier ZoL project is a prerequisite. Here is the original description. This addition will enable us to sync an open TXG to the main pool on demand. The functionality is similar to 'sync(2)' but 'zpool sync' will return when data has hit the main storage instead of potentially just the ZIL as is the case with the 'sync(2)' cmd. Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Author: Alek Pinchuk <apinchuk@datto.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=354377
* 10844 Serialize ZTHR operations to eliminate racesAndriy Gapon2019-10-167-130/+192
| | | | | | | | | | | | | | illumos/illumos-gate@6a316e1f6d32750bb8fcf2558dcb17b90ca580fd https://github.com/illumos/illumos-gate/commit/6a316e1f6d32750bb8fcf2558dcb17b90ca580fd https://www.illumos.org/issues/10844 ZoL 61c3391acc9 Serialize ZTHR operations to eliminate races Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Author: Serapheim Dimitropoulos <serapheim@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353637
* 10809 Performance optimization of AVL tree comparator functionsAndriy Gapon2019-10-1625-260/+163
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@c4ab0d3f46036e85ad0700125c5a83cc139f55a3 https://github.com/illumos/illumos-gate/commit/c4ab0d3f46036e85ad0700125c5a83cc139f55a3 https://www.illumos.org/issues/10809 Port ZoL ee36c709c3d Performance optimization of AVL tree comparator functions From the ZoL commit msg: perf: 2.75x faster ddt_entry_compare() First 256bits of ddt_key_t is a block checksum, which are expected to be close to random data. Hence, on average, comparison only needs to look at first few bytes of the keys. To reduce number of conditional jump instructions, the result is computed as: sign(memcmp(k1, k2)). Sign of an integer 'a' can be obtained as: `(0 < a) - (a < 0)` := {-1, 0, 1}, which is computed efficiently. Synthetic performance evaluation of original and new algorithm over 1G random keys on 2.6GHz Intel(R) Xeon(R) CPU E5-2660 v3: old 6.85789 s new 2.49089 s perf: 2.8x faster vdev_queue_offset_compare() and vdev_queue_timestamp_compare() Compute the result directly instead of using conditionals perf: zfs_range_compare() Speedup between 1.1x - 2.5x, depending on compiler version and optimization level. perf: spa_error_entry_compare() `bcmp()` is not suitable for comparator use. Use `memcmp()` instead. perf: 2.8x faster metaslab_compare() and metaslab_rangesize_compare() perf: 2.8x faster zil_bp_compare() Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Author: Gvozden Neskovic <neskovic@gmail.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353630
* 10842 Mutex leak in dsl_dataset_hold_obj()Andriy Gapon2019-10-161-0/+4
| | | | | | | | | | | | | | illumos/illumos-gate@ad027c0ff9612bff8f47b43d8561da627f80cd29 https://github.com/illumos/illumos-gate/commit/ad027c0ff9612bff8f47b43d8561da627f80cd29 https://www.illumos.org/issues/10842 ZoL d10b2f1d35b Mutex leak in dsl_dataset_hold_obj() Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com> Author: Jorgen Lundman <lundman@lundman.net> Notes: svn path=/vendor-sys/illumos/dist/; revision=353628
* 10841 predictive prefetch disabled on new pools until export/rebootAndriy Gapon2019-10-161-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@0ce4bbcb47d8f86307fb8d2c84fd0f4e070f576e https://github.com/illumos/illumos-gate/commit/0ce4bbcb47d8f86307fb8d2c84fd0f4e070f576e https://www.illumos.org/issues/10841 ZoL 944a37248a0 predictive prefetch disabled on new pools until export/reboot When a pool is initially created (by `zpool create`), predictive prefetch is inadvertently disabled, until the pool is export/import-ed, or the machine is rebooted. When device removal was introduced, we added some code to disable predictive prefetching until indirect vdevs have been loaded. This resulted in the "default state" of prefetch being disabled, until we proactively enable it after indirect vdevs are loaded. Unfortunately this resulted in a few bugs where in some code paths we neglect to enable predictive prefetch. The first of these was fixed by https://github.com/zfsonlinux/zfs/commit/20507534d4ede14d4dd82c99fc8d461704ce7419 This commit fixes another case where we also need to explicitly enable predictive prefetch, when the pool is initially created. Author: Matthew Ahrens <mahrens@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353626
* 10473 zfs(1M) missing cross-reference to zfs-program(1M)Andriy Gapon2019-10-161-3/+4
| | | | | | | | | | | | illumos/illumos-gate@736e6700391d17ab1494985a80076fc185722699 https://github.com/illumos/illumos-gate/commit/736e6700391d17ab1494985a80076fc185722699 https://www.illumos.org/issues/10473 Author: Jason King <jason.king@joyent.com> Notes: svn path=/vendor/illumos/dist/; revision=353623
* 9691 fat zap should prefetch when iteratingAndriy Gapon2019-10-165-8/+100
| | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@52abb70e073c2a88808c0d66fd810ba8c5080572 https://github.com/illumos/illumos-gate/commit/52abb70e073c2a88808c0d66fd810ba8c5080572 https://www.illumos.org/issues/9691 When iterating over a ZAP object, we're almost always certain to iterate over the entire object. If there are multiple leaf blocks, we can realize a performance win by issuing reads for all the leaf blocks in parallel when the iteration begins. For example, if we have 10,000 snapshots, "zfs destroy -nv pool/fs@1%9999" can take 30 minutes when the cache is cold. This change provides a >3x performance improvement, by issuing the reads for all ~64 blocks of each ZAP object in parallel. Author: Matthew Ahrens <mahrens@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353619
* 9425 allow channel programs to be stopped via signalsAndriy Gapon2019-10-167-89/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@d0cb1fb92629bc0283c88d4719df7285c1612700 https://github.com/illumos/illumos-gate/commit/d0cb1fb92629bc0283c88d4719df7285c1612700 https://www.illumos.org/issues/9425 Problem Statement ZFS Channel program scripts currently require a timeout, so that hung or long- running scripts return a timeout error instead of causing ZFS to get wedged. This limit can currently be set up to 100 million Lua instructions. Even with a limit in place, it would be desirable to have a sys admin (support engineer) be able to cancel a script that is taking a long time. Proposed Solution Make it possible to abort a channel program by sending an interrupt signal.In the underlying txg_wait_sync function, switch the cv_wait to a cv_wait_sig to catch the signal. Once a signal is encountered, the dsl_sync_task function can install a Lua hook that will get called before the Lua interpreter executes a new line of code. The dsl_sync_task can resume with a standard txg_wait_sync call and wait for the txg to complete. Meanwhile, the hook will abort the script and indicate that the channel program was canceled. The kernel returns a EINTR to indicate that the channel program run was canceled. Author: Don Brady <don.brady@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353617
* 9485 Optimize possible split block search spaceAndriy Gapon2019-10-162-101/+308
| | | | | | | | | | | | | | illumos/illumos-gate@a21fe349793c3805ec504bbe5e9acf06c2d63d7a https://github.com/illumos/illumos-gate/commit/a21fe349793c3805ec504bbe5e9acf06c2d63d7a https://www.illumos.org/issues/9485 Port this commit from ZoL: https://github.com/zfsonlinux/zfs/commit/4589f3ae4c1bb435777da8640eb915f3c713b14d Author: Brian Behlendorf <behlendorf1@llnl.gov> Notes: svn path=/vendor-sys/illumos/dist/; revision=353615
* 10731 zfs: NULL pointer errorsAndriy Gapon2019-10-168-42/+40
| | | | | | | | | | | | illumos/illumos-gate@dd328bf6d39366b8d7bde6a36114538fc14332dd https://github.com/illumos/illumos-gate/commit/dd328bf6d39366b8d7bde6a36114538fc14332dd https://www.illumos.org/issues/10731 Author: Toomas Soome <tsoome@me.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353613
* 10330 merge recent ZoL vdev and metaslab changesAndriy Gapon2019-10-166-92/+108
| | | | | | | | | | | | | | | | | illumos/illumos-gate@a0b03b161c4df3cfc54fbc741db09b3bdc23ffba https://github.com/illumos/illumos-gate/commit/a0b03b161c4df3cfc54fbc741db09b3bdc23ffba https://www.illumos.org/issues/10330 3 recent ZoL changes in the vdev and metaslab code which we can pull over: PR 8324 c853f382db 8324 Change target size of metaslabs from 256GB to 16GB PR 8290 b194fab0fb 8290 Factor metaslab_load_wait() in metaslab_load() PR 8286 419ba59145 8286 Update vdev_is_spacemap_addressable() for new spacemap encoding Author: Serapheim Dimitropoulos <serapheimd@gmail.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353611
* 10230 zfs mishandles partial writesAndriy Gapon2019-10-161-3/+8
| | | | | | | | | | | | | | | | | | | illumos/illumos-gate@b0ef425652e5cfce27df9fa5826a9cd64cee110a https://github.com/illumos/illumos-gate/commit/b0ef425652e5cfce27df9fa5826a9cd64cee110a https://www.illumos.org/issues/10230 The trinity fuzzer calls pwritev with an iovec that has one or more entries which point to some initial valid data and then the rest point to addresses which are not mapped. This yields EFAULT once the write hits the invalid address, but we do successfully complete some amount of writing. The zfs_write code does not handle this properly. It loses track of the error return from dmu_write_uio_dbuf and it has an invalid ASSERT which does not account for the partial write case. Author: Jerry Jelinek <jerry.jelinek@joyent.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353610
* 10165 libzpool: passing argument 1 to restrict-qualified parameter aliases ↵Andriy Gapon2019-10-161-4/+5
| | | | | | | | | | | | | | with argument 4 illumos/illumos-gate@f91fcf59ac2fd04f1816f3dcbc69a46d44276a65 https://github.com/illumos/illumos-gate/commit/f91fcf59ac2fd04f1816f3dcbc69a46d44276a65 https://www.illumos.org/issues/10165 Author: Toomas Soome <tsoome@me.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353608
* 10067 Miscellaneous man page typosAndriy Gapon2019-10-161-1/+1
| | | | | | | | | | | | | illumos/illumos-gate@e61d7e85ebb4a7361eeb10639b742a92e0bf5e55 https://github.com/illumos/illumos-gate/commit/e61d7e85ebb4a7361eeb10639b742a92e0bf5e55 https://www.illumos.org/issues/10067 fileystem - man1m/zfs.1m man1m/boot.1m Author: Peter Tribble <peter.tribble@gmail.com> Notes: svn path=/vendor/illumos/dist/; revision=353606
* 10154 zfs: cast between incompatible function typesAndriy Gapon2019-10-162-4/+6
| | | | | | | | | | | | illumos/illumos-gate@c62757b2b8b6c26589d7704d0ff20beb107fcd9a https://github.com/illumos/illumos-gate/commit/c62757b2b8b6c26589d7704d0ff20beb107fcd9a https://www.illumos.org/issues/10154 Author: Toomas Soome <tsoome@me.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353605
* 10343 ZoL: Prefix all refcount functions with zfs_Andriy Gapon2019-10-1527-389/+408
| | | | | | | | | | | | | | | | | | | illumos/illumos-gate@e914ace2e9d9bf2dbf9a1f1ce81cb776022096f5 https://github.com/illumos/illumos-gate/commit/e914ace2e9d9bf2dbf9a1f1ce81cb776022096f5 https://www.illumos.org/issues/10343 On the openzfs feature/porting matrix, this is listed as: prefix to refcount funcs/types Having these changes will make it easier to share other work across the different ZFS operating systems. PR 7963 424fd7c3e Prefix all refcount functions with zfs_ PR 7885 & 7932 c13060e47 Linux 4.19-rc3+ compat: Remove refcount_t compat PR 5823 & 5842 4859fe796 Linux 4.11 compat: avoid refcount_t name conflict Author: Tim Schumacher <timschumi@gmx.de> Notes: svn path=/vendor-sys/illumos/dist/; revision=353561
* 10572 10579 Fix race in dnode_check_slots_free()Andriy Gapon2019-10-155-11/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@aa02ea01948372a32cbf08bfc31c72c32e3fc81e https://github.com/illumos/illumos-gate/commit/aa02ea01948372a32cbf08bfc31c72c32e3fc81e 10572 Fix race in dnode_check_slots_free() https://www.illumos.org/issues/10572 The Fix from ZoL: Currently, dnode_check_slots_free() works by checking dn->dn_type in the dnode to determine if the dnode is reclaimable. However, there is a small window of time between dnode_free_sync() in the first call to dsl_dataset_sync() and when the useraccounting code is run when the type is set DMU_OT_NONE, but the dnode is not yet evictable, leading to crashes. This patch adds the ability for dnodes to track which txg they were last dirtied in and adds a check for this before performing the reclaim. This patch also corrects several instances when dn_dirty_link was treated as a list_node_t when it is technically a multilist_node_t. 10579 Don't allow dnode allocation if dn_holds != 0 https://www.illumos.org/issues/10579 The fix from ZoL: This patch simply fixes a small bug where dnode_hold_impl() could attempt to allocate a dnode that was in the process of being freed, but which still had active references. This patch simply adds the required check. Author: Tom Caputi <tcaputi@datto.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353558
* 10452 ZoL: merge in large dnode feature fixesAndriy Gapon2019-10-153-10/+9
| | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@946342a260bbae359b48bf142ec1fe40792ee862 https://github.com/illumos/illumos-gate/commit/946342a260bbae359b48bf142ec1fe40792ee862 https://www.illumos.org/issues/10452 illumos is missing a few small follow up ZoL bug fixes for the large dnode feature. We should pull those in. Those commits are in the ZoL tree as (newest to oldest): PR 8435 - 75d6b7ddca269542279975f716a343bb40a79baf - Add missing copyright notice to large_dnode tests PR 7433 - e14a32b1c844d924b9f093375c0badcf10f61741 - Fix object reclaim when using large dnodes PR 6616 - 48fbb9ddbf2281911560dfbc2821aa8b74127315 - Free objects when receiving full stream as clone PR 6695 - 39f56627ae988d09b4e3803c01c22b2026b2310e - receive_freeobjects() skips freeing some object Portions contributed by: Ned Bass <bass6@llnl.gov> Portions contributed by: Tom Caputi <tcaputi@datto.com> Author: Fabian Grünbichler <f.gruenbichler@proxmox.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=353551
* 10406 large_dnode changes broke zfs recv of legacy streamAndriy Gapon2019-08-152-5/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | illumos/illumos-gate@811964cd9f1fbae0fc3b93d116269e9b1fca090a https://github.com/illumos/illumos-gate/commit/811964cd9f1fbae0fc3b93d116269e9b1fca090a https://www.illumos.org/issues/10406 The large dnode changes from 8423 caused problems in zfs recv for a legacy stream. This manifests when attempting to mount the received stream, but the problem is in the receive code. We missed the following commit from ZoL which fixes this. commit da2feb42fb5c7a8c1e1cc67f7a880da9d8e97bc2 Author: Tom Caputi <tcaputi@datto.com> Date: Thu Jun 28 17:55:11 2018 -0400 Fix 'zfs recv' of non large_dnode send streams Currently, there is a bug where older send streams without the DMU_BACKUP_FEATURE_LARGE_DNODE flag are not handled correctly. The code in receive_object() fails to handle cases where drro->drr_dn_slots is set to 0, which is always the case when the sending code does not support this feature flag. This patch fixes the issue by ensuring that that a value of 0 is treated as DNODE_MIN_SLOTS. Author: Tom Caputi <tcaputi@datto.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=351075
* 8423 8199 7432 Implement large_dnode pool featureAndriy Gapon2019-08-1243-386/+1734
| | | | | | | | | | | | | | | | | | | | | | | | | 8423 Implement large_dnode pool feature 8199 multi-threaded dmu_object_alloc() 7432 Large dnode pool feature llumos/illumos-gate@54811da5ac6b517992fdc173df5d605e4e61fdc0 https://github.com/illumos/illumos-gate/commit/54811da5ac6b517992fdc173df5d605e4e61fdc0 https://www.illumos.org/issues/8423 https://www.illumos.org/issues/8199 https://www.illumos.org/issues/7432 ZoL issues: Improved dnode allocation #6564 Clean up large dnode code #6262 Fix dnode_hold() freeing dnode behavior #8172 Fix dnode allocation race #6414, #6439 Partial: Raw sends must be able to decrease nlevels #6821, #6864 Remove unnecessary txg syncs from receive_object() Closes #7197 Author: Toomas Soome <tsoome@me.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=350898
* 6585 sha512, skein, and edonr have an unenforced dependency on extensible ↵Andriy Gapon2019-08-122-6/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dataset illumos/illumos-gate@892586e8a147c02d7f4053cc405229a13e796928 https://github.com/illumos/illumos-gate/commit/892586e8a147c02d7f4053cc405229a13e796928 https://www.illumos.org/issues/6585 In any pool without the extensible dataset feature flag already enabled, creating a dataset with dedup set to use one of the new checksums would result in the following panic as soon as any data was added: panic[cpu0]/thread=ffffff0006761c40: feature_get_refcount(spa, feature, &refcount) != 48 (0x30 != 0x30), file: ../../common/fs/zfs/zfeature.c line 390 ffffff0006761830 fffffffffba8fbdd () ffffff0006761890 zfs:feature_do_action+11a () ffffff00067618c0 zfs:spa_feature_incr+1e () ffffff0006761920 zfs:dmu_object_zapify+b7 () ffffff00067619b0 zfs:dsl_dataset_activate_feature+97 () ffffff0006761a20 zfs:dsl_dataset_sync+ba () ffffff0006761ab0 zfs:dsl_pool_sync+153 () ffffff0006761b70 zfs:spa_sync+26e () ffffff0006761c20 zfs:txg_sync_thread+227 () ffffff0006761c30 unix:thread_start+8 () Inspection showed that feature->fi_feature was 7, which is the value of SPA_FEATURE_EXTENSIBLE_DATASET in the spa_feature enum. Testing shows that the panic can be prevented by explicitly setting extensible dataset as a dependency for the sha512, edonr, and skein feature flags. Alternatively, the new checksums code could possibly be changed to obviate the need for the dependency. Reviewed by: Matthew Ahrens <mahrens@delphix.com> Reviewed by: Richard Laager <rlaager@wiktel.com> Approved by: Robert Mustacchi <rm@joyent.com> Author: ilovezfs <ilovezfs@icloud.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=350896
* 9689 zfs range lock code should not be zpl-specificAlexander Motin2019-06-048-407/+441
| | | | | | | | | | | | illumos/illumos-gate@7931524763ef94dc16989451dddd206563d03bb4 Reviewed by: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com> Reviewed by: George Wilson <george.wilson@delphix.com> Approved by: Robert Mustacchi <rm@joyent.com> Author: Matthew Ahrens <mahrens@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=348596
* 9683 Allow bypassing devid in vdev_disk_open()Alexander Motin2019-06-031-1/+46
| | | | | | | | | | | | illumos/illumos-gate@6fe4f3002ca33af5ab7123ed78d81899dadf2fbb Reviewed by: Sara Hartse <sara.hartse@delphix.com> Reviewed by: George Wilson <george.wilson@delphix.com> Approved by: Dan McDonald <danmcd@joyent.com> Author: Pavel Zakharov <pavel.zakharov@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=348585
* 9847 leaking dd_clones (DMU_OT_DSL_CLONES) objectsAlexander Motin2019-06-033-5/+236
| | | | | | | | | | | | | illumos/illumos-gate@17fb938fd6cdce3ff1bb47dafda0774f742249a3 Reviewed by: George Wilson <george.wilson@delphix.com> Reviewed by: Serapheim Dimitropoulos <serapheim@delphix.com> Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov> Approved by: Richard Lowe <richlowe@richlowe.net> Author: Matthew Ahrens <mahrens@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=348583
* 9559 zfs diff handles files on delete queue in fromsnap poorlyAlexander Motin2019-06-031-7/+7
| | | | | | | | | | | illumos/illumos-gate@20633e304b57bc98f70fdb194081b7023adf527b Reviewed by: Joshua M. Clulow <josh@sysmgr.org> Approved by: Richard Lowe <richlowe@richlowe.net> Author: Paul Dagnelie <pcd@delphix.com> Notes: svn path=/vendor/illumos/dist/; revision=348580
* 9962 zil_commit should omit cache thrashAlexander Motin2019-06-036-76/+203
| | | | | | | | | | | | | | illumos/illumos-gate@cab3a55e158118937e07d059c46f1bc14d1f254d Reviewed by: Matt Ahrens <matt@delphix.com> Reviewed by: Brad Lewis <brad.lewis@delphix.com> Reviewed by: Patrick Mooney <patrick.mooney@joyent.com> Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com> Approved by: Joshua M. Clulow <josh@sysmgr.org> Author: Prakash Surya <prakash.surya@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=348578
* 9963 Seperate tunable for disabling ZIL vdev flushAlexander Motin2019-06-032-6/+14
| | | | | | | | | | | | | illumos/illumos-gate@f8fdf6812567ce1a3b76a6bee2ee3f04bd3fcaca Reviewed by: Matt Ahrens <matt@delphix.com> Reviewed by: Brad Lewis <brad.lewis@delphix.com> Reviewed by: Patrick Mooney <patrick.mooney@joyent.com> Approved by: Dan McDonald <danmcd@joyent.com> Author: Prakash Surya <prakash.surya@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=348576
* 9993 zil writes can get delayed in zio pipelineAlexander Motin2019-06-031-1/+2
| | | | | | | | | | | | | illumos/illumos-gate@2258ad0b755b24a55c6173b1e6bb6188389f72dd Reviewed by: Prakash Surya <prakash.surya@delphix.com> Reviewed by: Brad Lewis <brad.lewis@delphix.com> Reviewed by: Matt Ahrens <matt@delphix.com> Approved by: Dan McDonald <danmcd@joyent.com> Author: George Wilson <george.wilson@delphix.com> Notes: svn path=/vendor-sys/illumos/dist/; revision=348573
* 9880 Race in ZFS parallel mountAlexander Motin2019-06-031-3/+31
| | | | | | | | | | | | illumos/illumos-gate@bc4c0ff1343a311cc24933908ac6c4455af09031 Reviewed by: Jason King <jason.king@joyent.com> Reviewed by: Sebastien Roy <sebastien.roy@delphix.com> Approved by: Joshua M. Clulow <josh@sysmgr.org> Author: Andy Fiddaman <omnios@citrus-it.co.uk> Notes: svn path=/vendor/illumos/dist/; revision=348572
* 9466 add JSON output support to channel programsAlexander Motin2019-06-033-10/+30
| | | | | | | | | | | | | | illumos/illumos-gate@5267591016146502784860802129b16dab6f135c Reviewed by: Matt Ahrens <matt@delphix.com> Reviewed by: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com> Reviewed by: Sara Hartse <sara.hartse@delphix.com> Reviewed by: John Kennedy <john.kennedy@delphix.com> Approved by: Dan McDonald <danmcd@joyent.com> Author: Alek Pinchuk <apinchuk@datto.com> Notes: svn path=/vendor/illumos/dist/; revision=348568
* 9738 9112 broke third block copy allocations within one metaslab groupAlexander Motin2019-06-031-7/+5
| | | | | | | | | | | | illumos/illumos-gate@b86e7e3f0e50748bb5bb5cc91632d72ff17f08dd Reviewed by: Paul Dagnelie <pcd@delphix.com> Reviewed by: George Wilson <george.wilson@delphix.com> Approved by: Robert Mustacchi <rm@joyent.com> Author: Alexander Motin <mav@FreeBSD.org> Notes: svn path=/vendor-sys/illumos/dist/; revision=348556