[vbox-dev] [PATCH] fix file copy in shared folder on Solaris

[Life is hard, and then you die]

Attached is a somewhat hackish patch to fix the kernel panic in
https://www.virtualbox.org/ticket/9862 . There's probably a better
solution, but I'm not sure what it is. Together with the previous
rmdir patch it does make the shared folders usable again, though.

The patch is in the public domain.


  Cheers,

  Ronald

-------------- next part --------------

Hack to work around recursive mutex issue when copying files.

sffs_write() can pagefault in uiomove(), which in turn calls sffs_getpage().
This results in a panic due to a recursive mutex_enter. This hack just
detects this recursive enter in sffs_getpage() and avoids acquiring the
mutex in that case.

diff --git a/src/VBox/Additions/solaris/SharedFolders/vboxfs_vnode.c b/src/VBox/Additions/solaris/SharedFolders/vboxfs_vnode.c
index 0b7f838..c28729e 100644
--- a/src/VBox/Additions/solaris/SharedFolders/vboxfs_vnode.c
+++ b/src/VBox/Additions/solaris/SharedFolders/vboxfs_vnode.c
@@ -1604,6 +1604,7 @@ sffs_getpage(
 	)
 {
 	int error = 0;
+	int is_recursive = 0;
 	page_t **pageliststart = pagelist;
 	sfnode_t *node = VN2SFN(dvp);
 	ASSERT(node);
@@ -1625,12 +1626,21 @@ sffs_getpage(
 	if (protp)
 		*protp = PROT_ALL;
 
-	mutex_enter(&sffs_lock);
+	/*
+	 * The buffer passed to sffs_write may be mmap'd so we may get a
+	 * pagefault there, in which case we'll end up here with this thread
+	 * already owning the mutex. Furthermore, mutexes are not re-entrant.
+	 */
+	if (mutex_owner(&sffs_lock) == curthread)
+		is_recursive = 1;
+	else
+		mutex_enter(&sffs_lock);
 
 	/* Don't map pages past end of the file. */
 	if (off + len > node->sf_stat.sf_size + PAGEOFFSET)
 	{
-		mutex_exit(&sffs_lock);
+		if (!is_recursive)
+			mutex_exit(&sffs_lock);
 		return (EFAULT);
 	}
 
@@ -1647,7 +1657,8 @@ sffs_getpage(
 				page_unlock(*--pagelist);
 
 			*pagelist = NULL;
-			mutex_exit(&sffs_lock);
+			if (!is_recursive)
+				mutex_exit(&sffs_lock);
 			return (error);
 		}
 
@@ -1679,7 +1690,8 @@ sffs_getpage(
 	}
 
 	*pagelist = NULL;
-	mutex_exit(&sffs_lock);
+	if (!is_recursive)
+		mutex_exit(&sffs_lock);
 	return (error);
 }