Opened 15 months ago

Closed 15 months ago

Last modified 15 months ago

#30157 closed defect (fixed)

Cygwin: system sqlite3 modifies DLL search order

Reported by: embray Owned by:
Priority: major Milestone: sage-9.2
Component: porting: Cygwin Keywords:
Cc: Merged in:
Authors: Erik Bray Reviewers: Matthias Koeppe
Report Upstream: N/A Work issues:
Branch: 628e7d0 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description

Under normal operation Windows searches for DLLs in the following order:

  1. In the same directory as the main executable.
  2. In various standard system directories.
  3. Search $PATH.

On Cygwin, most DLLs are stored under /usr/bin, which is inserted early on the $PATH, but when in the Sage environment $SAGE_LOCAL/bin supersedes it.

However, I discovered that the Cygwin port of sqlite3 contains the following nasty patch, for reasons I can't be sure of:

@@ -47710,13 +48357,52 @@ SQLITE_API int sqlite3_os_init(void){
   assert( winSysInfo.dwAllocationGranularity>0 );
   assert( winSysInfo.dwPageSize>0 );

+#ifdef _WIN32
+  module = osGetModuleHandleW(L"CYGWIN1.DLL");
+  if( !module ){
+    module = osGetModuleHandleW(L"MSYS-2.0.DLL");
+  }
+  if( !module ){
+    module = osGetModuleHandleW(L"MSYS-1.0.DLL");
+  }
+  if( module ){
+    for( i=81; i<ArraySize(aSyscall); ++i ){
+        aSyscall[i].pCurrent = (SYSCALL) osGetProcAddressA(module,
+            aSyscall[i].zName);
+    }
+  }
+#endif
+
+#if SQLITE_OS_UNIX
+  sqlite3_os_unix_init();
+#endif
+
   sqlite3_vfs_register(&winVfs, 1);

+#if !defined(SQLITE_OMIT_LOAD_EXTENSION)
+  if( cygwin_conv_path ){
+    WCHAR buf[MAX_PATH];
+    cygwin_conv_path(CCP_POSIX_TO_WIN_W, "/usr/bin",
+        buf, MAX_PATH*sizeof(WCHAR));
+    osSetDllDirectoryW(buf);
+#ifdef _WIN32
+  }else if( cygwin_conv_to_full_win32_path ){
+    WCHAR buf[MAX_PATH];
+    char *buf1 = (char *)buf;
+    int i = MAX_PATH;
+    cygwin_conv_to_full_win32_path("/usr/bin", buf1);
+    while(--i>=0) buf[i] = buf1[i];
+    osSetDllDirectoryW(buf);
+#endif
+  }
+#endif
+
 #if defined(SQLITE_WIN32_HAS_WIDE)
   sqlite3_vfs_register(&winLongPathVfs, 0);
 #endif

The function SetDllDirectoryW is a system function which works a little like LD_LIBRARY_PATH or LD_PRELOAD allowing an application to insert a non-standard directory into the DLL search path.

Why sqlite3 is doing this I don't know. It has something to do with loading extensions apparently, but there's no reason it should assume I always want to load extensions from /usr/bin (e.g. what if I'm using a custom build of sqlite3 installed in /usr/local/bin.

It does this also when the library is first initialized, as opposed to just doing it before loading an extension and then unsetting it. Thus this change to DLL loading behavior affects the rest of the application for the lifetime of the application. It does get undone if sqlite3_shutdown() is called but this never happens normally.

How does this affect Sage? It's not even obvious that Sage uses sqlite3, but in fact IPython does to store its history, so a sqlite3 database is connected to when starting up the Sage/IPython REPL. This in turn impacts DLL search order for all libraries that haven't been loaded yet, and can cause Cygwin's system versions of those libraries to be privileged over any copies in Sage.

I think this might actually explain some related bugs I've seen in the past, but I'm not sure.

Change History (11)

comment:1 Changed 15 months ago by mkoeppe

sqlite3 is a dependency of python3. Not sure if it is only linked when the corresponding module is used, or in general

comment:2 Changed 15 months ago by embray

Only when the corresponding module is used.

comment:3 Changed 15 months ago by embray

  • Authors set to Erik Bray
  • Branch set to u/embray/ticket-30157
  • Commit set to 628e7d01166f21f480b0ae22361b9a29c4d854ef
  • Status changed from new to needs_review

It's not pretty, but it does fix the issue. Open to better ideas for what to name this and/or where to insert the fix.


New commits:

628e7d0Trac #30157: Undo sqlite3's munging of DLL search paths

comment:4 follow-up: Changed 15 months ago by mkoeppe

Perhaps this should also be reported as a bug to the cygwin maintainers?

comment:5 follow-up: Changed 15 months ago by dimpase

Isn't Sage using sqlite3 in src/sage/graphs/graph_database.py ?

comment:8 Changed 15 months ago by mkoeppe

  • Reviewers set to Matthias Koeppe
  • Status changed from needs_review to positive_review

comment:9 Changed 15 months ago by vbraun

  • Branch changed from u/embray/ticket-30157 to 628e7d01166f21f480b0ae22361b9a29c4d854ef
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:10 in reply to: ↑ 4 Changed 15 months ago by embray

  • Commit 628e7d01166f21f480b0ae22361b9a29c4d854ef deleted

Replying to mkoeppe:

Perhaps this should also be reported as a bug to the cygwin maintainers?

I did, the maintainer of the Cygwin package for sqlite3 agreed it's a bug and we discussed a possible fix: https://cygwin.com/pipermail/cygwin/2020-July/245536.html

comment:11 in reply to: ↑ 5 Changed 15 months ago by embray

Replying to dimpase:

Isn't Sage using sqlite3 in src/sage/graphs/graph_database.py ?

It's actually use during Sage start-up, but only when starting in the IPython REPL. It's used by IPython because it stores the interactive history in a sqlite database.

This is actually how I was able to narrow this down: because the issue didn't occur when I imported sage in a normal Python session, or ran something like sage -c 'print(1 + 1)'. It would only happen when starting with the interactive prompt, which was very bizarre.

Note: See TracTickets for help on using tickets.