Opened 2 years ago

Closed 2 years ago

Last modified 2 years ago

#30157 closed defect (fixed)

Cygwin: system sqlite3 modifies DLL search order

Reported by: Erik Bray Owned by:
Priority: major Milestone: sage-9.2
Component: porting: Cygwin Keywords:
Cc: Merged in:
Authors: Erik Bray Reviewers: Matthias Koeppe
Report Upstream: N/A Work issues:
Branch: 628e7d0 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges


Under normal operation Windows searches for DLLs in the following order:

  1. In the same directory as the main executable.
  2. In various standard system directories.
  3. Search $PATH.

On Cygwin, most DLLs are stored under /usr/bin, which is inserted early on the $PATH, but when in the Sage environment $SAGE_LOCAL/bin supersedes it.

However, I discovered that the Cygwin port of sqlite3 contains the following nasty patch, for reasons I can't be sure of:

@@ -47710,13 +48357,52 @@ SQLITE_API int sqlite3_os_init(void){
   assert( winSysInfo.dwAllocationGranularity>0 );
   assert( winSysInfo.dwPageSize>0 );

+#ifdef _WIN32
+  module = osGetModuleHandleW(L"CYGWIN1.DLL");
+  if( !module ){
+    module = osGetModuleHandleW(L"MSYS-2.0.DLL");
+  }
+  if( !module ){
+    module = osGetModuleHandleW(L"MSYS-1.0.DLL");
+  }
+  if( module ){
+    for( i=81; i<ArraySize(aSyscall); ++i ){
+        aSyscall[i].pCurrent = (SYSCALL) osGetProcAddressA(module,
+            aSyscall[i].zName);
+    }
+  }
+  sqlite3_os_unix_init();
   sqlite3_vfs_register(&winVfs, 1);

+  if( cygwin_conv_path ){
+    WCHAR buf[MAX_PATH];
+    cygwin_conv_path(CCP_POSIX_TO_WIN_W, "/usr/bin",
+        buf, MAX_PATH*sizeof(WCHAR));
+    osSetDllDirectoryW(buf);
+#ifdef _WIN32
+  }else if( cygwin_conv_to_full_win32_path ){
+    WCHAR buf[MAX_PATH];
+    char *buf1 = (char *)buf;
+    int i = MAX_PATH;
+    cygwin_conv_to_full_win32_path("/usr/bin", buf1);
+    while(--i>=0) buf[i] = buf1[i];
+    osSetDllDirectoryW(buf);
+  }
 #if defined(SQLITE_WIN32_HAS_WIDE)
   sqlite3_vfs_register(&winLongPathVfs, 0);

The function SetDllDirectoryW is a system function which works a little like LD_LIBRARY_PATH or LD_PRELOAD allowing an application to insert a non-standard directory into the DLL search path.

Why sqlite3 is doing this I don't know. It has something to do with loading extensions apparently, but there's no reason it should assume I always want to load extensions from /usr/bin (e.g. what if I'm using a custom build of sqlite3 installed in /usr/local/bin.

It does this also when the library is first initialized, as opposed to just doing it before loading an extension and then unsetting it. Thus this change to DLL loading behavior affects the rest of the application for the lifetime of the application. It does get undone if sqlite3_shutdown() is called but this never happens normally.

How does this affect Sage? It's not even obvious that Sage uses sqlite3, but in fact IPython does to store its history, so a sqlite3 database is connected to when starting up the Sage/IPython REPL. This in turn impacts DLL search order for all libraries that haven't been loaded yet, and can cause Cygwin's system versions of those libraries to be privileged over any copies in Sage.

I think this might actually explain some related bugs I've seen in the past, but I'm not sure.

Change History (11)

comment:1 Changed 2 years ago by Matthias Köppe

sqlite3 is a dependency of python3. Not sure if it is only linked when the corresponding module is used, or in general

comment:2 Changed 2 years ago by Erik Bray

Only when the corresponding module is used.

comment:3 Changed 2 years ago by Erik Bray

Authors: Erik Bray
Branch: u/embray/ticket-30157
Commit: 628e7d01166f21f480b0ae22361b9a29c4d854ef
Status: newneeds_review

It's not pretty, but it does fix the issue. Open to better ideas for what to name this and/or where to insert the fix.

New commits:

628e7d0Trac #30157: Undo sqlite3's munging of DLL search paths

comment:4 Changed 2 years ago by Matthias Köppe

Perhaps this should also be reported as a bug to the cygwin maintainers?

comment:5 Changed 2 years ago by Dima Pasechnik

Isn't Sage using sqlite3 in src/sage/graphs/ ?

comment:8 Changed 2 years ago by Matthias Köppe

Reviewers: Matthias Koeppe
Status: needs_reviewpositive_review

comment:9 Changed 2 years ago by Volker Braun

Branch: u/embray/ticket-30157628e7d01166f21f480b0ae22361b9a29c4d854ef
Resolution: fixed
Status: positive_reviewclosed

comment:10 in reply to:  4 Changed 2 years ago by Erik Bray

Commit: 628e7d01166f21f480b0ae22361b9a29c4d854ef

Replying to mkoeppe:

Perhaps this should also be reported as a bug to the cygwin maintainers?

I did, the maintainer of the Cygwin package for sqlite3 agreed it's a bug and we discussed a possible fix:

comment:11 in reply to:  5 Changed 2 years ago by Erik Bray

Replying to dimpase:

Isn't Sage using sqlite3 in src/sage/graphs/ ?

It's actually use during Sage start-up, but only when starting in the IPython REPL. It's used by IPython because it stores the interactive history in a sqlite database.

This is actually how I was able to narrow this down: because the issue didn't occur when I imported sage in a normal Python session, or ran something like sage -c 'print(1 + 1)'. It would only happen when starting with the interactive prompt, which was very bizarre.

Note: See TracTickets for help on using tickets.